Friday, August 12, 2011

Using RESTful APIs

Please note, this article is intended as an introduction to RESTful APIs and is has been simplified to target those who are new to programming. Comments, corrections, and feedback would not only be appreciated but greatly encouraged as I'm sure experts will find ways to improve this general overview. Happy hacking!


Problems with Non-RESTful APIs
APIs as we know (or application programming interfaces -- but no one really calls them that anymore... They're just 'APIs') are compact ways to use other people's code to extend the capabilities of a computer program. Traditionally, to use an API, one would install and then include/import the required software modules/packages within one's code and then interact this third party software via the programming interface defined by said package's author. Of course, this process requires (1) installation and importing of their code and (2) that you are using a programming language compatible with their libraries.


The RESTful Solution
RESTful APIs help solve this problem. Instead of downloading, installing, and including/importing someone else's package within your program (and hoping their software is supported by your language), you can leverage the HTTP as well as the properties of REST to achieve a more elegant solution.

The Big Picture
What if, instead of importing other peoples' libraries, these libraries were available and responsive as web-services? And what if your program could communicate with these designated web services using HTTP methods and have these services perform the requested computations and tasks for us? This is the underlying notion behind RESTful web APIs.

A good example would be, instead of buying a mill saw, some blades, a nice workbench, cutting wood in your house, and cleaning up the mess, you can instead send the wood off to a service which will just return cleanly cut pieces of wood exactly to your specifications.

If you read on, you will learn more about HTTP and REST, as well as see examples demonstrating how your program can make HTTP requests to use a RESTful web API.


History and Intro
There's been a lot of buzz recently about RESTful APIs, but what does it all mean? The term REST, or Representational State Transfer, is not new and dates back to the year 2000 when Roy Fielding was assisting in the writing of the HTTP 1.0 and 1.1 specs. While the idea of REST can be applied to many application layer protocols, it was originally designed parallel to HTTP and thus it makes sense to understand how HTTP works.


Understanding HTTP
The Hypertext Transfer Protocol (HTTP) is the underlying application layer protocol used for the World Wide Web. The HTTP spec defines several request methods over which communication can occur between participants: HEAD, GET, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, and PATCH. Every time you type a URL into your browser, that URL is actually translated and resolved into the name or physical address of a computer which will serve your request (or pass your request along to the correct server). When your request eventually reaches the correct destination server, a response will be generated and returned to you containing a response code and any resources which were requested. This response code provides insight as to the current state of the responder. This notion of responses and state are important to understanding the concept of REST (representational state transfer).

Understanding REST
The best resource for understanding REST is by reading Roy Fielding's PhD dissertation, Architectural Styles and the Design of Network-based Software Architectures. However, there are several blogs and articles which summarize Fielding's research and makes the information more accessible. Peter Laird makes an important clarification that REST, "is not a technology, a standard, or a product. REST is an architectural pattern that describes the underlying architecture of the World Wide Web and how it came to be such a massively scalable computer application" [Laird OnDemand].

Laird provides a nice summary:
In essence REST describes an architecture in which:
  • Application resources (objects, in the OO world) are exposed as URIs
  • HTTP requests are used to retrieve and update data on the server
  • The HTTP requests utilize the standard HTTP verbs (GET, POST, PUT, DELETE) to define the API operations, helping client developers by providing a consistent interaction model
Since I am not a networking expert, I will defer the the following three resources (in reading order):

1. tomayko (the basics)
2. technoracle (lite overview)
3. Laird OnDemand
General Wikipedia Supplemental
REST-style architectures consist of clients and servers. Clients initiate requests to servers; servers process requests and return appropriate responses. Requests and responses are built around the transfer of representations of resources. A resource can be essentially any coherent and meaningful concept that may be addressed. A representation of a resource is typically a document that captures the current or intended state of a resource.
The client begins sending requests when it is ready to make the transition to a new state. While one or more requests are outstanding, the client is considered to be in transition. The representation of each application state contains links that may be used next time the client chooses to initiate a new state transition.[7]

Submitting a GET Request over HTTP
Sounds nice, but how does it work? In this tutorial, I'll show you how anyone can use a RESTful API without installing any software packages (assuming you already have the python installed and urllib/2).

When you type a search on google, you are actually submitting a HTML form. A form is a special type of HTML tag which specifies instructions on how information should be passed from one computer (or server) to another, when an action such as 'submit' has been specified. Forms generally have two attribute fields, a method and an action. Your web browser examines the html form's attributes and uses this to determine how it should pass data to the recipient.

A html form may look like this:

Code:
<html>
    <head>
        <title>Mek's form example</title>
    </head>
    <body>
        <form method="GET" action="some-url">
            <input type="text" name="username" />
            <input type="password" name="password" />
            <input type="submit" name="submit" />
        </form>
    </body>
</html>
First, before anyone jumps at my throat, yes, I am aware the method of this form is GET and that a 'password' field is being used. Whenever you are constructing a form, you have several options regarding what 'method' to use when transmitting data from location A to B. Each has their own advantages and disadvantages. I will explain this in greater detail later, however, in nearly all cases, if you are transmitting secure data like credit card information, passwords, or private keys, you will NOT want to use GET as your HTTP method (and you will also want to use HTTPS to encrypt your request). I will explain why this is the case using the code above.

First, I'd like to explain what the code above does and what it means (and then I'll mention the difference between two of the methods of the HTTP protocol, GET and POST).

In the code above, we create a form, just like any HTML tag, but we specify two fields, a method and an action. The method determines how we send data from A to B, as I said before. The action determines who the recipient of the information.

In order to understand the difference between GET and POST, let's pretend the 'action' field in our code above was google, and not 'some-url'. I'll describe what's actually happening when a user (like you or I) visit the web page containing the code above up to the point where the form is submitted and information is returned.

1. A user goes to our website (with the code above) and enters in a username and password within the input fields.

2. The user presses the submit button

3. The user's web browser examine the form's attributes and determines how the information should be sent to the recipient (according to the method)

3. a) If the method is GET, the browser understands you would like to GET a new webpage as a response to your request. As we know, the way to get a new webpage (or the way we request a webpage) is by asking for a URL.

The browser constructs a URL from our data which looks like this:

Code:
something.com/something?username=xxx&password=1234

Anyways. You'll notice there is a question mark in the url. This question mark denotes the start of a query. A query is how special parameters are specified within a URL. They are used to tell the server you want specific pieces of information. In this case, we are passing the server two parameters through our URL (and we're asking to GET a webpage which fits that criteria). For a google search, when you type a query in the textbox, maybe we're searching for 'python' and press submit, your browser turns your request into something like:

Code:
google.com/#q=python
Why wouldn't you want to have a password sent to a server via GET? Notice using GET all the information transmitted is sent visibly through the URL, and you wouldn't want others to see your password. In what cases is it preferable to use GET? And what alternative is there to GET? Good questions!

GET is useful because it allows us to share links which contain very specific information. For instance, not only can I share a link to Google, but I can share a link containing the exact query I used to get a Google result! So there, GET actually is extremely useful. It's how we GET every web page we want! But when it comes to passing secure information, GET is not the best solution because it is passed visibly and the amount of information you can send within GET is (as per the specs) limited.
RFC 2068 states:
    Servers should be cautious about depending on URI lengths above 255 bytes, because some older client or proxy implementations may not properly support these lengths.

Therefore, we can use the POST method to send information to a server without it being directly visible in our URL. Instead of telling the destination server we want to GET a resource back, we are telling it we want to POST information to them. Note, while POST solves the problem of sending information without displaying it in the URL, the use of the POST method is not inherently secure. In order to perform secure POST requests, the request needs be encrypted and sent using HTTPS.

But wait, maybe you're thinking, "How come when I POST information like my username and password on Google, I GET a new page?". This is another good question. When you POST information to a server, the server will handle your POST request how ever it is designed. A POST doesn't have to direct you to a new website, however, traditionally this is preferable, so the recipient server will redirect you to the appropriate page.

Now that we understand the basics of HTTP requests, we'll write a simple python script which simulates a GET request to a web service acting as a RESTful API. We will request that this web service returns a JSON resource and will use python to capture the server's response so we can use this returned data!

Once you learn how to do this, you will be able to apply this knowledge use any RESTful API on the web! You will no longer be restricted by your programming language.

But there's one last thing I didn't discuss. Fine, we make a request to a server using their RESTful API, but what format is their returned resource in? Often time you can request the resource be sent in a specific format. Two of the more popular (and often default) formats are JSON and XML. In my opinion, dealing with XML is a pain in the arse, so I'm only going to discuss JSON.

In order to understand JSON (javaScript Object Notation -- actually a lot easier than it sounds) we must first learn about python dictionaries (associative arrays). Aw, but we were just getting to the good part! Let's see some python! Fine, we can hold off on learning how and why JSON is awesome when we get to the part in our Python code where the server has actually sent us the JSON code as a response.

So, without further ado, here's a snippet of python code which sends a GET request (without any data) to my website Baybo and retrieves a list of all the products on the platform. Oh, if you'd rather get info about our users instead, just replace the word 'products' in the url below with 'users':

Code:
import urllib2

url = "http://www.baybo.it/api/products"

# Now we have our url, let's open it in python
response = urllib2.urlopen(url)

# Now let's read that response into a variable
html = response.read()

# if you're feeling lazy, you could do this all in
# one line and just type:
# html = urllib2.urlopen(url).read()

# Now the variable 'html' contains our 
# JSON response but it's a String! Not JSON!

# Oh well, guess we have to learn about JSON now!

We got a string representation of the JSON data we want from the server, now we want to tell python to parse this data as JSON so we can interact with it.

In the last step we did

Code:
html = urllib2.urlopen(url).read()
In order to take this html data (in string form) and convert it to JSON, we will need to import the 'json' module (should already be installed on your machine with python) and use the json.loads() function. This is what the entire program would look like:

Code:
import urllib2
import json
html = urllib2.urlopen(url).read()
data = json.loads(html)
The variable 'data' is now an actual data structure within Python that we can natively access. In this case, the data the server transmitted to us is an array (a python list) of dictionaries. Uh oh, now we really need to learn what dictionaries are, or we won't be able to use this data effectively!

A dictionary is similar to a list/array in that it is a data structure for storing values. However, the method of indexing values and the values of an index are different than a list. In fact, the complexity analysis for the entire data structure is different! By this I mean, a dictionary (associative array) does not have the same complexity properties as a list: the computational cost (the number of steps) to perform insertion, deletion, and resizing over the data structure are different than lists.

So, before we tackle the complexity properties of dictionaries, what the heck is a dictionary? Think of a dictionary like... Well, a dictionary! Or a phone book, if you prefer. A dictionary is a collection (unordered) of key-value pairs. In a dictionary (the books we're used to) a dictionary word is used to reference a definition. If you want to know the definition for a word, use use the word as the key, find this word, and knowing the word will give you access to the definition. Same goes for a phone book. You use someone's name as the key, and with it can find someone's number (the value).

A dictionary in python looks like this:

Code:
mydict = { "key": 3,
           "mek": 13}
Values are access from a dictionary in a similar way as lists, however, the index is not a position (that is, not necessarily an integer), but a key. Here's an example of how I would access the value 13 in the above dictionary called 'mydict' using the key "mek" which is a string"

Code:
# This returns the value 3
mydict["mek"]
Ok! Dictionaries have very interesting properties with respect to complexity analysis... But I suspect you folks would rather continue the example right now and so something useful with your data / see a full example program. If you're interested in learning more about the data structures, let me know! This is useful information which can be applied to nearly any programming language and knowing this information is useful for CS job interviews and school.

Anyways. Where did we leave off in our program? Ah yes, you just turned our data into JSON format and python automatically loaded this JSON into a native python data structure as a list of dictionaries which we can use!

Code:
# data is a list of dictionaries
data = json.loads(html)

# The first element of the list is a dictionary.
# This will print out the dictionary so we can see
# what keys and values it has.
data[0]

# This tells us the keys for the dictionary are:
# "content", "name", "created", "slug", "price", "modified",
# "users_id", "currency", "avatar", "id".

# What does this mean? For every product on Baybo,
# We can get all this information! If we wanted,
# we could make our own website and advertise / affiliate
# products with tiny script.

# Q: How would we access the product id of the 1st product?
 # A: get the 0th product in the dictionary and 
#     get the value where the key is "id"
data[0]["id"]

# We could write a loop to iterate over every product
# and do something useful:

# This will print out the name of every product
for product in data:
    print data[product]["name"]

Here are some examples from the Python Documentation which explain httplib and urllib2 in greater depth. This resource also explains how you would accomplish such a request via POST, as opposed to GET.

Following these steps, you can use any RESTful API out there. I just used my personal RESTful API as an example, that way if you folks have any questions, I can help you solve your problems.

Until next time, good luck and happy hacking!

4 comments:

  1. There is nothing secure about sending password data over a POST request that does not use HTTPS. When using POST over HTTP all of the data is sent in clear text in the request body. Yes, it may not be visible to you in the URL, but anyone interested in that data on the same network as you can easily read it.

    Nothing in this post is really related to REST. You should probably read this http://www.amazon.com/RESTful-Web-Services-Leonard-Richardson/dp/0596529260

    ReplyDelete
  2. Thanks for the comment John! I updated the post to focus more on the basics of HTTP and REST and also provided free resources (including Roy Fielding's thesis on REST).

    The main objective of this article was to introduce the notion of RESTful web APIs and to provide an example on how one might be used.

    Also, while I never suggested POST was secure (only that it wasn't visible in the URL), you do mention a good point that HTTPS is needed to encrypt POSTed data -- the article has been updated to make this more clear.

    ReplyDelete
  3. Awesome blog man, the things you have mentioned above are really informative and are examples of your awesome writing skills and a good blog.
    John Wintermute

    ReplyDelete
  4. Thanks so much for your kind words Josiah! I took a look at your blog as well. Very excited by your line of work. My father has worked for several chemical companies doing R&D, much involving lubrication. I'd love to chat sometime and learn from you! Best wishes and happy holidays. :o)

    ReplyDelete