Flask by example 5 (How to build a simple REST API)
Welcome to part 5. In this part I’m going to show you how to build your own RESTful API with flask
The API would serve as the middle-man between the client (browser) and the web server and it’s the means by which data would be sent across both parties. We’re going to be passing the data along as JSON (Javascript object Notation).
At the end of this tutorial you’d have created an API that receives data as JSON, parses it and stores the information in the database, and also sends JSON back to the client whenever it’s queried.
The API would also be robust enough to respond with useful error messages (as JSON) whenever the client sends incomplete data to the server.
This is an overview on the structure of our application and the important technology/library used by each component.
From the diagram you can see that the API is the one who runs the show and provides a means for the other components to communicate with each other, including external services.
WHY DO WE NEED AN API
I’m not going to bore you out with all the numerous reasons to use an API, because you’ve probably read them online somewhere if you’ve not here are some.
The main reason why need an API is because client-side Javascript libraries like ReactJS cannot communicate directly with your database (which resides on the server) directly. Though Server side Javascript like NodeJS can do that.
Currently this is how our form declaration looks with React
With React
Without react we have this
When the handleSubmit
function in called a lot of “DOM changing” processes would have gone on in the other methods handleOptionAdd
, handleOptionChange
and handleTitleChange
(Adding options and setting the Title of the poll) that the external DOM wouldn’t know about. React has it’s own virtual DOM to efficiently keep track of all these changes. When you submit the form normally, you might find out that some information that show up in the form are missing on the server.
I urge you to experiment and submit the form you created in the previous part without using the onSubmit
handler and see what details are submitted to the server when your print request.form
in flask.
So in summary an API serves as a common interface for the client and the server to talk to each other reliably.
Enough of the talk let’s build an API.
The full code of some of some files won’t be pasted, Only the important part will be pasted so if you want to see the full source code at any point, feel free to checkout the github repo
Building the API
Our API is going to perform two basic functions at this stage:
- Receive data as a valid JSON string, parse it and store the information in the database
- Return data in as JSON back to our client who then decides on what he wants to do with the data. We're simply providing the raw data back and not forcing the presentation of the data on the client. we don't care about what the client wants to do with the data he could be using it for stastics and data analysis or to build an alternate version of our polling application
JSON STRUCTURE FOR GET REQUESTS
This is how we want to return polls back to the client when they make a request to get some or all the polls in our database. Let’s call this structure our Dream API
If the JSON structure looks scary to you, just think of it as a collection of python dictionaries and lists and the structure should become clearer to you.
You can also check the validity of this JSON and any JSON string online with JSON lint
Moving forward, we’re going to separate our API routes from our main application routes by prefixing the url with /api
hint: We’re goint to refactor the whole codebase in a future part of this series and make our API into a flask blueprint. if you don’t know what a blueprint is in flask this might be a good time to check it out
Now edit the votr.py
file to include a new route /api/polls
In the POST request, we aren’t really doing anything, we just get the JSON the user sent with request.get_json()
and then return the details back to show us that our API works. request.get_json()
is a helper method provided that flask that gets the JSON data from the request and converts it to a python dictionary for us. It’s similar to calling json.loads on a JSON string but does some extra things under the hood.
Earlier, our aim was to return a JSON string containing the poll and various details associated with it. This is exactly what we implemented in the GET request. To acheive that we queried the database for all the existing Topics and then for each topic we fetched the associated polls (Polls.query.filter_by(topic=topic)
), after that we appended the results in the format we wanted and then used another flask helper function called jsonify
to convert the string to JSON. jsonify is also similar to python’s json.dumps but like request.get_json
it does more than what meets the eye.
Okay our should work properly but we could still improve it by leveraging the power of SQLAlchemy and using an SQL join statement But how do you use a join statement when you’re not writing raw SQL…Simple use SQLAlchemy
Let’s fire up a terminal and play around with SQLAlchemy
If for some reason you have an empty database (your query returns a blank result), here’s a github gist to quickly create all the records in the database at this stage. Just run each line of code in the file in a python interactive shell
Neat! we were able to fetch all the topics and their associated options in a single query. Though you’re seeing only the title of the Topic (that’s __repr__
in action). those two items are actually SQLAlchemy objects that contain all the data you need, you just need to know how to extract them from the object.
Another amazing thing about SQLAlchemy is that we can see the query it generates behind the scenes
That shows us the General SQL statment it’s going to generate for us, but we can go a step further and see the specific query for the SQL dialect of a specific database.
For the experienced database admins this could be pretty useful, as ORM’s don’t generate the most efficient SQL query sometimes (but there’s no need for premature optimization at this level). And as a beginner you could also learn more about SQL as a language by inspecting the SQL generated by the SQLAlchemy statements you write.
Okay, so how do we convert this complex query object into a dictionary which can be jsonified.
To do that we’re going to define a method called to_json
in the Topics
Model that abstracts this functionality and simply gives us a custom dictionary representation of the object.
Update the Topics
model and add a to_json
method
Now we can simply call to_json
on any Topic
to get a custom dict
representation, which leaves us with a simpler list comprehension than we had before.
Wait? what about the poll state
This is the kind of JSON that our API will send back to us when we query it currently.
What happened to our dream API, weren’t we supposed to have the status of the poll included.
Yes we were but we can’t, because well…our database design is broken. Yes you read that right it’s broken. The status of the poll is supposed to be a column on the Topics
model, so once we set a Topic as closed all options (actually Poll
objects in our models) associated with it are automatically closed. We don’t need to go around marking all the Poll
objects as closed. It’s way easier to just close the Topic
object that all the Poll
objects are related to.
But you went ahead and brilliantly put it in the Polls
table (actually i misled you and congrats! you fell for it).
So after cursing yourself and me, how are you going to fix this mistake. Yes you’re interacting with your database with python through SQLAlchemy and you can easily cut and paste the status line into the Topics
model. But that’s not going to work.
SQLAlchemy on it’s own can only alter the schema of our database by adding new models, but we’ve found ourself in a situation where we need to delete a column from a table and then create that column in another table. (altering two tables).
SQLAlchemy and other ORM’s are only an abstraction not a direct replacement for SQL, so at the end of the day you’re still working with a database and SQL.
The database on it’s own part, knows nothing about python so it wouldn’t know when we made any change to the python file containing the models. The models.py
file needs a way need a way to tell the database hey one of my models just got updated. you need to update the correesponding table in the database to match our current state
This is where Flask-Migrate comes to the rescue and says step aside guys I’ll handle this.
Flask Migrate
Quoting the docs
Flask-Migrate is an extension that handles SQLAlchemy database migrations for Flask applications using Alembic. The database operations are made available through the Flask command-line interface or through the Flask-Script extension.
So what exactly is a migration?
Migrations (also known as ‘schema evolution’ or ‘mutations’) are simply a way of changing your database schema from one version into another.
Carrying out database migrations manually is very problematic, sure you could use a tool like phpMyAdmin or pgAdmin to alter your tables, but picture this scenario - You’ve done a lot of migrations manually with a graphical tool or from the command line and then you realized that your initial database design was better than the one you have currently, so you decide to go three steps back to the previous state of your database.
How do you go back when you have no previous history of your schema at that point in time? If you don’t have a good memory or you didn’t document that information down, you’re screwed. You either have to design the whole database again or some table(s) again depending on the size of your database and how much has changed.
This is what Flask Migrate (and other solutions for other web frameworks) are good at, with Flask migrate you can easily switch between older and newer versions of your database state at anytime. It’s just like a mini git
for your database.
It also makes it easier for you to switch databases, you could design and develop on an SQLite database database, and then deploy the application to a PostgreSQL database.
Let’s go on and install flask migrate from pip
Edit votr.py
and import the Migrate
class from flask_migrate
This is how votr.py
looks like now
Did you notice that at the end of the file we don’t have the if __name__ == "__main__"
condition again?
We don’t need that statement again because there is an alternate way to run flask applications (From Flask 0.11). This new method is more flexible and enables us to easily run the new migration commands Flask Migrate has exposed to us.
So you wont be running your application with python3 votr.py
anymore instead you’ll use the flask run
command, but for that to work properly we need to setup some basic info that tells flask how to find our application and how it should run it.
from your terminal type in the following
This set’s the flask app to votr and then turns on debug mode
To prevent yourself from typing this anytime you want to work on votr (possibly after a reboot) just add those two lines to your activate script for your virtualenv. located in the bin directory in your virtualenv’s root folder.
Anytime you run source/bin/activate
the correct app and the debug mode would be set.
To run the initial database migration, use this:
You’ll be instructed to edit the alembic file in the migrations folder. (You don’t really need to, for a simple database like yours)
Then generate the initial migration script with:
After doing this you can now make the required change to the Topics
model and also add the status field to the to_json
method (We also added a default value of 1
to the field so any poll that created is automatically opened for voting)
Now run
You should oops!
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "DROP": syntax error [SQL: 'ALTER TABLE polls DROP COLUMN status']
Don’t worry this problem isn’t because you did something wrong it’s because SQLite doesn’t support SQL alter statements fully. Gladly there is a workaroud in alembic > 0.7 to get around this by simulating alter statements for SQLite by:
- Renaming the table
- Creating a new table with the old name of the previous table that contains the new column
- Copying data over from the old table to the new one
- And finally dropping the old table
This procedure is called a Batch Migration and it’s not perfect in alembic and most times you’ll still need to edit the migrations file yourself to make it work. but don’t panic I’ll hold your hand as we go along.
Alembic is what actually handles the database migration, Flask Migrate is just a library that provides better integration with flask
Edit votr.py
and include the new parameter render_as_batch
, this tells alembic to use batch migrations from now on
Setting render_as_batch
to True
should prevent this issue from happening in subsequent migrations, but we didn’t apply this fix before the initial migration, so we have to manually edit the migration file and fix that.
Go the the migrations/versions
folder and you should see a python file with a random number as the name (mine was 347f4ec5eb5e_.py
, yours would be different), this is the migration script that was automatically generated by alembic. The migration script tells alembic exactly what to do when you run flask upgrade
. A new one is generated anytime you change your models and run flask migrate
.
If you’re familiar with alembic, you could easily write one yourself.
Open the file and Edit the upgrade function to look like this:
We aren’t going to touch the downgrade
function because we have no plans of going back to the previous schema of our database before the migration
After doing all this you should be able to run flask db upgrade
without any issues.
Hurray! you just did your first database migration!
In the next section we’re going to learn about some extra commands in the flask cli.
Flask CLI
Normally you’d start a python interactive shell by typing python
in the terminal.
Since Flask 0.11 the flask shell and some new commands where added to make working with flask easier.
Now you can start an interactive python shell with:
The flask shell
command starts up an interactive Python shell like you’d expect but it also sets up the correct application context and some variables for you. Now You don’t need to import votr
anymore it’s available to you as variable called app
Let’s update the previous polls we created earlier and make them all open for voting.
That’s it. All our existing polls are now open for voting
You can now run the application with
Testing the API
To test our API we need to be able to make requests
to it. Python already has an excellent module that has the same name requests
. you can also use curl
if you’re familiar with it.
Install requests
with
Based on what we discussed earlier about SQLAlchemy and Join statements, this is how our /api/polls
route looks like like now:
Open another Python shell, import the requests module and make a get request to the url:
Nice we got what we expected from our API.
The requests module even has a nifty json
method that converts the JSON string in the response to a python dictionary.
Let’s make another request, this time a POST requests with incomplete info, to test if our validation for POST requests work, and if we can eventually create a new poll when we supply the right information.
It worked! we tested our validation by supplying invalid JSON and we got a JSON response back telling us our request failed because a particular field was empty.
and then we finally gave it the right data and it succesfully created the poll. Let’s make a GET request to be sure the poll got saved in the database.
Congratulations! you’ve just built a fully functional RESTful API, that accepts JSON and also sends JSON data back when it’s queried
You’ve now provided a universal means for any client to talk to your application…The client could be a mobile app, a Desktop app or another website!
Another Migration
Now we want to make sure duplicate options don’t get stored in our database, to do that we’re going to make the name
field in the Options
table unique, and run another migration.
Change the field declaration in the Options
model to this:
Now run:
Edit the new migrations file and change the upgrade
method to
This seems like a lot work for something simple, but don’t get it wrong. Alembic isn’t as painful to work with as this. We don’t have to edit the migrations file for every migration we do (All these issues are as a result of us using SQLite and batch migrations), if you use a more robust database like MySQL or PostgreSQL you’ll hardly come across migration issues like this.
But it’s advised to always inspect the migration file before you do any migration because sometimes alembic doesn’t detect all the changes made in a model.
Link existing options
We’ve finally set a unique constraint on our Options
table so duplicate entries won’t be allowed.
Resend the previous poll we just created
You should get a JSONDecodeError
, because we didn’t return any valid JSON back To inform us that a certain option in the poll already existed. Rather SQLAlchemy raised an exception that couldn’t be converted into a valid JSON string. switch back to the flask terminal and you should see this:
This means our unique constraint worked!
But instead of returning a messsage back saying xxxx option already exists it’s better to link that existing option to the new polls. For example “Gareth bale” is a valid option for The “Fastest player in the world” and “The best footballer in the world” (hopefully when Messi and Cristiano Ronaldo retire).
We should only return a message that an option already exists when we’re trying to create an option directly not when a new option is part of a new poll. (Our API doesn’t allow us add new options, only the admin should do that).
So if an option exists in the database instead of failing with an IntegrityError
or returning a message back, let’s link the old option with the new poll to be created.
We only need to change one thing, the dictionary comprehension that creates Poll objects from Option objects
The list comprehension reads as follows Make a Poll object out of an option if that option doesn’t exist in the database (count == 0
), if exists in the database (else
) then get that option and make a Poll object from it.
Make another post request and it should tell us the Poll was inserted successfully, even though “Gareth Bale” was part of the options
and we should get a json string telling us that it was created successfully
Lastly we need to build a new API endpoint that returns a list of all the options we have in our database, so as our users are trying to create a poll. They can see the list of all the options in our database in a drop down
Add a new endpoint to the votr.py
Now add the to_json
method on the Options model
instead of sending the database id of the record back to the user we sent back a uuid (Universally unique identification), a randomly generated number instead. Why did we do this?
It’s for security reasons. You should expose as little information about the inner workings and structure of your application as possible (though this makes it harder to accomplish some tasks) to make it difficult for hackers to know how your application works and prevent attacks. Your application might be secure in all the common areas where web most web applications are vulnerable, but you might expose certain information that might assist hackers in gaining access to important information or hacking your web application totally.
In this case, we don’t want anyone to know the total number of options we have in our database (we would have exposed that information by returning the database user id in the json string).
Ideally, The uuid should be saved in the database at the point of creation, to avoid re-generating a new one everytime a request is made
Let’s test the new endpoint
What’s left is to make sure our users can vote
To achieve this we’ll make another api endpoint and this time send a PATCH
request. a PATCH request unlike post is used to update existing information and not to create a new one entirely
Conclusion
Well, well well that’s it for this part, if you followed this tutorial through the hoops of annoying database migrations with SQLite. Then you should take sometime and congratulate yourself because you’ve just built your first REST API with flask, expanded your knowledge about SQLAlchemy and also learnt a little about alembic and how to fix migrations that go wrong.
Our API isn’t complete yet. There is no link between the users and the polls they created or a way to track and prevent users from voting twice. Infact our API currently has no link to our Authentication system. This tutorial was meant to be a gentle introduction to building API’s with flask, so we’re still going to integrate our API with our auth system later on.
In the next part of this series, we’re going to re-visit ReactJS and hook our UI to all the API calls we’ve been making with the requests module.
If you don’t have any questions or observations, I’ll see you in the next part.