As a developer I have often come across situations where I have wanted to make a quick and dirty web server to allow people (or pieces of software) to post messages that can be seen from a common URL.
Maybe you want to send out a quick survey as an HTML form, or maybe you want to make some kind of a twitter-like messaging system for internal company use.
In my case I wanted to add a quick and dirty message-passing scheme to an existing internal continuous build web front-end so that users could send messages to each other about the state of the build. (“I’m fixing this crash”, or “No check-ins right now please”)
Essentially, I wanted something like a “micro blogging” application — a twiitter clone.
Searching Google for for “python twitter clone” there are plenty of hits and different directions this can take you. Right off the bat the thing that I notice is that they are almost all dependent on some framework — Flask, Django, Web2py, etc. Not wanting to learn all of the jargon associated with a framework or introduce extra unnecessary dependencies I set out to find out if there was an easy way to do this without tying myself in to a framework.
It turns out it’s not very hard to make something from scratch just using the BaseHTTPServer library that ships with Python, but I found it hard to find specific examples of a server that combined the features that I wanted:
- Taking POST’ed form data and storing it for later retreival
- Serving form data as JSON for easy use with an AJAX front end
- Persisting the data between server sessions
- Implementing queries to Add and Remove data via an HTML form or from Javascript
I came up with something that’s quite simple and and general purpose that I thought was worth sharing.
For a starting point, my reference was this site which shows how to make the BaseHTTPServer serve files and handle GET and POST request.
Since this is all very abstract, I’ll give you an idea where I’m going with this in the screenshot below:
This is the AJAX version of my Javascript front end. You can type in your name and the note that you want to post. When you press enter in the second textbox, it makes a POST request to the python server to store the data. Upon regular intervals, the Javascript does a GET query to get the full set of messages in JSON format and then updates the boxes at the bottom. So, this is a “single page web app” that you can run from two different machines (or just in two separate browser windows) and watch the messages appear on both browsers when they are entered from either one.
Here’s how I modified the do_POST member of the BaseHTTPHandler to store the form data as a dictionary
def do_POST(self):
form = cgi.FieldStorage(
fp=self.rfile,
headers=self.headers,
environ={'REQUEST_METHOD':'POST',
'CONTENT_TYPE':self.headers['Content-Type'],
})
if self.path=="/post":
formData = { key : form[key].value for key in form }
data = self.getStoredData()
data.append(formData)
self.setStoredData(data)
self.sendStoredDataAsJson()
return
The first line takes the POST’ed data and turns it in to a FieldStorage object. The only other part here that really merits explanation is the line
formData = { key : form[key].value for key in form }
which is done because the “form” variable is a dictionary where the values are objects of type MiniFieldStorage. For my purposes, all I really wanted was a dictionary with strings as the values, so the line above just unpacks the MiniFieldStorage.value and makes a dictionary from those.
the getStoredData and setStoredData functions are essentially just accessors that I created to a global list variable:
def setStoredData(self, data): global storedData print ( "setting data: " + str(data) ) storedData = data saveState() def getStoredData(self): global storedData return storedData
Since I wanted the data to persist between sessions of the server I added the call to saveState() each time the setter is called. saveState() and the corresponding loadState() which is called on startup are implemented using python’s very handle pickle serialization mechanism:
def saveState(): global storedData f = open(stateFile, "w") pickle.dump(storedData,f) f.close() def loadState(): global storedData if os.path.exists(stateFile): f = open(stateFile, "r") storedData = pickle.load(f) f.close() print ( "loaded " + str(storedData) )
In addition to the /post URL defined in the do_POST function, I also provided a /remove URL for getting rid of data elements that we no longer want:
if self.path=="/remove":
formData = { key : form[key].value for key in form }
# We should remove a data element if it matches all of the key-value pairs
# in the formData
def shouldRemove( storedFormData ):
return all ( (key in storedFormData and storedFormData[key]==formData[key]) for key in formData )
oldData = self.getStoredData()
newData = filter( lambda x: not shouldRemove(x), oldData )
self.setStoredData(list(newData))
self.sendStoredDataAsJson()
return
I did my initial testing using an old-style non-AJAX HTML form:
<h2>Static Form Interface - Boring!</h2> <div> <h3>Post a new record</h3> <form action="post" method="POST"> <p>Name<input name="name" type="text"></input></p> <p>Note<input name="note" type="text"></input></p> <input type="submit">Submit</input> </form> <hr> <h3>Remove a record</h3> <p>The server will remove all records which match the name you enter below.</p> <form action="remove" method="POST"> <p>Name<input name="name" type="text"></input></p> <input type="submit">Submit</input> </form> <hr>
On this interface, when you press the submit button it takes you away from that page and to another page with the response (sent as JSON)
After this was working I moved on to prototyping this as a single page app. The core of this is a ‘get’ request which is polled periodically. The request callback takes the json data returned from the Python server and uses JQuery to add a box for each data entry on the server (the grey blocks you see in the screenshot above):
function getData()
{
var url = "/data"
$.get( url, undefined, onDataReceived )
}
function onDataReceived( data )
{
// display pretty-printed json in HTML pre tag.
$("#data").text( JSON.stringify(data, undefined, 4) )
// display one div per entry in the data
cookedDiv = $("#cooked")
cookedDiv.empty()
if ( data )
{
for ( var i=0; i<data.length; i++ )
{
// Create a div containing the 'name' and 'note' fields
elementDiv = $("<div class='note'>").appendTo( cookedDiv )
elementDiv.html( data[i].name + "<br>" + data[i].note )
elementDiv.append("<hr>")
// add button to this div for removing this specific data entry.
var button = $("<button>").appendTo(elementDiv)
button.text("remove")
button.click( removeData(data[i].name, data[i].id) )
}
}
}
JQuery makes the POST queries to add and remove data elements straightforward:
function postData()
{
var data = {}
data["name"] = $("#name").val();
data["note"] = $("#note").val();
data["id"] = createUniqueID();
var url = "/post"
$.post( url, data )
}
function removeData( name, id )
{
return function()
{
var data = {}
if ( id != undefined )
{
data["id"] = id
}
data["name"] = name;
var url = "/remove"
$.post( url, data )
}
}
In the postData() function, I’m using JQuery to pull the values out of the form elements and stuff them into a javascript object called data, which is then shipped as the POST data.
the removeData() function works nearly the same way, except that it is a closure that binds the name/id of a specific note to the inner function, which is attached as the click handler of the remove button on each note. When clicked, the inner function posts the name/id to the /remove URL.
postData() adds an ‘id’ to each note to make them unique. To do this I used this implementation of a GUID generator I found on stack overflow, but any means of generating unique ID’s would suffice.
// GUID generator.
// Thanks broofa
// http://stackoverflow.com/questions/105034/create-guid-uuid-in-javascript
function createUniqueID()
{
var guid = 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function(c) {
var r = Math.random()*16|0, v = c == 'x' ? r : (r&0x3|0x8);
return v.toString(16);
});
return guid;
}
That’s the gist of it. To call this a “Twitter Clone” doesn’t really do justice to twitter, since it’s so minimal — it doesn’t even have user accounts! The point of this really is to make a simple mechanism for storing and persisting arbitrary form data. The HTML files above only contain fields for “name” and “note”, but the Python server is not bound to any particular set of form elements, so you could trivially add additional fields in the form without changing the server, and potentially extend this code for a wide range of applications.
The full source code is available on github:
https://github.com/uglycoyote/GenericFormDataServer
To run it, just run “python GenericFormDataServer.py” and browse to localhost:8080.
Fingertappings 