Niall's Blog

Pylons vs Pyramid

If you weren't already aware of it, the Pylons project has recently sprouted an all-new Python Web framework called Pyramid. Pyramid is kinda like Pylons 2.0 - it is a completely new codebase with no code-level backwards compatibility. It is actually based on repoz.bfg. However, many of the concepts are very similar to the older Pylons 1.x. Pyramid is where all the new development is happening, and it has fantastic code test coverage and documentation.

You can learn more about Pyramid in the Pyramid FAQ.

What Is Akhet

On its own, Pyramid is just a framework - a set of libraries you can use. Projects are most easily started from a template. A number of different templates are included, offering different persistence options, URL mappers and session implementations.

Akhet is a Pyramid project template (AKA scaffold) which tries to give you a more Pylons 1.x-like environment. Personally, I'm much more familiar with Pylons than Pyramid and so Akhet is a great place to start.

It is well worth your while reading the Akhet documentation. However, the subject of this post, working with MongoDB, takes advantage of a few changes I have made myself to my Akhet fork on BitBucket. I am hoping Mike Orr will accept my changes upstream, but for now you will need to build the egg from my sources. I will explain how in the next section.

Starting A Pyramid Project With MongoDB

First you need to create a virtual env for your project. Depending on your platform, you should be able to install the virtualenv tool through a package manager (sudo port install virtualenv-2.7 in Mac Ports, for example).

Install Pyramid

Once you have the virtual env tool installed, create the virtualenv, install Pyramid and its dependencies into it:

$ mkdir ~/projects/ahket-example
$ virtualenv --no-site-packages myenv
$ cd myenv
$ source bin/activate
$ easy_install pyramid

Install Akhet

Now you have Pyramid and all its dependencies installed. However, you still need Akhet and its dependencies like PyMongo etc. We are going to use my fork of Akhet, because it has MongoDB support at the moment.

In a new terminal, type the following (you will need Mercurial installed to fetch the sources from BitBucket):

$ cd ~/projects/akhet-example
$ hg clone https://bitbucket.org/niallohiggins/akhet
$ cd akhet
$ python setup.py sdist

Now you have the Akhet with MongoDB support module available. Install it in your virtual env (go back to original terminal, or re-source ~/projects/akhet-example/myenv/bin/activate):

$ pip install ~/projects/akhet-example/akhet/dist/Akhet-1.0.1.tar.gz

Create Pyramid Project with Akhet and MongoDB Support

Okay, great - you can create your Pyramid project now! We shall call it "mongofu".

$ cd ~/projects/akhet-example
$ paster create -t akhet mongofu
Selected and implied templates:
  Akhet#akhet  A Pylons-like Pyramid project

Enter project name: mongofu
Variables:
  egg:      mongofu
  package:  mongofu
  project:  mongofu
Enter sqlalchemy (Include SQLAlchemy configuration? (y/n)) [True]: n
Enter mongodb (Include MongoDB configuration? (y/n)) [False]: y

Install Akhet Dependencies

You are just about ready to go. You need to make sure your virtual environment has all the dependencies needed by your mongofu project (e.g. the PyMongo driver). To do so run the following:

$ cd mongofu ; python setup.py develop

Assuming all went well, you should now have a directory called "mongofu" with a whole bunch of stuff in it. The default configuration files tell Pyramid to connect to a MongoDB server on localhost, and a database called mydb. If you need to change that, simply edit the mongodb.url and mongodb.db_name settings in the INI-files.

Start The Development Server

Now try starting up a development server:

$ paster serve development.ini

Starting server in PID 6020.
serving on http://127.0.0.1:5000

Talking to MongoDB with Pyramid/Akhet

Now you are ready to write some MongoDB queries! Edit the file mongofu/handlers/main.py.
In the Main class, we can add a simple query to the "index" action method, referencing the request.db property - which is a handle to our MongoDB database:

class Main(base.Handler):
    @action(renderer="index.html")
    def index(self):
        # Do some logging.
        log.debug("testing logging; entered Main.index()")
        # MongoDB Query
        # can just as easily write request.db.mycollection.find()
        records = request.db['mycollection'].find()
        # Push a flash message if query param 'flash' is non-empty.
        if self.request.params.get("flash"):
            import random
            num = random.randint(0, 999999)
            message = "Random number of the day is:  %s." % num
            self.request.session.flash(message)
            # Normally you'd redirect at this point but we have nothing to
            # redirect to.
        # Return a dict of template variables for the renderer.
        return {"project":"mongofu"}

As you can see, we specify a collection in that database to query against. We're not doing anything with the results, but that is enough to demonstrate how to talk to MongoDB with Pyramid and Akhet.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

I've been playing with the recently-released HTTP API for accessing the Best Buy product catalog. While its a little strange to use at first, its actually pretty useful. One of the things I am interested in is online retail, specifically how to make Internet shopping easier. Lets imagine I am looking for information on a particular digital camera - the Nikon Coolpix S210.

A Python skeleton
First, lets get our little Python test harness together. Also, you are going to need your own Best Buy Remix API key. Here is a skeletal Python HTTP client:

import httplib

API_KEY=''

QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"
OPTS="?sort=name.desc&show=all&format=json"
c = httplib.HTTPConnection('api.remix.bestbuy.com')
c.request('GET', "%s%s&apiKey=%s" %(QUERY, OPTS, API_KEY))
r = c.getresponse()
data = r.read()
print data

Our first Best Buy query
Now lets try to write a sample query for the Nikon Coolpix S210. Although the Best Buy Remix API docs are a bit sparse, we can guess that items must have an attribute called 'name'. In fact they do! So lets try searching for the camera by name.

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Nikon Coolpix S210)"

$ python bb.py
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>400 - Bad Request</title>
 </head>
 <body>
  <h1>400 - Bad Request</h1>
 </body>
</html>

Gimme everything!
It turns out that Best Buy don't name their products in the most intuitive way. Lets try a wildcard on just `Coolpix' instead:

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Coolpix*)"

This time, we are going to get tons of data back, in JSON format. Best Buy remix defaults to XML, but I prefer JSON so I added the format=json parameter to the query. Ok, so now we have an overwhelming amount of data on Coolpix cameras - but we really just want information for the S210.

Best Buy's quirky product schema
Well, there is a solution. Best Buy don't store the model number in the `name' attribute - instead they store it in a separate `modelNumber' attribute. If we query for name=Coolpix* AND modelNumber=S210, we should get the expected result, finally:

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"

Et voila! Now Best Buy gives us back all the information it has about the Nikon Coolpix S210. This is pretty detailed stuff, including all those details like compatible memory formats, digital zoom, along with the price and availability. Very cool! Just for kicks, lets show the whole script to send a query to Best Buy, parse the JSON response, and finally print the price:

import httplib
import json

API_KEY=''
QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"
OPTS="?sort=name.desc&show=all&format=json"
c = httplib.HTTPConnection('api.remix.bestbuy.com')
c.request('GET', "%s%s&apiKey=%s" %(QUERY, OPTS, API_KEY))
r = c.getresponse()
data = r.read()

camera_info = json.loads(data)

print "price: %s"%(camera_info['products'][0]['regularPrice'])

$ python bb.py
price: 89.99

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

On ORMs
It so happens that I end up dealing with the Python ORM SQLObject pretty often. I don't really like ORMs very much, since in my experience they make those 80% of database things that are already easy to do with plain SQL easier, while making the other 20% of database things which are already hard impossible. They do save some boiler-plate, and let you express your schema and queries in Python (or whatever programming language you are using) instead of SQL - but this tends to break down at a certain level of complexity, and just gets in the way. You end up doing a huge amount of wrangling with the ORM to do something which is very simple in plain old SQL. Fundamentally, SQL was designed as a declarative query language for the relational model and not to represent object hierarchies in the same way programming languages do, hence ORMs are always going to be a nasty hack in my opinion.

SQLObject vs. SQLAlchemy
I actually much prefer SQLAlchemy to SQLObject. SQLAlchemy has a more explicit divide between its various components - you aren't forced to use the ORM stuff if you don't want to. It can be used just as a handy database abstraction layer with programmatic SQL and connection pooling and so on if you want. And if you truly want to go the full ORM mapping route, they provide that too. For truly tricky things, SQLAlchemy will be happy to provide you with a DB-API 2.0 cursor so that you can execute whatever custom SQL you wish.

Monolithic vs. modular
This is my main problem with SQLObject - its very difficult to figure out how to get at the underlying database connection. I don't know how its possible to use the connection pooling and programmatic SQL builder without using the ORM but perhaps it is doable. The documentation for SQLObject is far inferior to the documentation of SQLAlchemy I'm sorry to say. Just try to figure out how to use transactions reliably with SQLObject! Even when I managed to put together some code which according to the documentation should work, SQLObject decided to interleave the actions in separate transactions. With SQLAlchemy I never had this problem.

Getting at the cursor
While seemingly undocumented, it is in fact possible to get the underlying driver's connection object, and from there grab a DB-API cursor. The pattern is:

# Set up the SQLObject connection as usual
connection = connectionForURI('sqlite:/:memory:')
# Grab the database connection
dbobj = connection.getConnection()
# Get a cursor from the low-level driver
cursor = dbobj.cursor()
# 
cursor.close()

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

SQLite is an extremely useful little database. It has a nifty bunch of features and is super simple to set up. Using SQLite reduces the cost of developing and maintaining a powerful SQL database even more than traditional free RDBMS' like PostgreSQL and MySQL. Your database is simply an on-disk file - no need to configure user accounts, connection strings or any of that stuff. While this has its limitations, I have found that SQLite is more than sufficient for many web applications. These days I like to write my web applications in Pylons. Python has a full-featured SQLite module which conforms to DB-API 2.0. To use SQLite in a Pylons application, all you need to do is open the SQLite database file. I use a little convenience function to do this:

import sqlite3
def open_db():
    conn = sqlite3.connect(config['sqlite_db_file'])
    return conn

[app:main]
use = egg:MyApp
full_stack = true
cache_dir = %(here)s/data
beaker.session.key = mykey
beaker.session.secret = somesecret
sqlite_db_file = /path/to/your/sqlite.db

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Freebase Suggest is a little JavaScript (and jQuery) widget for adding Freebase's auto-complete and search API reconciliation features to any of your text boxes. While this might sound a bit strange or pointless initially, its actually incredibly useful in many applications Imagine you are running a site for your own music reviews. You have a nice web form for writing the reviews, which includes fields for artist name and album. It can be a pain sometimes to remember the exact album title or band name, and you probably find yourself having to double-check this a few times before posting your review. However, chances are high that your artist and album are already available in Freebase, and this is where the Freebase Suggest widget comes in. You simply attach the widget to your artist field, and your album field. Now by default, you can auto-complete to any topic in Freebase. This is pretty good, but wait there's more! You can trivially nail down the types returned by the auto-complete service to just be, say, /music/album. You could do the same for your musical artist box by limiting to /music/artist. To see what this Freebase Suggest widget looks like in this configuration, check out some of the examples. So now you know how to add Freebase Suggest to your web forms to reduce friction on input. You also know how to nail down the auto-complete results to a specific type. But what if you have a single input box which accepts, say a more generic 'media title'. This could be a film name, a TV show or a music album. Again, it would be really nice here to be able to use Freebase Suggest to leverage the already-entered and verified data of Freebase to ease input in your form - but how do you make it work with multiple types? This is something I wanted to do in one of my own applications today, and its very easy! In order to limit your Freebase Suggest results to a specific type, you already have some code setting up the ac_param dictionary passed to the widget. That code probably looks something like this:

var o = { type:'/music/album' };
$(id).freebaseSuggest({ac_param:o});
$(id).bind("fb-select", suggest_cb);

var o = { type:'/music/album', 
          type:'/film/film',
          type:'/tv/tv_program' };
$(id).freebaseSuggest({ac_param:o});
$(id).bind("fb-select", suggest_cb);

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.