MongoSF 2011

I was at the MongoSF 2011 conference in May. Very much enjoyed the 1-day conference, made some great connections and the 10gen folks hooked me up with O'Reilly to work on a book "MongoDB and Python", which I hope to have out in a couple of months.

My talk: MongoDB with Python, Pylons, and Pyramid

I was very happy to be invited to speak at MongoSF. My talk covers using Python with MongoDB, goes into a little bit of detail on the Web frameworks Pylons and Pyramid. Watch the video.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

 

Pylons vs Pyramid

If you weren't already aware of it, the Pylons project has recently sprouted an all-new Python Web framework called Pyramid. Pyramid is kinda like Pylons 2.0 - it is a completely new codebase with no code-level backwards compatibility. It is actually based on repoz.bfg. However, many of the concepts are very similar to the older Pylons 1.x. Pyramid is where all the new development is happening, and it has fantastic code test coverage and documentation.

You can learn more about Pyramid in the Pyramid FAQ.

What Is Akhet

On its own, Pyramid is just a framework - a set of libraries you can use. Projects are most easily started from a template. A number of different templates are included, offering different persistence options, URL mappers and session implementations.

Akhet is a Pyramid project template (AKA scaffold) which tries to give you a more Pylons 1.x-like environment. Personally, I'm much more familiar with Pylons than Pyramid and so Akhet is a great place to start.

It is well worth your while reading the Akhet documentation. However, the subject of this post, working with MongoDB, takes advantage of a few changes I have made myself to my Akhet fork on BitBucket. I am hoping Mike Orr will accept my changes upstream, but for now you will need to build the egg from my sources. I will explain how in the next section.

Starting A Pyramid Project With MongoDB

First you need to create a virtual env for your project. Depending on your platform, you should be able to install the virtualenv tool through a package manager (sudo port install virtualenv-2.7 in Mac Ports, for example).

Install Pyramid

Once you have the virtual env tool installed, create the virtualenv, install Pyramid and its dependencies into it:

$ mkdir ~/projects/ahket-example
$ virtualenv --no-site-packages myenv
$ cd myenv
$ source bin/activate
$ easy_install pyramid

Install Akhet

Now you have Pyramid and all its dependencies installed. However, you still need Akhet and its dependencies like PyMongo etc. We are going to use my fork of Akhet, because it has MongoDB support at the moment.

In a new terminal, type the following (you will need Mercurial installed to fetch the sources from BitBucket):

$ cd ~/projects/akhet-example
$ hg clone https://bitbucket.org/niallohiggins/akhet
$ cd akhet
$ python setup.py sdist

Now you have the Akhet with MongoDB support module available. Install it in your virtual env (go back to original terminal, or re-source ~/projects/akhet-example/myenv/bin/activate):

$ pip install ~/projects/akhet-example/akhet/dist/Akhet-1.0.1.tar.gz

Create Pyramid Project with Akhet and MongoDB Support

Okay, great - you can create your Pyramid project now! We shall call it "mongofu".

$ cd ~/projects/akhet-example
$ paster create -t akhet mongofu
Selected and implied templates:
  Akhet#akhet  A Pylons-like Pyramid project

Enter project name: mongofu
Variables:
  egg:      mongofu
  package:  mongofu
  project:  mongofu
Enter sqlalchemy (Include SQLAlchemy configuration? (y/n)) [True]: n
Enter mongodb (Include MongoDB configuration? (y/n)) [False]: y

Install Akhet Dependencies

You are just about ready to go. You need to make sure your virtual environment has all the dependencies needed by your mongofu project (e.g. the PyMongo driver). To do so run the following:

$ cd mongofu ; python setup.py develop

Assuming all went well, you should now have a directory called "mongofu" with a whole bunch of stuff in it. The default configuration files tell Pyramid to connect to a MongoDB server on localhost, and a database called mydb. If you need to change that, simply edit the mongodb.url and mongodb.db_name settings in the INI-files.

Start The Development Server

Now try starting up a development server:

$ paster serve development.ini
If things are working correctly, you should see a message like this on your terminal:
Starting server in PID 6020.
serving on http://127.0.0.1:5000

Talking to MongoDB with Pyramid/Akhet

Now you are ready to write some MongoDB queries! Edit the file mongofu/handlers/main.py.
In the Main class, we can add a simple query to the "index" action method, referencing the request.db property - which is a handle to our MongoDB database:

class Main(base.Handler):
    @action(renderer="index.html")
    def index(self):
        # Do some logging.
        log.debug("testing logging; entered Main.index()")
        # MongoDB Query
        # can just as easily write request.db.mycollection.find()
        records = request.db['mycollection'].find()
        # Push a flash message if query param 'flash' is non-empty.
        if self.request.params.get("flash"):
            import random
            num = random.randint(0, 999999)
            message = "Random number of the day is:  %s." % num
            self.request.session.flash(message)
            # Normally you'd redirect at this point but we have nothing to
            # redirect to.
        # Return a dict of template variables for the renderer.
        return {"project":"mongofu"}

As you can see, we specify a collection in that database to query against. We're not doing anything with the results, but that is enough to demonstrate how to talk to MongoDB with Pyramid and Akhet.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

MongoDB + Pylons at PyCon 2011

March 18, 2011 at 05:04 PM | categories: Python, MongoDB, Cloud | View Comments |

PyCon 2011

I was at the awesome PyCon 2011 last week, where I had a fantastic time. Really great conference - tons of incredible talks and wonderful conversation. Definitely the best technical conference I've attended. Everyone I met was super friendly, down-to-earth and just all-around fun.

Had the pleasure to hang out with the Pylons/Pyramid crew who are a great bunch. I will certainly be going in 2012, when the event is scheduled to be held in Santa Clara. Plan to stay for the sprints next time!

My talk: MongoDB + Pylons at Catch.com

I was very happy to be able to present a talk on some of my work at Catch.com, where I designed and implemented the platform backend using Pylons and MongoDB. The video of my talk is embedded below:

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

mongodb logo

MasterSlaveConnection: Works in Pymongo 1.6
In my previous article Python, MongoDB and Pylons, Connection handles and all that lark I described the code and configuration we use at Snaptic for working with Pylons and MongoDB. We use the MasterSlaveConnection class to automatically handle read/write splitting, because we expect to be running a single master/multiple slaves configuration in production.

Broken in Pymongo 1.7
Well, yesterday I upgraded from Pymongo 1.6 to Pymongo 1.7 (just released) and found that unfortunately all the MongoDB stuff breaks, throwing TypeError exceptions when issuing even a simple find_one() query. You will see an error like this one: TypeError: 'Database' object is not callable. If you meant to call the 'document_class' method on a 'Collection' object it is failing because no such method exists.

But fixed in tip
After talking with Mike Dirolf (Pymongo author) on the MongoDB IRC channel, he came up with a fix within a few minutes. Although I don't think its in a release yet, if you grab this commit to Pymongo from github, MasterSlaveConnection should work again. He also filed this bug to come up with a more general solution. Cheers to Mike for his super quick turnaround on fixing the bug!

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

mongodb logo

I've been doing a bunch of hacking with Pylons and MongoDB recently for some backend stuff at Snaptic. Right now we are using Paster as the webserver and the Pymongo driver. This all works fine and is pretty straightforward to set up - but there are a couple of subtleties.

MongoDB vs SQLAlchemy

If you've ever used Pylons with SQLAlchemy, then you've probably noticed the integration is quite good. All of the glue to get access via the global Session object is done for you. With MongoDB and Pylons, the integration isn't quite there yet. You have to set this up yourself.

Gotta have some replication

MongoDB doesn't give you the same kind of single-server durability guarantees that a RDBMS like MySQL or PostgreSQL does, so you pretty much have to use some kind of replication in production. I'm expecting a master/slave configuration, and Pymongo has native support for read/write splitting (writes go to the master, reads go to the slave(s)), so I've used that in my Pylons integration code.

Getting a handle to the database

The basic idea is that you define a list of masters and slaves in your Pylons config file, and some code in lib/app_globals.py sets up a global handle to the connection object.

Here is what I have at the moment, in the __init__ method in lib/app_globals.py:

# in production, we will write to the master and read from the slaves
mongo.master_host = localhost
mongo.master_port = 27017
# to supply multiple slaves, use comma as the separator.
mongo.slave_hosts = localhost
mongo.slave_ports = 27017

mongo.db = foo

Now, to get a handle to the db object from a controller context, you just do the following:
db = self._py_object.app_globals.db
# now I can go wild and run:
# db.fooCollection.find_one()
If you need a handle to the connection object instead, no problem:
db_conn = self._py_object.app_globals.db_conn
# now I can run:
# db_conn.end_request() if I want.
There might be more elegant ways to do this, but this seems to work fine for us. I will write about connection pooling and end_request() next time. Feel free to comment if I left anything out or you have questions!

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

Next Page ยป