You are probably already aware of Node.JS, unless you've been living under a rock for the past year. It's a pretty ingenious project combining the extremely fast Google V8 JavaScript engine with a highly-optimized cross-platform asynchronous networking library (libuv). The next two paragraphs will be some rambling opinions on Node.JS coming from a Python and C hacker. Then we shall get down to writing tests.

Node.JS makes it possible to write very efficient async network demons in a higher-level language than C without taking a huge performance hit. V8 is extremely fast and libuv and the NodeJS core library have been heavily optimized for minimal number of system calls. Also, since absolutely everything in the ecosystem is asynchronous from the start, you don't have to worry about things like how you can effectively talk to your database because your MySQL driver does blocking IO. Node.JS is a good fit for people who would like to leverage asynchronous IO in their program but don't want the headaches of programming in C or using a slower language like Python or Ruby which also carry a legacy of synchronous-only library code.

While I originally really disliked JavaScript as a language before learning Node.JS, it has since become a lot more enjoyable. I have found its support for object literals, closures and anonymous functions to be a great fit for async network programming. Furthermore the lightweight syntax and CommonJS module system have grown on me. Finally, I am loving the philosophy of a minimalist standard library married with an awesome, Github-integrated, local-by-default package system, npm. I also recommend using the underscore library to anybody who finds themselves missing the list and set operations from other languages.

Testing in Node.JS

Node.JS is very new, still maturing and pre 1.0 release. I initially found it hard to know where to begin with writing tests. Which test runner is standard? How do you do assertions? What about dependency injection / mocking? Are there easy ways to set up functional (ie full stack) tests?

Tip 1 - Choosing a Test Runner

It turns out that there are a whole bunch of JavaScript and Node.JS-specific test runners/frameworks. The major ones I'm aware of:

• Nodeunit - Simple, lightweight test runner for Node and browser. Supports setup and teardown and all the basic stuff you'd expect.
• Mocha - Similar to Nodeunit, few more bells and whistles (like a test-creation API) along with a zillion output formatters.
• Vows - BDD framework which I couldn't really get into personally but is clearly quite powerful and popular.

I ended up choosing Mocha and haven't regretted it. It is easy to use from the command line (reflects test status via exit code), I can use regular expressions to select narrower sets of tests and it has an API for defining tests programmatically along with machine-readable output formats. You can use a few different vocabularies (BDD, TDD, etc) to write your tests. It reminds me a bit of Nose from the Python world, but not quite as full-featured (yet). Oh, and Mocha makes it easy to test asynchronous code, too. Just remember to use the done() function, otherwise you can end up with strange bugs in your tests :-)

Tip 2 - Choosing an Assertion Library for Node.JS

In Python, the standard library contains an assertion library and often the test runner will provide some more wrappers, for example either nose.tools or unittest.TestCase. In Node, the standard library contains some basic assert functions. If you want more than that (e.g. some convenience wrappers) you'll need to look for an additional assertion library. It doesn't seem like the test runner people are in the business of providing assertion shortcuts at the moment.

I ended up going with should.js which patches the Object prototype. This enables you to write code like:

// copied from example at https://github.com/visionmedia/should.js
var user = {
      name: 'tj'
    , pets: ['tobi', 'loki', 'jane', 'bandit']
};
user.should.have.property('name', 'tj');
user.should.have.property('pets').with.lengthOf(4);

To be honest I'm not sure I like this style because it is a bit wordy and doesn't work with null or undefined. To assert that a variable is null or undefined, you have to use something else. However should.js provides some useful shortcuts so I've stuck with it.

I also came across Chai which offers the same "should" / BDD-type stuff that should.js offers, in addition to having a couple of other modes, including "assert"-style. The Chai assert-style adds a few nice convenience methods over the Node.JS stdlib assert module.

Next time I might just use the standard Node assert module or try Chai's convenience wrappers. I am not feeling amazing benefits to the should.js BDD style right now.

Tip 3 - Mocking / Stubbing / Dependency Injection in Node.JS

Let's imagine you have written a module which exports a number of functions. Those functions depend on another module which you use for communicating with an external resource - a good example is a database. Consider the following two modules:

// database.js - functions for talking to persistence store
exports.insert = function(key, value, cb) {
  // ... implementation ...
}
exports.query = function(key, cb) {
  // ... implementation ...
}

// user.js - functions to manage users of the system
var database = require('./database.js');
exports.get_user = function(username, cb) {
  database.query("user_" + username, function(err, user) {
      if (err) throw err;
      var res = {};
      // ... do some additional processing then call cb() ...
  });
}

You want to test user.js without actually talking to a real database. You want to do this to avoid the performance penalty and additional complexity involved in actually testing against a real database. Note that you may also choose to have further integration/functional/insert-jargon-here tests which do actually talk to a real database, but that isn't the case I'm talking about here.

People use different terms to describe the act of replacing the implementations of the "database.js" functions when under test. Mocking, stubbing, sandboxing, dependency injection - all terms I have heard people use. I am used to using the Python Mock module for this so I tend to use "mock". I don't think it really matters.

I searched around for a few different mocking libraries, but I ended up finding things which would build mock objects but wouldn't actually handle the cross-module dependency injection problem. For example, Sinon.js is great for building fake objects and then asserting that methods have been called a specific number of times or with particular arguments. However, it won't enable you to stub out module dependencies.

Eventually I stumbled upon @felixge's node-sandboxed-module which does exactly what I was looking for. It uses the Node.js VM module to run a module in a totally new V8 context, and then load in replacement required modules. For example, I have a module which communicates with the Heroku HTTP API. I wish to mock out actual HTTP responses from the API so that my tests run faster and I avoid being IP banned for exceeding an API call limit. My Heroku module lives in a file called "heroku.js" and it uses @mikeal's Request HTTP client library. So I use node-sandboxed-module to inject a Mock-Request instance to fake out the server response:

it('should correctly set the SSH key and persist details in the user object',
function(done) {
  // Mock the real user object with an object literal
  var user_obj = {heroku: [], save: function(cb) {cb(null, this)}};
  // Fake out the HTTP response: https://github.com/nodejitsu/mock-request
  var monkey_request = mockrequest.mock({
    protocol: "https",
    host: "api.heroku.com"
  })
    .post('/user/keys?')
    .respond(200)
    .run();
    // Make the result of "require('request')" in the heroku module under
    // test be monkey_request
  var heroku = sandboxed_module.require('../heroku.js', {
    requires: {'request':monkey_request}
  });

  // Call the function under test and assert that the expected
  // side-effects have been realized.
  heroku.setup_account_integration(user_obj, TEST_USER_API_KEY,
    function(err, user_obj) {
    user_obj.heroku[0].api_key.should.eql(TEST_USER_API_KEY);
    user_obj.heroku[0].pubkey.length.should.be.above(0);
    user_obj.heroku[0].privkey.length.should.be.above(0);
    user_obj.heroku[0].account_id.should.match(/^.*@.*$/);
    done();
  });
});

Conclusion

Compared to other environments (Rails, Pyramid, Django, etc) there is a bit more leg work to getting your tests started with Node.JS. Other environments offer a bit more of a unified, pre-baked testing setup. However, it is perfectly feasable to write a very comprehensive test suite for your application, employing unit tests, integration tests (optionally with mocks) and fullblown, full-stack functional tests.

I'm sure that in the relatively near future, people in the Node community will begin settling on tools and libraries and things will consolidate a bit more. This should make it easier for newcomers like myself to know where to start. Tests FTW!

Final note: Make sure your application supports npm test :-)

An Interview with Startup Chile Company Founder James Kennedy in Santiago

James Kennedy is the founder of Mineral Rights Worldwide. He has been a participant in the the Startup Chile program for over a year, and I spoke to him in detail about his experiences while visiting Santiago. You can find him on Twitter as @JamesKennedy.

Please install the Flash Plugin

Transcript

Niall: We're sitting here in a german beer hall in Santiago, Chile and I'm here with James Kennedy who is a fellow Irishman from Dublin, whose the founder of Mineral Rights Worldwide and he's doing the Startup Chile program. So James, tell us a bit about what you're doing here.

James: Ok well right now I'm working on Mineral Rights Worldwide, I've been here for a year and I came down as part of startup Chile last November, I'm kind of staying on in the place because I like it and I see a couple of opportunities here and now what I'm doing is building a mineral rights marketplace where yo ucan buy and sell gold and silver mines more or less. so yeah, thats what i'm doing here

Niall: Cool, yeah Chile have a lot of commodities and natural resources so it seems like it makes sense why you are doing such a thing here. So you're from Dublin, Ireland - tell us a bit about how you got into the program in the first place and what was attractive about Startup Chile.

James: Sure so about two years ago I left Ireland, I read a book called "The 4 Hour Work Week" and I wanted to work for myself, I was kinda a bootstrapper working with my girlfriend at the time, couldn't really afford to live in Ireland off our startup, which was PieHole.ie, which was like a marketplace for voiceovers. So we decanted to Argentina, lived in Argentina for a while, kind of built up the business, then we were kind of floating around - South Africa, London, and then eventually San Fran - and it was when we were in San Fran that I met Diego, one of the promoters of Startup Chile. He had this brand new idea which was basically one of the fouders of the program had studied at Stanford, he had done his MBA there, and he kind of wanted to come up with ways to promote entrepreneurship in what is traditionally a risk averse country. So he decided that rather than the traditional approach - you could say the Russian approach - is to buy a big technology transfer park beside a university, and spend a couple of billion dollars doing that, which has kind of been proven not to work. Because people think they can copy the Stanford model, they think, oh Stanford was there and they had companies in a park beside it and therefore Silicon Valley was created. But not really, thats coincidence, what really creates it is the people not the buildings. So his idea was ok why don't you just give a bunch of foreigners some money to come to Chile and artificially create this startup environment, this culture of, you know, its ok to create new business as opposed to just graduating from college and going to work for a bank. Which is a little bit what happens down here traditionally. So anyway, he gave us that whole pitch and he said, listen you guys should apply , we applied, not really expecting to get accepted but we got in, so basically within two months - I remember clearly I was sitting in a hotel room in Boulder and two things happened in the same day. I got a tax bill for about 3 times more than I had in my bank account and then we were about to jump out the window and then about 40 minutes later we got a letter from Startup Chile saying we'd been accepted.

Niall: Good timing!

James: Exactly. So I might not be here, on the bigger scale, if it wasn't for Startup chile. So anyway we ditched all our plans - we were going to go sit on the beach in Mexico for 3 months, instead we packed up all our bags and arrived in the airport in Chile, and we met all the Startup Chile team. We were kind of half expecting for it to be some elaborate scam, like if it was from Nigeria I don't know, like, oh yeah, fly down here, will give you forty thousand dollars, and it wasn't totally like that in the end, by and large we did get the forty thousand dollars and we basically poured that into what was a kind of a dude business and it meant that we were able to hire a couple of people to do our sales, it meant we were able to take a few more risks with paid advertising, do some more experimentation that we couldn't afford to do and it meant now a year later we have four people working on it and it's doing reasonably well and basically my way of looking at it is it maybe sped up our progress by about four years, because they gave you forty grand but they didn't expect any equity so we basically had nothing to lose.

Niall: Right, well that kind of goes to the next thing, which is when I first heard about Startup Chile which was a couple of years ago on Tech Crunch or something like that, it almost seemed too good to be true - you know, the deal supposedly is you get forty thousand US dollars and they take no equity which is obviously great for anyone who wants to maintain control - but how does it really work? Is there any catch - could you talk a little bit about that?

James: Its not totally - they don't give you forty thousand dollars, they subsidise you for forty thousand dollars, and there is a subtle difference there. The upside is like you said they don't take any equity , the downside is that they basically reimburse you for the money that you spend - so you spend a couple of grand on this and that and then they give the money back to you about six weeks later. So it means that you do need to have a float of about 5-10k in order to come down here to start spending the money, but once you have that its fine. And so lon gas you're spending the money on legitimate business expenses - it's very much like when you go on a business trip and you're supposed to submit your business expenses when you get back - it's very much like that. Someone will look over it and say, ok I see that you spent money on KY jelly, how were you using that in your business? How did it really help you? And if you can't come up with a good reason then they won't give you the money back. But that aside, really, the point is that you're just supposed to be here.

Niall: So you have to spend 24 weeks in Chile, that's part of it right, so you have to be here for 6 months and do you necessarily have to have a company here or do all your founders have to be here or - details like that, how does that work?

James: No, so you don't have to register a Chilean company, although worldwide income for the first three years is tax free, so you may want to consider doing that so you can avoid paying tax for three years, depending upon your own domestic tax situation. But you don't have to create any entity here - you can do it through any American or European entity you already have set up. You can send down one co-foudner, as long as you have someone here on the ground, you are supposed to send down a founder, it's not good enough to send down an employee. But you can name someone as a founder and send them down and they can use that money inside or outside the country, so you could hire developers in San Fran if you wanted to or you can outsource your work wherever the hell you like, just so long as you have an invoice. And then you're expected to basically go to networking events and mix with the locals, and kind of the things you might want to do anyway if you are looking to hire some people or make some contacts, so there is a little bit of expectation that you get out there and mingle, it's not like they fine you if you don't do that. So you can come here, sit in your apartment for six months and piss off afterwards and noone will care. And plenty of people have done that. And some people have come down here and just enjoyed their time here, networked a lot, create some connections with Chile - other people come down here and they're more focused, they're just hammering away all the time, so there's a mixture of people and I'm not really sure that one is more valid than the other.

Niall: Right, because either way you could be contributing to the scene here, bringing your own connections. And on a little bit of a different note, how've you found living here, in terms of the social life, obviously Spanish is the language here, is the language barrier an issue, is there any issue say with the banking system, you know, moving money in and out of the country, or regulation - how have you found all these kinds of things?

James: Well the team themselves handle all the visa stuff really well - within a week or two of being here I had a bank account and a visa for a year. So all the admin side of stuff is taken care of. Spanish, if you're single the best way is to find a Chilean chick or a Chilean dude to help you practice, and I can definitely see the people who have done that are a lot better at Spanish, so if that's an option, that'd be the recommended way to learn your Spanish. But there are plenty of schools down here. Obviously you can survive, I mean there's a bit of an ecosystem of English speaking people because of Startup Chile - you could basically not speak a word of Spanish for the 6 months and you could get away with it, but obviously it's a lot more fun if you give it a go.

Niall: And what do you think of Santiago as a place to live? Cost of living, the kinds of amenities available. I mean I've been here for a few days and it seems very well run and very modern in fact. What's your impression having been here for a year?

James: Well before I came to South America I literally didn't know what to expect. I was expecting to like fly into a two bit airport and see two guys beating the crap out of donkies outside but that was obviously completely ignorant and when you get here you've seen yourself, it's basically the standard of any American city. It has fantastic transport system, great underground, it's a very comfortable place, it's very western, you can get your burger, you can get your Starbucks - it's just like anywhere in the world. There is a little more culture clash in that there's a difference between the rich and the poor here. You don't really see it that much but you might see it more than you'd be used to somewhere like London or somewhere like that. But it's definitely a very comfortable place to live.

Niall: Yeah and it's very walkable too, I've been walking around a lot on foot and it has all these beautiful parks that are all very well maintained.

James: Yeah, and it's afterall a very rich country, they produce 60% of the world's copper and it means that services are very good and they have a lot of money to spend on services. You've probably seen a lot of the architecture in terms of the more modern buildings, it's pretty impressive, it's fancier than anything you'd see in a lot of other places. It's the richest place in South America and it definitely feels that way to be honest with you.

Niall: Yeah, certainly compared to Buenos Aires, you can feel there's been a lot more investment in the city. And so would you recommend the Startup Chile program to - I guess you could call them aspiring entrepreneurs? What kind of people should check out the program? Who do you think it suits?

James: I think there's a couple of different types of people it suits very well. One is if you are a tech guy and you're in the early stages of getting your prototype, your minimal viable product let's say, out and just hang out for six months and put your head down and get that done. Because your cost of living is going to be lower. I mean it's not super low compared to the rest of South America, it's like, about equivalent to about a mid to small-sized American city - I've heard people compare it to that. But of course Startup Chile give you forty grand.

Niall: That goes quite far.

James: Yeah, like you'll live off about 1,500 dollars a month quite comfortably, that's going out to eat whenever you want etc. So I'd recommend the program to tech guys who just need time to build their product. I would say bring your tech co-founders with you, so if you have a team convince them all to come down here, work here, build out the product - there are good designers here they'll help you with the design side of it or maybe some of the UX but the technical side of it - I wouldn't necessarily bank on finding a technical person to help you with it. Like I've seen people who are Biz Dev quote un quote come down here and kind of struggle trying to find someone to build their product. The other kind of group - it worked well for us, we were kind of a bootstrapping business, and it just meant that we were able to turn things up to eleven, without giving away any equity. So if you've got something that's making a couple of hundred or a couple of thousand dollars a month and you've just always wondered, well what would happen if I focused on this more full-time - then that would be a great thing to do as well. I mean they are looking for the mythical billion dollar business down here, and in terms of furhter funding, there are further sources of so-called free money, so you can get a further eighty thousand dollars of funding again for no equity if you have a good idea.

Niall: So I know that we were talking about this, you've spent several months in the Bay Area, and you're quite familiar with the scene there. Do you have any comments on how it would stack up or compare - or what's the opportunity cost of being here in Santiago, Chile versus say San Francisco , California. What are your thoughts there?

James: Well the biggest thing for me is like, an Irish guy, when I went to San Francisco I was just another wannabe tech guy. I was just one of the crowd. Which is good in that there are people there with similar interests but it is hard to stand out. Now if you compare that with down here, Startup Chile is a far more focused group of people, ironically I've made much better contacts with the Bay Area here in Chile than I had when I was in San Fran.

Niall: That's very interesting.

James: Because like people like yourself, you know, we met up because you heard I was here, we had a few beers, and that's happened several times where peopel are seeking me out to find out about Startup Chile. And also the connections you get inside Chile are pretty phenomenal. I've seen several people here start selling into companies here that they just wouldn't get a look in the door with in the States or Europe. They just wouldn't be able to get the meeting. But because you're part of the program you can get a foot in the door quite easily. One guy, he's done a deal with CNN Chile which means he has the CNN brand to use elsewhere, another guy has done a deal with like the largest retail conglomeration in South America. So, comparing the two, I'm getting far more out of it down here, because up there I'm kind of a nobody.

Niall: Sort of a small pond, big fish. That's a great summary and certainly answers a lot of my questions coming from the Bay Area. So James can be found as @JamesKennedy on Twitter and he's more than happy to answer any questions you may have. Thanks a lot James.

James: Cheers!

Over the past month or so, I have been using JQuery Mobile to build a HTML5 mobile/tablet front-end for a strength training website I've developed. While I have built native Android and iOS applications before (in Java and Objective-C, respectively) the promise of a single codebase which would work on most major platforms, as well as the ability to iterate quickly in HTML and JavaScript, was very appealing.

JQuery Mobile & HTML5

To build a mobile app using HTML5 and JavaSript, you will need to write quite a bit of JavaScript. However, the UI controls and handling for a touch device are quite different from a standard web application. You will therefore want to take advantage of some kind of existing mobile HTML5/JavaScript framework (unless you have a lot of time and want to write everything from scratch yourself). There are a number of choices out there: jQTouch, Sencha Touch etc.

I have been a fan of JQuery for many years, enjoying its minimalist philosophy and the excellent core and community-contributed features and plugins. JQuery made JavaScript development tolerable to me. So, having heard good things about JQuery Mobile from Chris McDonough (Author of the Pyramid Python Web Framework), I figured I'd give it a try.

JQuery Mobile & HTML5: 3 Advantages

1. Easy to learn and Quick to Iterate:

   After reading the JQuery Mobile docs and also reading the JQuery Mobile book from O'Reilly, I was able to build a working, rough version of my app over a weekend. That is with no prior HTML5 / JQuery Mobile experience. Compared to Android and iOS, it is far faster to build out your UI and logic using JQuery Mobile and HTML5 than either platform natively.

   I found the learning curve for Apple's Interface Builder quite steep, and it also takes time to learn the quirks of the Android layout system. Furthermore, it is quite complicated to hook up a list view (ListView on Android, UITableView on iOS) to a remote datasource, and have it looking pretty, with native code. Using JQuery Mobile, I was able to leverage all my existing JavaScript and HTML/CSS knowledge to do this very quickly. I didn't have to learn a whole new abstraction with adapters, delegates etc. I just wrote JQuery code.

2. Avoid App Store Approval Madness & Debug Build Hell:

   One of the most painful things in developing for mobile, which by definition means being on iOS, is dealing with the Apple App Store approval process. In order to get a native app to your iOS users, you must go through a lengthy process (days, possibly weeks). Not only must you jump through this hoop on initial release, but for each upgrade, also. This adds greatly to the complexity of the QA and release cycle, in addition to simply adding time. Since JQuery Mobile apps are merely a special type of web app, they inherit all the wonderful qualities of that environment: As soon as the user reloads your site, they are "upgraded" to the latest version. Pushing bugfixes and new features is instantaneous. Even on Android - which with much less stringent Market requirements is the Wild West compared to the Apple environment - it is still nice to simply have the users upgraded without even being aware it is happening.

   An extension to this is how easy it is to distribute beta/test builds to users. Simply tell them to point their browser at a URL and they have it! No DRM madness for iOS nor even APK distribution with Android to think about.

3. Cross-platform & cross-device:

   The fact that my app worked immediately on both Android and iOS - along with many other platforms - is a massive advantage. As an individual developer, it is a huge effort to maintain a separate code base for each platform. Writing a quality mobile application for a single platform is a full-time job, having to re-do that for each platform takes a lot of resources. The fact that my app will work on both my Android and iOS devices equally well is a huge bonus.

   Furthermore, especially with the massive proliferation of Android devices in all shapes and sizes, making your app look presentable on a huge variety of screen resolutions is a real challenge. For serious Android developers, device fragmentation in terms of screen size (scaling from the downright tiny to quite large) actually represents a considerable development cost. With the browser rendering your application in a way that looks reasonable on every device, you don't have to worry about this to anywhere near the same extent.

JQuery Mobile & HTML5: 3 Disadvantages

1. Slower Than Native:

   The biggest drawback in my opinion is that even on the latest Android and iOS hardware (dual core Tegra 2 Android phone, dual core iPad2 tablet), JQuery Mobile applications feel noticeably slower than native. Especially on Android, where the browser is surprisingly (since Google is a web-focused company) far slower and buggier than on iOS. I haven't yet tested my app on older Android devices but it may be downright unusable there (e.g. G1 Android phone).

   I believe that in the next 12-24 months hardware will catch up sufficiently (e.g. quad-core devices coming in 2011) that this will be less of an issue. But today, it is a definite drawback. Also, if you are solely targetting iOS, you can count on the performance of the browser being at least reasonable (unlike Android, BlackBerry, etc).

2. Quirky:

   JQuery Mobile is still beta software, and as such I have encountered bugs. That being said, the team is extremely responsive in addressing issues on GitHub. I think one of the big problems is simply quirks across various browsers on different platforms. I am certain that this will be ironed out over time. The applications also can look quirky in a visual sense - while I think the JQuery Mobile team have done a great job with their widgets and themes, it is still fairly distinct from native. How much this really matters to users is not clear, but it is something to be aware of.

3. Limited Capabilities vs Native:

   Obviously, JavaScript running in a browser does not have full access to many features of the device. A common example being the camera. However, tools like PhoneGap help greatly with many common issues. Indeed, I have started working on deploying versions of my app for iOS and Android with PhoneGap - using native Facebook bindings - and am quite impressed. I will be writing about my experiences with PhoneGap and native bindings in future posts.

Conclusion

To sum it up, I believe that JQuery Mobile and HTML5 is a viable platform for mobile application development. However, it does not (yet) suit all classes of applications. For simple content display and data entry-type apps (as opposed to very rich multimedia/gaming applications) it can be a great force multiplier compared with native. I certainly am happy with how it has turned out for my app - I simply do not have the time to maintain a port of my strength tracker app for both Android and iOS.

Over the course of the next 1-2 years, as hardware gets faster and devices proliferate, I believe HTML5 (and JQuery Mobile, PhoneGap, etc) will be even more important technologies in the mobile app space.

I was at the MongoSF 2011 conference in May. Very much enjoyed the 1-day conference, made some great connections and the 10gen folks hooked me up with O'Reilly to work on a book "MongoDB and Python", which I hope to have out in a couple of months.

My talk: MongoDB with Python, Pylons, and Pyramid

I was very happy to be invited to speak at MongoSF. My talk covers using Python with MongoDB, goes into a little bit of detail on the Web frameworks Pylons and Pyramid. Watch the video.

Pylons vs Pyramid

If you weren't already aware of it, the Pylons project has recently sprouted an all-new Python Web framework called Pyramid. Pyramid is kinda like Pylons 2.0 - it is a completely new codebase with no code-level backwards compatibility. It is actually based on repoz.bfg. However, many of the concepts are very similar to the older Pylons 1.x. Pyramid is where all the new development is happening, and it has fantastic code test coverage and documentation.

You can learn more about Pyramid in the Pyramid FAQ.

What Is Akhet

On its own, Pyramid is just a framework - a set of libraries you can use. Projects are most easily started from a template. A number of different templates are included, offering different persistence options, URL mappers and session implementations.

Akhet is a Pyramid project template (AKA scaffold) which tries to give you a more Pylons 1.x-like environment. Personally, I'm much more familiar with Pylons than Pyramid and so Akhet is a great place to start.

It is well worth your while reading the Akhet documentation. However, the subject of this post, working with MongoDB, takes advantage of a few changes I have made myself to my Akhet fork on BitBucket. I am hoping Mike Orr will accept my changes upstream, but for now you will need to build the egg from my sources. I will explain how in the next section.

Starting A Pyramid Project With MongoDB

First you need to create a virtual env for your project. Depending on your platform, you should be able to install the virtualenv tool through a package manager (sudo port install virtualenv-2.7 in Mac Ports, for example).

Install Pyramid

Once you have the virtual env tool installed, create the virtualenv, install Pyramid and its dependencies into it:

$ mkdir ~/projects/ahket-example
$ virtualenv --no-site-packages myenv
$ cd myenv
$ source bin/activate
$ easy_install pyramid

Install Akhet

Now you have Pyramid and all its dependencies installed. However, you still need Akhet and its dependencies like PyMongo etc. We are going to use my fork of Akhet, because it has MongoDB support at the moment.

In a new terminal, type the following (you will need Mercurial installed to fetch the sources from BitBucket):

$ cd ~/projects/akhet-example
$ hg clone https://bitbucket.org/niallohiggins/akhet
$ cd akhet
$ python setup.py sdist

Now you have the Akhet with MongoDB support module available. Install it in your virtual env (go back to original terminal, or re-source ~/projects/akhet-example/myenv/bin/activate):

$ pip install ~/projects/akhet-example/akhet/dist/Akhet-1.0.1.tar.gz

Create Pyramid Project with Akhet and MongoDB Support

Okay, great - you can create your Pyramid project now! We shall call it "mongofu".

$ cd ~/projects/akhet-example
$ paster create -t akhet mongofu
Selected and implied templates:
  Akhet#akhet  A Pylons-like Pyramid project

Enter project name: mongofu
Variables:
  egg:      mongofu
  package:  mongofu
  project:  mongofu
Enter sqlalchemy (Include SQLAlchemy configuration? (y/n)) [True]: n
Enter mongodb (Include MongoDB configuration? (y/n)) [False]: y

Install Akhet Dependencies

You are just about ready to go. You need to make sure your virtual environment has all the dependencies needed by your mongofu project (e.g. the PyMongo driver). To do so run the following:

$ cd mongofu ; python setup.py develop

Assuming all went well, you should now have a directory called "mongofu" with a whole bunch of stuff in it. The default configuration files tell Pyramid to connect to a MongoDB server on localhost, and a database called mydb. If you need to change that, simply edit the mongodb.url and mongodb.db_name settings in the INI-files.

Start The Development Server

Now try starting up a development server:

$ paster serve development.ini

Starting server in PID 6020.
serving on http://127.0.0.1:5000

Talking to MongoDB with Pyramid/Akhet

Now you are ready to write some MongoDB queries! Edit the file mongofu/handlers/main.py.
In the Main class, we can add a simple query to the "index" action method, referencing the request.db property - which is a handle to our MongoDB database:

class Main(base.Handler):
    @action(renderer="index.html")
    def index(self):
        # Do some logging.
        log.debug("testing logging; entered Main.index()")
        # MongoDB Query
        # can just as easily write request.db.mycollection.find()
        records = request.db['mycollection'].find()
        # Push a flash message if query param 'flash' is non-empty.
        if self.request.params.get("flash"):
            import random
            num = random.randint(0, 999999)
            message = "Random number of the day is:  %s." % num
            self.request.session.flash(message)
            # Normally you'd redirect at this point but we have nothing to
            # redirect to.
        # Return a dict of template variables for the renderer.
        return {"project":"mongofu"}

As you can see, we specify a collection in that database to query against. We're not doing anything with the results, but that is enough to demonstrate how to talk to MongoDB with Pyramid and Akhet.

Today Samsung announced a redesigned Galaxy Tab 10.1 and 8.9. After the iPad2 announcement earlier this month pretty much killed the first round of over-priced, under-achieving Android tablets (here's looking at you, $800 Xoom) it is great to see Samsung coming out with a much more competitive offering.

Competitive Tablet Pricing

One of the most disappointing things about the Motorola Xoom was simply the rather insane $800 price tag. The Xoom is a nice piece of hardware don't get me wrong, but there is no way to justify the 60% higher price over the entry-level 16G iPad2. While they're releasing a $599 32G WiFi-only version in the next couple of weeks - which is somewhat competitive with the $599 32G iPad 2 - it feels like too little too late.

Furthermore, the honest truth is that Honeycomb isn't yet ready for primetime - with a shortage of apps and a beta-quality OS that often experiences crashes. Of course, Google are going to fix those bugs over the coming months and slick, native Honeycomb apps are under frantic development at this very moment. Still though, that doesn't change the situation today: Tablets are all about the apps, and it is hard to ignore the 1 year head-start of the Apple iPad tablet app ecosystem.

For this reason, it is fantastic to see Samsung understand that they simply cannot charge more than Apple and are releasing models at $499 and $469 for 16G WiFi-Only 10.1" and 8.9" tabs respectively.

Form Factor

I find it refreshing to see Samsung releasing an 8.9" tab in addition to the more iPad-like 10.1" offering. I've found the 7" Nook Color to be a real joy to hold with one hand unlike iPad and Xoom which are just a bit too big for that. Am hopeful that the 8.9" model is just about right for one-handed use also.

Display Technology

Samsung are well-known for their screen technology and their latest development is Super PLS or Plane-to-Line-Switching for LCD displays. Compared to current In-Plane-Switching (IPS) panels, as found in the Motorola DROID or the iPad 2, Super PLS displays offer about a 100% improvement in viewing angles and a 15% improvement in brightness. Furthermore, they are available in 1280x800 resolution as opposed to the iPad 2's lower 1024x768.

It will be interesting to see if the screen alone could make the Samsung devices quite a bit more compelling. Apple has already been receiving criticism for giving the iPad 2 the same display as their first-gen iPad instead of offering Retina Display which they feature in their iPhone 4.

Video playback: Tegra 2 or Exynos?

As I wrote in my previous article, 3 Surprising Things About Video On Android Tablets, video is an obvious killer app for tablets but the Tegra 2 chip used in many of the Honeycomb devices has some severe limitations in video playback.

I speculated in that post that Samsung's then-rumoured 8.9" device might use their own Exynos CPU in place of Tegra 2 and it appears that may now indeed be the case. Samsung have so far been vague about the specifics of what chip they are putting into the device saying only that it has a dual-core processor. Analysts suspect this is due to shortages in their own CPU which may be made up on a per-region basis by some models having Tegra 2.

If the 8.9" Galaxy Tab does ship with Exynos as opposed to Tegra 2 and carries DivX certification, that could make it a far superior device for video playback. It would beat both the current crop of Honeycomb tablets in addition to the Apple iPad2 as much more efficient video streams would be supported. Beyond higher quality streaming, it would also largely eliminate the need for time consuming and tedious re-encoding of video files prior to playback on the device.

Conclusion

To wrap up, the Samsung announcement today is certainly exciting. The Samsung devices look like they are shaping up to be competitive with Apple's iPad2 in industrial engineering, features and price.

However, with a June 8th US release date for the 10.1" model and a still-vague date for the more interesting 8.9" tab we are still left with quite a wait for a truly compelling and competitive Honeycomb offering.

I was at the awesome PyCon 2011 last week, where I had a fantastic time. Really great conference - tons of incredible talks and wonderful conversation. Definitely the best technical conference I've attended. Everyone I met was super friendly, down-to-earth and just all-around fun.

Had the pleasure to hang out with the Pylons/Pyramid crew who are a great bunch. I will certainly be going in 2012, when the event is scheduled to be held in Santa Clara. Plan to stay for the sprints next time!

My talk: MongoDB + Pylons at Catch.com

I was very happy to be able to present a talk on some of my work at Catch.com, where I designed and implemented the platform backend using Pylons and MongoDB. The video of my talk is embedded below:

After a decade or so of various failed or underwhelming attempts, it is now blatantly obvious that Apple kicked off the consumer tablet revolution with their launch of iPad in 2010. By getting the hardware, pricing, UI and application ecosystem right, they have managed to sell 15 million units in 2010 alone (April - December), bringing in $9.5 billion.

An obvious killer app for this kind of device is video. At the distance you hold a tablet it has the same apparent size as a massive TV screen. The easy-to-hold and carry form factor, the long battery life, the high quality screen and speakers - and apps like Hulu Plus, Netflix, ABC and CBS pretty much have the TV and movie situation covered, with more services and apps coming all the time, to both iOS and Android devices.

In the not-too-distant future, practically everyone will have a tablet, and everyone will want to watch video on them. Awesome!

1) NVIDIA Tegra2 Chip Performance Surprises: Not What They Claim To Be

You've probably noticed the recent explosion in Android handsets and tablets powered by the NVIDIA Tegra 2 chip. It offers extremely impressive performance with a dual core ARM Cortex-A90 clocked at 1Ghz, and powers the latest devices including the Motorola Xoom, Atrix, Bionic and Samsung Galaxy Tab 10.1 along with a host of others just-released or near-release.

Tegra 2's Dirty Little Secret
What you may not be aware of amidst all the marketing is how poorly these Tegra 2 chips decode video. Tegra 2 is not capable of playing just any old video format. While technically it supports H.264 at 720p and 1080p, it specifically only supports the "Baseline" profile which means two reference frames and no B frames.

This means you must take your existing HD video files and re-encode them into a special format using a program like handbrake. According to Anandtech, that puts a 2 hour movie at around 8 GB - too big to fit as a single file on a typical microSD card formatted with FAT32.

Additionally, with those kinds of space requirements, streaming 1080p HD video for viewing on your device quickly becomes extremely bandwidth intensive (~10mbps). Almost certainly too high for most consumer Internet connections - according to DSL reports average Internet connection speed in USA is 4.7Mbps.

In other words, your brand spanking new, top-of-the-line 2011 dual core 1Ghz tablet cannot simply play any 720p/1080p HD video file you throw at it. Unlike a cheap PC or media center, in order to watch video on your shiny new Tegra 2-based tablet you:

• Almost certainly will have to re-encode your video specially for the device.
• Need a ton of storage space, approx. 8G per movie, possibly having the video split across multiple files.
• If streaming true HD video in 1080p baseline profile, need a very fast (~10mbps) Internet connection. (FWIW BBC iPlayer HD streams are 720p @ 3.2 MBPS. These HD streams are not available on iPad since that only supports baseline too!)

Ouch! Rumour suggests this is one of the reasons why HTPC (Home Theater PC) manufacturer Boxee ditched the Tegra 2 platform for the Intel Atom.

2) Samsung's Hummingbird CPU: DivX Certification

An interesting tidbit about Samsung's Android devices: Many of them - including 7" Galaxy Tab and the Galaxy S phone - are DivX certified! What does this mean? It means they can play pretty much any old HD video file (at least 720p high profile) you throw at them without the friction of painful, space-wasting conversion processes!

What about the soon-to-be-released Galaxy Tab 10.1? We don't know exactly, however all information so far suggests it will be powered by the same NVIDIA Tegra 2 chip. Unless Samsung does something custom, this device will have the same video limitations as the Motorola Xoom.

However, there is speculation that the other tablets Samsung is planning - in particular an 8.9" device - may use the next generation Samsung CPU named Exynos. This dual-core CPU could well have the same DivX certification as the Hummingbird devices, offering these devices superior video playback capabilities compared to Tegra 2.

This may mean that consumers interested in good video playback support should hold off on the otherwise very promising Galaxy Tab 10.1 and wait for an Exynos-powered device instead.

3) Tegra 2 Honeycomb Exclusivity: But Won't Last Long

Android 3.0 AKA Honeycomb is the first tablet-optimized version of the OS. If you want to be able to truly take advantage of the extra screen real estate afforded by the tablet form-factor, you will want a device with Honeycomb.

The first round of Android 3.0 tablets are all powered by Tegra 2. According to the SVP of Cellular Products at Qualcomm the reason for this is that Google partnered with Motorola on the initial launch devices. For whatever reason, Motorola chose Tegra. However, Honeycomb will soon be ported to and running on processors from other vendors.

As soon as Google open-sources Honeycomb, we should see a flood of Android 3.0 tablets from vendors like Acer, Asus, HTC, Samsung and others with much better video playback support - which means much better value for consumers!

MasterSlaveConnection: Works in Pymongo 1.6
In my previous article Python, MongoDB and Pylons, Connection handles and all that lark I described the code and configuration we use at Snaptic for working with Pylons and MongoDB. We use the MasterSlaveConnection class to automatically handle read/write splitting, because we expect to be running a single master/multiple slaves configuration in production.

Broken in Pymongo 1.7
Well, yesterday I upgraded from Pymongo 1.6 to Pymongo 1.7 (just released) and found that unfortunately all the MongoDB stuff breaks, throwing TypeError exceptions when issuing even a simple find_one() query. You will see an error like this one: TypeError: 'Database' object is not callable. If you meant to call the 'document_class' method on a 'Collection' object it is failing because no such method exists.

But fixed in tip
After talking with Mike Dirolf (Pymongo author) on the MongoDB IRC channel, he came up with a fix within a few minutes. Although I don't think its in a release yet, if you grab this commit to Pymongo from github, MasterSlaveConnection should work again. He also filed this bug to come up with a more general solution. Cheers to Mike for his super quick turnaround on fixing the bug!

I've been doing a bunch of hacking with Pylons and MongoDB recently for some backend stuff at Snaptic. Right now we are using Paster as the webserver and the Pymongo driver. This all works fine and is pretty straightforward to set up - but there are a couple of subtleties.

If you've ever used Pylons with SQLAlchemy, then you've probably noticed the integration is quite good. All of the glue to get access via the global Session object is done for you. With MongoDB and Pylons, the integration isn't quite there yet. You have to set this up yourself.

MongoDB doesn't give you the same kind of single-server durability guarantees that a RDBMS like MySQL or PostgreSQL does, so you pretty much have to use some kind of replication in production. I'm expecting a master/slave configuration, and Pymongo has native support for read/write splitting (writes go to the master, reads go to the slave(s)), so I've used that in my Pylons integration code.

The basic idea is that you define a list of masters and slaves in your Pylons config file, and some code in lib/app_globals.py sets up a global handle to the connection object.

Here is what I have at the moment, in the __init__ method in lib/app_globals.py:

# in production, we will write to the master and read from the slaves
mongo.master_host = localhost
mongo.master_port = 27017
# to supply multiple slaves, use comma as the separator.
mongo.slave_hosts = localhost
mongo.slave_ports = 27017

mongo.db = foo

db = self._py_object.app_globals.db
# now I can go wild and run:
# db.fooCollection.find_one()

db_conn = self._py_object.app_globals.db_conn
# now I can run:
# db_conn.end_request() if I want.