Pylons tip #5 - Streaming static files

November 04, 2009 at 07:15 PM | categories: Technical, Python | View Comments |

Pylons makes it super easy to return data to a client. You just return a string from your controller method!

class HelloController(BaseController):

    def index(self):
        return 'Hello World!'

Very nice. However, what if you want to serve up a potentially quite large file to the client? Sure, you could read the file into memory, and then return the entire buffer, but that is not very efficient. If you have a multi-megabyte file, you end up wasting lots of memory. What you want to do, actually, is read a chunk of the file at a time, and then send that. So instead of reading the entire file into memory and returning it in a single go, you do lots of little chunks. Simple conceptually. How do you do this in Pylons? Thankfully you can do this with a Python generator. Instead of returning a buffer, you return a generator:
class ImageController(BaseController):

    def index(self, id):
        ''' Stream local image contents to client '''
        try:
            imgf = open("%s/%s" %(config['image_data_dir'],
                os.path.basename(id)), 'r')
        except IOError:
            abort(404)
        def stream_img():
            chunk = imgf.read(1024)
            while chunk:
                yield chunk
                chunk = imgf.read(1024)
            imgf.close()
        return stream_img()
This works quite nicely. Hope that helps!

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

Using OpenBSD's OpenSMTPd for Email

October 31, 2009 at 01:52 PM | categories: Technical, UNIX | View Comments |

As many readers may be aware, the venerable Sendmail has been the default mail daemon in OpenBSD for years. This is largely because it is the only reasonable BSD-licensed mail server around. Personally, I have never trusted Sendmail enough to use it on any of my hosts - despite the fact that it has been audited by the OpenBSD team. It has a Byzantine configuration which I could never figure out, and perhaps more importantly has a terrible security record, owing at least partly to its monolithic single-process design. So I've always used either Qmail or more recently

Postfix is a free and open source mail transfer agent (MTA), a computer program for the routing and delivery of email. It is intended as...
Qmail has a very strange license which prevents it even being in the OpenBSD ports system. Postfix is not BSD-licensed, and so cannot be included in the base system. This means that running Postfix can be a little bit of extra work, since you have to deal with installing and upgrading packages. Wouldn't it be nice if there was a modern, simple, secure SMTP daemon in base? Now there is. New in OpenBSD 4.6 is the latest secure SMTP daemon on the block, OpenSMTPd. Turning on OpenSMTPd Sendmail is still the default MTA in base. You must follow these instructions to enable OpenSMTPd on your system:
smtpd is not enabled by default. In order to use it as the system mailer, ensure the mail queue is empty, then stop sendmail(8): # pkill sendmail Modify the current mailwrapper(8) settings by editing /etc/mailer.conf: sendmail /usr/sbin/smtpctl send-mail /usr/sbin/smtpctl mailq /usr/sbin/smtpctl makemap /usr/libexec/smtpd/makemap newaliases /usr/libexec/smtpd/makemap Rebuild the aliases database, and enable the daemon: # newaliases # echo "sendmail_flags=NO" >> /etc/rc.conf.local # echo "smtpd_flags=" >> /etc/rc.conf.local # smtpd
Note that while debugging your setup, you might find running smtpd in verbose foreground mode via `smtpd -dv' useful. OpenSMTPd configuration I'm very impressed at how simple and clean the OpenSMTPd configuration is. Check out the docs here. There are some more docs and example configs at Calomel.org. It still took me a little while to figure out a few things, so I thought I'd post my configurations to help others. Using OpenSMTPd as a Backup MX I've been using Postfix as a backup MX for unworkable.org. I decided to try OpenSMTPd in this role instead.
listen on lo0
listen on bnx0

map "aliases" { source db "/etc/mail/aliases.db" }

accept from all for local deliver to mbox
accept for all relay

accept from all for domain "unworkable.org" relay
The configuration is pretty straight forward once you are aware that the default 'from' is 'local' - that is why its necessary to add `accept from all' to accept mail from the outside world. Relaying mail to another SMTP server for delivery (nullmailer) with SSL I use Mutt as my MUA. Mutt assumes you have a local MTA to deliver mail. This means you need to use something like nullmailer or msmtp. Until now. My ISP (sonic.net) doesn't let me use port 25, so I have to relay to their SMTP server to send mail,
listen on lo0

map aliases { source db "/etc/mail/aliases.db" }
map secrets { source db "/etc/mail/secrets.db" }

accept for local deliver to maildir
accept for all relay via smtp.sonic.net ssl enable auth
The /etc/mail/secrets.db file is generated from a map, /etc/mail/secrets. This file includes your username and password - check out the smtpd.conf manual page for details.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

jQuery Freebase Suggest In Place

October 27, 2009 at 08:56 PM | categories: Technical, JavaScript | View Comments |

I just finished my hack of JQuery In Place Editor to work with Freebase Suggest. I call it "JQuery Suggest In Place". Its pretty much the same as JQuery Edit In Place, except it is stripped down a bit to only give you an input text field with a bound Freebase Suggest. You can pass along a type (or array of types) and you get back the results by supplying a callback function. Here's an example of how it works with a City/Town selector:

Click Me!
And the code to do the above:




Download the source to my jquery.suggestinplace.js hack, under BSD license.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

Read a file line by line in C - secure fgets idiom

October 03, 2009 at 02:57 PM | categories: Technical, C, UNIX | View Comments |

A pretty common thing to do in any program is read a file line-by-line. In other interpreted or managed languages this is trivial, the standard libraries will make it super easy for you. Just look at how simple it is to do this in Python or Perl or even Shell. In C its a little more complicated, because you have to think about how much memory you need up front, and also the standard library is kind of crufty. You always have to worry about overflowing buffers or dereferencing a NULL pointer. buffer-overflow However, there is a nice libc function available in BSD-derived platforms (including Mac OS X) - fgetln(3). This function makes it nice and easy to read arbitrary-length lines from a file in a safe way. Unfortunately, its not available in GNU libc - that is to say, if you use this function, your program won't compile on Linux. Its not a trivial libc function to port - unlike say strlcpy - since it relies on private details of the FILE structure. These private details don't happen to be the same in glibc, so it doesn't work out of the box. While GNU libc doesn't provide fgetln(3), it does provide its own similar function, getline(3). Of course, if you use this function - which is a bit uglier than fgetln(3) in my opinion - your program won't work on BSD libc systems. So basically, neither of these functions are usable if you want your program to be reasonably portable. Pretty much everything I write in C I want to work on at least Linux, the BSDs, and Mac OS X. You could write your own line reading function on top of ANSI C. Or you might be able to get away with using the existing ANSI function, fgets(3). You need to be careful with fgets, however. You can easily introduce bugs if you aren't careful to cover all the error cases. The other big problem with fgets is that you need to know the maximum length of lines you are going to read in advance, otherwise you'll end up with truncation. In most applications, you can get away with a kilobyte or two on the stack for each line and be ok. In some places, it could be a deal killer though. Anyway, here is a good idiom for using fgets:

char buf[MAXLINELEN];
while (fgets(buf, MAXLINELEN, ifp) != NULL) {
    buf[strcspn(buf, "\n")] = '\0';
    if (buf[0] == '\0')
        continue;
}
The explanation of why you use strcspn() can be found in the OpenBSD manual page.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

Last month I started Py Web SF, the San Francisco Python & Web Technology meet-up. The idea is 1-2 conversation-style presentations of about 30 minutes with a group of 10-20 people. My hope is to have a more intimate group than the very good Bay Piggies (which I highly recommend). With a small group, it is possible to have more interaction, discussion and collaboration. In a typical lecture/audience format, people unfortunately tend to switch into "passive listener" mode.

pywebsf

June meet-up
Anyway, the first meet-up went extremely well - we had 15 people show up, which was a perfect number for the space. Shannon -jj Behrens gave an excellent talk on building RESTful Web services and Marius Eriksen - in fact a colleague from the OpenBSD project - gave an awesome talk on GeoDjango. Slides for both talks are online, of course.

July meet-up
Metaweb Technologies presenting a comparison of Django and Pylons. Then we have Alec Flett, another Metaweb'er, speaking about all the issues involved in scaling Python web applications.

Check it out
If you are interested in checking out the event, its July 28th, 6pm @ SF Main Public Library’s Stong Room. Full details can be found at pywebsf.org. Or if you are interested in giving a talk, just let me know.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

Next Page »