Niall's Blog

A pretty common thing to do in any program is read a file line-by-line. In other interpreted or managed languages this is trivial, the standard libraries will make it super easy for you. Just look at how simple it is to do this in Python or Perl or even Shell. In C its a little more complicated, because you have to think about how much memory you need up front, and also the standard library is kind of crufty. You always have to worry about overflowing buffers or dereferencing a NULL pointer. However, there is a nice libc function available in BSD-derived platforms (including Mac OS X) - fgetln(3). This function makes it nice and easy to read arbitrary-length lines from a file in a safe way. Unfortunately, its not available in GNU libc - that is to say, if you use this function, your program won't compile on Linux. Its not a trivial libc function to port - unlike say strlcpy - since it relies on private details of the FILE structure. These private details don't happen to be the same in glibc, so it doesn't work out of the box. While GNU libc doesn't provide fgetln(3), it does provide its own similar function, getline(3). Of course, if you use this function - which is a bit uglier than fgetln(3) in my opinion - your program won't work on BSD libc systems. So basically, neither of these functions are usable if you want your program to be reasonably portable. Pretty much everything I write in C I want to work on at least Linux, the BSDs, and Mac OS X. You could write your own line reading function on top of ANSI C. Or you might be able to get away with using the existing ANSI function, fgets(3). You need to be careful with fgets, however. You can easily introduce bugs if you aren't careful to cover all the error cases. The other big problem with fgets is that you need to know the maximum length of lines you are going to read in advance, otherwise you'll end up with truncation. In most applications, you can get away with a kilobyte or two on the stack for each line and be ok. In some places, it could be a deal killer though. Anyway, here is a good idiom for using fgets:

char buf[MAXLINELEN];
while (fgets(buf, MAXLINELEN, ifp) != NULL) {
    buf[strcspn(buf, "\n")] = '\0';
    if (buf[0] == '\0')
        continue;
}

The explanation of why you use strcspn() can be found in the OpenBSD manual page.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet

Last month I started Py Web SF, the San Francisco Python & Web Technology meet-up. The idea is 1-2 conversation-style presentations of about 30 minutes with a group of 10-20 people. My hope is to have a more intimate group than the very good Bay Piggies (which I highly recommend). With a small group, it is possible to have more interaction, discussion and collaboration. In a typical lecture/audience format, people unfortunately tend to switch into "passive listener" mode.

June meet-up
Anyway, the first meet-up went extremely well - we had 15 people show up, which was a perfect number for the space. Shannon -jj Behrens gave an excellent talk on building RESTful Web services and Marius Eriksen - in fact a colleague from the OpenBSD project - gave an awesome talk on GeoDjango. Slides for both talks are online, of course.

July meet-up
Metaweb Technologies presenting a comparison of Django and Pylons. Then we have Alec Flett, another Metaweb'er, speaking about all the issues involved in scaling Python web applications.

Check it out
If you are interested in checking out the event, its July 28th, 6pm @ SF Main Public Library’s Stong Room. Full details can be found at pywebsf.org. Or if you are interested in giving a talk, just let me know.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet

I started using tmux today. It's a terminal multiplexer / task switcher for UNIX-likes, very much in the same vein as GNU Screen. However, it's a from-scratch implementation, designed to be clean, sane and easy to configure. The more liberal 3-clause BSD license is a plus also, since it means that OpenBSD has been able to integrate it into the source tree, so that it's available out of the box.

Comparison with GNU Screen
I've been a heavy screen user for many years - almost all my work is done on remote screen sessions. However, screen configuration has always been essentially black magic to me. For this reason, tmux and its nice manual page is a breath of fresh air. `tmux list-commands' is very straight forward and easy to grok. Furthermore, I like that everything in tmux is scriptable from the command line - you can run commands like `tmux resize-pane-up -t comms' to resize the pane on a session called 'comms'.

The other thing I really like about tmux is its default status bar. Some people might hate this, but I find it very useful to have a clock and a list of windows along with the process executing in them. This took quite some work to set up to my liking in GNU screen, but the default in tmux is great.

My config
One thing I don't much like is the default of C-b as the 'prefix' command. I suppose this makes some sense, since the author doesn't want to clobber GNU screen key bindings. Perhaps he will consider changing it to C-a, like in GNU screen, in the future. In any case, this isn't hard to change. Also, I am constantly using C-a C-a to switch back to the previous window - the default for this action in tmux is C-b l. Much less friendly in my opinion - of course, it's also easy to change!

So here are the contents of my $HOME/.tmux.conf:

set -g prefix C-a
bind-key C-a last-window

Getting tmux
I'm sure that packages exist for most operating systems. You can grab the source from http://tmux.sourceforge.net/. On OpenBSD, you can simply run `pkg_add -i tmux' to get the binary on your system.

UPDATE
Since OpenBSD 4.6, tmux is part of the base system. This means that if you are running OpenBSD 4.6 or later, you don't need to install any packages in order to get tmux.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet

Lots of people have a small home network. Usually you have a combo box which acts as a router/firewall/file server. Then you have a couple of other machines hooked up, and you share the Internet using NAT. A private DNS server is helpful in this kind of scenario for two reasons:

• Recursive resolver cache can speed up common DNS lookups.
• Private authoritative resolver lets you easily refer to machine in your home by name, instead of remembering IP addresses.

The DNS Dichotomy For many years there has been a dichotomy in the DNS server implementation world, pretty much between the ISC's BIND and just about everything else. The essence of this dichotomy is that BIND integrates both djbdns, has one tool - tinydns - for the authoritative portion while another - dnscache - implements the caching recursive resolver functionality. Convenience costs you security The monolithic BIND approach has certain limited benefits - mainly that it is convenient to configure and install a private DNS server which acts both as a cache and as an authority for the private domain. Unfortunately, this design has severe implications for the robustness of the software. It serves both to increase complexity within a single process while ignoring the principle of least privilege. Essentially, BIND is a horribly complicated beast, with serious security vulnerabilities being found pretty often - and even the smallest security flaw can result in major problems due to the single process design. Alternative approaches [caption id="attachment_487" align="aligncenter" width="250" caption="Unbound: A modern, secure DNS server"][/caption] While djbdns might be one of the better-known BIND alternatives, I recently came across Unbound, a BSD licensed recursive resolver. One of the authors of Unbound is also an OpenBSD developer, which inspires confidence in the security of the software. Unbound also does simple authoritative resolution One of the nifty features of Unbound is that you can very simply configure it to act as an authority for your private domains. Due to this feature, you can have a single daemon on your home network router acting as both a cache and server for your local domain. This is very nice. In fact, I have found the Unbound configuration format to be considerably nicer to deal with than that of BIND. Setup under OpenBSD This describes how I set up Unbound on my OpenBSD machine - it should be a pretty similar procedure on most other operating systems.

# install the package
$ sudo pkg_add -i unbound

Now you have the binaries on disk, you can edit the configuration to set up your private domain. Unbound runs as a recursive resolver out of the box, so this is just about all the configuration you'll need to do.

# edit the config
$ sudo vi /var/unbound/etc/unbound.conf

For a single machine, add the following under 'server', replacing 'inet' with the desired name of your local domain, and 'joust' with the name of your machine:

    local-zone: "inet." static
    local-data: "joust.inet. IN A 192.168.1.1"

Since you want the DNS server to be accessible from other machines, you probably want it to listen on 0.0.0.0 (all available interfaces). Make sure you have some kind of firewall in place before you do this, though - you don't want to let random Internet hosts query your DNS server:

    interface: 0.0.0.0
    # Make sure you have a packet filter to block queries from the Internet.
    # Alternatively, set this only for your local network.
    access-control: 0.0.0.0/0 allow

Now you can start up Unbound:

$ sudo /usr/local/sbin/unbound

And of course you probably want it to come up on boot, so follow these instructions:

$ pkg_info -D unbound
Information for inst:unbound-1.2.1p0

Install notice:
You should add:

    syslogd_flags="${syslogd_flags} -a /var/unbound/dev/log"

to /etc/rc.conf.local to create a syslog socket in the unbound chroot.

You may also want to add the following to /etc/rc.local to start unbound
at boot:

        if [ -x /usr/local/sbin/unbound ]; then
                echo -n ' unbound'; /usr/local/sbin/unbound 
        fi

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet