I run OpenBSD on all my machines. I think its a great operating system with excellent range of features and all the components fit together nicely. One of my favourite things about OpenBSD is the highly aggressive release schedule. While a stable release is cut every 6 months, Theo is producing complete, full builds of the system for most architectures from CVS HEAD on a nearly daily basis. The entire ports tree is baked into binary packages very frequently too - although since this is much more time-consuming it is more like a full package build appears on mirrors every week or two. Such releases are called 'snapshots'.

In any case, I don't run OpenBSD 'release' or 'stable' builds on any of my machines - I run snapshots everywhere. So I am frequently downloading new snapshots. While its not exactly difficult to mirror a directory via FTP by hand, I wrote a small Python program to do it for me. The Python program has a few nice options. It defaults to using ftp.openbsd.org as the mirror, but this can be trivially overriden by the -m flag. I typically use -m rt.fm, rt.fm I have found to be an excellent mirror for the USA. The program also automatically detects the architecture of the machine you are running on - but you can override this via the -a flag. It also doesn't download the very large ISO images which are built along with the snapshots. Finally, once it has completed downloading everything, it will (if there is an MD5 file present) verify the MD5 checksums of each downloaded file.

This isn't a complicated program, but I find it useful, and I thought I'd share. Here it is in its 80-odd line entirety (or download it here).

#!/usr/bin/env python
# $Id: autosnap.py,v 1.5 2008/11/19 04:20:25 niallo Exp $

import fnmatch
import ftplib
import getopt
import hashlib
import os
import sys


# list of files not to download - globs supported
FILE_EXCEPT = ['*.iso*']

def usage():
    print >> sys.stderr, "autosnap.py [-a arch] [-d drop dir] [-m mirror] [-p path]"

def main():
    ftp = ftplib.FTP(MIRROR)
    ftp.cwd("%s/%s" %(PATH, ARCH))
    files = ftp.nlst()
    remove = []
    for p in FILE_EXCEPT:
        remove.extend(fnmatch.filter(files, p))
    for r in remove:
    for f in files:
        print "fetching file %s" %(f)
        ftp.retrbinary("RETR %s" %(f), open("%s/%s" %(DROP_DIR, f), 'wb').write, 4096)
    if 'MD5' in files:
        print "Verifying MD5sums"
        f = open("%s/MD5" %(DROP_DIR), "r")
        md5sums = {}
        for line in f:
            filename = line[line.index('(')+1:line.index(')')]
            if filename in files:
                hash = line.split('=')[1].strip()
                md5sums[filename] = str(hash)
        good = 0
        for filename in md5sums.keys():
            f = open("%s/%s" %(DROP_DIR, filename), "r")
            d = f.read()
            m = hashlib.md5()
            digest = m.hexdigest()
            if digest == md5sums[filename]:
                print "%s OK" %(filename)
                good += 1
                print "%s FAIL" %(filename)
        print "%d/%d files verified OK" %(good, len(md5sums))
        if good == len(md5sums):
if __name__ == "__main__":
        opts, args = getopt.getopt(sys.argv[1:], "a:d:m:p:")
    except getopt.GetoptError:
    for o, a in opts:
        if o == "-a":
            ARCH = a
        if o == "-d":
            DROP_DIR = a
        if o == "-m":
            MIRROR = a
        if o == "-p":
            PATH = a

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

blog comments powered by Disqus