Niall O'Higgins

Py Web SF: The San Francisco Python & Web Technology Meet-up

Fri, 24 Jul 2009 19:29:47 PDT

Last month I started Py Web SF, the San Francisco Python & Web Technology meet-up. The idea is 1-2 conversation-style presentations of about 30 minutes with a group of 10-20 people. My hope is to have a more intimate group than the very good Bay Piggies (which I highly recommend). With a small group, it is possible to have more interaction, discussion and collaboration. In a typical lecture/audience format, people unfortunately tend to switch into "passive listener" mode.

June meet-up
Anyway, the first meet-up went extremely well - we had 15 people show up, which was a perfect number for the space. Shannon -jj Behrens gave an excellent talk on building RESTful Web services and Marius Eriksen - in fact a colleague from the OpenBSD project - gave an awesome talk on GeoDjango. Slides for both talks are online, of course.

July meet-up
Metaweb Technologies presenting a comparison of Django and Pylons. Then we have Alec Flett, another Metaweb'er, speaking about all the issues involved in scaling Python web applications.

Check it out
If you are interested in checking out the event, its July 28th, 6pm @ SF Main Public Libraryâ€™s Stong Room. Full details can be found at pywebsf.org. Or if you are interested in giving a talk, just let me know.

Windows, P2P, network freezes and the TCP `half-open' state

Mon, 18 May 2009 21:27:23 PDT

Have you ever tried to run a very network-intensive P2P application on Windows XP SP2 or higher? If so, you may have encountered very strange behaviour with the Windows TCP/IP network stack. Specifically, you won't be able to open any new TCP/IP connections, so web-browsing, email checking, SSH, etc will all be basically unusable. At the same time, P2P transfers might be running fine for all you can tell. I spent some time trouble-shooting all aspects of my set up including testing the router, my Internet connection, and even re-installing Windows a couple of times - all to no avail. Finally I figured out the source of the problem, and how to fix it! It turns out that this behaviour is due to a feature introduced in Windows XP Service Pack 2 which limits the number of TCP/IP sockets you can have in the 'half-open' state to 10. A connection in the TCP half-open state means that one side of the socket has disappeared and stopped responding, without properly notifying the other side. After 10 such connections, the Windows kernel queues up all new socket connection attempts and you will experience this network "freeze" behaviour. Apparently the reason for Microsoft having this low default is to slow the spread of virii and limit infected hosts from participating in DoS attacks etc. Unfortunately, P2P applications in particular are very negatively affected by this low default, because they are constantly opening new connections to peers, they very quickly fill up the default limit of 10 half-open connections. The good news is that once you realise this is the problem you are encountering, its trivial to fix. There is a very simple patch which raises this default to any value, with a recommended setting of 100. I have used this patch a couple of times, setting the default to 100, and it has completely solved the issue for me. Yet another annoying, niggling thing to install on a fresh Windows system! Oh well.

Mount remote filesystems via SSH on Windows with free software

Tue, 12 May 2009 20:06:44 PDT

I often use Windows as a terminal to my various UNIX systems. Sometimes its helpful to run proprietary software - and I don't have time/inclination to mess around with half-baked emulators/ports/binary blobs/whatevers under Linux. I either run a completely open system like OpenBSD or I run Windows. Anyway, I never use Windows to do any real work. I always shell into a remote system to actually get things done. Either PuTTY or - if you prefer real OpenSSH like me - OpenSSH via Cygwin/X work fine for getting a terminal. WinSCP or Cygwin's OpenSSH for scp(1) are good for file-transfer under most circumstances. However, one of the nice things about Windows is the Explorer shell. It - and its KDE knock-off - are useful for certain file management operations. Why not leverage it? So I started looking for a way to mount remote filesystems via SSH, so that they appear as native Windows volumes. And I found a way to do it, for free. Dokan user mode filesystem for Windows Dokan is basically FUSE for Windows. That's all dandy and there are plenty of useful FUSE filesystems out there, like this one which uses my BitTorrent implementation. Whats cool about Dokan is they also do an SSH FS implementation. Is it hard to set up I figured this thing was surely going to be a PITA and probably not work to boot. In fact, you just install three things - some Microsoft runtime library, the main Dokan library, and Dokan SSHFS - and there you go. There is simple GUI app to set up remote mounts that supports all the things you'd expect, saving sessions, both password and public key authentication. Does it actually work Yes, although it doesn't seem to support symlinks. A symlink to a directory on a remote system appears as a file under Dokan. So no $HOME/public_html for you - oh well. Final thoughts Its fun to look at your horribly un-organised UNIX home directory in Windows, and see just how messy it is. Almost makes me want to start cleaning things up. But then I remember I know how to use locate(1) and find(1).