Niall's Blog

Have you ever tried to run a very network-intensive P2P application on Windows XP SP2 or higher? If so, you may have encountered very strange behaviour with the Windows TCP/IP network stack. Specifically, you won't be able to open any new TCP/IP connections, so web-browsing, email checking, SSH, etc will all be basically unusable. At the same time, P2P transfers might be running fine for all you can tell. I spent some time trouble-shooting all aspects of my set up including testing the router, my Internet connection, and even re-installing Windows a couple of times - all to no avail. Finally I figured out the source of the problem, and how to fix it! It turns out that this behaviour is due to a feature introduced in Windows XP Service Pack 2 which limits the number of TCP/IP sockets you can have in the 'half-open' state to 10. A connection in the TCP half-open state means that one side of the socket has disappeared and stopped responding, without properly notifying the other side. After 10 such connections, the Windows kernel queues up all new socket connection attempts and you will experience this network "freeze" behaviour. Apparently the reason for Microsoft having this low default is to slow the spread of virii and limit infected hosts from participating in DoS attacks etc. Unfortunately, P2P applications in particular are very negatively affected by this low default, because they are constantly opening new connections to peers, they very quickly fill up the default limit of 10 half-open connections. The good news is that once you realise this is the problem you are encountering, its trivial to fix. There is a very simple patch which raises this default to any value, with a recommended setting of 100. I have used this patch a couple of times, setting the default to 100, and it has completely solved the issue for me. Yet another annoying, niggling thing to install on a fresh Windows system! Oh well.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet

My C BitTorrent implementation, Unworkable, used to be hosted on an anonymous CVS repository I had running on my server at home. This was fine, until I reinstalled the machine from scratch and didn't feel like setting up the whole anonymous CVS access again. Its a pretty painful process, unfortunately, although there is this guide to setting up anonymous CVS. No public VCS, bad for an Open Source project So, for a while, there was no public source control for Unworkable, which sucked. Its difficult and cumbersome for other developers to write diffs and track changes without one. While I generally like to maintain full control of the source code hosting, I've had good experiences with Google Code before. They don't show ads, and their interface is clean and to the point, unlike say SourceForge, which is covered in ads and various nonsense. Anyway, I had a CVS repository and I first wanted to convert it to Subversion, including all the history, then I wanted to import that into Google Code. Converting from CVS to Subversion The first thing to do was to convert my existing CVS repository to Subversion. There is a nice tool specifically for this, cvs2svn. It is in fact very easy to use, at least in my basic case - I only work with HEAD or in SVN terminology, trunk. I simply ran:

cvs2svn --trunk-only --svnrepos ~/unworkable-svn ~/unworkable-cvs

Et voila, I have a shiny new Subversion repository in ~/unworkable-svn, with full history. Importing Subversion repository to Google Code Google Code lets you import an existing Subversion repository pretty easily, as long as you have an empty project. When your Google Code project is created, it will be set to revision 1. In Subversion-land, revision 0 is sort of magic, and so you will need to overwrite it to properly import your existing repository. Google give you a place to do this, but its slightly confusing because they don't put it under 'Administer'. In order to reset your repository you must:

• Log into your Google Code project with administrator privileges.
• Browse to the 'Source' page (either 'Checkout' or 'Browse' but not 'Changes').
• Scroll to the bottom of the page, and click the 'reset this repository' link, which is sort of hidden.
• Choose the option "Did you just start this project and do you want to 'svnsync' content from an existing repository into this project?"
• Click the big "Reset Repository" button which has a big red warning label beside it.

Now you are ready to import your repository. You will use the 'svnsync' tool included in Subversion 1.4 and up to do this. There are two commands, one which takes a path to both the Google Code repository and your repository, and another which takes just a path to the Google Code repository. The first one will run quite quickly, the second one can take a while as it imports each individual revision,

# This command takes both the path to your Google Code repository
# and the path to the repository you want to import
svnsync init --username YOURUSERNAME 
    https://YOURPROJECT.googlecode.com/svn file:///path/to/localrepos
# This command takes just the path to the Google Code repository.
# It will take a while to complete.
svnsync sync --username YOURUSERNAME https://YOURPROJECT.googlecode.com/svn

Once you've done that, your code is imported. Enjoy!

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet

For quite a long time now, OpenBSD has, among numerous exploit mitigation techniques, had a very strict mmap()-based malloc() implementation. Recently re-written by Otto Moerbeek, it is even harsher now. I find that this feature makes OpenBSD one of the best platforms to develop C programs on. If you have a double-free, use-after-free, off-by-one, or other typical mistake in your program, chances are OpenBSD's omalloc will trip up on it eventually. Especially on a strict-alignment, long-pointer architecture like sparc64, running it under OpenBSD is a great way to gain confidence that your program is solid. Anyway, recently Otto has made OpenBSD's malloc even stricter. I upgraded my home machine over the weekend to the latest snapshot with these commits, and I'm currently running a bunch of Unworkable BitTorrent download processes on it to make sure my code still holds up. So far so good! If you have some C code you care about, I'd recommend taking the time to run it under OpenBSD for a while - you might find you catch some bugs which even Valgrind missed. Enjoy!

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet

Over at the Peer to Peer Research Institute, we have performed analysis on over 128,959 film torrents. From this data, we were able to extract the most torrented films of 2007. Without futher ado, here is the list:

1. Harry Potter and the Order of the Phoenix
2. 300
3. Transformers
4. Ghost Rider
5. Ratatouille
6. Shrek The Third
7. The Simpsons Movie
8. Sunshine
9. I am Legend
10. I now pronounce you Chuck and Larry

You can expect more interesting data extracted from the dynamic world of P2P file sharing on this blog in the future.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet

I have just tagged, packaged and announced version 0.4 of my BitTorrent implementation, Unworkable. Here are the release notes:

• Implemented sending peer keep-alives.
• Trace log now contains timestamps.
• Make us more tolerant of intermittent tracker failures.
• Added support for Arch Linux.
• Fixed an off-by-four bug which could cause segfaults on some platforms.
• Fix zero padding in peer id generation.
• Overall code reduction and re-factoring plus improvements to documentation.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Tweet