Niall O'Higgins

Windows, P2P, network freezes and the TCP `half-open' state

Mon, 18 May 2009 21:27:23 PDT

Have you ever tried to run a very network-intensive P2P application on Windows XP SP2 or higher? If so, you may have encountered very strange behaviour with the Windows TCP/IP network stack. Specifically, you won't be able to open any new TCP/IP connections, so web-browsing, email checking, SSH, etc will all be basically unusable. At the same time, P2P transfers might be running fine for all you can tell. I spent some time trouble-shooting all aspects of my set up including testing the router, my Internet connection, and even re-installing Windows a couple of times - all to no avail. Finally I figured out the source of the problem, and how to fix it! It turns out that this behaviour is due to a feature introduced in Windows XP Service Pack 2 which limits the number of TCP/IP sockets you can have in the 'half-open' state to 10. A connection in the TCP half-open state means that one side of the socket has disappeared and stopped responding, without properly notifying the other side. After 10 such connections, the Windows kernel queues up all new socket connection attempts and you will experience this network "freeze" behaviour. Apparently the reason for Microsoft having this low default is to slow the spread of virii and limit infected hosts from participating in DoS attacks etc. Unfortunately, P2P applications in particular are very negatively affected by this low default, because they are constantly opening new connections to peers, they very quickly fill up the default limit of 10 half-open connections. The good news is that once you realise this is the problem you are encountering, its trivial to fix. There is a very simple patch which raises this default to any value, with a recommended setting of 100. I have used this patch a couple of times, setting the default to 100, and it has completely solved the issue for me. Yet another annoying, niggling thing to install on a fresh Windows system! Oh well.

Importing a CVS repository to Google Code Subversion

Wed, 06 May 2009 21:11:17 PDT

My C BitTorrent implementation, Unworkable, used to be hosted on an anonymous CVS repository I had running on my server at home. This was fine, until I reinstalled the machine from scratch and didn't feel like setting up the whole anonymous CVS access again. Its a pretty painful process, unfortunately, although there is this guide to setting up anonymous CVS. No public VCS, bad for an Open Source project So, for a while, there was no public source control for Unworkable, which sucked. Its difficult and cumbersome for other developers to write diffs and track changes without one. While I generally like to maintain full control of the source code hosting, I've had good experiences with Google Code before. They don't show ads, and their interface is clean and to the point, unlike say SourceForge, which is covered in ads and various nonsense. Anyway, I had a CVS repository and I first wanted to convert it to Subversion, including all the history, then I wanted to import that into Google Code. Converting from CVS to Subversion The first thing to do was to convert my existing CVS repository to Subversion. There is a nice tool specifically for this, cvs2svn. It is in fact very easy to use, at least in my basic case - I only work with HEAD or in SVN terminology, trunk. I simply ran:

cvs2svn --trunk-only --svnrepos ~/unworkable-svn ~/unworkable-cvs

Et voila, I have a shiny new Subversion repository in ~/unworkable-svn, with full history. Importing Subversion repository to Google Code Google Code lets you import an existing Subversion repository pretty easily, as long as you have an empty project. When your Google Code project is created, it will be set to revision 1. In Subversion-land, revision 0 is sort of magic, and so you will need to overwrite it to properly import your existing repository. Google give you a place to do this, but its slightly confusing because they don't put it under 'Administer'. In order to reset your repository you must:

Log into your Google Code project with administrator privileges.
Browse to the 'Source' page (either 'Checkout' or 'Browse' but not 'Changes').
Scroll to the bottom of the page, and click the 'reset this repository' link, which is sort of hidden.
Choose the option "Did you just start this project and do you want to 'svnsync' content from an existing repository into this project?"
Click the big "Reset Repository" button which has a big red warning label beside it.

Now you are ready to import your repository. You will use the 'svnsync' tool included in Subversion 1.4 and up to do this. There are two commands, one which takes a path to both the Google Code repository and your repository, and another which takes just a path to the Google Code repository. The first one will run quite quickly, the second one can take a while as it imports each individual revision,

# This command takes both the path to your Google Code repository
# and the path to the repository you want to import
svnsync init --username YOURUSERNAME 
    https://YOURPROJECT.googlecode.com/svn file:///path/to/localrepos
# This command takes just the path to the Google Code repository.
# It will take a while to complete.
svnsync sync --username YOURUSERNAME https://YOURPROJECT.googlecode.com/svn

Once you've done that, your code is imported. Enjoy!

OpenBSD's omalloc: Bug and buffer overflow detection

Wed, 03 Dec 2008 18:30:37 PST

For quite a long time now, OpenBSD has, among numerous exploit mitigation techniques, had a very strict mmap()-based malloc() implementation. Recently re-written by Otto Moerbeek, it is even harsher now. I find that this feature makes OpenBSD one of the best platforms to develop C programs on. If you have a double-free, use-after-free, off-by-one, or other typical mistake in your program, chances are OpenBSD's omalloc will trip up on it eventually. Especially on a strict-alignment, long-pointer architecture like sparc64, running it under OpenBSD is a great way to gain confidence that your program is solid. Anyway, recently Otto has made OpenBSD's malloc even stricter. I upgraded my home machine over the weekend to the latest snapshot with these commits, and I'm currently running a bunch of Unworkable BitTorrent download processes on it to make sure my code still holds up. So far so good! If you have some C code you care about, I'd recommend taking the time to run it under OpenBSD for a while - you might find you catch some bugs which even Valgrind missed. Enjoy!

Top 10 Torrented Films of 2007

Tue, 18 Mar 2008 14:46:02 PDT

Over at the Peer to Peer Research Institute, we have performed analysis on over 128,959 film torrents. From this data, we were able to extract the most torrented films of 2007. Without futher ado, here is the list:

Harry Potter and the Order of the Phoenix
300
Transformers
Ghost Rider
Ratatouille
Shrek The Third
The Simpsons Movie
Sunshine
I am Legend
I now pronounce you Chuck and Larry

You can expect more interesting data extracted from the dynamic world of P2P file sharing on this blog in the future.

Unworkable 0.4 released

Mon, 07 Jan 2008 22:45:10 PST

I have just tagged, packaged and announced version 0.4 of my BitTorrent implementation, Unworkable. Here are the release notes:

Implemented sending peer keep-alives.
Trace log now contains timestamps.
Make us more tolerant of intermittent tracker failures.
Added support for Arch Linux.
Fixed an off-by-four bug which could cause segfaults on some platforms.
Fix zero padding in peer id generation.
Overall code reduction and re-factoring plus improvements to documentation.

BitTorrent Strategies: The Beginning

Thu, 27 Dec 2007 07:30:21 PST

To follow up on my last post on the bittorrent end-game, I'm going to write about a strategy to bootstrap a torrent download. I am talking here about the case where you start a download with no existing data, in other words, from scratch. As I described in one of my earlier articles about the basics of the BitTorrent protocol, torrents are divided into units called pieces. The torrent meta data contains the SHA-1 checksums for each piece, so that we can hash each piece once we have downloaded it, for verification purposes. Pieces are downloaded on a block-by-block basis - typically the block size is sixteen kilobytes (16KB). Another important thing to understand is that BitTorrent peers use a "tit-for-tat" transfer scheduling algorithm. That is, peers will be rewarded by other peers for uploading data to them. It is therefore important that a bootstrapping peer have at least a small number of complete pieces as soon as possible, in order to share them, and hence be rewarded by the "tit-for-tat" algorithm. The bootstrapping peer wants to get at least a few pieces to share as soon as possible. Additionally, pieces are usually downloaded in rarest-first order. This ensures that rare pieces are replicated sufficiently so that holes do not appear in the torrent as peers leave the swarm. One of the design considerations of BitTorrent - vs other distributed data mechanisms - is to have as reliable replication as possible. It is highly undesirable in BitTorrent for even a single piece to disappear, as this could render the final file(s) completely unusable. This contrasts heavily with a file distribution system such as Amazon's Dynamo, where it may be quite acceptable for data to disappear - is it really a huge deal if a user loses the contents of their shopping cart in an outage? In any case, the desire to ensure replication of the rarest pieces in a torrent is overridden by the need to get a small number of pieces as a peer is bootstrapping. To this end, the initial pieces are downloaded in random order as opposed to rarest-first order. In practice this means the bootstrapping peer should get its initial data quite a bit faster. The number of pieces to download before switching to rarest-first order is suggested to be four. As a general point, the peer will aim to complete a piece fully before requesting blocks belonging to another piece. Alternately put, if the peer has a few blocks in one piece, it will concentrate on downloading the remaining blocks belonging to that piece before starting on another piece.

BitTorrent Strategies: The End Game

Wed, 26 Dec 2007 23:51:55 PST

Downloads in BitTorrent take place according to a number of strategies, which map to stages. Initiating a torrent download has one strategy, normal operation has another strategy, and finally pulling down the last remaining pieces has yet another strategy. The End Game is the name for the final download strategy - there is a tendency for the last few pieces of a torrent to download quite slowly. To avoid this, many BitTorrent implementations issue requests for the same remaining blocks to all its peers. When a block comes in from one peer, you send CANCEL messages to all the other peers requested from, in order to save bandwidth. Its cheaper to send a CANCEL message than to receive the full block and just discard it. However, there is no formal definition of when to enter End Game Mode. In my BitTorrent implementation, Unworkable, I quite stupidly used a percentage-based threshold. When only 5% of the torrent download was incomplete, Unworkable would enter End Game Mode. I didn't notice how stupid this was until I got around to testing with a large torrent download, a Fedora Core 8 DVD image which is 4G in size or so. It turns out that 5% of 4G is quite a bit of data, and requesting each bit of it from every single peer is extremely inefficient. So I did some more research to find a more effective way to detect when to enter End Game. I found two popular definitions:

All blocks have been requested
Number of blocks in transit is greater than number of blocks left, and no more than 20

I didn't understand the choice of the number 20 in method 2, so I decided to go with option 1), which I did understand. In Unworkable CVS HEAD (and in the 0.4 release, coming in a week or two), the ill-informed percentage-based threshold is gone and End Game Mode is entered when all blocks are requested.

BitTorrent Distributed Hash Table (DHT) or Trackerless BitTorrent I

Mon, 24 Dec 2007 18:12:06 PST

One of the more interesting extensions to the BitTorrent protocol has been the introduction of a distributed hash table implementation. As mentioned in my previous article on the basics of the BitTorrent protocol, traditionally BitTorrent relies upon a centralised "tracker" application - which runs over standard HTTP - in order to facilitate contacting peers and so on. The requirement for a centralised tracker is obviously a major barrier to fully decentralised operation, and a problem in terms of BitTorrent's resistance to tracker outage (perhaps even caused by legal actions). In part one of this article I'm going to look a bit at the network side of BitTorrent's DHT. The official BitTorrent DHT specification states that the protocol is based on Kademilia. In BitTorrent, DHT is mostly separated from the original protocol. Peers listen on an additional port, using a UDP protocol, to issue network searches and so forth. The DHT protocol is known as KRPC and consists of three message types - query ("q"), response ("r") and error ("e"). There are four queries:

PING
FIND_NODE
GET_PEERS
ANNOUNCE_PEER

Each KRPC message is formatted in B-Encode, with various key-value pairs encoded in a dictionary structure. Each node participating in the DHT has its own routing table containing contact information for nodes "near" to it (according to a mathematical 'distance function'). This routing table is used to produce responses to queries. We will go more into details of how this works in the next article. The TCP BitTorrent peer protocol itself has only been very slightly extended in order to take advantage of this new DHT functionality. During the handshake, peers may indicate that they support DHT. If the receiver also supports DHT, it should send a new message called PORT (id is 0x09) with the port number of its UDP DHT service, encoded as a 16-bit value. The receiver of this message will then try to send a PING message to the peer on this port, and if it gets a reply, routing table adjustments etc should take place. Those are the very basics of BitTorrent DHT from a network perspective. I will write more about the details of the algorithms used for routing and how the consistent hashing is employed in part two.

Decoupled Python GUI Construction, or BitTorrent visualisation

Sat, 22 Dec 2007 02:07:25 PST

While in general I appreciate very simple, no-nonsense user interfaces for applications that work efficiently on the console and so can be used via SSH, there are times when increased visualisation is very useful. Specifically with regard to my BitTorrent client, Unworkable, the default user interface is exceedingly simple. Inspired by the ubiquitous scp program on UNIX, the idea is that it should be just as simple to download a file via BitTorrent as via SSH or FTP/HTTP. Indeed, I borrowed the progressmeter.c source code from OpenSSH (a great repository of nice code). Have a look at this screenshot of Unworkable running on Windows to get an idea of what it looks like. While I'm pretty happy with the current console user interface, it is obviously very limited in how much information it can display. To overcome this, I have for a long time had an extremely verbose trace mechanism. Pass the -t option to Unworkable, and it will write a detailed log. This is incredibly useful for debugging, but it suffers from the opposite problem - information overload. Its quite difficult for a human to parse scrolling pages of ASCII text and spot patterns. It can take considerable analysis to determine exactly why the program is making the decisions it is. This is the main motivation for adding support for a graphical user interface - information can be much more easily distilled into graphs and other visualisations which make it much easier to understand, at a glance, what is going on. For some examples of the kind of things I have in mind, take a look at the Azureus screenshots page. This brings me to the question of how to nicely add a GUI to my application. I strongly want to keep the existing simple, console interface, and don't want to bloat the application with additional external dependencies for widget sets and so forth. I decided to go with an IPC mechanism, to split the GUI from Unworkable entirely. While considering IPC mechanisms, I figured why not simply use TCP/IP. While in general the GUI is going to be running on the same host as Unworkable, this at least gives the option for operation over a network. So, I have added a simple "control server" to Unworkable, which currently has a fairly basic ASCII protocol. At the moment, communication is unidirectional - Unworkable only pushes out some event notification messages. There is no mechanism for clients to send instructions back, since this isn't required for visualisation just yet and that is my first priority. I have started a client implementation in Python. "Why Python?", you may ask. Python has pretty good networking support, good UI toolkit support and good multi-platform support. I'm also pretty experienced with the language, and I find it very fast to write applications with. Since the GUI itself performs practically no hard computation nor I/O, the performance penalties of a higher-level language are hardly a concern. In closing, the formula of having the application split into a C program which does the performance-sensitive stuff, exposing a simple ASCII protocol over TCP/IP, while implementing the user interface in Python, permits maintaining a lean and efficient core application with a slick graphical user interface.

Unworkable 0.3 released

Thu, 20 Dec 2007 13:41:04 PST

I have just tagged, packaged and announced version 0.3 of my BitTorrent implementation, Unworkable. My goal with Unworkable is to make releases frequently - hopefully twice a month or so - with incremental improvements each release. The hope is that each release should be of a higher quality than the last. Therefore I try to test new features well and ensure the stability is at least as good as the previous release. I also try to run tests across a wide variety of platforms (Solaris, OpenBSD, Linux, Windows, Mac OS X, etc). Anyway, here's whats new in this version:

Fixed a subtle bug in download strategy
Removed numerous format specifier bugs by bringing source in line with C99.
Major refactoring and code cleanup.
Added initial implementation of a TCP/IP "control server"
Checked in some initial work towards a decoupled Python UI.
Portability improvements to build and run on Windows (Cygwin).
Build and runtime testing on Fedora 7 and Gentoo Linux.