Today, we are very happy to release it as open source. You can download Twitterslurp from its Google Code project page at http://twitterslurp.googlecode.com/.

Since last month, I’ve made a lot of changes to improve the quality (and ease of configuration) of the Twitterslurp code. Twitterslurp’s error handling has been improved, and I added the ability to start and stop the tweet stream and show more than the most recent 20 tweets. Our graphics team also created a spiffy logo.

Yesterday and today, Twitterslurp has been driving a video wall of tweets at the Personal Democracy Forum conference in NYC. The conference, which just ended, had over 17,000 tweets in the last two days.

Previously, we ran test versions of Twitterslurp during mysqlconf and php|tek, and officially on behalf of the Dutch PHP Conference. Twitterslurp started as a project for a client to allow them to track tweets, and give members of their website rewards for tweeting with a particular hashtag.

We’ve also set up a copy of Twitterslurp tracking itself.

We’d love for you to check out Twitterslurp, and we’re open to any and all feedback.

I’ve since added a lot of features, including a set of graphs, and we’ve set up a version of Twitterslurp for php|tek 2009.

Twitterslurp consists of two parts. The first is a PHP cron job that queries Twitter for new tweets matching a search string and adds them to a database. The second is JavaScript code that interfaces with a PHP script that searches the database for new tweets. In a nutshell, the JavaScript examines the web page for placeholders that represent different types of data (tweets, the leaderboard, and the graphs), and then posts to the PHP script with a JSON object detailing the data it wants. Once PHP responds, also with a JSON object, the JavaScript takes that data, formats it, and updates the page with the new content. The web page updates automatically based on an interval specified in the response object, so in the unlikely event the server came under heavy load, we would be able to easily throttle requests.

The new feature I’m really happy with is the stats page I’ve added, using jQuery and a graphs library called flot.

Flot uses the canvas tag introduced in Safari, and later supported by Firefox and Opera (and Chrome), to draw 2D images (graphs, in this case) within the browser. IE is supported with a library that implements canvas support on top of VML.

Flot is pretty simple to learn and use, thanks to its detailed API documentation and helpful set of examples. It actually has relatively few API functions. For the most part, you implement graphs simply by building a JavaScript object with the graph data and configuration options you want, and tell flot where to put the graph. (Twitterslurp builds this object in PHP and gives it to the JavaScript code via a JSON object.) With a few hours of work, I was able to put together the four graphs shown on the stats page, all of which have a rollover that shows the value of individual data points.

To lessen the server load, the stats page updates less frequently, and the graph data is cached on the server via APC, so the graph data doesn’t have to be regenerated on every request.

Please feel free to take a look, and let me know if you have any comments or suggestions, either here, or via Twitter (@jbafford).

We’re planning on taking what we’ve learned with Twitterslurp for php|tek, clean up the code (which wasn’t intended on providing graph data), adding a few more new features, and releasing it with an open source license in June. Stay tuned for more details…

The slides are now available for download.

Also, I want to thank everyone who showed up. It was great talking to all of you!

While the space savings are easily apparent, and I’ve seen demonstrations of how much faster search operations are, I could not recall anyone ever doing an analysis of how long it takes to insert IPs as strings or integers into the database. Naturally, I had to determine an answer to this so I’d know for future reference.

My findings show that, beyond the disk space savings and the much faster index queries possible with integer-based IP addresses, even with the database doing the string-to-numeric conversion, it is 9-12% faster to store IP addresses into the database as integers, rather than strings.

Background

There are two ways you can store an IPv4 address in a database. In its native format, an IP address is a four-byte long integer, and can be represented as an int unsigned.

In its human-readable format, an IP address is a string, with a minimum length of 7 characters (0.0.0.0) and a maximum length of 15 (255.255.255.255). This gives it an average length (assuming uniform random distribution) of 13.28 characters. Accordingly, one would store it in a database field of type varchar(15). In order for the database to keep track of exactly how much data is in the column, an additional byte of data must be added to store the length of the string. This brings the actual data-storage costs of an IP represented as a string to an average of 14.28 bytes (assuming the characters can be represented by one byte per character, as with latin1 or utf8).

This means that storing an IP address as a string requires, on average, about 10 bytes of extra data. In an application that saves access logs, that 10 bytes of data will eventually add up, and that should be reason enough to store IPs as an integer, rather than a string.

Disk Space Is Cheap, So No Big Deal, Right?

There are other costs associated with having larger data fields. If the column is indexed, the index will be larger as well. Larger indexes tend to perform slower than smaller indexes. Additionally, while disk space is plentiful and cheap, RAM is considerably more limited, so more memory will be used to cache the data or indexes, potentially pushing other, more valuable content out of the cache.

Further, while disks have been gradually getting faster, it still takes a relatively long time to read data from (and even longer to write to) a disk, and CPUs have gotten faster much more quickly than disks. The more data that has to be moved around, the more time the CPU wastes moving that data instead of performing more interesting work.

So, What’s the Actual Cost?

I wrote a PHP script that generates and inserts 1,000,000 random IP addresses into four mysql tables and timed the results. All tests were done with PHP 5.3alpha2 and MySQL 5.0.67 on a 2.16 GHz MacBook Pro (Intel Core Duo), with MyISAM tables.

The test uses four tables as described below. Each table has three columns: an id int unsigned not null auto_increment primary key column, and ip and s as described below:

The PHP benchmark script generates 1 million random IP addresses, and then inserts that data into MySQL To make MySQL do as much work as possible, for the inetbench_long tables, the sql query string uses INET_ATON() to convert the IP to an integer, rather than attempting to do it in PHP.

The results:

MySQL is adding 1 byte of overhead per row to inetbench_long, 10 bytes/row to inetbench_long_dynamic, and an average of 5 bytes/row of overhead to inetbench_varchar and inetbench_varchar_utf8.

The s column was added to test the performance difference of fixed vs. dynamic row storage, but it turns out there’s not much difference.

You can download the code used to generate these results here (2 kb zip).

Analysis

Storing IPs as a string, besides requiring more disk space, takes 9% longer than storing them as integers, even with the overhead of converting the IP from a string to an integer. If the table uses utf8 encoding, it’s 12% slower. (This should not be surprising: UTF-8 is inherently slower to process than a strictly 8-bit encoding.) Storing data as an integer in a table with a dynamic row length is not appreciably slower. The indexes on the string tables are 63% larger.

For good measure, I tested a few select queries against the tables. The results were actually somewhat interesting.

Doing a search where you’re looking for a specific IP, or a range of IPs that can be satisfied with a like clause resulted in no significant difference between the integer and varchar storage. (This was somewhat surprising to me.)


select benchmark(10000000, (select count(*) from inetbench_long where ip between inet_aton('172.0.0.0') and inet_aton('172.255.255.255')));
1 row in set (0.41 sec)

select benchmark(10000000, (select count(*) from inetbench_varchar where ip like '172.%'));
1 row in set (0.42 sec)


However, as expected, when the search range can’t be represented with a simple like query, the speed difference between numeric and string indexes really show:


select benchmark(35000000, (select count(*) from inetbench_long where ip between inet_aton('172.16.0.0') and inet_aton('172.31.255.255')));
1 row in set (1.43 sec)

select count(*) from inetbench_varchar_utf8 where inet_aton(ip) between inet_aton('172.16.0.0') and inet_aton('172.31.255.255');
1 row in set (1.72 sec)


This results in an integer search that’s about 35 million times faster than the string search. Also, the utf8 table is about 17% slower than the latin1 table (which isn’t too surprising, given UTF-8′s overhead).

In Conclusion…

With “only” a difference of 12 microseconds per insert query, it may not make sense to change an existing database if you’re not doing many queries against stored IPs. If you’re doing IP range queries, though, you probably want to convert your tables. Any new development should be storing IP addresses in the database as integers by default. The space and time savings are worth it.

Once IPv6 becomes more prevalent, the savings will only become larger: a 128-bit (16 byte) IPv6 address can be up to 39 characters long when represented in a “human readable” format. (Storing IPv6 addresses in the database is going to be a bit more difficult, as MySQL doesn’t have a native 16-byte-wide data type.)

The load on MySQL when inserting integer IPs could likely be slightly reduced by doing that conversion in your application, rather than using MySQL’s INET_ATON() function.

One of the challenges with displaying a map with lots of markers is that if there are too many markers in one location, the placemarks themselves quickly become an unmanageable sea that obscures the map. In some cases, this is the desired effect, but in our case, we wanted to present a cleaner map that showed a single marker that provided an indication of the number of people present in an area. (Less markers also has the advantage of loading faster.)

One solution to this problem is, instead of showing all of the markers all at once, you can group all of the markers that are close together into one. But, how do you determine if two locations are “close together” so that you can do that aggregation?

Geohash

One way is to use a geohash, which takes a latitude and longitude coordinate and, using a hashing algorithm, mixes the two numbers together into a a string that represents the position. More accurately, it represents a box which contains the coordinate in question. For example, the coordinates of the TBG office (38.9193540 °N, 77.0705180 °W) is represented by the geohash dqcjqjncwv2t9. This string has an interesting property: as you remove characters from the string, you get a less accurate representation of the location – the box gets larger. With a list of addresses stored in geohash format, you can do a search for all of the addresses whose geohashes all start with the same characters.

However, because of the way geohashes are encoded, different geohashes have different shapes, there is some overlap between different geohashes, and in some cases, removing a character from a geohash can result in a larger shape on the map that doesn’t cover the entirety of the more specific geohash. As a result, we determined that they weren’t suitable for our needs: because of the overlap, it was possible for one location to appear to be in the space covered by multiple geohashes, thus potentially leading to unreliable counting.

Quadtree

Some further searching led me to maptiler.org, which provided an alternate answer. Since the Earth is a sphere, it can be broken up into quadrants (corresponding to the four hemispheres: north west, north east, south west, and south east), which (with some stretching) are square. Each of those quadrants can be subdivided into subquadrants, and those into more subquadrants, each of which contains one quarter of the total area covered by the quadrant that contains it.

In fact, this is how Google Maps map tiles work. When zoomed all the way out, the entire earth is represented as a 256 x 256 pixel map tile image. Each larger zoom level contains four times the number of map tiles as the previous zoom level, with no overlap. At the maximum zoom level (20), the entire Earth is represented by 4¹⁹ = 274,877,906,944 256 x 256 pixel images.

How does that help us? Well, if we number each of the quadrants (0 = NW, 1 = NE, 2 = SW, 3 = SE), and add a digit for each sub-quadrant we “zoom in” to, we get a string that represents a square on the map at a zoom level equivalent to the length of the string. For example, the coordinates of the TBG office can be represented as 0320100322312003121. If we remove a character from the hash, it’s guaranteed that the larger space contains all the possible locations in the smaller space. There’s no overlap between hashes of the same length. This means we’re guaranteed the accurate counting we wanted. And we can very conveniently scale our searches of the database of addresses with the zoom level of the map window we’re viewing the results in.

Pulling it all Together: GlobalMapTiles

On the client’s map page, whenever the map position or zoom level changes, we send a few ajax queries to the web server with quadtree codes covering the map window. The server obligingly searches the database for the number of people in the area represented by the code, and returns a result list containing the number of people in the quadtree a level or two deeper. We then take the results, convert the quadtree code back into a coordinate on the map, and add a marker at its center.

As luck would have it, there’s a Python script, globalmaptiles.py on the maptiler.org page that handles the conversion from latitude and longitude to these quadtree coordinates. Although Python is a fine language (and some of the most fun code I’ve written has been in Python), we needed this code in PHP and JavaScript. But that’s ok. The code is almost entirely simple mathematical operations, and that’s very easy to port from Python to JavaScript and PHP. This port, plus some additions of my own to facilitate turning a quadtree string back into its bounding coordinates on the map is what we’re releasing today.

This is, of course, only half of the solution. We now have a system by which we can identify locations on the map which are nearby and aggregate them together, but we still need a way to pull that information out of the database and send it on-demand to the web browser to be displayed. That’s a more interesting problem (and solution). I’m still refining that code and adding additional functionality, and we’ll release that soon, when we’re satisfied with it.

As the original implementation was licensed with the X11 license, GlobalMapTiles is also licensed under the X11 license. For more details on how to use it, and to download, see the Aggregate Map Tools page.