Today, we are very happy to release it as open source. You can download Twitterslurp from its Google Code project page at http://twitterslurp.googlecode.com/.

Since last month, I’ve made a lot of changes to improve the quality (and ease of configuration) of the Twitterslurp code. Twitterslurp’s error handling has been improved, and I added the ability to start and stop the tweet stream and show more than the most recent 20 tweets. Our graphics team also created a spiffy logo.

Yesterday and today, Twitterslurp has been driving a video wall of tweets at the Personal Democracy Forum conference in NYC. The conference, which just ended, had over 17,000 tweets in the last two days.

Previously, we ran test versions of Twitterslurp during mysqlconf and php|tek, and officially on behalf of the Dutch PHP Conference. Twitterslurp started as a project for a client to allow them to track tweets, and give members of their website rewards for tweeting with a particular hashtag.

We’ve also set up a copy of Twitterslurp tracking itself.

We’d love for you to check out Twitterslurp, and we’re open to any and all feedback.

One of the challenges with displaying a map with lots of markers is that if there are too many markers in one location, the placemarks themselves quickly become an unmanageable sea that obscures the map. In some cases, this is the desired effect, but in our case, we wanted to present a cleaner map that showed a single marker that provided an indication of the number of people present in an area. (Less markers also has the advantage of loading faster.)

One solution to this problem is, instead of showing all of the markers all at once, you can group all of the markers that are close together into one. But, how do you determine if two locations are “close together” so that you can do that aggregation?

Geohash

One way is to use a geohash, which takes a latitude and longitude coordinate and, using a hashing algorithm, mixes the two numbers together into a a string that represents the position. More accurately, it represents a box which contains the coordinate in question. For example, the coordinates of the TBG office (38.9193540 °N, 77.0705180 °W) is represented by the geohash dqcjqjncwv2t9. This string has an interesting property: as you remove characters from the string, you get a less accurate representation of the location – the box gets larger. With a list of addresses stored in geohash format, you can do a search for all of the addresses whose geohashes all start with the same characters.

However, because of the way geohashes are encoded, different geohashes have different shapes, there is some overlap between different geohashes, and in some cases, removing a character from a geohash can result in a larger shape on the map that doesn’t cover the entirety of the more specific geohash. As a result, we determined that they weren’t suitable for our needs: because of the overlap, it was possible for one location to appear to be in the space covered by multiple geohashes, thus potentially leading to unreliable counting.

Quadtree

Some further searching led me to maptiler.org, which provided an alternate answer. Since the Earth is a sphere, it can be broken up into quadrants (corresponding to the four hemispheres: north west, north east, south west, and south east), which (with some stretching) are square. Each of those quadrants can be subdivided into subquadrants, and those into more subquadrants, each of which contains one quarter of the total area covered by the quadrant that contains it.

In fact, this is how Google Maps map tiles work. When zoomed all the way out, the entire earth is represented as a 256 x 256 pixel map tile image. Each larger zoom level contains four times the number of map tiles as the previous zoom level, with no overlap. At the maximum zoom level (20), the entire Earth is represented by 4¹⁹ = 274,877,906,944 256 x 256 pixel images.

How does that help us? Well, if we number each of the quadrants (0 = NW, 1 = NE, 2 = SW, 3 = SE), and add a digit for each sub-quadrant we “zoom in” to, we get a string that represents a square on the map at a zoom level equivalent to the length of the string. For example, the coordinates of the TBG office can be represented as 0320100322312003121. If we remove a character from the hash, it’s guaranteed that the larger space contains all the possible locations in the smaller space. There’s no overlap between hashes of the same length. This means we’re guaranteed the accurate counting we wanted. And we can very conveniently scale our searches of the database of addresses with the zoom level of the map window we’re viewing the results in.

Pulling it all Together: GlobalMapTiles

On the client’s map page, whenever the map position or zoom level changes, we send a few ajax queries to the web server with quadtree codes covering the map window. The server obligingly searches the database for the number of people in the area represented by the code, and returns a result list containing the number of people in the quadtree a level or two deeper. We then take the results, convert the quadtree code back into a coordinate on the map, and add a marker at its center.

As luck would have it, there’s a Python script, globalmaptiles.py on the maptiler.org page that handles the conversion from latitude and longitude to these quadtree coordinates. Although Python is a fine language (and some of the most fun code I’ve written has been in Python), we needed this code in PHP and JavaScript. But that’s ok. The code is almost entirely simple mathematical operations, and that’s very easy to port from Python to JavaScript and PHP. This port, plus some additions of my own to facilitate turning a quadtree string back into its bounding coordinates on the map is what we’re releasing today.

This is, of course, only half of the solution. We now have a system by which we can identify locations on the map which are nearby and aggregate them together, but we still need a way to pull that information out of the database and send it on-demand to the web browser to be displayed. That’s a more interesting problem (and solution). I’m still refining that code and adding additional functionality, and we’ll release that soon, when we’re satisfied with it.

As the original implementation was licensed with the X11 license, GlobalMapTiles is also licensed under the X11 license. For more details on how to use it, and to download, see the Aggregate Map Tools page.

I wrote xarfs because I wanted an easy way to be able to inspect Mac OS X Installer package files, which as of Mac OS X 10.5, began to be xar archives. In particular, I wanted to be able to lsbom the package manifest in order to see what files were being modified. (In retrospect, it probably would have been easier, though less satisfying, to have written a simple tool that extracts only the package manifest and lists its contents.)

Although xarfs provides easy access to view the contents of a xar archive, it has some pretty significant limitations. Besides having a rather clumsy user interface, and presenting only a read-only interface to xar files, it handles large files spectacularly poorly, decompressing them entirely into memory regardless of how much of the file is being accessed. (This is a limitation in the xar API.) It’s also written in Objective C using MacFUSE’s framework. This rather limits its portability; an implementation using the regular FUSE API would probably allow xarfs to work on Linux.

If anyone’s interested in this, it’s probably possible to resolve some of its deficiencies and make it a lot more useful. Let me know if you find it helpful.

xarfs is licensed under the three-clause BSD license. For more information, or to download, see the xarfs page.

Slow does this by taking advantage of two signals, SIGSTOP and SIGCONT, which respectively stop and re-start a unix process’ execution.

I’ve used this tool to prevent long-running cpu-bound processes from consuming an entire cpu to improve performance of other processes on the machine. I’ve also used it to slow part of a two-part data import process where temporary data files could be created much faster than they could be processed and removed, running the risk of low disk space conditions.

Slow is released under the GPL v3. For documentation, more information, and download links for slow and its source code, please see slow’s web page.

As is common knowledge, Safari is based on the open-source WebKit rendering engine, which releases nightly builds. For reference, the current WebKit version is 520.1+.

Because WebKit is open source, using WebKit’s version history tracker, it’s possible to determine the following interesting information:

* The Safari-6528 branch was created on January 29th based on the state of the WebKit trunk at r40289, from January 27th.

* There have been 75 merges of code (presumably bug fixes) into the 6528 branch since then. The last of these merges was r41097 (merging r41091) on February 19.

* The release is tagged in WebKit svn as r41182, the/releases/Apple/Safari 4 Public Beta tag.

This means that the Safari 4 beta is based roughly on the state of WebKit as it was at the end of January, one month behind current development. Considering the importance of WebKit to Safari in general, and Mac OS X as a whole, a one month delay for QA purposes is probably pretty good.

Kudos to all of the people working on WebKit! Keep up the good work!

Fortunately, the bug has been fixed in Mac OS X 10.5, which does not have a lookupd service, but it was present in Mac OS X 10.3 and 10.4.

Unlockupd was first released in 2004, and it really should have been released under an open source back then. Better late than never, though. At this point, it’s really more of an historical relic than a useful program, but hopefully someone will find some use with the code.

More information is available on Unlockupd’s web page, including download links for Unlockupd and its source code. Unlockupd is licensed under the GPL v3.

Long overdue, this new website and blog are now live. I’ll be talking mostly about PHP and related technology — I’ve spent the last almost ten years writing PHP (and JavaScript) code, as well as a smattering of Perl, Python, and other languages, after many prior years writing in C/C++. Expect some Mac talk as well, on occasion.

To “celebrate” the new website, throughout this week, I’ll be doing a number of (in some cases, very belated) open source releases of some software I’ve worked on over the years. There will be a somewhat Mac focus to some of these releases, but even it you don’t have a Mac, hopefully, you’ll still find them interesting enough to follow along, and I’ll get to the PHP-related content soon enough.

If you have questions or comments, feel free to comment here, or send me an email from the feedback form via the button at the bottom of the page.