Kriston and I built Lady Ada’s Game of Life kit over at HacDC a while back, and it roused the interest of a few folks. Sommer asked that I be on the lookout for another kit that might be a good candidate for learning to solder. I eventually suggested another Lady Ada kit: the MintyBoost, a simple circuit that lets you top up your USB gadgets’ batteries through the use of a pair of AAs. By the time I got my act together, six(!) people had expressed interest in building the kit, pushing supplies of my tools (somewhat scarce) and electronics knowledge (decidedly meager) to their limits.

But it went surprisingly well! Everyone’s kit worked immediately; in fact, they even seemed to work with the iPhone 3GSes that were on hand — something that the MintyBoost can’t necessarily be relied upon to do.

I think that my favorite part was watching everyone mess around with the Dremel. That’s some advanced nerdery for you.

I think that folks had fun; we might do this again, perhaps with more of an Arduino bent. If you want in, let me know.

Every DC Metro rider owes Michael Perkins a debt of gratitude. For a while now he’s been not only covering WMATA [UPDATE: link fixed], but jumping through the hoops necessary to uncover details of their contracts, budgets and other documents (and working on tech projects besides!).

Over the weekend he published his latest acquisition: the WMATA/Nextbus contract. As you might imagine, it takes the reader on a harrowing emotional rollercoaster. But before we get to that, let’s build some dramatic tension.

PREVIOUSLY, ON “LICENSING QUESTIONS RELATED TO NEXTBUS DATA”

After pulling the service due to concerns about its reliability, WMATA finally redeployed Nextbus last year. This happened not too long after the release of WMATA’s scheduling information in the open GTFS format — combined, the two excited a lot of interest in transit-enthusiast developers like myself.

But under what circumstances could the Nextbus data be used? The answer was not at all clear. Fortunately, Nextbus representatives began to pop up in relevant comment threads. Unfortunately, it soon became clear that there are two companies called Nextbus — one of them is “Nextbus, Inc.” (which I’ll call Nextbus) and one is “Nextbus Information Systems” (which I’ll call NIS). I emailed back and forth with representatives from both. First, my emails with Alex Orloff from NIS:

Me:

Alex, another question for you: there’s a lot of online confusion surrounding the relationship between Nextbus, Inc. and Nextbus Information Systems. Can you clarify the relationship between the two? Do you both have contracts with WMATA? And am I right in thinking that you’re the company responsible for the Washington, DC-centric Nextbus app in the iPhone store? I’m seeing folks show up in blog comment sections who claim to be affiliated with these two companies, and who are asserting contradictory things.

Alex:

When the transaction that created NextBus Inc. as a wholly owned subsidiary of Grey Island Systems occurred, NextBus Information Systems kept what is called the “distribution business” piece of NextBus, which includes all the things you would associate with distribution of data – licensing to third parties, delivery to cell phones, etc. The NextBus DC app on the App Store is indeed ours. We understand that there is some confusion about the relationship and we have encouraged NextBus Inc. to put a web page on nextbus.com helping people to understand our role. We think that that would be the simplest way to clear up any confusion – don’t you think ?

Me:

Much of the conversation is occurring at the Greater Greater Washington blog. And yes, I agree that it would be helpful if Nextbus clarified its relationship to your organization and the bounds of its arrangement with WMATA.

Am I right in thinking that Nextbus Information Systems has no formal arrangement with WMATA, but rather possesses the rights to license the distribution of some of the data emerging from the Nextbus/WMATA agreement?

Sorry if this is muddying the waters — just trying to get to the bottom of where I and other DC-area devs stand.

Me:

Aha I see the discussion now, I had seen that post a few days ago but Mike Smith’s comment was not up then. Did he send you any information directly ? I’m curious as to what he said if you have contacted him directly. We are definitely trying to get NextBus to be more open about our relationship with them and our rights. Many developers are asking the same questions, and like you, many are confused about what’s what.

We do not have an agreement with WMATA directly, our rights come from the agreement that was part of the sale to Grey Island, which reserved exclusive rights to distribution of the predictions, in particular to cell phones but also advertising, licensing, etc. in the United States (and UK). So our agreement covers any transit agency that NextBus tracks here in the US.

Next, my correspondence with Mike Smith of Nextbus.

Me:

Thank you for contributing to the comment thread at Greater Greater Washington. I hope you won’t mind clarifying the distinction between your organization and Nextbus Information Systems (NIS). I’ve been in touch with Alex Orloff of NIS, and from my discussions with him have developer the understanding that Nextbus Inc. is contracted with WMATA, but that Nextbus Information Systems — a wholly distinct entity — retains the rights to distribute the resulting data to mobile devices, via an API, and perhaps in other ways.

I and others in the DC developer/transit community find all this quite confusing. I have a number of questions that I hope you won’t mind answering. I think they’ll go a long way toward clarifying the situation:

1. Is the relationship between Nextbus Inc. and Nextbus Information Services that I have described correct?

2. If NIS has the right to distribute the data, am I correct in assuming they have access to your servers? Mr. Orloff indicated a desire to have Nextbus Inc. clarify the relationship between the two firms at nextbus.com, leading me to believe that you have control over the domain; but surely he has *some* access to your systems if he is able to offer an API view of the data, as he indicated.

3. If NIS owns the rights to distribute the data but does not control the servers, what steps is Nextbus Inc. obligated to take in order to restrict third parties’ access to the publicly available data at e.g. wmata.nextbus.com?

4. Does Nextbus assert ownership over non-predictive data produced as part of its contract with WMATA (e.g. routeconfig files)? What is its position on noncommercial use of such data?

Thanks very much for your help in clarifying these issues.

Mike:

It would be inappropriate for me to weigh in on what distribution rights that NextBus Information Systems (NBIS) does or does not have, since they are indeed a separate entity. But I can tell you that the transit agencies own the data and have legal control over it.

Sorry I cannot be of more help.

Mike

I wasn’t able to get more out of either of them. So here’s my understanding of the situation: at some point, Nextbus sold the main part of its business to Grey Island Systems. An independent portion remained, however, called Nextbus Information Systems, and it retained the rights to license prediction information generated by past — and future! — deals with the part now owned by Grey Island. The Grey Island/Nextbus folks say that the transit agencies they contract with own full rights to the data, same as NIS, but they’re understandably hesitant to step on the NIS folks’ toes (lest they get sued, presumably). Mr. Orloff was very helpful, but made it clear that I’d need to purchase a license before distributing any Nextbus-based application. Mr. Smith was also helpful, but not really empowered to speak about the licensing situation.

WHICH BRINGS US TO THE CONTRACT

First: Grey Island Systems is listed as the parent company of the Nextbus that did business with WMATA. So far so good. But then things take a discouraging turn:

But wait! There’s reason for hope: (WMATA is “the Authority” in this excerpt):

And finally, vindication (CIS = Customer Information System, i.e. Nextbus):

It’s possible that I’m missing something, but at this point I think my pre-contract understanding has been validated: WMATA has full rights to the data, which means it can give the data away if it wants to. Now we just need WMATA to give the all-clear! We may also need them to mirror the data or otherwise ensure that Nextbus can’t complain that we’re unfairly hammering their servers. I’ve got some ideas about how to proceed, and I’ve got a serendipitous meeting happening in the next few days — more soon, I hope!

Matt has taken my offhanded complaint about DC beer prices and placed it within the context of social justice, noting that DC’s high wages account for its high beer prices (our average drink prices are comparable to New York’s; our median annual wage, at $57.1k, is somewhat higher than New York’s, at $52.8k; both are significantly above the national median of $42.3k). His basic project is all to the good: thus is the charge of any thoughtful person concerned with the common weal and/or our ability to drink away worries about the state it’s in.

However! I object to his specific analysis of the situation for two reasons.

The first concerns the relevance of the wage figures Matt cites — I think the average DC beer-buyer is poorer than those numbers imply.  Matt’s reliance on pan-workforce statistics is understandable, but still insufficient for the task at hand. One needs to look at the bar-going slice of the population in order to characterize the market for happy hour beer. Sadly, the BLS does not consider this a focus of their work. However! Both New York and DC are on the high end of the marriage age spectrum, so let’s assume that bar-going rates are roughly similar for both populations. I contend that DC’s drinking class is likely to be impoverished relative to New York’s, for two reasons. First, DC’s young professionals are likely to have a substantially larger average debt burden than those of New York, given that we have twice the incidence of graduate degrees and, one imagines, student loans (UPDATE: proof!). Second: while I don’t have stats to back it up, the incidence of unpaid internships in this city must be abnormally high, even when compared to New York’s wage-depressed economy of starry-eyed small-town dreamers.

The second objection is based on evidence that the supply of alcohol in DC is artificially constrained, producing higher prices than would otherwise be found. I’m pretty sure that Matt’s aware of this effect, because I think I stole the idea from him. Namely: the terrible regulatory situation faced by DC bars. A search for retailers licensed to sell alcohol for on-premises consumption in New York, NY yields 14,718 records. An extremely generous summing of DC’s licensed alcohol sellers — I included everyone but grocery stores, liquor stores and caterers — shows 1,039 licensees. Normalizing this to population is challenging: liquor licenses aren’t administered at the MSA level, and it should be obvious that a larger proportion of DC’s happy hour attendees are from the suburbs than is the case for New York. But Wikipedia puts the workday population of DC at around a million, so let’s run with that number. If we do, we can see that DC has 1.039 milli-watering holes per capita (mWHPC), versus a robust 1.760 mWHPC for New York City (using per capita rates across the entire population — not just the bar-going segment — because of our our assumption that the demographics (if not the economics) of bar attendance are similar in both cities). Even using the formal population of DC (591,833) leaves New York in the lead, with a DC score of 1.756. And this is to say nothing of New York’s later last call, which provides the city’s residents with 8.3% more drinking hours per bar (marginal though they may be).  Normalizing the mWHPC score to DC’s last call, New York emerges with a commanding 1.91 mWHPC_w(2AM) — nearly twice as much as DC!*

The significance of this disparity is bolstered by looking at alcohol consumption stats.  These are actually pretty hard to track down on an MSA level, and obviously state figures won’t do when making comparisons to DC. This admittedly-dated survey is about the best I could find, and shows a rate of alcohol use over a 30-day window that’s five percentage points higher for DC than New York. All else being equal, one would assume that the per-capita quantity of bars would be higher in harder-drinking cities.  Yet in this case we see exactly the opposite.

This all suggests to me that wages aren’t the whole story — at least when that story is told in comparison to New York — and that regulatory forces play a significant role in distorting the DC market for alcohol by keeping supply artificially low relative to demand.

* As I went to sleep last night I realized that, unless something has changed since I last stayed out late and DC bars are now open for 24 of the day’s 26 hours, my math is wrong. The real percentage increase caused by a later last call should probably be substantially higher, even though a contrary effect should be introduced by applying something like a discount rate to the extra hours themselves.  So I can’t really propose a firm number, though using no discount and assuming drinking begins at 5pm would give you an adjustment weight of 1.22; it therefore seems safe to say that New York’s weighted score should be no more than 2.15.

No word yet from WMATA, but I did end up writing a script to grab NextBus’s routeconfig data (download here). Then I tried to match each NextBus-defined stop with the closest one in the GTFS dataset. Some stats*:

• NextBus’s dataset tracks 711 unique stop IDs. GTFS has 10,380.
• Using this function to measure distance, the average space between matched stops is 164 feet. The smallest is 11 feet. The largest is 9/10ths of a kilometer.
• 218 NextBus stops wound up sharing the same GTFS stop.

All in all, pretty bad — this level of data quality is clearly unusable. My GIS skills are weak; this may be my own stupid fault. I’ll consult with some experts and see what I might be doing wrong. But the basic distance-matching idea is pretty straightforward, so I’m not terrifically optimistic. It’s possible that data quality is just going to really, really stink — to be sure, this is not particularly encouraging. Here’s hoping we can get a proper lookup table out of WMATA or NextBus. Otherwise I don’t see a great alternative to manual intervention.

* These numbers ignore the routes that NextBus tracks but which GTFS does not; those are B99, F99, L99, NH1, P99, REX, S80 and S91 (they appear to be shuttles and the like). I haven’t yet identified the routes that are in GTFS but not tracked by NextBus.

UPDATE: Scratch that. I’ve hit a roadblock related to serial communication in DD-WRT. If you’re only interested in using Python for socket communication and simple stuff, the below text is still useful. But I’ve concluded that if you want serial, DD-WRT is not the way to go. I’ve moved on to OpenWRT — you can see the first step in this process here.

You might remember that I’ve done a bunch of work on integrating the Fonera and the Arduino. It’s a handy setup for microcontroller hobbyists: a reflashed router provides a wireless network interface that’s much cheaper than the purpose-built alternatives available for the Arduino, and one which, by virtue of its Linux firmware, is also considerably more powerful. The downsides are its size and power requirements, but for most applications those aren’t big concerns.

Last time around I just ran a cron job on the router that periodically fetched data from a web server and shoved it to the Arduino using the busybox-based command line tools that come with the DD-WRT firmware (wget, bash, stty). For my next project, those tools aren’t going to cut it. I need two way communication between the Arduino and the Fonera, and I need less latency and more flexibility than those command line tools can offer. I need a proper scripting language.

Unfortunately, things are a bit cramped on the Fonera. There’s an apt-get-like system for installing add-on packages to OpenWRT-based firmwares — it’s called ipkg — but it’s broken on the Fonera, and there’s very little space for the installation of such packages.

You can get around this, but it’s a little tricky. First: realize that ipkg is only sort of broken (on the newer firmwares, at least). You can’t update the list of packages. You can locate a package, copy it over and do an “ipkg install filename“. You’ve just gotta find the right one. (Also: note that I’ve read that using ipkg to remove items on the Fonera can badly screw up your filesystem. Better to remove them manually and/or reformat your JFFS partition.)

Packages with “mini” in their name are probably your best bet. Now you just need to find a compatible one. From the DD-WRT List of Supported Devices we can see that the Fonera uses an Atheros chipset. Start running Google searches like this one and you should be able to find what you need. Here, I’ll make it easy for you: this is the ipkg I used to install python on the Fonera. Copy it over to the Fonera with scp or something similar, then run “ipkg install python-mini_2.5.1-2_mips.ipk”. Success!

Well, until you try to do something with it. I needed three things out of this python installation: simple client/server network functionality (the socket module); the ability to fire off shell processes (the os module); and the ability to communicate with the Arduino over a serial link (the pySerial project).

The first is pretty simple. For reasons that I haven’t bothered to figure out, the socket module is named _socket in the minipython distribution. Adjust your scripts to do an “import _socket as socket” instead of “import socket” and you should be fine. I was able to run the first two examples (client and server) on this page, anyway (debugging the other end of the connection with netcat on my laptop). Good enough for me!

Firing other programs runs into problems. Try to import the os module and you’ll get some complaints about a missing UserDict class. I believe that this class wrapper has been deprecated; I cobbled this UserDict.py together from some Google searches, and after placing it in the lib-dynload path (on my router it’s /jffs/usr/lib/python2.5/lib-dynload/), I can import the os module successfully. I have not tested it thoroughly. os.system(‘touch WHATEVER’) works fine, but beyond that I can’t make any guarantees.

Finally, there’s the question of serial communication. This is actually harder than is may sound: there’s a hardware issue we need to work out before we worry about pySerial at all.

The comments on my Dorkbot post revealed a problem I didn’t know I had: the serial voltage of the Arduino is different from the serial voltage of the Fonera. The former uses 5v, the latter uses 3.3v. This is a pretty common situation, actually, and it’s generally solved through the use of a chip like the MAX232. Fortunately, we can get away with even less. The Arduino’s designers anticipated this problem and made the device able to recognize 3.3v serial input, so signals from the Fonera to the Arduino should work fine. This is why my Dorkbot project worked even though I hadn’t accounted for the voltage mismatch.

But the voltage differential of signals sent from the Arduino to the Fonera needs to be explicitly dealt with — we can’t rely on the Fonera to handle the higher voltage as gracefully as the Arduino does. Fortunately, shedding DC voltage is a lot easier than boosting it, so this is also a pretty easy problem to solve. We just need to make a voltage divider circuit that goes from 5v to 3.3v. It’s basically two resistors, simply arranged. Easy! Alternately, you might be able to step the voltage down using the magic voltage-dropping properties of diodes, but I haven’t bothered to try.

Alright! So our signals are aligned. Back to the software problems. pySerial requires the os module, but we’ve already solved that module’s UserDict problem; on to the next headache. This one is called termios, and it’s a compiled library, not just a python class. It’s commonly included with python, but apparently not with the stripped-down version that we’re using. That’s a drag. We need a version of this library that has been compiled for the Fonera’s MIPS processor. Compiling such a thing on a non-MIPS processor (like your computer) is possible, but cross-compilation is fairly involved to set up. I wanted to just find a compiled version and drop it in.

Once again, googling around downloads.openwrt.org turned out to be a good idea. But don’t go looking for termios — you won’t find it. Instead, I looked for a beefier python ipkg — one designed for more powerful machines, and which contains all of the normal libraries that ship with python. I found it, downloaded it and then uncompressed it (ipkgs are just tarball archives with a specific layout). That yielded two more tarballs; I opened the one named “data.tar.gz”. Within the directories spawned by that I found termios.so. I took that file, dropped it into the aforementioned /jffs/usr/lib/python2.5/lib-dynload/ directory, and was immediately able to import pyserial. Well, okay, not immediately — the first version of the python ipkg that I grabbed was built for mipsel processors, not mips, and gave me a helpful error message when I tried to import it. Once I had the right binary everything went surprisingly smoothly. I haven’t currently got the necessary hardware set up to properly test pySerial, but will report back when I do.

For those of you trying to follow my steps exactly, here’s the termios.so that I’m using. This method should work more generally for selectively loading python libraries that you need onto your Fonera.

Incidentally, all of this is a massive overengineering of what’s necessary to finish my Artomatic project: those requirements could probably be satisfied with a wireless doorbell kit from Logan Hardware and a couple of 555 ICs. But I want this system to ultimately work over the internet, so for me it’s worthwhile to create an IP-capable device.

I’ve committed myself to getting something ready in time for Artomatic, and that probably means that my GTFS explorations will be delayed for a bit. It’s probably for the best, given the timing of the iPhone 3 SDK.

On the other hand, the folks who run the Chinatown bus frown on the use of soldering irons while in transit, so I did spend a little time writing scripts to chew through the dataset. At this point I have 85,921 CSV files delineating exactly where each bus or train is supposed to be on a second-by-second basis during a typical weekday. Well, alright, not exactly: when a unit is between two stops I just interpolate its position linearly — “as the crow flies”, in other words. Still, sort of neat. Fun fact: under the current schedule, the last trip of weekday Metro service concludes on exactly the 100,000th second of the day (which is actually part of the next day, but the trip started before midnight).

Anyway, by counting the lines in each file, I was able to generate the following graph. It shows pretty much what you’d expect: there are a hell of a lot of active trips during rush hour. Still, it feels good to have wrangled with a dataset of this size and produced an actual image from it. Also: there must be well over a thousand Metro and Metrobus operators out there! It explains a few things: given those numbers, you can see how it’s essentially a statistical certainty that on any given day one of them will punch someone in a mascot costume.

Sunlight Labs has been running its now-annual Apps for America contest, and this morning I’ve been paging through a bunch of the entries. There’s some pretty cool stuff! I haven’t made it all the way through the 40-some submissions — and frankly, I’m just trying to orient myself, not delve into the apps with attention to make solid judgments about which ones are worthwhile (this is where you notice that the views expressed here have no bearing on the contest or its judging, are not representative of my eployer, are completely misinformed, etc.). But I feel like pointing out two entries in particular.

First, Buddy, partly because I have a soft spot for chatbots but mostly because I find the robotic logo both adorable and terrifying.

And second, there’s Filibusted, which my filibuster-hating friends may find interesting, or at least encouraging.

But do check out the rest of the entries — there’s some really good stuff in there. Although I should probably warn my fellow D.C. residents that repeatedly stopping yourself from entering your own zip code when searching for information about congressional representation is kind of depressing.

Man! There are few things I like more than getting back from a good show at the Black Cat. Chris suggested we go see his buddy’s CD release party, and it was unexpectedly great. With the exception of the City Veins’ horn-heavy, guest-star-laden offering, it was probably the best local CD release observation I’ve attended.

A big part of that was Olivia Mancini, whose opener status continues to perplex me. She’s got stage presence, she’s got pipes, she can play and, most importantly, she’s got the songs to make all of the aforementioned matter. All I can hope is that she’s on deck for the next doo-woppy pop/rock slot — between the Pipettes and the Ting Tings it seems like the market can bear one such act every year or two. Olivia certainly deserves that level of… well, if not stardom, then whatever replaced it in the current music industry landscape.

Olivia Mancini & the Mates – Jealous Type

Next up: The Caribbean. I spent a large part of their set retrieving beer, so I’m probably not in a good position to judge them. What I heard I mostly liked, but I think it’s safe to say that they’re not really my thing.

Finally there was Impossible Hair. And while they were oversold — if pressed, I would rate their haircuts as merely unlikely — they proved to be an excellent rock band. It was a tight performance filled with really, really good songs. Recommended! Would buy a t-shirt from again. Go see ‘em if you have a chance — looks like your next opportunity closer than Baltimore will be May 13 at Velvet. Unless you’re planning a trip to Europe, that is.

Impossible Hair – What is the Secret of Impossible Hair?

(which seems to be their entire album, unfortunately — I wish they offered some individual tracks on their site)

Last night I finally found a moment to begin playing around with the WMATA GTFS data. Well, that’s not quite right: I tried to use the Django ORM to load the dataset when it was first released, but a memory leak killed the process when I left it to run overnight — something that doesn’t bode particularly well for our use of the ORM at work.

Last night I shoved the data it into a barebones MySQL database and added a few convenience columns to make it easier to work with the embedded time and date information. The scripts I used to do this take a little while to run (a few hours, perhaps, on the index-creation step — this could doubtless be sped up by moving the math into the SQL itself). And my use of MySQL is somewhat dumb, as the GIS capabilities of Postgres make it a more obvious choice. But if you’re just looking to get started with this stuff, this will at least save you from having to write the CREATE TABLE statements yourself. It’s all up on GitHub — have at it.

The next step for me is to generate a second-by-second snapshot of where every bus in the city is throughout the day, then feed that into Processing to make a movie like this one. Maybe I’ll even add a soundtrack this time — anybody feel like doing some field recordings of Metrobuses?

Kriston asked Ryan and me to write about Larry Van Dyne’s article on historic preservation for DCist. This was at least half of a good idea: Ryan always has interesting things to say about the business of running cities. And for my part, I imagine Kriston was counting on me to bring my own trademark brand of techno-philistinism (“Why preserve buildings when we can just make models of them in Second Life?”).

The joke’s on him, though. The past couple of years’ worth of Ryan’s writing have been so convincing that whatever opinions I may have originally had on these matters have been almost entirely replaced with his — this happens an embarrassing amount when you have smart friends who write a lot. I’m afraid that the essay I produced is consequently just a half-baked version of the one that Ryan wrote. On the upside, I was glad to see Ryan take the potshot at the FBI building that I’d barely restrained myself from making (it’s so awful). So really, he wins on all fronts — go have a read.