Notwithstanding last night’s late-breaking pumpkin seed roasting and the Obama jack o’lantern rotting in my front yard (thanks, neighbor), it’s probably time to let go of Halloween 2012. But it was a good year, marked, in some subtle ways, by more and more friends indicating that their crazed devotion to the holiday is just as deep as my own. I find this immensely encouraging.

The only down notes: the continued lack of a Fickeween-competitive party venue (although: tremendous thanks to Megan, Peter, Kate and Brant for leaping into the breach so ably; I just miss running my own party); and the unavoidable aging and en-lamening that makes it less and less likely that we’ll find one while it still matters. There was also the hurricane’s cancellation of Sunlight’s Halloween Open House (which is a good time, though I am a little saddened, even if only by the symbolism, to have been forced to reprogram my decorations and general spooky zeal away from a big huge mess of a dance party and toward a professional event).

But nevermind all that. As with last year, I redirected my enthusiasm toward my costume, and it all turned out pretty well.  Dave got the best shot of it (apologies to him and everyone else whose Instagrams I’m about to rip off):

Here’s a pretty good (almost) full-body shot:

I’m proud of the engineering behind this one, so excuse me while I indulge myself with a walk-through.  First, the easy stuff:

I veered further off-model than last year. In part this was out of simple necessity: Mr. Freeze has gone through a bunch of different character designs; and the realities of fitting a dome over a human head (especially my outsize noggin) require different spacing for things than in the cartoons. But it also reflected preference: I don’t actually like the gray torso in the animated series original all that much, and didn’t feel like trying to implement a design as focused on conveying a sense of boxy roboticism.

But the gloves are a legitimately nice touch from the original design. Victor Fries is a scientist; scientists wear purple nitrile gloves. Easy!

The cuffs are made from some cheap jersey sheets I bought on Amazon, rolled into cords and covered in colored duct tape. I had a lot of success last year with colored duct tape, and it proved really easy to work with this year, too — when something only has to work for one night, it becomes a really fantastically easy and versatile fabrication technique. The whole thing is secured by adhesive velcro, a shift from last year’s use of taped-in-place neodymium magnets. Magnet fetishism is fun but not all that practical, it turns out.

The mantle is posterboard (which these days is cheapo, paper-thin garbage! scandalous) and more tape. The cut-off bottom of a stackable garden cloche adds some structure and stability. Incidentally, I had no idea cloches were a thing until Fancy Hands turned them up in response to a costume-related request I made. Thanks, guys! The top dome, as you might imagine, is an uncut cloche from the same three-pack.

On to the electrics! I’m pretty proud of this. The LEDs are mounted on an adhesive-backed strip that I got off of ebay. These things are absurdly cheap — they’re sold all the time for about $1/meter — and are very pleasant to work with. All of the resistors are in place, the adhesive backing makes mounting easy, and you can cut them to various lengths with a pair of scissors. Highly recommended.

I terminated the power leads in a barrel connector, the other end of  which I ran down to a belt I contructed, again, out of colored duct tape and velcro. The buckle ornament was a $2 5K potentiometer from Radioshack (pre-soldering shot here; here’s the front). It’s a circuit simple enough that you might’ve built it in elementary school, but it did the job nicely. LEDs’ perceived brightness level doesn’t scale linearly with current, meaning that there was a bit of a brightness cliff right in the middle of the potentiometer dial. This worked out fine, since I mostly just wanted to pulse the display in time to music or to surprise people, not fade it smoothly — having a quick transition point was desirable for this. If you wanted to do it “right” you’d probably want to move things to a little AVR, setting the brightness with PWM control of the duty cycle rather than simple resistance. Actually, a 555 in astable mode might work very well for this, now that I think about it (though the AVR would allow for some cool automatic effects). But this was basically fine, and dead-simple. And having a costume that involves integrated circuits sounds like a good goal for next year.

The LED strip industry was basically invented to help stupid men make their cars look tacky, so it all runs on 12V. This made a sealed lead acid battery a natural choice:

Pros: the aforementioned voltage; cheapness; safety and simplicity of charging relative to other chemistries like lithium. The downside: weight. But this thing was only 3 or 4 pounds, and I don’t think I used more than a fraction of its charge over the course of the night. A cut-up black nylon backpack let me carry it around fairly comfortably (though getting it mounted on the backpack required some creativity).

Also worth noting: the duct tape pouches for credit cards, ID and my iPhone. I’m rather proud of those last-minute additions. More support for the flexibility of the velcro/duct-tape combo. I had spare coin cell batteries taped in various places, too — more on that in a second.

My freeze gun was the last, and most half-assed, thing that I tackled. But it was also the most ambitious.

That’s a bicycle brake lever and cable that a gentleman at Bicycle Space sold me while he replaced my wheel. Pretty cheap! Going to a bike shop was a good idea: I wouldn’t have known that I needed little metal collars for the cable housing, for instance, but they threw ‘em in for free, plus they provided some valuable installation guidance.

The inside of the gun is PVC, which was probably overkill. But, as I said, this was ambitious, because everything was connected to an actual pressure vessel. In this case, a fire extinguisher. I drilled holes through the two handles, then terminated the cable with a couple of bolts tightened in an appropriate spot. The carabiner is there to hold the safety pin in place. The whole thing was connected to the freeze gun by both the brake cable and a pressure hose and clamp.

Did it work? Well, sort of. The trigger mechanism worked great! But I didn’t appreciate just how little charge is in a conventional fire extinguisher, or how focused its design is on moving yellowy fire retardant powder around. The powder got stuck in the hose (and spread everywhere — very glad I did this outside), and the pressure was discharged after two quick test bursts. Boo. My dreams of an actual freeze gun: dashed. If I had to do it all over again, I would probably look at a CO2 extinguisher, particularly since we have a 25 lb tank at work that I could probably have used to refill it after any test firing. Ah well — sloppiness on my part.

The final bit was the goggles. These were a big hit:

They were also pretty simple to make. You can see the construction here. I had bought a bunch of CR2032 cells off Ebay a while ago for my bike lights. Protip:  batteries are one of many electronic-y commodities that are super-cheap on ebay; in this case, just 5% of the retail CVS/Radioshack price, if not better.

So I had a bunch of these things hanging around, and a few nice holders (I don’t even remember what project that was for). I added a couple of cheapo Radioshack switches and some not-so-cheapo Radioshack red LEDs (the LED Ebay/Radioshack price difference is perhaps the most astounding and offensive). I threaded things through the side ports on a pair of cut-rate welding goggles that helpfully came with a set of clear lenses in addition to their impractically dark default lenses. I scuffed the LEDs with sandpaper to make the light diffuse better, and bang: I was done! I didn’t even need a resistor — the battery’s internal resistance is enough. Each eye is basically an LED throwie, but with two LEDs instead of one and no magnet. I changed the batteries once during the night, but this was mostly because I’d been running down the originals a lot in the preceding weeks (the goggles were the first thing I built).

Like I said, these were a big hit — several people said they wanted a pair. If I did it again, I would choose a smaller switch and move to surface mount LEDs (and perhaps more of them) so that the light sources wouldn’t be so individually noticeable. The LED strips I used for the mantle/dome might be a good choice, actually, if you could pare down the sides. Better still, you might use bicolor LEDs and let people swap between modes. Hmm…

As is probably clear, I get way into this stuff. But this was much less work than it probably looks like. I spent a Saturday afternoon on the last-minute gun bit (which included some overkill LEDs of its own), but the rest of it was done in three evenings’ worth of puttering around the apartment while watching Battlestar Galactica.

The biggest revelation, for me, was understanding how much of successful fabrication is about knowing what to source and where. Knowing about the existence of the LED strips made everything much easier; knowing to get them from Ebay instead of an auto shop made it cheap. Same goes for having the batteries and holders lying around, and the right crimping connector tabs for the battery — something I’d bought years ago for who knows what, and managed to remember I had on hand. Pretty much everything else came from Amazon Prime or the hardware store around the corner.

Basically, I felt like Adam Savage seems to in this video:

I’m much less talented, and have much less impressive tools (just some electronics bric-a-brak; meager compared to a guy who has a band saw on hand). But I can relate to the experience of seeing how a system is going to fit together, and building it with surprising speed thanks to the investments you made in parts (or yourself) in the past. That’s a very nice feeling — one of the best things about engineering, if you ask me, even if it’s in service of something as silly as a Halloween costume.

I’ve been meaning to write about the Breaking Bad season premiere all week; now with the next episode about to air, my window for having anyone care about the viability of last week’s (spoiler alert!) magnet caper is rapidly closing. Nevertheless!

I was curious to know whether the episode’s scheme — which centers around the use of a salvage yard electromagnet to erase a laptop drive from outside a police station’s walls — was at all plausible. I imagine the Mythbusters will tackle this in highly entertaining fashion a year or so from now, but I wanted quicker answers. Conveniently enough, my friend M works at a related job, figuring out the chemistry that allows hard drive platters to be coated with various metals. So I wrote her and asked if she knew anything about whether this was plausible. Not specifically, she said. But! Her all-around science talent and experience provided some promising leads:

I’m not sure how feasible this is. You would need to generate a strong enough field, get the field close enough, and also sustain the field long enough. I think that article mentioned that the power from the batteries were an issue and I think that is the biggest obstacle for a “portable” system. Not sure if the car batteries could sustain the juice long enough for the hard drive to get completely erased. Also distance is an issue. I think the field drops off exponentially away from the source, and other materials in the way (like the building wall) can dissipate the field depending on its dielectric properties. I don’t know how well computer hard drives are shielded or what strength field you’d need to erase one, but it could be possible. Do you know what you would need for that in terms of strength and time? Seconds? Minutes?

We never erased a computer when I was in grad school, but we always kept metal and electronics outside the 5Gauss line when working near magnets. In the fringe field of a couple Gauss outside this, it was strong enough to erase subway tickets but not credit cards and definitely not a computer. To get a feel for lengthscales, a magnet of ~90,000G had dissipated to 5G by about 7-9ft away from the magnet. Metal would not get pulled from our hands toward the magnet unless we were within ~3 feet away.

I would be curious to find out how strong a field you need to lift a car though. I thought those junkyard magnets you have to be really close to the surface before it actually sticks?

This led me to some productive Googling — well, productive in a certain sense — that turned up a few more interesting details. The following has been written by a guy who never even took enough physics to get through Maxwell’s equations. Still, I think it’s not too hard to reach a plausible conclusion through some back-of-the-envelopery.

There are basically two considerations that M is pointing to: field strength and how easy it is to erase a given type of magnetic media.

On the field strength side, the news is not good for Walt and Jesse. Unlike most emissive sources (light bulbs; radioactive materials), magnetic field strength declines with the cube of distance rather than the square. Exactly why has something to do with the nonexistence of magnetic monopoles (outside of Star Control 2 anyway) and seems to be one of those mind-bending situations where reality knuckles under to some particularly inescapable math. But the upshot is that magnetic fields get weaker very, very quickly as your distance from them increases — faster than your experience with other radiative sources might make you think.

But how strong would the field be at its source, anyway? Here it’s tough to say: salvage magnets seem to be specced by how much scrap iron they can lift, not the precise attributes of the fields they generate. But MRI machines top out around 30,000 Gauss. Is a salvage magnet more powerful? M subsequently warned me about reading too much into the fancy cryogenic cooling of an MRI’s superconducting magnet versus the air-cooled conventional tech in the salvage magnet. They’re different machines built for different things, with very different field shapes, she stressed. All this is true. Still, to me it seems at least unlikely that a salvage magnet could outpace an MRI machine. And judging by the example distances and field strengths in M’s email, it would clearly need to.

Then there’s the question of how much of a field it takes to erase a hard drive. I know a little bit about the considerations here, having looked into magstripe reader technology back when I was fooling around with Metro’s farecards. The ease with which a magnetic medium can be altered is called its coercivity, and as M hints, there are high- and low-coercivity magstripe standards (for any that care, WMATA’s farecards are low-coercivity, and I think not even digital; based on my abortive experiments with them, I believe that they use an acoustic encoding scheme, though I’m not positive).

Anyway! How hard is it to flip a bit on a hard drive platter? Things get tricky here — coercivity is measured in Oersted rather than Gauss, and concerns the B component of a magnetic field rather than the H component (actually, neither are measured in those pre-SI units any more, but “Gauss” and “Oersted” sound a lot cooler than “amp-meter”). (H and B are linearly related based on some constants specific to each material (the fields are functionally identical outside the domain of a given magnetic medium), so all the above business about field strength still applies). Quantifying the coercivity of a typical hard drive — to say nothing of the magnetic shielding effect of the case and other junk around it — is not something I’ve been able to do.

But we have some circumstantial evidence. For one thing, any dedicated nerd will tell you that a broken hard drive is a great source for extremely powerful neodymium magnets. These have nothing to do with flipping the bits on the disk (they’re in place for the voice coil that positions the read/write head over the platter). But it does seem safe to say that having a very powerful magnet — powerful enough that, given a pair of ‘em, you’ll have a hell of a time separating them with just your hands — mere centimeters away from a hard drive platter is not enough to influence the data on the disk one bit, even as it whirls through the magnet’s field at several thousand RPM. It therefore also seems pretty safe to say that you would need a noticeably strong magnetic field outside the device before data loss became an issue. In the show, of course, stuff flies all over the place, so in this respect, at least, Breaking Bad’s verisimilitude isn’t in question.

Finally, I am a little more optimistic about the viability of a battery power source than M is. This kind of project is a great way to wind up with a bunch of burning, half-melted plastic tubs of acid and lead (a horrifying clean-up problem, but I suppose Walt’s seen worse), but an array of lead-acid batteries really can deliver an impressive amount of juice (turning over an engine takes quite a lot of it). Judging from the afore-linked salvage magnet vendor’s site it looks like the show’s creators settled on a realistic voltage; and indeed, Vince Gilligan has said that this was something the writers wasted a bunch of time worrying about.

All in all, though, I think Walter and Jesse probably should’ve stayed in the chemistry lab rather than wandering over to the physics department: for all of Mike’s talk about the evidence room’s impregnability, it sure looked like it was just a cinderblock wall. I suspect some explosives and incendiaries would’ve done a better job of killing the data on that hard drive than an electromagnet could. After all, there’s a reason why geeks tend to talk about degaussing wands for sanitizing videotape, and thermite for securing old hard drives:

Will Wilkinson is too kind to me, but too cruel in general:

[The] hyperventilating false drama about never-delivered transformative change is by no means unique to the tech beat. Here on the politics blogs, we’re only too happy to remind our readers that every coming election is the most important election in a generation, that the fate of our civilisation depends upon which of two barely discernible politicians’ cronies get paid. If we can’t generate a narrative with live-or-die stakes out of meaningless developments in public-opinion polls, then we’ve got nothing worthwhile to offer. Reflecting too often upon the ultimate triviality of almost everything we write about does no good for technology or politics writers, or for their readers. The illusion that the next thing will be truly meaningful has always meant more to us than the reality of the next thing. I agree with Mr Lee that there is something quite sad in the way Mr Madrigal, after having discovered that he has been reporting on nothing of significance, does not then go on to draw the well-warranted conclusion that he has wasted some of the best years of his youth foolishly yammering on about ephemera, but instead doubles down and declares “we all better hope that the iPhone 5 has some crazy surprises in store for us later this year”. But it’s only sad because life is sad. Really, why not roll the rock back up the hill?

I am rarely out-gloomed, but I think this is one such instance. So let me present a case for technology being meaningful. I think it’s possible! Anyone who knows me can tell you that, contra my somewhat embittered bloggy pronouncements, I love technology. I mess around with Arduino on weekends; I obsessively amass, modchip and then fail to actually play game consoles; I spent my holiday building a programmable array of Christmas lights; and I can put my hand on a Digikey packing slip without leaving my bed (though this last credential is perhaps as much about messiness as it is about geekiness).

The point is that I believe in this stuff. Information technology, in particular, is incredibly powerful and democratically accessible, and I genuinely think it can improve our society. When you see me getting upset about the tech industry, it’s because I feel that others have lost sight of this. They’re making this inspiring thing I love into a silly business school game, or they’re making ignorant promises — on my behalf, it feels like! — about things they don’t understand and which won’t ever come to pass. Loudmouths are distracting from good work done humbly. Fuck those guys; I hate ‘em. I wish they would shut up and go away. But since they won’t, we might as well get on with things.

If you’re someone with technical skills, hopefully you you will prove to be better at ignoring those people than I have. Aside from that, I’d like to talk about the ways that I feel a career making technology can be meaningful. Because I really do believe it’s possible; I would hate the people who know me, who work with me, to read this blog and conclude that I feel otherwise.

Not, mind you, that your job has to define you. There’s nothing wrong with doing an honest day’s work and coming home to enjoy your family, or partner, or dog. Pick up a hobby. Enjoy your vacations. In a few short decades you will only exist as the memories of your loved ones. A few more and you’ll be nothing more than a couple of kilobytes in the Mormons’ genealogical databases. I wish I had a better deal to offer, but by all accounts history is relentless, and it seems assured that rocking back and forth muttering/tweeting about “innovation” and “disruption” will be no charm against it. The important thing is to try not to waste the time you have on stupid bullshit.

I should warn you: this will be grandiose and sappy. To wit:

Improve the World

Yes, the hi-tech, still-quite-expensive things that you build will mostly be used by rich people. That’s just a for-now thing, though. Smartphone adoption is already better than home broadband penetration. Speaking very conservatively, in two generations, everyone in America will be using this technology. In four, I’d bet on everyone in the world using it. And in the meantime, you can push on the decisionmakers. Correcting asymmetries of information can ameliorate asymmetries of power, despite the occasional troublingly counterintuitive result. Look at what Public Laboratory is doing: democratizing technology to make it possible for ordinary people to monitor and — hopefully — legally defend the quality of their environment. I’m admittedly biased, but I find their work incredibly inspiring.

A lot of efficiency gains are made possible by better information — dynamic energy pricing systems, car- and bike-sharing fleets, programmable thermostats. Information technology has a real role to play in keeping the earth habitable.

But you don’t have to know anything about IR filters or weather balloons or Arduino to make a difference. Designing a webform that serves the needs of some fraction of a social worker’s clients, freeing resources for others: that’s work that isn’t flashy, but is truly important. By way of example, my friend Chris helped build a clearinghouse of performance data for the microfinance sector, and though I know the day-to-day development experience was nearly indistinguishable from any other CMS-project-hell, it still seems to me a very fine thing to have done. I’m sure that toiling on the EHR problem is even more mind-numbing, and yet it’s unquestionably of huge potential importance. Writing a line of code can feel very distant from the act of directly alleviating human suffering, but that distance is and will continue to shrink.

Create Knowledge

The callowness and innumeracy of those promoting the Big Data brand almost defies belief, but (I should remind myself more frequently) it’s important not to let this distort your perspective. Yes, there are dopes who don’t understand that a properly selected sample of their (inevitably clickstream or social media) data could get them the same “insights” (always insights) as their massive Hadoop infrastructure. Plus it would let them use scientific-looking error bars, which I bet they would enjoy.

But there really are problems in need of solving which are bigger than human cognition. The gulf between the people who think their FitBits will extend their lifespan and the people working on actual computational biology problems is vast, but those willing to traverse it should be celebrated. There are archives to be digitized, regressions to be run, extraterrestrial radio signals to be processed. There are more disciplines than I can imagine that could make use of our skills if only they were introduced to them.

Make Art

All of this stuff is changing us, and we’re going to need to spend some time figuring out how — particularly as the energies, quantities and general magnitudes of the things we can manipulate grow ever more threateningly huge. Somehow we’re going to have to give this old monkey brain the slip.

That would be the pragmatic case, but maybe it’s foolish to try to mount one. What better thing could there be to spend your time on than making beauty? Besides, you’d be hard-pressed to read much of rhizome.org or the (now-defunct) New Aesthetic Tumblr or the increasingly philosophically-minded indie game scene and not come away convinced that a bunch of exciting, fast-moving (and yes, somewhat insufferable) conversations are reaching crescendo right now. It’s getting to be the part of the party where you have to shout to be heard, and either everyone will start to dance or there’ll be a fight or we’ll get up on the roof. Something interesting is sure to happen — it probably already is, in fact.

Try, At The Very Least, Not To Hurt Anyone

There are a few subdisciplines that you should probably stay away from. “Neuromarketing,” Zynga-style games, Klout scores and other algorithmic approaches to eliminating human agency, dignity and/or equality strike me as basically evil, and though the trend they represent is probably unstoppable, I sure wouldn’t want to be associated with it. Ditto becoming one of the quants designing the HFT engines of tomorrow, or one of the parasites that make their living off of SEO.

On the more benign/less high-skill end of the spectrum, coupon sites are starting to look less like a positive-sum marketing interaction and more like a system for skimming small businesses’ revenue. This model has been deployed to arguably good effect in the past (newspapers! Gmail!), but this latter phase seems to merely be subsidizing my fellow yuppies’ lifestyles in a sort of bizarrely regressive retail sales tax scheme. If you have the economic freedom to choose, I’m confident that you’ll be able to find something more productive to do with your time and talents.

If You Absolutely Must Play The Startup Game

Understand that you’re unlikely to come up with a million dollar idea solely by sticking together free software like so many legos, hoping that lightning will strike and you’ll wake up to a valuable population of users who are now pleasantly locked into your product by network effects and/or transition costs. Sure, it happens — for now, Instagram still counts as an example rather than a punchline — but a lotto ticket offers only slightly worse odds, and requires you to spend much less time fiddling with Keynote. It’s simply too easy for other, smarter people to have the same idea and build it. Competitive markets are good for consumers and bad for entrepreneurs.

But if the startup dream compels you, I would suggest two things.

First, realize that ICT makes information cheaper. That’s it, really. If you want to earn money with this technology, you should look for tractable problem areas where information is still expensive.

Second, connect your project to the logistical nightmare that is the real world. Ship physical goods, install a bikesharing fleet, go meet with the bureaucracy to get the data you need for your business intelligence site. These things are hard to do without leaving the house (or at least picking up the phone), and consequently fewer of them are being done. Another handy heuristic bucket: pursue ideas that require capital for things other than loft space, foosball tables and your bar tab. The low hanging fruit has been plucked, in other words. Reach higher. It’ll certainly be more interesting, and you might even improve your odds.

I’m a Lucky Guy

I’ve already copped to not being a startup guy myself. I guess I should probably acknowledge that I’m not a particularly cheerful person, either. But while I have admittedly made some terrible decisions, my professional choices haven’t been half bad, if I do say so myself. I’m extremely grateful to have the opportunity I do: one that affords me the chance to do work that I count as meaningful across a couple of the above dimensions.

I can’t guarantee you’ll have the good fortune I’ve had in finding a fulfilling way to spend your workdays, but I do wish you luck at not wasting your time.

Spencer is right: this Wired piece about DARPA’s Nexus 7 initiative is very good. Nexus 7 is an ambitious data processing effort meant to synthesize both traditional signals (e.g. vehicle tracking data) and unorthodox signals (market fruit prices seem to be their favorite example) into useful intelligence through sophisticated analytic techniques taken from the social sciences.

And it’s a pretty good reminder of why I’m wary of the Big Data movement.  These were my two favorite bits:

On the surface, there wasn’t much to it: just a graph of violence in the Jalalabad region, and a plot of those fruit prices. When the level of violence was stable — reliably low, or reliably high — so were those prices. Fruit sellers knew what to expect. But when there were sudden swings in the number of attacks, the prices shot up.

Therefore, the Nexus 7 team said, you could use the fruit as an indirect indicator of instability.

The reaction was less than rapturous.

“Right from the start, I’m like: Oh. My. God,” one of the people who attended a Nexus 7 presentation tells Danger Room. “A high school kid could do that.”

Afterward, Dugan presented the pilot as a triumph — a “big breakthrough” that impressed a bevy of four-star generals.

Privately, she was underwhelmed. Dugan was looking for projects that could save troops’ lives, and maybe even bend the direction of the war. By that standard, fruit-price swings seemed pretty inconsequential.

But the presenters maintained an aura of confidence. Oh, this is just a test. Give us more data sources, they said, and we’ll make better connections. We’ve got the hardware: a cloud computing platform that would soak up all kinds of classified and open source intelligence data. We’ve got the software: these social science PhDs and counterinsurgency veterans, who can figure out how to apply that data to rebuild Afghanistan.

and:

“One assumed there was some secret mound of data to be exploited. But it’s just not true.”

I’ve fallen prey to this temptation: thinking that your mastery of awesome tools means you’re about to do some awesome stuff (perhaps via some cleverly counterintuitive Freakonomic insight). Unfortunately, it’s not that easy. You actually need to have a great idea before great things will happen, and it’s difficult to come up with great ideas unless you both know and care — deeply — about the topic you’re planning to examine.

It’s important to acknowledge that the story of Nexus 7 seems to be told, to some extent, from the perspective of people in the military establishment who feel insulted or threatened by the project.  But that in itself is telling: it’s never a good idea to enter a field of inquiry with the assumption that those who preceded you were well-meaning simpletons — particularly when your reasons for thinking so boil down to a difference in the complexity of your tools.

I think this same story is about to unfold in the tech industry, albeit with a more cheerful tone.  Consider this recent post from Read Write Web about the explosion in job listings mentioning the phrase “data scientist”:

“Right now, everybody with data knows that there’s value in there, that they should be doing something,” says Edd Dumbill, program chair for Strata, O’Reilly’s new conference on Data. ”Trouble is, nobody’s entirely clear on the next steps, but they do know that a data scientist can help frame questions and transform data into useful insight.”

They don’t “know” this. They’re assuming it.  And this leaves me worried, because the ability to draw meaning from mountains of information is almost always going to depend on the specific question being examined more so than the tools being used or the investigator’s level of enthusiasm for the idea of quantitative analysis.

It’s not that I don’t believe in the techniques and tools that have these folks so excited.  It’s not even that I think nothing will come of data-rich firms applying quantitative analytic techniques. These things have got me excited, too!  I’m trying to make sure we take advantage of the same kinds of tools at work.  Still, there’s no substitute for good ideas.

To me, this wave of hype doesn’t seem much different from the one that occurred at the start of the last decade.  ”Look at the power of webservers and online payment processing!” we exclaimed. “Can you imagine the benefits they’ll yield when applied to the problem of selling pet food?”

Those things are powerful. But that’s beside the point.

About three weeks ago I finally received my Wakemate. A part of the burgeoning quantified self movement and yet another example of a product made possible by the last half-decade’s debut of cheap silicon accelerometers, it’s exactly the kind of thing you’d expect me to buy.

Wakemate is built on three ideas borrowed from sleep research. First: we experience a recurring cycle of sleep states during a night’s rest. Pretty much everyone’s aware of this, if only because it was part of an episode of Star Trek. Over the course of a night you spend progressively less time in a deep sleep state, and more in light states where dreaming occurs.

Second: these sleep states are measurable using a technique called actigraphy. As this paper explains, during sleep the motion of your non-dominant wrist seems to correlate pretty well with more precise measures of sleep state. You can get a decent measurement of sleep state just by tracking what your left hand is up to.

Third: your level of grogginess upon waking varies depending on which part of your sleep cycle you’re in when your alarm goes off. This is known as sleep inertia, and the WM’s creators have a few paper excerpts about it here.

The Wakemate folks took these three ideas and combined them — in a way sure to elicit much (potentially justified) tongue-clucking from sleep researchers — into a product. Put on a wristband, load a program on your phone, and set a twenty-minute window during which you’d like to wake up. The device keeps watch during that time period for moments when you seem to be in a light sleep state, doing its best to find one and rouse you in a way that minimizes grogginess (if it doesn’t find one, it’ll wake you up at the end of the time window). The idea’s so clever that I barely care whether it works.

Snakebit

I first heard about all of this from my colleague Kevin back in February of last year. It sounded like an interesting idea, and for just $5 you could reserve your place in line for the device (it ultimately cost me $50; it’s now selling for $60). Wakemate is a Y Combinator startup, and its founders went through a semi-hilarious series of problems as they tried to ship their first product. Bad wristbands. Delayed electronics. Problems with Apple certification. The thing finally arrived, months late; the next day I got an email warning me that the included power adapter might burn my house down. And for the first week or so, the app only woke me up at the end of the 20-minute window — at the fail-safe point — seemingly because it wasn’t able to communicate with the wristband (I had to reboot the latter unit multiple times to get the night’s data downloaded). With the exception of the charger (any USB adapter will do), all of these problems have been fixed. But it was a bumpy ride. Kevin still hasn’t received his.

Surprisingly Plausible

Here’s the source data from last Thursday’s sleep, and Wakemate’s classification of that data into sleep states.

This seems kind of reasonable! Check out the huge spike at the beginning of the accelerometer time series. That’s when I was still awake and reading. Over the course of the night I went through about four cycles, spending less time in deep sleep each iteration. You can see four clusters of movement data, too. This isn’t the cleanest night’s worth of data — I didn’t feel like clicking through all of them to find the tidiest — but as I’ve looked at these over the past few weeks, I haven’t yet seen any patterns that seemed implausible either in terms of the reported sleep cycle pattern or its correlation to the underlying movement data.

Does It Work?

At first I was a bit disappointed: the central gimmick of the WM didn’t seem to be working. If anything, I seemed to be groggier than usual when I woke up. But as I already mentioned, I eventually realized that the alarm was only going off at the end of the twenty minute window. I emailed WM’s extremely responsive support line and was told that the issue had already been fixed in software and was just waiting on Apple certification. Happily enough, I was able to download the update by that evening. And although the days since have seen a suspicious number of wakings during the first minute of the alarm period, I’m actually surprised to report that it might be working. I’m still plenty groggy during the minute or two when I futz with the alarm (and report my level of alertness using the software slider). But I’ll be damned if I don’t seem to snap out of it sooner than usual.

On the other hand, this may not have anything to do with the timing of the alarm: it might just be that I’m getting more sleep. Which brings me to the best thing about Wakemate.

Data Porn

I was most excited for the alarm functionality, but the analytics package that WM provides has proven to be its most compelling feature. Your nightly sleep data is uploaded each morning and placed into an attractive interface. You can easily find information about time spent asleep, how long it took you to fall asleep, and how many times you woke up in the night. It’ll also show you how your recent performance in these areas compares to your career average, and to that of the entire population of WM users.

You can also tag each night’s sleep when you set the alarm — did you read before bed? go to the gym? drink alcohol? — and perform comparisons between tags.

Perhaps less helpfully, WM provides a “Sleep Score”. I can’t find any detailed information about how this is calculated — I suspect that this opacity is intentional, both to allow the formula to be tweaked and to keep users from trying to game it. And while it’s sort of amusing to have competitive sleeping leaderboards (how does Justin Sweetman sleep so virtuosically?), the scores seem to me to be basically bullshit. I tend to score highest when I’ve gone to bed late and with alcohol in my system; as you might guess, my scores don’t correlate very well with how rested I feel. You seem to be penalized for “low quality” sleep, even if it means more sleep — in other words, collapsing from exhaustion and sleeping like a corpse for three hours might earn you a higher sleep score than getting a normal night’s rest.

Since I’m on a bit of an Excel kick, here’s a plot of my sleep scores versus minutes asleep (WM recently added the ability to download your data as a CSV, which is nice of them).

Admittedly, I don’t yet really have enough data for that trend line to be meaningful. But I have my suspicions.

Still, I’ve actually found the product to be worthwhile, not just as an interesting exercise in navel-gazing. For instance, it turns out there’s a reason my Sundays aren’t very productive:

I honestly had no idea I was getting so little rest on weekends.

In general, I’d say that it’s been surprising and useful to have the amount of time I spend asleep quantified. I’ve always needed a relatively large amount of rest in order to function. I have nothing but admiration (and jealousy) for those of you who get five hours a night, hop out of bed, write a thousand words and run a half marathon. But I just can’t do it. At the absolute depths of puberty/hibernation my body, when left to its own devices, was helping itself to twelve or thirteen hours of sleep a night. That’s thankfully not necessary any more, but I’m certainly not at my best when I get less than eight hours.

Wakemate has actually been useful for telling me when I’m not taking very good care of myself, and has provided a small but real incentive for paying attention to when I should call it a night. Admittedly, you can see that incentive diminishing in the above graph as the novelty of the WM wears off. Still, I’ve found the information useful.

Anyway, if it sounds appealing, you might want to give it a try — although until I’m more convinced of the alarm’s utility, I’d suggest considering the FitBit as well. I haven’t tried FB, but in addition to sleep analysis it quantifies your activity during the day, which might be interesting. It hasn’t got any anti-sleep-inertia alarm functionality, but perhaps that’ll be added later.

There were two good pieces on NPR this morning discussing the reaction to Wikileaks.

First, there’s this story, which briefly gets at some of the concerns about Twitter as a platform for activism that I tried to express in this piece. The point Shirky makes is an important one: so much vital expression now happens in privately-controlled mediums that nominal speech rights are often a bit beside the point. This, combined with the fact that we seem to be relitigating the “should shield laws apply to bloggers?” question, makes me think that we’re collectively more confused about how you can and ought to be able to speak on the internet than I would’ve expected. Hopefully we’ll muddle our way through to a productive conclusion.

Second, they did a piece discussing the costs of prosecuting participants in Operation Payback, and it’s also worth a read. I’m sympathetic to the basic dilemma: going after the participants is a lot like calling in Interpol for a vandalism case. Individual actors are only responsible for substantial damage when considered together; and nobody’s very happy about the idea of locking up bored rich kids. Still, it’s worth keeping in mind that this is what a hard computer crime problem looks like. It’s not thrilling to me to read that the FBI and DOJ are basically shrugging their shoulders and saying, “Eh, that sounds too hard.” This is all the more galling when these agencies are bothering to pursue computer crime enforcement agendas around intellectual property. Like it or not, distributed international attacks are the class of problem most in need of solving.

One other thing I’d add: keep an eye on how useful the “cybersecurity” community makes itself during this process. My guess is that they’ll keep their traps shut (or at least spend their time gleefully fretting about what kind of funding requests Stuxnet will necessitate) while the “computer security” community does the hard work of grappling with the Operation Payback DDoS. On the other hand, I suppose the cybersec guys were probably the ones behind DDoSing Wikileaks, so, y’know, your tax dollars at work.

Third: about that DDoS. I’m not sure what to say about it, really. I find the argument that it can be considered civil disobedience to be more compelling than I would’ve expected. I’m somewhat sympathetic to arguments like the one Tim makes here (or that’s discussed in this comment). But I think the idea that past instances of civil disobedience were done in an orderly manner by uniformly thoughtful people who did their best not to inconvenience anyone else is probably wishful thinking. I’m no expert on this stuff, but that strikes me as the sort of perceptual shift that happens after a movement is vindicated by history. You’re going to have to piss some people off. Otherwise you might as well go schedule a protest march for all the good it’ll do you.

And although I’m not particularly sympathetic to Payback’s targeting of Amazon — cowardly though the firm’s behavior may have been, too many others rely on the AWS infrastructure, and there are plenty of hosts out there — the payment-processing and DNS control points now being counter-attacked by Operation Payback really do represent worrying concentrations of power. These systems are controlled by entities that are immune to public oversight, yet seem to be completely compliant when state agents ask them to restrict their customers’ liberty. That’s a recipe that should worry libertarians a lot more than it seems to.

On the other hand, there’s Gawker. And although it’s probably a mistake to conflate all of these actors, there do seem to be some connections. The “Anonymous” community is becoming a sort of petulant digital Fight Club that’s going to be very difficult to combat, and which behaves in a chaotic and unprincipled manner. That they’re targeting those who dare to talk about them is pretty dismaying; that the only way to respond seems likely to be a combination of quiet state cooption of ISPs and throwing children in jail — that’s incredibly depressing.

Alex Payne points to a blog post by Ben Laurie that discusses Diaspora and Haystack, and how projects like these can attract huge amounts of press, only to flame out as their charismatic founders’ incompetence is revealed.

I agree with Ben’s post, but it’s worth being a bit more explicit about what allows these situations to arise: the quality of most tech journalism is abysmal. I mean really inexcusably bad. Mainstream publications regularly assign writers to cover the software industry that have a level of understanding regarding the field that would be unacceptable in an intern. The most esteemed practitioners in the tech press are either focused on the consumer electronic user experience or are building personal brands around faith-based tech triumphalist movements.

In this sort of environment, it should be no surprise that an embarrassing hype cycle can emerge — one that talented self-promoters will use to enhance their status and wealth. I find it difficult to assign all that much blame to those self-promoters: the whole problem is that they don’t know any better. What more can we expect? Besides, it’s very easy to start believing your own bullshit once people with seemingly-meaningful professional credentials start validating it. Self-promoters will self promote; it’s not realistic to expect them to be the ones providing diligence.

I suspect that the problem may have to do with the structure of the industry: if you know much about it, you’re probably going to be able to make more money participating in it than writing about it. I don’t know enough about finance to really judge, but it seems as though that press sector suffers from a similar systemic disability — certainly all can agree that the financial press didn’t cover itself in glory in advance of the recent financial crisis. Once that story became big enough, talented generalist journalists filtered in and did the job properly.

But unless and until the skill premium for the software industry diminishes relative to journalism I’m not sure there’s a good way to align incentives in a way that fixes this problem. The best we can do is to recognize that the journalists who wrote excitedly about Haystack and Diaspora made a mistake; they were fooled, and they wasted our time. There’s no need to tar and feather anyone, but their credibility needs to suffer if we want this situation to improve.

Maybe we don’t need it to improve! It’s not that important, to be perfectly honest. But it sure does bug the hell out of me.

Way back when, I wrote a Google Maps application for DCist that overlaid the DC Metro system on the usual GMaps tiles. People found it useful — me especially, since I think it helped me land a job at EchoDitto.  Its only real innovation was some simple, hacked-up geometry that would horrify a cartographer, but which allowed me to make an attractive map that recalled the more stylized WMATA map.  It wasn’t rocket science, but I still occasionally get emails from developers asking me how I did it (which is slightly bizarre, given that the code is right there for them to see).

In 2007 the GMaps API got an update, and I converted the project into something called a mapplet. I had to rewrite a few things, but it was more or less the same.  The main difference was that mapplets were used through the maps.google.com interface — you could add a bunch at the same time, but you could still use Local Search and permalinking and comments about businesses and other Googly innovations from within the interface.  I didn’t have to implement any of that stuff!  Instead, users could simply have their polished Google Maps experience supplemented by my modest mapplet.  Handy.

Unfortunately, over the last few weeks I’ve started receiving reports that the mapplet’s behaving weirdly.  Load the mapplet, then do a search for something — the station markers will disappear, and sometimes some of the lines that are supposed to connect them will, too.  It looked to me like an event handler had started working differently, so I went to investigate.

Alas!  It turns out that v2 of the API has been deprecated.  They’re on to v3 (not so bad) and they’ve discontinued the mapplet platform entirely (bad)!

I can still make the lines appear on a Google Map.  But I don’t think I can do it on the maps.google.com interface.  This is a drag: I don’t think the thing’s half as useful as a standalone product as it is when it supplements search functionality.  And I really don’t want to reimplement the entire maps.google.com interface (even though, yes, they expose the API for their local search stuff).

So! Developers! Anyone out there dealt with this? I’m not eager to dump a huge amount of time back into this project — a project that’s increasingly unnecessary thanks to Google Transit and the addition of transit stations to the GMaps tileset, but which is still useful when you’re working at a modestly wide zoom level.  But it would be nice to get things working again.

Kevin Drum elects to take security advice from Microsoft. This is not a good idea!

In a nutshell: MS says that users ignore warnings about unsigned encryption keys, which makes those warnings useless.  Some browsers, like Firefox, make it really difficult to ignore unsigned keys, but that’s annoying, and MS says we should abandon such efforts.

This is wrong.  Those warnings are saying: “The URL you entered means that you’ve asked for a snoop-proof connection, so okay, your connection to this server is encrypted; however, I can’t verify that this server is who it’s claiming to be.”  Your conversation with the server will be private, but you could be subject to a so-called man-in-the-middle attack, whereby someone hijacks the local network segment you’re on and starts speaking on behalf of, say, bankofamerica.com.  The magic of the certificate authority system means that they can’t do this without generating a warning.

There is one caveat: if they simply don’t try to use encryption, the warning won’t be generated.  This is one of the reasons why you’re supposed to check for https:// in the URL whenever you submit sensitive information; not just so that your password isn’t available to everyone else on the Bolt Bus (though that’s a good reason, too), but also so that anyone pretending to be a server they’re not will get caught.  Unfortunately, people aren’t very good at checking for that little s in the URL or that little key icon or whatever other little security indicator your browser provides, so the bad guys just direct their victims to unsecured sites.

That’s too bad, but it isn’t a sign that warnings about unsigned keys are bad ideas.  Actually, the fact that phishers have drifted away from that link in the security chain means that it’s working properly.  Weakening it isn’t any kind of solution.  MS security researchers would be better-served by spending more time thinking about how to get people to notice when they ought to be using a secure site.

The American Prospect on Trivium!  Worlds colliding!  It’s great!

I read the article, though, and found myself disagreeing with its author, Marisa Meltzer.  How can you provide an accounting of Tumblr’s success and not use the word “Gawker” once?  The service was launched in New York in the right kind of scene, was pitched in the right kind of way, attracted the right kind of writers, and grew from there. Like so many successful products, you can’t simply run Tumblr’s featureset backward through the perfectly-deterministic mechanism of the imagined market and arrive at a proper accounting of why it succeeded and its competitors failed.

I agree that it’s worth talking about the subtle mechanics of the site — how there’s a thumbs-up mechanism but no thumbs-down; I think the consolidation of reader and blog into a single form is a genuinely interesting idea. But Meltzer’s account seemed to me a bit like explaining the success of the iPod via a discussion of its clickwheel.