stupid is cheap
So I guess that Glenn Greenwald and Megan are having a fight over what stories the media covers. This seems a little odd, but I guess she yelled at him, so now his commenters are yelling at her. And while I don't want to align myself with the people who comment on her site, who tend to be (for one reason if not another) some of the worst people on the planet, it's worth remembering just how lousy the metrics data Megan refers to can be.
Consider this admittedly irritatingly-formatted conversation about the Arbitron ratings. Basically: over the last decade or so Arbitron, the one big radio ratings agency, recorded the declining popularity of rock radio, and stations like DC's own WHFS were shuttered (boo!). Then Arbitron developed a new technology that passively monitors what people listen to instead of relying on them to accurately report their listening habits. It turned out that rock radio was considerably more popular than anyone thought — rock listeners were just unusually bad at filling out Arbitron's surveys. With that factor removed, rock stations are now once again popping up in cities that have rolled out the new monitoring technology.
Now, while I certainly don't want to imply that listening to Smash Mouth correlates with intelligence, in general I think it's probably true that it's harder to collect tracking metrics on the sorts of educated, sophisticated consumers that Greenwald's commenters presumably consider themselves to be. Their time will tend to be more valuable, meaning your ratings company will be able to afford to buy less of it. And they may have less time to consume media, meaning there's less collectible data available per capita — even if you don't have to pay for it. They also may be quicker to notice that you're tracking or otherwise inconveniencing them and take steps to stop you. Show of hands — how many people are reading this with Adblock Plus installed?
From listening to Emily talk about her work it's clear that avoiding sampling error is hard enough when the attempt is made in a rigorous, well-funded academic setting. And of course the situation here is considerably worse. Ratings companies like Nielsen, Arbitron and the like have a financial incentive to sample poorly (or as poorly as they can get away with). And, from my cursory understanding of their industry (I read DCRTV! I tell clients which web analytic reports to buy!), it doesn't seem as though the firms sell themselves with their highly scientific methods so much as they sell themselves by being the only game in town.
So it doesn't seem totally outrageous to me to suggest that metrics-driven editorial decisions may favor idiotic stories even more than the economists' magical intersecting lines say they should. I think the internet has helped reverse this trend, but America is still probably being fed stupider content than it ought to be. And, lest you think I'm just creating elaborate justifications for my own tastes' supremacy, keep in mind that I say all this as a guy who ordered a pay-per-view wrestling event less than a month ago.
Comments
One hand raised
Mmmm . . . this is true for some kinds of data, but the data that I'm talking about is mostly television, which is captured by Nielsen/Cable boxes/Tivo; and internet clicks. There are a bunch of different ways of doing print usability, but the best ones watch the readers read the damn thing.
I'm the last person to say that market research is a killer app . . . but there are mutually reinforcing sources of data, some of them usuing passive monitoring, that all say exactly the same thing.
Nielsen television ratings have gotten better as they've been automated, it's true. But you're still looking at similar problems related to which portion of users will choose to participate in those programs.
As for web analytics -- I wouldn't rely on them too much. Most people looking at those reports can't tell you the difference between a visit and a hit, much less how a week's worth of visitors can add up to less than the sum of the encompassed daily visitors. And when it comes to demographic analysis you're either looking at a self-selected survey or something like Hitwise/Comscore, which is, um, less than transparent.
But you're right, automated measures of content popularity aren't always as terrible as other measures have historically been. Still, it's always hard to disentangle the subject matter from placement, promotion and all the rest.