Centerfield

Long Tail Controversy and Explanations

Dr. Elberse of Harvard caused a ruckus in an article in the current Harvard Business Journal issue about an interesting new study she's been involved in ( Wall Street Journal summary here), as far as I know the only one to look at how online buying patterns are changing. She also raises a ruckus with her title, "Should You Invest in the Long Tail," which is chosen to discredit Chris Anderson's Long Tail book.

I see her title as attacking a straw man, personally. My sense of the Long Tail isn't about ignoring what's popular, as she implies, but rather understanding that stocking the top ten of anything is getting you less market share than it did pre-Internet. But, Chris Anderson is perfectly able to take care of himself, as he shows. As is Anita in response. They're also more fun than my verson.

The Long Tail is of particular interest to me, because I studied it personally, and was involved in a way to overcome it somewhat in one arena, web caching. The Long Tail is a newfangled network-researchy name for an oldfangled problem: why libraries need to be big to keep more than a handful of their clients happy, and why you can never keep all your customers happy with any finite-sized library. Even in the Library of Congress, some people will want books only published abroad.

A mathematician named Zipf came up with a mathematical model that he designed to model the frequency of linguistic word use. It decays slowly, to reflect the fact that, although "the" and "and" are way popular, you also see "arthroscopic" and "quantitative" in real use as well. The law turns out to apply to libraries and web caches as well.

A cache is a bunch of commonly used stuff kept close by for quick access. They've traditionally done well indeed in computers, speeding up processors and disks alot. A web cache is that idea applied to the Web, except it doesn't actually work as well. And that's because what people look at on the web follows Zipf's Law. You need to continuouly prefetch the entire Web onto your web cache to get the kind of improvement you get with disk caches.

I was involved with a couple of measures to work around this a bit. One measure was to support efficient push and look at how much different policies for pushing pages accessed by cache users would help. Another was, observing that caches work better the more people they have going through them, to implement a scalable way of having big cache clouds work together. Our work showed each of those could've been a big improvement, but would still not bring you to the domain of disk cache improvements. So, *I* certainly saw fit to invest in the Long Tail. The answer to Elberse's question, of course, depends on what you're doing.

So what did Dr. Elberse et al see over time? That the long tail on the overall curve tails off at the end and becomes more concentrated at the moat interesting stuff. The highest 10%, for example, becomes more popular and the lowest 10% becomes less popular. That's very interesting stuff, and it's something to think about and research more. The work raises one additional question for me: if you have a certain number of marketing or cache slots to fill, will they become more or less effective over time? Notice that's a different question.

Another thing to think about is that books' title popularity is still long-tailed, despite having had centuries to concentrate.

Hat tip and good call on the Elberse/Anderson controversy at Marginal Revolution.

Canadian Human Rights Commission

I haven't yet gotten around to grumbling about the anti-human-rights Canadian Human Rights Commission. In a free society, the right way to deal with hate speech is to laugh at it and teach your kids that words can never hurt them, not to take away freedom.

There is some good news. The CHRC decided to dismiss the charges against Mark Steyn.

But it's not enough to be letting them go. With the cost of lawyers and the difficulty of dealing with bad PR what it is, simply being charged is a problem. This avenue of charges should be shut down.

No, Slavery Wasn't Competive With Free Labor

Megan Mcardle has a stunner of a bad-assumptions post up. It relies on a book claiming slaves are more efficient than free labor, based on limited slaveholder sources.

I can see why Fogel's controversial: because he neglects to pay attention to some important parts of the facts out there. He just looks at how ONE slaveholder saw it. As one reviewer wrote, "Time on the Cross is seriously flawed. While the intent behind it is honest, the outcome of this project is a gold mine for Confederate apologists."

Well, I'm not surprised that most slaveholders BELIEVED slavery was more efficient, Reality is a different question. At the very least, he should've looked at many other kinds of evidence.

Other whoppers excerpted from Fogel in McCardle's post:

1. . . . The purhcase of a slave was generally a highly profitable investment which yielded rates of return that compared favorably with the most outstanding investment opportunies in manufacturing.

Slaves were certainly profitable investments, or slavery woudldn't've needed Lincoln to put it out of its misery. But, more profitable than manufacturing? Then why was the average northerner richer than the average southerner? Magic? It flies against economic basics as well - prepared goods are usually more profitable than farming, because they have value added.

2. The slave system was not economically moribund on the eve of the Civil War. . . .

3. Slaveowners were not becoming pessimistic about the future of their system during the decade that preceded the Civil War. . . .

Unbelievable - points the evidence I've seen agrees with!

4. Slave agriculture was not inefficient compared with free agriculture. . . .

6. Slaves employed in industry compared favorably with free workers in diligence and efficiency. Far from declining, the demand for slaves was actually increasing more rapidly in urban areas than in the countryside.

Several majorities of entire states decided to abandon slavery because they found it unproductive for their uses (both farm and nonfarm). Were they all just making it up? And how does the first half of (6) follow from the second half?

7. The belief that slave-breeding, sexual exploitation, and promiscuity destroyed the black family is a myt. . . .

8. The material (not psychological) conditions of the lives of slaves compared favorably with those of free industrial workers. ...

9. ... Over the course of his lifetime, the typical slave field hand received about 90% of the income he produced. . . .

Those don't much resemble what the slaves had to say, do they? Shouldn't he've read some of their accounts before passing judgement on slavery's efficiency and effects?

10. Far from stagnating, the economy of the south grew quite rapidly. Between 1840 and 186, per capita income increased more rapidly in teh South than in th rest of the nation. By 1860 the South attained a level of per caita income which was high by the standard of the time. Indeed, a country as advanced as Italy did not achieve the same level of per-capita income until the eve of World War II.

This is just sad spin; I guess we aren't comparing against the North because we'd lose; and the rapid increase just supports the point, as it's easier for weak economies to improve quickly, as Fogel oughtta understand as an economist.

Maybe Fogel's just from an alternate time line where Nazi Europe at its height, with tens of millions of slaves, approached free Allied production levels, and where the USSR was a food exporter.

It's clear there are some people who wish slavery was more powerful than free labor, like, say, Steve Stirling, author of the Draka novels, in which a slave society takes over the world (never mind that it never seems to work that way in practice). I hope Fogel's not in that basket. Another explanation is that historians are often "captured" by the people they study. I read one TR history that backed his resource imperialism, now considered evil by most societies.

Back online

The blog was hacked over the weekend, but thanks to William Swann's efforts (with an assist from Rick Heller) we are back up and running. Consider this an open thread.

Irish Blogger Charged For Blogging

"Belfast blogger Alan Murray is facing charges of harassment for writing about public figures on his blog".

Friday open thread

I will start with this hilarious video.

Via Andrew Sullivan.

Headline: Obama and Clinton Together in Unity

Whoever thought to put Hillary Clinton and Barack Obama in Unity, NH is freaking brilliant.

Speaking of freaking brilliant... Barack Obama would have to be crazier than a s%*& house rat not to put HRC on the national ticket. We would be talking about a 45 - 48 state landslide and a political mandate that could lead to energy independence, dramatic healthcare reform, immigration reform, and many other accomplishments that have been sunk in gridlock for the last twenty years.

I got to admit, the thought of it geeks me out a bit. Maybe even enough to vote for Senator "I was for public financing before I was against it." Yes, after years of hating her guts, I have become quite the "Hillblazer. " I found her push at the end of the nomination process to be courageous, heoroic, and down right inspiring.

There Is No EPA Document, There Is No EPA Document

The White House refused to accept an EPA response to a SCOTUS decision it had lost. The slashdot first post, by Inglix the Mad, says it all:

Frankly I'm pretty sure my boss would give me the sack for that sort of BS.

Only The Party In Power Need Apply

Yeah, it's always been easier for the party in power to get jobs, but this, I think, takes us back to 19C standards. Since around 1900, there's been a notion of the professional bureaucracy, to be filled by the most professionally qualified candidates rather than the most, er, politically qualified.

I think historians will be arguing where on the list of worst 10 Presidents he belongs. I don't think he's as bad as Nixon, since there's no evidence he tried to hack democracy. Although, something along those lines, unverifiable eVote machines, was foisted on us during his first term, I see no evidence Bush made tha happen, though. Before 9/11, I don't believe Bush would've let this happen.

Prosecutions?

Now that subpoenas have dragged corroborating evidence that the high leadership did want torture, do you think there should be prosecutions of Rumsfeld, Sanchez, and/pr other responsible officials?

How about against Bush after he leaves office on charges of breaking the FISA laws?