What the Hell Does "Looking for Patterns" Actually Mean?

I was just asking this on Twitter -- every defender of the NSA program says they're "just looking for patterns."

I know what looking for patterns means in real life. I do not know what it means as far as higher mathematics and large databases. And few seem to be interested in explaining -- the NSA's defenders don't, choosing a euphemism rather than saying "data-mining." At least if they said "data-mining" we could look that up and see what it's about.

The media doesn't. Obama has settled on a vague euphemism to describe this and so the media decides he knows best. If Obama thinks it's best not to explain to the public, then the media will follow his lead.

As someone who's not on the full-on Libertarian pole on this issue, my own take on it will be fact-specific. That is, I could envision scenarios where I might support this, and I can imagine scenarios where I would strongly oppose it. The details, then, are important to me.

They're not important to some -- full Libertarians are ideologically against this, no matter what the actual process might be; and Obama partisans are of course now the foremost advocates for an intrusive surveillance state, no matter how intrusive that surveillance might be.

But to some of us, the on-the-fencers, we really need to know what the hell exactly it is we're talking about before we can render any judgment.

I am just coming to this, but I'll link two articles (and a third as a bonus) about data-mining.

This clever, cute piece pretends to be written from the point of view of one of King George III's intelligence agents based in Boston circa 1770, and uses just one tiny snatch of information -- real historical records of what clubs certain Colonial trouble-makers belonged to -- and then uses data-mining techniques using her Difference Engine (mechanical, primitive computer) to determine that one suspected Traitor called Paul Revere is absolutely central in terms of the "connections" between the trouble-makers, and scores near the very top of the list in terms of his connections to people who themselves have a lot of connections. Per this simple data-mining, this Royal Intelligence Report points to Paul Revere as a likely high-ranking leader of a Colonial "terrorist cell."

This Forbes piece looks at Target's data-mining figuring out that a young female customer was pregnant before her family did. Based on data-mining, they had seen a pattern: Woman buys unscented lotions and supplement like magnesium, and then, four months later, starts shopping for a crib and baby clothes. Seeing that one particular woman was buying unscented lotions and magnesium, they sent her a flier offering her great deals on cribs and strollers, which made her dad pretty angry, as he thought Target was encouraging her to get pregnant.

A few weeks after angrily complaining to Target, he apologized to the store, stating that perhaps there were a few things going on under his roof that he wasn't previously apprised of.

Now, that article in turn is just a digest of a single anecdote from a much-longer New York Times piece on corporate marketing data-mining and its power, which runs nine pages, and which I have not had a chance to read yet. (But will. But I thought I should get this up. Why should you wait on me, if you're interested?)

Obviously these meager little hints are the beginning of knowledge, not the end of it, but I did think it was about time I started to learn exactly what these people mean when they say they're "looking for patterns."

And Wikipedia, Of Course: It's written in that Wikipedia way, but I imagine I'll be referring to this too as I try to figure out what we're actually talking about here.

And here's an official government document about the technique, which I assume, being authored by the government, is primarily designed to hide information rather than reveal it.

posted by Ace at 03:10 PM

| Access Comments