too cheap for NSA

Siobhan Gorman of the Wall Street Journal has some very interesting background on how NSA scuttled a software program that wasn't as appealing as its much more expensive alternative.

The current NSA model relies largely on amassing as much data as it can obtain and trying to sort through it all later.

In its place, the presidentially appointed review panel suggested a drastic and fundamental change in the 20th of 46 recommendations in its report released Wednesday: "Software that would allow…intelligence agencies more easily to conduct targeted information acquisition rather than bulk-data collection."

 The panel proposed a feasibility study. But former NSA officials say such a transition is certainly doable. "That's exactly what we did," says former NSA official Ed Loomis. "It's not only feasible—the government threw away the software that did it."

Mr. Loomis said he and his colleagues developed just such a program 15 years ago. It was designed to cheaply search an array of data sets—wherever they happened to be—without first importing all the data into an NSA-held system.

The program helped spies conduct targeted searches of large amounts of data and included a number of privacy protections that performed well in pilot tests. But the program, known as ThinThread, lost an internal bureaucratic fight and wasn't deployed.

Bill Binney, another member of the ThinThread team, said ThinThread was also handicapped because it was too cheap. With a $3 million price tag, the program couldn't compete with a $4 billion program called Trailblazer that was backed by major contractors.

Maybe there's a better justification for NSA's decision, but a lot of bureaucratic battles in government are lost because officials think that bigger is better and expensive programs are harder to kill.