Hadoop Wins Converts Outside Silicon Valley

Two years ago, when the Detroit Crime Commission began collecting and analyzing the social media posts of suspected criminals, it found Excel wasn’t up to the task. The 11-person agency began using Hadoop, a software suite developed in the early 2000s to help Web giants such as Google and Yahoo! store and analyze mountains of data. “Several million pieces of content is a lot and cannot easily be analyzed within any Excel spreadsheet, so we needed to do something a lot more robust,” says Lyle Dungy, director for intelligence at the Detroit Crime Commission. The software has already helped reveal a relationship between two suspected criminal organizations. “There’s so much digital evidence that’s out there. Most agencies are not exploiting it,” says Dungy.

A lot of technology never makes it out of Silicon Valley, let alone worms its way into a small city agency. Yet in the decade since Hadoop was developed, a cottage industry has emerged around the open-source software. The agricultural giant Monsanto relies on Hadoop to analyze and predict weather patterns, while the Indian government uses it to store information on more than 500 million citizens for its national identity registry. India’s biometric database, said to be the world’s largest, is so robust it can handle as many as 4 million logins per minute.