Big Things

Dear Linux Pro Reader,

As everyone knows, we journalists are always looking for the next big thing. High-tech journalists are especially attuned to this quest, because what is high tech but the history of the next big thing unfolding?

After that inspiring introduction, you probably think now I'm going to tell you what the next big thing is. Actually, though, what I really want to tell you is what the next big thing isn't – at least not so far: Hadoop.

That might surprise you if you've been reading computer journals for the past couple years. After all, the Hadoop data analysis framework embodies the promise of Big Data, and Big Data is (what have we been calling it?) a revolution. Of course, high-tech revolutions come and go at the speed of Moore's law, but this one's a real revolution, right? Actually, a recent study from Gartner suggests that the truth about the Hadoop revolution might be a little too complicated for a tweet or headline.

According to Gartner, "Despite considerable hype and reported successes for early responders, 54 percent of survey respondents report no plans to invest at this time, while only 18 percent have plans to invest in Hadoop over the next two years." [1] Only 26 percent of survey respondents said they were currently either "deploying, piloting, or experimenting with Hadoop."

Only 26 percent means the world isn't on fire quite yet. All this is no diss on Hadoop or other implementations of the MapReduce model, which really is an amazing bit of tech. I guess the diss is more on the publishing industry, and also, perhaps, the investment and financial services industries, which also play a part in the culture of the next big thing.

Hadoop has been chugging along. Companies are using it; software vendors are integrating it with their products; development proceeds apace. The problem isn't with programmers, the integrators, and the service sellers. The problem is with those of us who concern ourselves with figuring out what Hadoop means – how we position it in our constantly evolving picture of the world.

The fact is, Big Data is great for Facebook and Google, but it isn't for everybody. Maybe it would be for everybody if everybody had the budget and the expertise to explore the possibilities, plus the patience and resolve to build the necessary business processes, but reality slows things down a bit. Also, any technology that requires a new vision isn't implemented within any company unless it also comes with:

• A champion within the company, who has sufficient clout to make the change happen
• Proof of a tangible benefit – as in, "this will make money" or "this will save us money" (as opposed to intangible benefits, such as "our company will be in the vanguard of a shining new information era")

In the case of Hadoop, I have a feeling that, for some companies, the champions are easier to find than the proof. Crunching all your company's workforce data, customer data, and IT log data through sophisticated data analysis is a really cool idea for any company (because information is power and power is money), but is it worth it? In the corporate corridors where such decisions are made, the players are all well aware that their careers depend upon walking that careful line between "this will save us money" and "this might save us money."

So, maybe the future of everything doesn't belong to Hadoop and Big Data – at least not yet. But at least Hadoop is doing better than some of the other "next big things" that have mesmerized journalists in the past, like Google Wave and Microsoft Kin. We in the press will keep looking for the next big thing because no one seems to want to read about the last big thing or the next little thing. Sometimes we might get it wrong, or only half right, but it is all about the chase.