Signal in the Social Noise

Eric Schmidt famously said in 2010 that every two days we generate more data that we have up to 2003. To clarify:

  • dawn of civilization -> 2003 = X amount of data
  • this past weekend = X amount of data

That’s a mind-bending statement.

For a long time, middle management in large organizations has lamented the tornados of data that it’s faced with processing on a near daily basis.  Middle management’s primary responsibility in large organizations is processing and interpreting vast swaths of information in order to report to upper management for strategic decision-making. We had to invent middle management in the business world just to process data.

And if that wasn’t enough, now we seem to have as much data in other spheres of our lives: social, personal, professional.  Everything is being documented and quantified now. The silver lining, of course,  is that our ability to process huge amounts of data is also in its infancy. As storage costs continue to decline and as more data stores are hosted on cloud-based systems, the possibilities are nearly infinite for us to track, process, analyze and interpret larger and larger pools of data than we ever have previously.

Facebook’s new graph search is an early attempt to address the need for a new kind of search, one that’s built on the people we know.

For me, the followup question is about untapped opportunity.  What haven’t we started to process that can help change the way we do business and the way we live our lives?  Is there any signal left in the social media noise that we haven’t found? I assume that graph search will produce a ton of value that hasn’t been unlocked: restaurants, travel, workplace searches will all become more interesting. But are there insights that it won’t pick up?

Here’s another question. If we use Google to search information about the world and we will presumably use Facebook’s graphsearch for information about our friends, what will we use in the future to search for information about ourselves?  Who is going to store our personal data, process it and feed insights back to us and our doctors?  I doubt that Google or Facebook are going to provide that service. I think we’re going to want that information in a ‘closed’ system, or at least a one way street (pull data from other services but never let my weight history, blood history, dietary habits, emotions, measurements or other personal information out into the real world).

Not that this is a new idea, but I believe that we’re on the cusp of tremendous innovation around data processing, and I’m personally excited to be here for the ride.