Why Big Data? Here’s Why I’m Interested

I just had my 2nd conversation this morning before coffee about this fabulous Economist special report on Big Data: Data Data Everywhere. The person I was corresponding with asked me why I was interested in this topic. Here’s my answer. If this is something you’re interested in, I’d love to know what it is about Big Data that captures your interest, too.

What got me excited is just that this is a topic I think is fascinating. I’ll tell you frankly: I think in big data there lies a lot of hidden patterns that represent both opportunities for action and for reflection. At RWW we’re working on trying to find ways to mine data to find news first (we’ve got some interesting methods employed already) and personally, I think the world is an awfully unfair mess and I’m hoping that data analysis will help illuminate some of the hows and the whys. Like the way that real estate redlining was exposed back in the day by cross referencing census data around racial demographics and housing loan data. That illuminated systematic discrimination against black families in applying for home loans in certain parts of town. So too I think we’ll find a lot of undeniable proof of injustices and clues for how we might deal with them in big data today.

How about you? Are you interested in Big Data? Where does your interest come from?

Related: Check out Ta-Nehisi Coate’s critical analysis of one of the most prominent recent examples of social media data analyzed. I’m still reading it, myself.

  • Big Data is more than just Wolfram Alpha. It is the entire http realm. Anything that is publicly fetchable via http is game (though if you rely on Google API, you may not be seing the whole story). If you stick to Twitter API, you’ll only see noteworthy links to data (you’ll have to parse the text and that’s a TOUGH problem), but you’ll also be affected by the “echo chamber” effect. The dialog here and elsewhere discussing big data is the first steps. Once developers, hackers, collaborators begin hacking away at the problem, if even to prove it is possible with a subset of the BigData, we’ll see results soon after.

    Thanks for sharing, keep the discussion flowing, and hope to see more soon!

  • Altan Khendup

    I share and like your concepts as well. In my recent projects taking data from a variety of sources which may seem unrelated and cross referencing them creates new and fascinating insights, and uncovers patterns that were undiscovered. Overall companies are having a lot of growing interest in this concept though are wrestling a lot with exactly how to tackle it. I am sure RWW has lots of interesting findings as well 🙂

  • marko anderson

    Agree. Fasten your sealtbelts, its going to be an enjoyable ride 🙂 An intro to something i was wrote recently….

    The world is digital. Nearly everything we do leaves personal digital trails in our wake; from communications, to media consumption, to purchases, to financial transactions, to healthcare. We are already seeing some of the early tangible benefits that this data can generate for us such as, customer loyalty rewards, book and music recommendations, and better medical prognosis. This rapidly increasing digitization of our activities is creating an ever-expanding pool of raw material from which we could potentially realize great benefit for ourselves and others. As this data is becoming more connected via the internet and accessible via open standards, it is a natural progression that more extensive collections of our data will be aggregated into rich personal data repositories. These personal data assets will not exist in isolation, but will be integrated parts of much larger data platforms. These large-scale user data platforms will enable exciting new experiences and interfaces that will allow us to gain deeper insights about ourselves, and our complex interconnections to the people and things around us. The current benefits we are seeing now are trivial compared to what will be possible in the near future when massive data platforms emerge with the ability to aggregate vast quantities and varieties of real-time data about nearly everything. These systems will be the catalyst for an evolutionary shift in our most basic and fundamental understanding about ourselves and our connections to the world we live in.

  • Big data
    – Archiving (big problem)
    – The speed of technology going obsolete before we can process everything, let alone understand it.
    – We are incapable of properly understanding or analyzing vast amounts of data.
    – Which data is meaningful and how can we spot it? Is semantics enough?

    We have a mega problem.

  • marko anderson

    mega problem = mega opportunity

  • I’m really interested in matching up what people search for and what they say in status updates with what they do in real life. I think that cracking the predictive analysis nut will do a lot of good toward ensuring that consumers have exactly what they want – be it information, products, or services – exactly when and where they want it.

  • We already have more data than we can usefully deal with in most instances. BI seems to have hit it’s limits and we need to find new ways to extract weak signals[1]. Tesco’s (fairly) recent efforts to integrate weather forecasts[2] with till data is a pointer in the right direction. A rise of 10C, for example, led to a 300% uplift in sales of barbecue meat and a 50% increase in sales of lettuce.

    The challenge we have, which DC CROWLEY points to, is finding the data that matters. Not all information is of equal value[3]. As Chris Hall points out, the earlier we can get to a customer, the likely we are to be able to service their need exactly when they want it. (Though I still see this as reactive rather than predictive — the key difference is that we’re using externally sensed data to identify the client need, rather than mining trends from internally manufactured data.)

    There’s a whole thread on this that ping-pongs between my blog, and Andy Mulholland’s blog at Capgemini. You can find my side of the discussion here:

    http://peter.evans-greenwood.com/category/focus/value-of-information-focus/

    Oh — and when did “Big Data” become a proper noun 🙂

    r.

    PEG

    1. Why scanning more data will not necessarily help BI
    http://peter.evans-greenwood.com/2009/12/22/why-scanning-more-data-will-not-necessarily-help-bi/

    2. Tesco’s looking outside the building to predict customer needs
    http://peter.evans-greenwood.com/2009/09/09/tesco-looking-outside-the-building-to-predict-customer-needs/

    3. Inside vs. outside: the value of information
    http://peter.evans-greenwood.com/2009/08/31/inside-vs-outside/

  • WOW – what interesting comments. I hope we see some others weigh in as well. Just tremendous. Marko: On customer loyalty rewards: that alone is a tremendous amount of data that belongs to it’s creator: you/us — When we get access to that data (probably not through the rewarding organization, but through a service like Blippy) we can begin to make some cool decisions based on it.

  • Pingback: In Case You Missed It. Weekend Tech Reading()

  • I got interested in big data after watching the last TED video on Open Standards and data, I was so excited about it i even posted an article on my blog about it!

    After some research i found out that it’s actually our right to demand data from companies out there (some which i won’t mention and that have not opened up their data to anyone)

    What was particularly interesting to see is the connections that linked data offers us and that’s what people and companies can tap into to provide rich & helpful experience to so many people out there.

    In the future I’m sure we can see data interopability between our favorite social networks, because at this stage that isn’t totally possible yet.

    Excuse my lack of in the comment. This is something really exciting. BTW everyone just because you’re small, doesn’t mean you can’t contribute – Try starting here: http://www.okfn.org/

  • I’m pretty interested in big data too. I’ve Benny fascinated with the implications hi h to me are glaring and yet most people seem relatively un impressed by the stagering nature of what’s staring us in the face.

    The whole world and everything we know is being irreversably transfigured and these changes are coming nearly instantaneously.

    I’m interested also in the concept of our progression as having an exponential growth curve rather than a linear one.

    Google “the singularity” for some interesting reading. Some of it is doom and gloom, just skip that crap and read the crazy speculations.

    Oh and follow me on Twitter. =P

    -Joshua

  • Good article. Take a look at BigDataNews.com — we’re starting to get a good set of articles and community sharing, and should be announcing a 2nd corporate sponsor this week. If you’d be interested in posting articles I can set up an author account for you.

    Sincerely, Brett

    Brett Sheppard
    Editor, BigDataNews.com

  • Amy

    I got interested in big data after watching the last TED video on Open Standards and data, I was so excited about it i even posted an article on my blog about it!

    After some research i found out that it’s actually our right to demand data from companies out there (some which i won’t mention and that have not opened up their data to anyone)

    What was particularly interesting to see is the connections that linked data offers us and that’s what people and companies can tap into to provide rich & helpful experience to so many people out there.

    In the future I’m sure we can see data interopability between our favorite social networks, because at this stage that isn’t totally possible yet.

    Excuse my lack of in the comment. This is something really exciting. BTW everyone just because you’re small, doesn’t mean you can’t contribute – Try starting here: http://www.okfn.org/