Why “Big Data” Scares Me

Is it just me or is talk of “big data” suddenly everywhere these days? It kind of reminds me of a few years ago when everything was about “the cloud”. I’m still trying to figure out what exactly that was all about and whether I’ve missed the boat on something that is critical for my business and personal life. Sometimes it’s really hard to get beyond the tech-talk and discover the practical, value-added applications of any given solution.

Since measurement is a critical part of my life, I couldn’t help but perk up my ears when I first heard the term “big data”. Since then, it seems as though big data is being discussed everywhere. It wasn’t long before I just had to find out what big data really is and, more importantly, why anyone would/should want it.

My search took me much deeper into the tech world than I commonly like to go so, in an effort to spare you the trouble, here’s what I found out:


What is “Big Data”?

While “big data” refers to large amounts of data, there are additional attributes that cause data to be categorized using this terminology. Experts have come to classify big data using the “3V’s” (though some people argue that there are actually 4V’s): volume, velocity, and variety.



Big data involves huge amounts of information. The attraction of large data sets is that they give you the ability to develop better understanding of certain aspects of your business (for example, customer behavior) and stronger statistical models with better predictive power. As well as leveraging more data points, big data gives you the opportunity to include more contributing factors in the modeling process – theoretically giving you more robust predictive models. However, the volume of data in these data sets poses a problem for conventional IT platforms. While they may have the capacity to store large amounts of data, the problem is that they don’t have the capacity to process it. As a result, if you are going to play with big data, you need a new IT platform that enables the processing of a huge data load.



Velocity refers to the speed at which data flows into an organization. While handling fast moving data has been a reality for some time in certain industries (e.g. stock trading), the ability to make quick sense of large sets of fluctuating data hasn’t been widely available. However, this is on the verge of changing. This is good news for industries where, for example, being able to make sense of customer buying behavior and adjusting the range of products available to the customer tomorrow rather than next season represents a significant competitive advantage. Essentially, big data is fast and readily available data. While many existing platforms allow you to stream data into bulk storage for later processing, big data platforms provide a speedy feedback loop, shortening the time from data point creation to business decision. This matters because the faster the feedback loop, the greater the potential for operational and strategic high performance.



As you know, data rarely comes to you in a format that your IT systems can handle. Usually data needs to be put into an organized format – by fields and according to formatting “rules”. An unfortunate by-product of this data transfer process is a loss of data – some of it just gets thrown out for one reason or another. The act of cleaning data in this way means that you may get rid of something that is important without knowing that it’s important (and, because it is lost, you’ll never have a way of discovering that it is in fact important). When you are looking at big data sets, a common feature is that the data sources are extremely varied. This means that data points are coming in in different formats (e.g. text, video/graphic, data points, etc.) and most of it won’t fit nicely into your data analysis application – not without a lot of work that is. Big data processing platforms can handle and make use of all of the unstructured data in a big data set – nothing gets lost because there’s no translation. Being able to eliminate the risk of missing out on previously unknown important data is a critical benefit of big data processing platforms.


Basically “big data” refers to large sets of fast moving data that exist in a variety of formats.


Big data also involves new processing platforms that can integrate all the data included in the large data set to create more robust predictive models and provide you with insights faster so that you can make better business decisions more quickly. The touted business benefit of big data is that it can enable better, faster fact-based business decisions at all levels of an organization, thus offering any company that uses big data the opportunity to transform greater insights into better customer/stakeholder service and/or a competitive advantage in the marketplace.


All of this sounds great “in theory” but I’ll be honest, I feel like I’ve seen this movie before. In recent memory, I personally saw it in the early days of CRM, and I see it regularly with the balanced scorecard. That is, we get confused thinking that we are implementing a tool or, worse, an IT solution – full stop. Just sort out the IT infrastructure requirements to support the platform, map the data inputs, design the results reports, and flip the switch on the application. Why are we always surprised when things don’t turn out the way we expected? I can predict how this big data thing will turn out for many of the companies that adopt it – a few years from now they’ll wonder why the promised ROI on their big data investment never materialized.

Sorry to be so negative but this is what really scares me about the “big data” frenzy – organizations will jump into the deep end without putting all the essentials for success in place before diving in. Some will swim but many will end up sinking.


The Key to Success with Big Data

As it is with any good IT project, it’s important to begin by defining a question or business problem you think big data could help you with. To maximize impact, be sure to ask a critical question that is directly related to your key business value drivers or your key differentiator in the marketplace. It’s also important to make sure that your question relates directly to your primary customer value proposition and supporting business model. For example, if your primary CVP is operational excellence, you might want to leverage big data to discover ways to drive more operational efficiencies across your organization and deliver a more consistent customer experience at every interface with your company. On the other hand, if your primary CVP is customer intimacy, you might want to leverage big data to develop deeper insights into the buying behaviors and preferences of individual clients and customer groups to pro-actively present new targeted product offerings to them at a time that maximizes the chance of purchase. Whatever your big business question is, it’s important to figure this out before plunging ahead with big data.

Since big data requires new IT platforms, there’s the whole matter of selecting the right one(s). This must be driven by a variety of factors including business requirements, budget, the existing (and vision for) IT environment, etc. The key to success is to ensure that IT and the business collaborate fully during the spec development, platform selection and infrastructure implementation processes. It is critical that the business doesn’t abdicate its responsibility to participate in this entire process AND it is equally critical that IT balances the business question with the technical aspects of this process. A breakdown at this stage of the project is the root cause of countless system implementations that have failed to live up to expectations.

Getting the most out of big data requires an investment in new organizational skills and capabilities. More that building fluency in data analytics, companies that wish to play with big data must develop and/or hire people who are skilled in data science. Writer Edd Dumbill describes data science as an emerging discipline that combines math with programming and scientific instinct. Success with big data requires the willingness to develop (and/or buy) this capability and then provide these employees with the supports they need to produce the insights you want to get from big data.

Most importantly (and most neglected in my opinion) is the leadership will to create the culture and mindset required to understand, communicate, and take action on the results of the analysis of big data. It is critical not to underestimate the investment required to develop an openness to listen to what data is telling you, talk about learnings across your organization in a productive way, leverage insights to look at problems in new and innovative ways, and to take unexpected and possibly counter-intuitive actions. Even if an organization does everything else right, a failure to do the hard work related to culture change will still translate into a sub-optimized big data investment.



The proponents of big data like to talk about the big benefits that are possible. What company wouldn’t want to get on board an opportunity to play their industry’s version of Moneyball? As an executive, you’d be crazy not to be tempted. Unfortunately, history is littered with IT trends and solutions that didn’t deliver – sometimes because the IT platform didn’t perform as expected or wasn’t as user friendly as advertised. However, in my experience, it’s a lack of attention to the success factors outlined above that usually brings these initiatives down.

And this is what really scares me about “big data”. Adopting big data is going to be a big investment for any company that chooses to go this route and based on past evidence, the risk is high that it will not end well for most of them.


Is your company thinking about going the big data route? Before you do, (1) be sure to take the time to clarify the business reason for adopting big data, and (2) be sure that you really understand, and can commit to, the non-IT, organizational investments that will be required to achieve success. With these in mind, take the time to re-assess the full costs and benefits of big data for your organization. If you take this approach to evaluating the opportunity big data may offer to your company, you’ll have a better chance of actually realizing the benefits you expect to get for your big data investment.