To capitalize on Big Data, companies must learn to sprint. A few weeks ago, I met with several corporate executives and educators to discuss the future of information technology in the health care industry. Not surprisingly, most agreed that the companies that embrace and overcome the challenges of Big Data will be better positioned to lead this industry over the long term. Much of the discussion focused on three popular measures of Big Data: Volume (a measure of scale), Velocity (a measure of speed), and Variety (a measure of form). To my surprise, however, it was the challenge of velocity that this group most associated with business value and competitive advantage.
Big Data velocity refers to the speed at which data is created, made available, ingested by those who want it, and processed to gain valuable insights. As our available data sources have evolved over the last three decades (see “What makes today’s data ‘Big’?”), so too have velocity expectations.
Velocity expectations were much lower in the 1990s, when most available data came from a company’s own internal systems. Many of the Business Intelligence systems I was responsible for back then took almost 24 hours to collect and completely process data — and some took up to a week!
The rapid growth of the internet and social media, starting in the 2000s, provides opportunities to understand consumer interests and personalize interactions with individuals. Effective personalization requires predicting an individual’s interests by analyzing recent online behaviors. In this scenario, behaviors observed within the last few minutes are often much more insightful than those captured days ago. For example, you may have used Google to search for a particular product online, only to have your future internet travels bombarded with advertising for that very product. Odds are, you are more likely to buy a product you searched for a few minutes ago than you are to buy one you quit searching for days ago.
Now, let’s turn that example around and say you are the manufacturer of a certain product and you want to know in near real-time who is searching for your product, on which sites they are searching for it, if they are buying it, and if they like it after the purchase. The internet and social media sites such as Google, Facebook and Twitter can provide valuable insight about these questions. But consider the rate at which these sites generate data today. According to Go-Globe, a leading internet solution provider, in just one minute Google conducts more than 2.3 million searches. In just one minute, more than 3 million items are shared on Facebook. In just one minute, nearly a half million tweets are sent via Twitter. This happens every minute of every day. The velocity of this data is so high that many are unable to leverage it effectively.
In the 2010s, we are seeing a rapid increase in internet-connected devices and the use of equipment sensors to monitor and send an alert about problems, gather statistics, and measure trends. Cisco Systems, a leading networking technology company, suggests that sensors alone will generate about 40 percent of all data by 2020. (for more on Big Data volume, see “How ‘Big’ is today’s data?”). A single jet engine currently generates about 1 terabyte of data per flight. We may soon have as many as 200 sensors in our everyday vehicles, each capturing real-time information about how the vehicle is performing, how we’re operating the vehicle, and how the vehicle interacts with other vehicles in close proximity. As you may have seen in recent commercials, refrigerators can now tell us when we need to buy milk. According to the World Economic Forum, within the next five years more than 5 billion people and 50 billion devices will be connected. These devices will generate data at an unprecedented rate.
Thinking back to my meeting with the health care industry, I shouldn’t have been surprised that they so strongly associated Big Data velocity with business value and competitive advantage. The aim of health care is not only to help us when we are sick, but to also assist us in managing our health on an ongoing basis. This will require devices that can capture complete and accurate patient data, including consumer-focused technologies, and are capable of communicating via the internet or other electronic networks. These devices will create a tremendous amount of data, which health care providers must continuously ingest into their systems in order to monitor our health and drive better health outcomes. Like the health care professionals in that room said, companies that learn to effectively manage the velocity of this data will be better positioned to lead the health care industry over the long term.
Dr. Rick Brattin is an assistant professor of Computer Information Systems at Missouri State University. He has 25 years of experience in data analytics, business intelligence and information governance with Fortune 100 companies. Email:rickbrattin@missouristate.edu.
This article appeared in the August 13, 2016 edition of the News-Leader and can be accessed online here.