Big Data, that corpus of global digital information characterized by velocity, variety, and volume—with contributions from just about every being and machine on the planet—has achieved such a scope and speed of growth that any attempt to quantify it is outdated as soon as it’s measured. If, in the last year, Amazon sold 636 items per second on Amazon Prime Day, YouTube saw 300 hours of video uploaded by users every minute, and Google handled 3.5 billion searches per day, then count on 2018 to bring more of the same.

If 2011 was the year that “data” began answering to “Big Data,” Tamara Dull, director of emerging technologies for SAS’ Best Practices team, says that in 2017, “IoT ripped the ‘big’ right off Big Data’s face.” Dull says, “The story isn’t as much about Big Data, but rather, the Big Data technologies that allow organizations to store and process all kinds of data—structured, semi-structured, and unstructured—at a fraction of the cost and time of traditional technologies.”

That change in emphasis means a permanent shift to figuring out the best way to pull out what’s needed for the people who need it—and doing so in a way that supports their decision making without requiring a data-science degree. It means integrating disparate datasets to create more personalized end-user experiences. As 5G wireless moves closer to fruition, there will be implications for when, where, and how Big Data is accessed by an increasingly mobile userbase. And while pushing data to the edges for faster decision making has its appeal, concerns around data security and privacy abound, especially after a year when data breaches dominated headlines and set consumers on edge.

With Big Data growth getting a boost thanks to such things as your organization’s customer relationship management (CRM), the U.S. president’s Twitter account, and your mother-in-law’s smart thermostat, there’s no time to lose in tackling these challenges.


The Big Data Year in Review

The growth of the Internet of Things (IoT) in commercial and industrial usage is one of the main drivers of Big Data’s expansion in uncharted directions. Stan Lequin, VP of consulting services for Insight, a technology provider of hardware, software, and service solutions to business and government clients, says, “We work with companies that have been around for 200 years, who are now able to pull client endpoint data and analyze it for the first time,” thanks to IoT. “It’s made predictive maintenance and preventative maintenance easier and created new as-a-service possibilities that are entirely new revenue streams.”

Dull sees the IoT having an impact across the board, with manufacturing, transportation, utilities, and healthcare leading the way. She says, “Any organization, regardless of size or shape, can now ask questions like, What does ‘data-driven’ look like for us? What stories are locked inside our data? And can we make money with our data?”

Who gets to dive into the data to answer those questions has changed. “The democratization of Big Data means that we are seeing data push out to the edges,” says Lequin. “Business intelligence for the masses has really picked up over the past year.” He credits an increase in data accessibility within organizations, easier access to external data sources thanks to APIs, and tools that put visualization and analytical capability into the hands of decision makers.

With regard to security concerns in 2017, Dull points out that the devices networked in the IoT don’t necessarily come with robust built-in security, and that puts the onus on consumers. “We can no longer assume that a manufacturer or an app developer is going to do everything they can to make our experience safe and secure. It is now our responsibility as consumers to become amateur security geeks and privacy freaks.” The upside? IoT manufacturers that create safe, secure devices will have a competitive edge.


A Look Ahead at Big Data

Lequin believes that for Big Data in 2018, all the key capabilities are in place, but their adoption and evolution will speed up. “There will be more accessibility; it will be easier to plug in both internal and external data sources—and the database tools with which it’s all done will be easier to work with,” he says. Dull says there’s an urgency to getting the IoT’s contribution to Big Data right. She says, “I’ve been keeping my eye on three ‘S’ developments for IoT: security, standards, and skills. If these three areas don’t get addressed properly—and sooner rather than later—then it’s game over for IoT.”

Personalization will continue to gain importance, according to Craig Smith, CEO and founder of Trinity Insight, an optimization agency that assists ecommerce brands with managing data, digital marketing, and user-experience efforts. “A key part of Big Data is activation in the customer journey,” says Smith. He cites the example of a shopper buying a pair of children’s cleats in-store and providing an email address during the checkout process. Effective personalization might come in the form of an emailed offer for a complimentary ebook on football, a Facebook sidebar ad for football helmets, and a football equipment catalog in the mailbox 12 months later. Due to the complexity of integrating datasets such as account information, web analytics, and search behaviors, Smith says that type of implementation isn’t mainstream yet. “But it will be par for the course in 5 years.”

With the growing importance of mobile usage, all eyes are on 5G, the next-gen network system that will be characterized by higher speeds and capacity and lower latency than existing cellular systems—or at least they should be. “We’re shocked at how much people don’t know about 5G,” says Lequin. “Our expectation is that 5G will be a part of every client conversation we have in late 2019 and 2020. We know it will create a lot more accessibility.”

Finally, Lequin points to one challenge that pre-dates Big Data by approximately 200,000 years: an aversion to change. “Organizations are averse to change; it’s hard to think of as-a-service revenue streams that bring in revenue monthly rather than all at once.” It’s why Lequin says that to get the most value from their Big Data, organizations need one thing above all: blue-sky visionaries who are internal champions and can guide that transformation from analog to digital.