Did Big Data Deliver Business Transformation & Improved CX?

It’s been one decade since the “Big Data Era” began (and to much acclaim!). Analysts asked, What if we could manage massive volumes and varieties of data? What insights and new abilities might we unlock? Ten years later, there’s no doubt much progress has been made. Yet the question remains: How much value have organizations derived from big data?

This was the question that I posed recently to CIOs. In this article, we’ll take stock of what big data has achieved from a c-suite perspective (with special attention to business transformation and customer experience.)

Big Data as an Enabler of Digital Transformation

Big data technologies have been foundational to digital transformation. “Organizations should be using data today to fuel their improvement efforts, regardless of whether they’re driven by digital, business, or something else,” says Miami University CIO David Seidl.

Aligning business goals to data usage is a key foundation. “There is no digital transformation without business-aligned data control,” Capgemini Executive Steve Jones adds. “The architectural model of big data is fundamental to this.”

Yet, while big data is an enabler of digital transformation, data alone won’t transform an organization. Everything comes down to what organizations do with data. How will you use data in your business to improve experiences and stay competitive? “Core digital transformation practices should be focused on customer and employee experiences, technology + data competitive differentiators, and business model evolution,” former CIO Isaac Sacolick advises.

These goals will evolve with time! Data leaders should keep in mind that becoming data-driven is more of a journey, and less of a destination.

So, What did Big Data Achieve?

CIOs have clear opinions about what big data achieved and failed to achieve.

Some CIOs suggest that big data was largely marketing spin from companies trying to sell data tools. However, “Big data did lead to improved technology in the form of data platforms and was often the first step into the cloud outside of SaaS platforms,” Quickbase CIO Deb Gildersleeve points out. However, “big data didn’t magically fix bad data,” Gildersleeve laments. “It just highlighted how bad underlying data was.” For modern companies awash in customer data and focused on CX, this challenge persists.

Many CIOs argue the rise of big data pushed people to use data more proactively for business decision-making. Big data got“ more leaders and people in the organization to use data, analytics, and machine learning in their decision making,” says former CIO Isaac Sacolick. That said, CIOs also claim that in many cases big data failed to deliver proclaimed results. Blindspots and silos left vital gaps empty.. And where data was available, the ability to access and interpret it proved problematic.

Big data can grow too big fast. Left unchecked, data lakes became data swamps. Some data lake implementations required expensive ‘cleansing pumps’ to make them navigable again. Such ‘pumps’ were a sign that implementers had failed to align business assumptions with technical reality.

“There is a major difference between collecting big data and being able to do something meaningful with it,” says analyst Jack Gold. “The collection part is easy. Analysis and action remain difficult.”

Skills take time to develop in these new data landscapes. “In many cases, big data failed to make us better at using data,” Seidl argues. “Often, we just had more of it. It didn’t make the data better in many cases either. But as a positive, it did mean we started talking about data and wanting data, and we’re starting to make it a part of our culture.”

After decades of tech and data debt, an easy “data hygiene” button is still not available. And many still want to claim that “data is the new oil” without addressing the data literacy gap. How do we translate this “oil” into better customer experiences? How do we use it to transform a legacy business into a competitive one? Without the right skillsets, no value can be created from data.

New Big Data Concepts vs Cloud Delivered Databases?

So, what has the emergence of cloud databases done to change big data? For starters, the cloud has made data more affordable. “Cloud has not replaced big data but lowered the cost of entry,” says Gildersleeve. “Setting up Hadoop on-premises was a huge undertaking. But with the cloud, you can take a small project and test it out on new platforms with a smaller budget to start. see that it works before going all-in.”

“Spark, Tensorflow, Apache Kafka, et cetera, are all out found in cloud databases,” points out Jones. “File-based storage of data is the norm even under more relational models. , Graph databases, document stores, file stores, relational stores all now exist, each addressing different challenges.” In this way, the cloud has democratized access to some of the best outputs of big data.

For many, the cloud offers more efficient analysis “under the hood.” Seidl says, “We just run slices of them in hyperscale-provider space instead of in a local data center or under a desk. I remember one implementation of Snowflake made the on-premises problems go away because of how things worked. Its scale and design were magical in the scenario we faced. I think concealing the magic is one of the cool things about cloud for many organizations.” Contrast this with the large on-prem installations of the past: “Nearly every large organization adopted Hadoop and transitioning away from its pain and expense,” says analyst Daniel Kirsch.

The cloud also offers a new era of flexibility. A key challenge of legacy approaches involved data quality. How could you ensure data was valid and accurate, and then follow through on new insights with action? Solving this isn’t just about scale. It’s about choice and the ability to select the tools you need, even if those tools come from different vendors.

Value Achieved from Big Data

So what has big data achieved? “It got people realizing that data is a business tool, and that technologists are the custodians of that data,” points out New Zealand CIO Anthony McMahon. And while data leaders own data quality, it’s up to analysts and business leaders to determine the course of action. “The CIO is key in making sure data is available, accessible, and secure. But it is not up to them to make decisions from it.”

“I think awareness of data and the potential for its utility is actually the core big win for big data so far,” agrees Seidl. “There are a lot of other wins, The ability to handle higher volumes, greater velocity, and greater variety of data. This includes the ability to handle large volumes of unstructured data.”

“Big data added agility into a managed platform in a way that old school data warehouses just couldn’t,” stresses Jones. “It matured parallel compute models for data processing at scale and accessibility of skilled developers and data science teams.”

Where Should Big Data Go from Here?

Big data techniques must evolve to handle, consume, and clarify data. “One of the challenges with big data is going from a massive amount of data to impactful bits of small data,” claims Kirsch. Big data by itself does not equal money or insights. But as the cloud matures, storage is relatively cheap and the processing power to work on big data is easily available.

For this reason, Jones asserts it is “not where big data goes, it is more about where data goes. The future is the business control of data based on a managed platform where the CIO provides provisioning and availability. Big data broke the old school data warehouse model, but it was a transition, not an end.”The rise of big data motivated business leaders to invest in data technology and practices. As data management matures,, CIOs say it is important to consider data ethics too.

“We need to deal with ethics, data quality, data democratization and access,” says Seidl. We also need to “learn about both better AI/ML/analysis tools and understanding the implicit and explicit biases that exist within them.” To improve ML and Ethics, data literacy training is critical. Clearly, it is easy to create a bias in training data. Implicit bias is a big issue, as is defining what constitutes ethical, appropriate use of data.

Parting Words

Data is an obstacle that can become a competitive advantage… with the right work. And much of that work, with the right tech stack in place, is about people and process. “In moving organizations from data-rich to data-driven, the issues that remain are people and education,” points out CIO Myra Davis. Data storytelling, she stresses, is important as well. CIOs today need to be great communicators, who are able to tell stories with data — forging alignment across the business. Otherwise, business leaders will never grok that the value buried in their data.