Data Strategy and Decentralization: A Data Architect’s View

In a world as interconnected as ours, it’s difficult to overstate the value of secure information exchange. Blockchain technology delivers a secure record of transaction, using cryptography to interlink records. It’s the basis for cryptocurrency, but also has applications in virtually every industry, from finance (capital markets) to retail (supply chain management) and health sciences (medical drug development). Small wonder, then, that investment is growing. In 2021, global spending on blockchain amounted to $6.6 billion and will grow to reach nearly $19 billion in 2024.

How are blockchain organizations tackling data management? To learn the answer, we sat down with Karla Kirton, Data Architect at Blockdaemon, a blockchain company, to discuss data strategy, decentralization, and how implementing Alation has supported them. Here’s a recap of our discussion.

What is your data strategy and how did you begin to implement it?

Karla Kirton, Data Architect at Blockdaemon: The first question I was asked when I started at Blockdaemon was “how would we approach data?”

We had the obvious choices in front of us: centralized vs. decentralized. So, we started looking at what our use cases were going to be, what we wanted to do with the data, and whether we wanted an offensive or defensive data strategy, or even a mix of both. All of that helped us to decide on a decentralized approach, which was a natural fit for our industry.

Then we looked at the structure of the teams to make sure that we were working in domains and squads, to match that decentralized approach. We looked at how we wanted to use data, how we wanted to make sure it is self-service to people and that it scales, whilst making it understandable and discoverable.

We decided to split the approach into three categories: people, policies, and technology. We started with the policies…. We then moved onto making sure that from a technology perspective our tech stack was what we needed. And, we have now moved on to getting people engaged with those two other aspects – ensuring that they understand the tech and policies, and understanding how they interact with the data – which is where Alation came in.

What are the goals of your data team?

Karla: Well, they’re twofold. We wanted to introduce a Kappa architecture to serve not only our operational data needs, but our analytical data needs too.

And we wanted to do this through one data architecture where possible, so that we didn’t duplicate processes. We made data available to those that needed it and the experts that are there.

It’s a platform-focused architecture, which means that the data experts and the domain team, who know the data the best, can direct their focus towards optimizing the data platform and making it available to the rest of the business.

So, once you’ve considered that, you are essentially building data as a product. And as soon as you start talking about data as a product, it needs to be discoverable, understandable, accessible, secure.

Where does data mesh fit into your plans?

Karla: Since we decided we wanted to decentralize our data, we inadvertently went down the data mesh route without necessarily picking it from the start. It was just a natural fit with what we were trying to do. We were trying to make data available to those who need it, so we wanted it to be self-serve and make the data discoverable.

It meant that domain teams would own their data and we would build a platform that supports them to do just that, while still allowing the business users to access the data themselves. Essentially, we were producing data as a product.

Secondly, we decided we wanted a more decentralized (or federated) approach to data governance as well, because of the capacity for scalability.

We wanted to make sure that the people who understand the data are also aware of the principles that they are subject to. So, they are contributing to their own data governance, and feel empowered to do so and we’re not just defining it top down.

Was there anything you took from data mesh, which helped your data strategy?

Karla: When you start looking at the principles of data as a product, it needs to be discoverable, understandable, accessible, and secure. All these principles of data as a product help you to build your data catalog and understand the framework that your data has to meet.

When I initially set out to do this, I wanted domain areas focused on their own data, but those areas need to be discoverable, which they weren’t. So, we decided we needed what I’ve called the discovery layer.

And, that’s what Alation has provided us. With the data catalog becoming our discovery layer, which has only empowered data mesh further. If no one knows how to access, understand, or discover that data, a data mesh will only become more siloed. It needs that area where people can come together to understand that information.

How else has Alation helped?

Karla: We originally brought in Alation to make our domain areas discoverable. But, when we started looking at what we needed from a data catalog it was metadata management, lineage, data asset registry and more. All these things factored into what we needed from our solution. And, Alation ticked a lot of our boxes!

The big thing that Alation offered us though, was the ability to introduce and assign data stewards. This allowed us to involve more people in our data strategy, more contributors to the database of discoverability and accessibility information, all of whom can see and interact with the data. This ability to interact with the data brings our data strategy to life much more than when people simply view it as just being a string of concepts.

Alation was the way that we introduced data stewards into Blockdaemon. This helped us showcase a tangible use case of what users should be doing and helped us to sell the idea to them.

As a solution, Alation also gave us the ability to connect data sources to people, and vice versa, and has got everybody speaking the same business language, which is so underrated, and has been so helpful overall.

Why did you choose Alation and what was the selection process?

Karla: Initially we gathered the requirements for what we needed from a data catalog, which included metadata, governance, lineage etc. We considered a wide range of tools looking at each of them from the people, policy, and technology perspective that I mentioned earlier, and also looked at which one would fit best with our existing technology stack – to make sure that the integration was as seamless as possible.

So, the fact that Alation would fit into place relatively effortlessly was a big reason why we eventually decided on it.

The engagement process played a big role too. We felt like Alation listened to us and that our needs were heard from the very beginning, which we didn’t find with some other vendors. Alation engaged with us as a company to understand what our path was and how we could work collaboratively.

What has your experience working with Alation been like these past two months?

Karla: We started with a proof of value, a 14-day trial, which was extremely helpful because we could still connect that to real use cases within the business. During that time, we got into a proof-of-value environment and got users from the business to create an account so that they could see the tool features and give feedback on them as well.

We also had the 12-week Rightstart program where we had all the relevant training, and we engaged all of the right stakeholders throughout different parts of the business to take the suitable training modules too. We had support throughout that entire process, to make sure that our installation was correct and we’re just coming to the end of that now. So now we’ve moved on to the customer success plan to make sure we have an adoption strategy.

So, we’ve gone from proof of value to a fully supported production instance within two months and now we’re starting on the adoption and the rollout. There’s been plenty of support along the way and I think a member of every team throughout Alation has been involved.

What does the rest of your data stack look like?

Karla: As I mentioned before, we’re aiming for a Kappa architecture within Blockdaemon. So, we’re streaming first through our Kafka Data Pipeline within the data tech stack, and then through BigQuery and Looker, our main serving layer and visualisation tools, respectively.

We design our data products to be consumer facing. So, the streaming platform will look at a mutable log of all events and will feed that information down into BigQuery. Then we can create consumer data sources, which will be specific to people’s use cases. And, if there are some dashboarding requirements, the data is fed from BigQuery to Looker.

What advice do you have for people who are just starting their data strategy journey?

Karla: To start, you need to understand what your business plans to do, and its vision for data. This could manifest as a defensive strategy to meet regulatory requirements. If you want to innovate, create products, or change the marketplace then you might be looking at an offensive data strategy, or it could also be somewhere in-between. But understand that you have different requirements depending on which route you go down.

Then you need to define your data approach, understand whether a more centralized or decentralized approach is better, and go from there. But regardless, you still need all of your data to be viewable in one place, and you need to understand your data too.

For example, if you opt for a more centralized approach, understanding is often a challenge. So, if nothing else, use a tool like Alation to help out.

What’s next for Blockdaemon from a data perspective?

Karla: It’s adoption, embedding, and creating our data products. So, now that we have all of the tools that we need, and we have the concepts: people, process, and technology… we actually need to put them into action and create our data products.

Curious to hear more from Alation customers? Learn how European energy company Vattenfall harnesses the power of data to become fossil free.