Alation 2021.1: Architecting a Platform for Data Intelligence with Junaid Saiyed, SVP of Engineering at Alation

Before we jump into what’s new with Alation 2021.1, can you talk about what it means for Alation to be a platform?

I’m happy to. Alation has evolved a great deal since pioneering the data catalog category. Initially, Alation was a solution that made it dramatically easier to find, understand, and trust data. Now, Alation is a platform for a broad range of data intelligence use cases — from data search & discovery to data governance to cloud migration.

Can you discuss what is going on “underneath the hood” to make the platform work?

Underpinning Alation is broad and deep connectivity. Alation makes it easy to connect to anything in the organization and get critical context from those connections. We’ve also made Alation open, enabling third-party and customer developers to add full-blown connectivity to a particular system.

Let’s start with broad and deep connectivity. What was the evolution there?

In 2020, we made a huge investment in our APIs. Creating APIs for connectors is a huge challenge. Security setups are exotic and have become more stringent with a slew of new standards to address. On top of that, conflicts occur due to libraries and accessed resources on the filesystem, like Kerberos configuration. Now, we have solved these challenges and gotten our APIs to the level where third parties and customers can leverage capabilities in a programmatic and granular fashion.

But connectivity isn’t only about the breadth of connectors — the depth of connectivity matters just as much. Depth comes in many forms, but much of it is powered by three forms of metadata: technical, behavioral, and provenance.

Technical metadata tells you a lot about the data, including the names of columns and tables and what schema they are a part of. Unique to Alation is the ability to leverage behavioral metadata, which tells you how the data is being used, who is using it, and much more. On top of that, provenance metadata enables lineage, telling you where data has come from, and what has been created using that data. We have created best practices for dealing with hundreds of millions or billions of pieces of each of these types of metadata and made it accessible to data consumers, giving them an incredible amount of intelligence on the data.

Can you talk about the openness of the platform?

With Alation 2021.1 and the introduction of the Open Connector Framework, we have given customers and partners the ability to create their own connectors and applications that leverage the full breadth and depth of the platform.

Now the entire platform is infused with openness throughout, enabling developers to add full-blown connectivity to virtually any system. They can make their own connectors and applications as deep as they want by leveraging any combination of technical metadata, behavioral metadata, and lineage.

To help them, we provide well-defined APIs to ingest metadata and starter kits that help developers create connectors with ease. On top of that, we enable developers to package their connectors as containers, enabling them to scale as well as share their connectors for others to use.

What does the connectivity and openness mean for partners and customers?

The Alation Data Catalog supports a wide range of data intelligence use cases, and we have strengthened that ecosystem with our partners. For instance, our partnership with Tableau makes it easier for analysts to find trusted dashboards, and our partnerships with Snowflake and AWS make it easier to migrate data to the cloud. We are laying the foundation for a marketplace where customers and certified partners can create connectors and applications that add to that already broad ecosystem.

This is exciting news, Junaid. What’s next? What are some of the engineering challenges that your team is excited to tackle?

Our customers are using the Alation Data Catalog to address a range of data intelligence use cases. And they’re doing it at scale. We have deployments today with thousands of users and metadata being ingested and analyzed for millions of columns. Our teams have built a robust, high performance platform that holds up well under these conditions.

As increasing demands are placed on the data catalog, the “scaling imperative” becomes even more pronounced. Enabling outcomes for our customers – and doing it at scale – is our passion and it’s part of what makes Alation a great place to work. Our Engineering teams on both sides of the Atlantic have been growing at 50% with this singular focus, and we have many positions open today across Engineering and other departments.

Last but not the least Alation started it’s journey to be a cloud native architecture in 2020 and we plan on finishing that transition in 2021. So if you like technologies like k8s, containers, service mesh, again we have a lot of positions open.