Alation Announces 2021.4 Release: Interview on Column-Level Lineage with Jason Ma, Senior Director of Product Management

Hi Jason, thanks for taking the time to speak with me. Which 21.4 update are you most excited about?

Of course, Talo, happy to speak with you. With this release, we’ve made some nice enhancements to our data lineage capabilities. We’ve invested in two areas to help our customers get more lineage into the data catalog. First, we enhanced our automated lineage capabilities. Second, we created a process to empower those with tribal knowledge to curate that lineage manually.

Can you talk more about column-level lineage (CLL)? Why should customers upgrade?

This has been a big ask from our customers, and something we’re extremely excited to introduce. With 2021.4, we’ll more deeply introspect the query logs and automatically populate the column-level lineage.

Because of our deep partnership with Snowflake, we are introducing this feature first with Snowflake. But in our next releases you’ll see a rolling thunder approach as we release this capability for more and more sources.

Robust Upgrades for Snowflake Users

Why should Snowflake users be excited about these lineage updates?

Simply put? Automation lightens the Snowflake user’s burden. With automated column-level lineage for Snowflake, we’ll parse the query logs and those column-level attributes. The catalog will then populate those column attributes in the lineage graph.

Why are these updates important?

Compliance is no longer an option; it’s an imperative. And stewards need these lineage details for their organizations to pass compliance audits. But lineage maintenance is often tough and tedious to do manually, so automating this process will free up a lot of folks for more productive, meaningful work.

Can you talk more about manual lineage? Why is it valuable?

Automated column level lineage does a lot of heavy lifting. But there are edge cases where the lineage cannot be automatically captured. For those situations, we are excited to introduce manual lineage.

Manual lineage empowers users to stitch together the lineage directly from the UI, which empowers stewards to input their tribal knowledge directly. Manual lineage also inherits all the rich functionalities of our normal lineage, so users can propagate deprecation and even conduct impact analysis, just like they would with lineage captured automatically.

We’re rolling out a beta version of manual lineage with this release, and it will be available to all customers in our next 2022.1 release early next year.

External Tables Create a Shared View of the Data Lake

How else does this release support Snowflake users?

In addition to adding support for Snowflake column-level lineage, we’ve also added the capability to automatically extract and catalog Snowflake external tables. We’ve seen external tables become popular with our customers, who use them to provide a normalized relational schema on top of their data lake. Essentially, external tables create a shared view of the data lake, a single pane of glass everyone can reference.

With 21.4, not only can you catalog those external tables — you can also query against them from Alation. This is powerful, as customers can discover insights in their data lake using our intelligent SQL editor, Compose.

Alation will also surface key details about every table: The catalog page will automatically provide a pointer to the file that created the external table, or the source file (Parquet, Avro, JSON, etc). These are valuable details for compliance officers, data stewards, and basically anyone curious about the data asset’s origin.

The Future of Alation

That’s a lot to take in! And I know these are just a small piece of all the upgrades you and your team have made in 21.4. Looking into the future, what enhancements are coming up?

That’s a loaded question. But lineage and governance will remain a core focus.

We’re definitely going to do much more with our other technology partners, like Google, SAP, and AWS. Right now, we’re beta-testing automated column lineage on these platforms before a wider rollout.

Wow! It’s clear you and the product teams have been hard at work. Thank you so much for taking the time to speak with me today, Jason. I can’t wait to see what’s next.