Overview¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Available from Alation version 2024.3.4
The OCF Connector for dbt Gen2 is developed by Alation and is available for download from the Connector Hub on Alation Customer Portal. Ask an Alation admin with access to Customer Portal to download the connector from the Connectors section (Customer Portal > Connectors).
Use the connector to catalog dbt as a data source on Alation Cloud Service instances. It extracts projects, models, model columns, and job runs (jobs executed for a given model in an instance). After the metadata is extracted, it is represented in the data catalog as a hierarchy of catalog pages. Alation users can leverage the full catalog functionality to search for and find the extracted metadata, curate the corresponding catalog pages, create documentation about the data source, and exchange information about it.
Team¶
You may require assistance from the following administrators to install and configure this connector in Alation:
Alation Server administrator:
Installs the connector.
Enables and configures extraction from dbt ELT source.
dbt administrator:
Provides the API URLs required to access dbt and extract metadata using dbt Admin and Discovery API
Provides Account ID required for the API URL
Provide Access Token for use with dbt APIs
Scope¶
The table below shows which metadata objects are extracted by this connector and which operations are supported.
Feature |
Scope |
Availability |
|---|---|---|
Authentication |
||
dbt access token |
Authentication using the access token obtained from dbt account settings |
Yes |
Metadata Extraction (MDE) |
||
Default MDE |
Extraction of metadata using dbt APIs. |
Yes |
Filter MDE |
Extraction of selected metadata based on filters. |
Yes |
Extracted metadata objects |
||
Model descriptions |
Description of the models within a dbt project. The model description is available in the model catalog page. |
Yes |
Model column descriptions |
Description of the model columns. Model column descriptions is available in the model catalog page. |
Yes |
Test Status |
Included as Data Quality for impacted tables. |
Yes |
Lineage |
||
Jinja code extraction |
Extraction of Jinja code from dbt into dataflow content |
Yes |
Automatic lineage generation |
Auto-calculation of source lineage based on the metadata extracted from dbt |
Yes |
Column-level lineage |
Calculation of lineage data at the column level. |
Yes |
Table-level lineage |
Calculation of lineage data at the table level. |
Yes |
dbt Gen2 Data Architecture¶
The dbt source architecture contains Projects from the associated RDBMS data sources. Each Project contains models and models in turn have model columns and list of jobs run.
Use the Alation UI to understand and navigate through dbt metadata hierarchy.
Catalog Page¶
After extraction, the catalog page for a Project, model, model columns, and jobs are displayed as shown here:
Here is an example of a project catalog and model catalog page.
On the catalog page for a project, Alation displays the RDBMS data sources associated with Alation that are involved in extraction.
The catalog page for a project displays the list of models and the Job Runs table. The Source System information section on the catalog page displays the linked RDBMS data sources associated with an environment.
Based on the model execution, Alation displays the linked tables in an environment associated with the extracted RDBMS data source on the respective model catalog page. The Job Status table. For a given model, this table displays the latest job execution summary for an environment.
Similarly, the catalog page for model columns display the associated attributes from the RDBMS data source and the Source System information section displays the columns extracted from the respective tables.
Lineage¶
To view the lineage for the tables associated with a model, select the table from the Lineage tab on the model catalog page.
By default, the catalog page displays the lineage for the first table listed in the Source System information section.
To learn more about viewing lineage in the Alation catalog, see Discover Lineage.
For information on how to configure Lineage using Alation user interface, see Configure Lineage.
Data Health¶
To view the health information for any table with associated Data Health rule, navigate to the source table using the link on the Source System information section on the model catalog page and click Data Health. Any failure in model execution gets propagated to downstream objects in the lineage graph making it easy to find the root cause of the failure. Here’s an example:
To understand more about viewing data health within the Alation data catalog, see View Data Health.
Supported Data Sources¶
The dbt Gen2 OCF connector supports the extraction of metadata and lineage of following data sources:
Snowflake
Redshift
PostgreSQL
Google BigQuery
Databricks Unity Catalog
Databricks on AWS
Azure Databricks
Databricks on Google Cloud