Overview

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

To get the enhanced connector for Universal DS, contact the Forward Deployed Engineering team at Alation.

The enhanced Universal DS OCF Connector simplifies loading metadata from DS sources into Alation, overcoming challenges like networking blocks and rigid file formats. It offers flexible integration via CSV files from external systems or cloud storage, providing a secure, efficient solution for managing complex data environments.

The Universal DS OCF provides a powerful solution for easily generating DS data sources in Alation by reading metadata from CSV files. Administrators have full control over how the file’s columns are managed directly in Alation’s user-friendly interface, using familiar OCF settings.

With the Universal DS OCF, you can capture key technical metadata, including:

  • Catalog/database names

  • Schemas

  • Tables

  • Columns

  • Data types

Team

The following administrators are required to install this connector:

  • DS Admin:

    • Provides the Universal DS CSV file post extraction

  • Alation Server Admin:

    • Installs the connector

    • Creates a Universal DS source

    • Configures the Universal DS source

Scope

The table below describes which metadata objects are extracted by this connector and which operations are supported.

Feature

Scope

Supported

Metadata extraction

Default MDE

Extraction of metadata from CSV

Yes

Report Lineage

Intra-system lineage

Yes

Cross System Lineage

End-to-end lineage at the system level

Manual via API

Column Level Lineage

Column level lineage refers to the ability to trace columns across data sources in the “Lineage” tab for a catalogued object

Manual via API

Object Usage Statistics

Popularity of Schema, Tables and Columns

No

Selective Extraction

Ability to apply filters

No

Full Extraction

Ability to run full extraction on the CSV instance

Yes

Extracted metadata objects

Schema

Data source schema displayed

Yes

Table

All Tables displayed

Yes

Column

All Columns displayed

Yes

Primary Keys

Primary keys can be identified

Yes

Foreign Keys

Foreign keys can be identified

Yes

File based extraction of Query History

Ability to load file based extraction of Query History (CSV)

Yes

File Storage Location

Ability to load file Storage Location (S3, Local, Blob) (CSV)

Yes

Object Mapping

The diagram and table below describe which metadata objects are extracted by this connector and how they are mapped to objects in Alation.

../../../_images/UniversalDSMappings.png

Column location configuration

The following columns are required in the CSV file that is being uploaded into Alation.

  • Catalog Name Column (1)

  • Schema Name Column (2)

  • Schema Comments Column (10)

  • Table Name Column (3)

  • Table Type Column (4)

  • Table Comments Column (11)

  • Column Name Column (7)

  • Column Type Column (5)

  • Column Datatype Column (6)

  • Column Position Column (8)

  • Column Nullable Column (9)

  • Column Comments Column (12)

  • Primary Key Column (13)

  • Foreign Key Column (14)