DQP Operation¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
This section describes how to create DQP rules and schedules. You will need the URL and login credentials for your Alation Service Manager (ASM).
Login to ASM¶
In your browser, navigate to your Alation Service Manager (ASM) instance. Login to ASM by clicking the user icon, top right:
Provide your login credentials for ASM and click LOGIN:
On ASM’s homepage, select Data Quality Processor:
Alternatively use ASM’s left navigation menu by expanding the Data Quality Processor option and clicking Home:
The left navigation has three DQP options:
Home - The DQ rules and schedules that you create appear here.
Define - For creating new DQ rules.
DQP Settings - For setting and testing the Information Query ID for the relevant data source.
DQP Home¶
DQP’s home page is where you manage the DQ rules, alert thresholds, and scheduling.
DQP Rules¶
In the DQ Rules list the following cells can be edited:
Good - DQ score values at or below this will be classified as “Good” (or green) status.
Warn - DQ score values between “Good” and “Warn” will be classified as “Warning” (or amber) status. Finally, all values above this will be classified as “Alert” (or red) status.
Is Active - Enable or disable the DQ rule.
Note
The DQ rules must be associated with source data that is a cataloged Alation data source.
DQP Scheduling¶
In the Currently Scheduled DQP Rules list further down the home page you will see job schedules. The schedule controls DQP’s execution frequency. When DQP executes it does three things:
DQP pulls the DQ results from the published Compose query in Alation. The query itself is also typically running on a schedule.
DQP processes the DQ results data against the defined rules on the DQP home page.
DQP pushes the DQ results back to Alation.
It is recommended DQP’s schedule is sync’d with the Compose query schedule in Alation. DQP’s cron job should not run more frequently then the Compose schedule, and DQP should run just after the Compose query completes.
Check the box alongside a schedule to reveal controls to reconfigure rules, run DQP manually on-demand, as well as view and download DQP’s logs.
Define New DQ Rules¶
To create a new DQ Rule, either click the Create DQ Rules button at the bottom of the page, or click Data Quality Processor > Define in the left navigation.
Choose the relevant data source. In the following example we’ll choose Snowflake:
The Define New Rules page is displayed:
Follow these steps to create a new rule:
Enter the Alation datasource IDs for each database shown in the table on the right. In the above example, we have Snowflake databases named DQ_DEMO_DATA and COMPANY_DATA. In Alation we can identify the Snowflake data source ID by navigating to the data source and examining the URL.
Fill out the values for the Good and Warn thresholds.
Good: Metric values at or below this value are consider to be acceptable, and so no alert or warning is produced.
Warn: Metric values between “Good” and “Warn” would be marked as a Warning using a yellow “!” triangle icon in the UI.
all values above this would be marked as an Alert using a red “!” circle icon in the UI.
Note
If an “inverted” threshold is required, for example when you require an alert when a value is zero (or “high values good, low values bad”) then make the Good value higher than Warn value. For example if you were to configure a threshold for ORDER.ITEMCOUNT to be Good = 5 and Warn = 1 then anything less than 1 would be an Alert.
Press Enter after entering the desired value in the cell.
Tip: Use the tab key to move to the next editable cell and shift+tab to move to the previous editable cell.
Alternatively you can download the rules table as a CSV file, edit it, and then re-upload it.
Scheduling¶
Enter a cron schedule expression. For examples of cron schedule expressions please refer to For example, to pull DQ results once daily at midnight use 0 0 * * * as shown:
