dbt™ Transformation Logs

This pipeline can be used to request and retrieve logs generated for all errors and successful runs of dbt™ transformations performed on your data. The pipeline will return data in the form of a table. The name of the table generated will be the same as the Dataset Name given by you while configuring the pipeline. The table created can be viewed from two places on the DataChannel Console: The 'Data Preview' tab of the configured pipelines; and also from the 'Data Explorer' tab on the sidebar.

The table contains the fields: start_time, end_time, status, id, project_name, account_name, errors, dbt_project_id and model_type. The status column will indicate whether the run was successful or erroneous.For erroneous runs the details will be visible in the errors column in JSON format. This can be understood from the following sample table:

logs dbt transformation tables schematic

Data Pipelines Details

Data Pipeline: Select dbt Transformation Logs from the dropdown

Setting Parameters

Parameter Description Values

Parameter	Description	Values
Fetch Mode	*Optional* Specifies the manner in which logs data will be fetched from DataChannel : FULL will fetch the entire object and INCREMENTAL will only fetch the new records after the latest update.It is suggested that the pipeline be executed in incremental mode.	{Full, Incremental} *Default Value:* INCREMENTAL
Insert Mode	*Required* This refers to the manner in which data will get updated in the data warehouse, with 'Upsert' selected, the data will be upserted (only new records or records with changes) and with 'Append' selected, all data fetched will be inserted. Selecting 'Replace' will ensure the table is dropped and recreated with fresh data on each run. Recommended to use "Upsert" option unless there is a specific requirement.	`Upsert`, `Append`, `Replace` *Default Value:* UPSERT

Fetch Mode

Optional

Specifies the manner in which logs data will be fetched from DataChannel : FULL will fetch the entire object and INCREMENTAL will only fetch the new records after the latest update.It is suggested that the pipeline be executed in incremental mode.

{Full, Incremental}

Default Value: INCREMENTAL

Insert Mode

Required

This refers to the manner in which data will get updated in the data warehouse, with 'Upsert' selected, the data will be upserted (only new records or records with changes) and with 'Append' selected, all data fetched will be inserted. Selecting 'Replace' will ensure the table is dropped and recreated with fresh data on each run. Recommended to use "Upsert" option unless there is a specific requirement.

Upsert, Append, Replace

Default Value: UPSERT

Datapipeline Scheduling

Scheduling specifies the frequency with which data will get updated in the data warehouse. You can choose between Manual Run, Normal Scheduling or Advance Scheduling.

Manual Run: If scheduling is not required, you can use the toggle to run the pipeline manually.
Normal Scheduling: Use the dropdown to select an interval-based hourly, monthly, weekly, or daily frequency.
Advance Scheduling: Set schedules fine-grained at the level of Months, Days, Hours, and Minutes.

Detailed explanation on scheduling of pipelines can be found here

Dataset & Name

Dataset Name: Key in the Dataset Name(also serves as the table name in your data warehouse).Keep in mind, that the name should be unique across the account and the data source. Special characters (except underscore _) and blank spaces are not allowed. It is best to follow a consistent naming scheme for future search to locate the tables.
Dataset Description: Enter a short description (optional) describing the dataset being fetched by this particular pipeline.
Notifications: Choose the events for which you’d like to be notified: whether "ERROR ONLY" or "ERROR AND SUCCESS".

Once you have finished click on Finish to save it. Read more about naming and saving your pipelines including the option to save them as templates here

Still have Questions?

We’ll be happy to help you with any questions you might have! Send us an email at info@datachannel.co.

Subscribe to our Newsletter for latest updates at DataChannel.