Data Transformations
DataChannel Transformations provide everything data teams need to convert raw data into a target format that can be fed into operational systems in your selected destination or to run analytics/BI operations. Once a data warehouse is configured within DataChannel, users can schedule transformations to automatically update tables / views / calculated fields whenever new data is loaded into the destination.
DataChannel Transformations follow an ELT (extract, load, transform) model. Since transformations happen in the destination, your raw data is always available along with the transformed data. This implies that if a transformation fails, you will not lose data. If your organization’s analytical needs change, you can edit your transformations and run them again on the raw data.
Transformations are simply SQL scripts that are executed in the destination based on specific events or conditions. The main purpose of transformations is to map data into a specific shape that will be easier or faster to use downstream in the BI or analytical tools.
| DataChannel does not parse the SQL query for logical or semantic correctness and just executes the provided SQL script on the schedule. Please check the SQL scripts carefully before configuring the transformations. |
Run conditions
DataChannel transformations allow you to execute SQL scripts based on a chosen run condition.
We support the following run conditions:
-
Scheduling (running a transformation periodically)
-
Scheduling with a dependency (running a transformation only when pipeline(s) has finished running successfully)
-
Manual Runs
| If a transformation is already running or is being tested, it cannot be run or tested again. Another attempt to run or test the transformation will fail immediately. After the existing transformation completes, it can be run again. |
Scheduled transformations
Scheduled transformations run periodically based on the frequency you’ve selected. You may set a frequency between 15 minutes and 24 hours.
Scheduled transformations (with dependency)
Dependant transformations run only when the pipeline(s) the transformation is dependant upon has finished executing successfully. When you create a dependant transformation, you select which pipelines the transformation is dependant on. We check the status of the execution of those pipelines before we run the transformations. If they are still running or had some error, the scheduled run will be skipped and will get executed at the next scheduled interval.
Still have Questions?
We’ll be happy to help you with any questions you might have! Send us an email at info@datachannel.co.
Subscribe to our Newsletter for latest updates at DataChannel.