Data Integration with Cloud Data Fusion

Module 1) Introduction to data integration and Cloud Data Fusion  

• Understand the need for data integration,
• List the situations/cases where data integration can help businesses,
• List the available data integration platforms and tools,
• Identify the challenges with data integration
• Understand the use of Cloud Data Fusion as a data integration platform
• Create a Cloud Data Fusion instance,
• Familiarise with core framework and major components in Cloud Data Fusion

Module 2) Building Pipelines

• Understand Cloud Data Fusion architecture
• Define what a data pipeline is
• Understand the DAG representation of a data pipeline,
• Learn to use Pipeline Studio and its components
• Design a simple pipeline using Pipeline Studio,
• Deploy and execute a pipeline

Module 3) Designing complex pipelines

• Perform branching, merging, and join operations.
• Execute pipeline with runtime arguments using macros.
• Work with error handlers.
• Execute pre- and post-pipeline executions with help of actions and notifications.
• Schedule pipelines for execution.
• Import and export existing pipelines.

Module 4) Pipeline execution environment

• Understand the composition of an execution environment.
• Configure your pipeline’s execution environment, logging, and metrics.
• Understand concepts like compute profile and provisioner.
• Create a compute profile.
• Create pipeline alerts.
• Monitor the pipeline under execution.

Module 5) Building Transformations and Preparing Data with Wrangler

• Understand the use of Wrangler and its main components.
• Transform data using Wrangler UI.
• Transform data using directives/CLI methods.
• Create and use user-defined directives.

Module 6) Connectors and streaming pipelines

• Connectors
• DLP
• Reference architecture for streaming applications Building streaming pipelines

Module 7) Metadata and data lineage

• List types of metadata.
• Differentiate between business, technical, and operational metadata.
• Understand what data lineage is. Understand the importance of maintaining data lineage.
• Differentiate between metadata and data lineage