Detailed Course Outline
Module 1 - Introduction to Cloud Composer
Topics
- Data Engineer's need for Workflow Orchestration
- Introduction to Apache Airflow
- Cloud Composer
- Environment Setup
- Using the Composer and Airflow
Objectives
- Explore Apache Airflow and Cloud Composer.
- Provision Cloud Composer instances.
- Explore the Airflow and Composer UIs.
Activities
- Lab: Provisioning Cloud Composer
Module 2 - Creating and managing DAGs
Topics
- DAG structure and best practices
- Common operators
- Dependencies, trigger rules, and flow control
- Integration of Airflow and Google Cloud Services
Objectives
- Write DAGs.
- Explore common Airflow operators.
- Manage triggers, dependencies, and flow control.
- Integrate Airflow with Google Cloud Services.
Activities
- Lab: Assembling a Data Processing Workflow
Module 3 - Advanced Airflow techniques and best practices
Topics
- Advanced Airflow features
- Debugging DAGs
- Performance and scalability
- Security and Access Control
- Observability and monitoring
Objectives
- Leverage advanced Airflow features.
- Debug DAGs.
- Observe and monitor your running DAGs
Activities
- Lab: Extending and Monitoring DAGs