Data Engineering Foundations in Python

Module 1: Getting Started

⏳ ~30 minutes

• Topics: Introduction to data engineering roles, team structures, and GCP setup.

• Hands‑on: Set up GCP and review data engineering lifecycle stages.

Module 2: Team Structures

⏳ ~45 minutes

• Topics: Understand embedded vs centralized data teams and role responsibilities.

• Hands‑on: Quiz to assess team configuration strategies.

Module 3: Data Lifecycle & Cloud Arch

⏳ ~1h 15m

• Topics: End‑to‑end lifecycle, data lakes, warehouses, architecture patterns (Lambda/Kappa).

• Hands‑on: Quiz on architecture models and data lifecycle checkpoints.

Module 4: Data Ingestion

⏳ ~1h 30m

• Topics: Batch vs streaming ingestion, CDC, APIs, file systems, pandas/PySpark pipes.

• Hands‑on: Quizzes and code pads for ingestion pipelines.

Module 5: Data Modeling & SQL

⏳ ~1h

• Topics: Dimensional modeling (Kimball), DDL/DML, SQL query lifecycle in BigQuery.

• Hands‑on: Solve SQL challenges in a BigQuery sandbox.

Module 6: Orchestration Tools

⏳ ~1h 30m

• Topics: DAG design in Airflow, overview of Dagster and dbt.

• Hands‑on: Create full DAG pipelines and build dbt workflows.

Module 7: Data Quality

⏳ ~45 minutes

• Topics: Schema validation, testing with Avro/Protobuf, dbt checks.

• Hands‑on: Setup and test quality pipelines with quizzes.

Module 8: Capstone & Epilogue

⏳ ~30 minutes

• Topics: End‑to‑end Formula‑1 data pipeline, billing management, next steps.

• Hands‑on: Build capstone pipeline and review GCP billing setup.

Get certificate