Organizations

1 results for Orchestration
  • Project Overview

    This repository demonstrates workflow orchestration for data engineering pipelines using Kestra. It guides users through building, running, and scheduling data pipelines that extract, transform, and load (ETL) data both locally (with PostgreSQL) and in the cloud (with Google Cloud Platform). The project is hands-on and includes conceptual explanations, infrastructure setup, and several example pipeline flows.


    Key Concepts

    • Workflow Orchestration: Automating and managing complex workflows with dependencies, retries, logging, and monitoring.
    • Kestra: An orchestration platform with a user-friendly UI and YAML-based workflow definitions (called “flows”).
    • Data Lake & Data Warehouse: Demonstrates moving data from raw storage (GCS) to structured analytics (BigQuery).