Roadmap

• Repo, mise (databricks CLI / terraform / python) + uv venv
• This documentation site live (auto-deployed via GitHub Actions)
• Databricks Free Edition workspace; Unity Catalog + Delta schema — Terraform IaC applied & verified (3 catalogs, 7 schemas, landing volume, live PHI-gating grants)

• FHIR-shaped NDJSON landed in bronze (seeded synthetic generator)
• Mess-injector (schema drift, dupes, unit drift, missingness — deterministic seed)
• bronze→silver: flatten FHIR; de-id (PHI dropped + assertion); standardize LOINC/ICD-10, mmol/L→mg/dL
• dbt silver→gold: dim_patient, fct_observation, mart_condition_outcomes
• 29 dbt tests on the silver/gold gate, passing; marts backed by a MetricFlow semantic layer (7 metrics)
• Feast feature store — materialized offline→online (sqlite), parity-proven
• Vector index + RAG over clinical notes — real pgvector (BGE-small 384-d, HNSW cosine) with TF-IDF fallback
• Demo surgery-risk model (MLflow, ROC-AUC 0.825 at Phase 1)
• Single end-to-end run (make run) + Airflow DAG mirroring it

• Land the OMOP CDM (person, condition_occurrence, measurement) with concept mapping + tests
• Add claims (837/835) + PRO surveys + wearable batch as sources (cleaned at silver)
• Expand marts & features — mart_cost_outcomes; 20-feature store across 4 sources

• Wearable stream via Spark Structured Streaming — real Kafka source (local Docker KRaft), parity-proven identical (15,169 events, file == kafka)
• PySpark-at-scale transform with a window function (7-obs rolling pain per patient)

• Lineage + data dictionary auto-generated from dbt → Data Catalog
• Governance model (PHI classification, de-id boundary, Unity Catalog mapping)
• Drift monitoring (PSI) on the feature store
• Decision records (ADRs) in repo docs/adr/ + vault

Wire the local pipeline to write Delta into Unity Catalog on the live Free Edition workspace, with end-to-end row-count + DQ parity verified against local DuckDB.

• UC object graph applied & verified via Terraform: 3 catalogs, 7 schemas, landing volume, live PHI-gating grants
• Bronze → Delta: backend abstraction; 8 Delta tables in vitals_bronze.raw.*; row-count parity for all 8 sources
• Silver → Delta (the PHI boundary on UC): de-id + conform ported to Spark; de-id assertion + full DQ parity verified
• Gold via dbt-databricks: 10 models + 26 tests pass on Databricks; all 11 gold tables match local row counts
• Production deploy path: a Databricks Asset Bundle ships gold as a scheduled serverless job, verified TERMINATED SUCCESS

• Full-medallion python_wheel_task job: one scheduled serverless run does generate → bronze → silver → gold → drift, no laptop; verified TERMINATED SUCCESS, with on_failure email alerts
• MetricFlow semantic layer over the marts: 7 composable metrics declared in YAML, parity-proven vs the dbt marts
• Great Expectations gates the silver DQ contract in CI: coded-vocabulary value-sets, PHI boundary check, ranges + key uniqueness
• Hermetic CI quality gate: ruff + unit tests + full local pipeline + the GE silver gate, on every push