Architecture

Vitals follows a medallion lifecycle (bronze → silver → gold) with a healthcare layer overlaid — the part a generic ETL project doesn't have.

Why three gold stores

Analytics, classical ML, and LLM/RAG need different shapes of the same clean data:

• Analytics marts — dimensional fct_/dim_ models plus a MetricFlow semantic layer (declarative, composable metrics: surgery rate, conservative spend, adherence — one definition shared by BI, cohort analysis, and ad-hoc queries). The trusted serving layer. Local DuckDB; dialect-fixed and validated live on the Databricks serverless job.
• Feature store (Feast) — entity = patient; 20 time-windowed aggregates spanning four source types. Online store (sqlite) for low-latency inference; offline store (file) for point-in-time historical training joins (leakage-safe). Both paths materialized and parity-checked.
• Vector index (pgvector) — 390 clinical notes indexed with fastembed bge-small-en-v1.5 (384-d, HNSW cosine) in a local Docker pgvector instance. TF-IDF is the clone-and-run fallback when Docker is down. Retrieval-only; the demo proves the data is AI-ready, not an LLM project.

The healthcare layer

• Standardized vocabularies. ICD-10 (diagnoses), SNOMED CT (problems), LOINC (labs/observations), RxNorm (medications) — mapped on the way into a recognizable OMOP Common Data Model.
• Data-quality contracts. Validity, completeness (the silent-bias killer in health), unit consistency, uniqueness, and timeliness — enforced as contracts at the silver gate. Great Expectations validates coded-vocabulary value-sets in a suite of 14 expectations; all 14 pass (make dq). This gate runs in CI and exits non-zero on any violation.

Production deployment

The medallion runs two ways by design: clone-and-run on DuckDB (no creds, no network, no Spark cluster — the reproducible baseline any reviewer can run), and as a scheduled Databricks serverless job. A Databricks Asset Bundle ships the full pipeline — medallion_ingest → gold_dbt → drift_monitor. Verified TERMINATED SUCCESS; bronze = 28,816 rows, silver = 27,402.

terminal
$ make setup && make run   # DuckDB, no creds
$ make bundle-deploy       # ship serverless job
$ make dq                  # 14/14 Great Expectations

The wearable stream reads from a real local Kafka broker (Docker, single-node KRaft), not just a file source. Parity is proven: make stream-parity runs both the file and Kafka paths through the shared clean_wearables transform and asserts identical cleaned output (15,169 events, file == kafka).

Tooling