Vitals  /  Dev Log

Dev Log

A running, dated log of what was built and what was learned — newest first.

2026-07-02

Real Kafka stream source ✅

  • Wearable stream now reads format("kafka") from a local Docker KRaft broker (single-node, no Zookeeper). A kafka-python producer publishes 15,169 events to the wearables topic; the Spark consumer runs the same clean_wearables transform → same parquet sink as the file path.
  • make stream-parity confirms cleaned output is identical (15,169 events, file == kafka) — concrete proof the source swap changes nothing downstream.
2026-07-01

Great Expectations silver DQ gate ✅

  • Great Expectations (GX Core 1.x) is now the gating DQ contract for silver. A code-defined suite validates coded-vocabulary value-sets, the PHI boundary column set, and ranges + key uniqueness — exits non-zero on any violation.
  • CI runs it after make build: the gate can't be skipped. Complements the descriptive dq_report.json; dbt tests still gate gold.
2026-06-30

Feast feature store made real ✅

  • Feast was scaffolded but never applied. Now materialized offline→online (sqlite): get_online_features (low-latency inference) + point-in-time historical retrieval (the leakage-safe training join). Both paths parity-checked against the offline parquet.
2026-06-30

Full-medallion job on Databricks ✅

  • Bronze + silver are now in the Asset Bundle job as a python_wheel_task. One scheduled serverless run does generate → bronze Delta → silver Delta → gold (dbt + 29 tests) → drift monitor. Verified TERMINATED SUCCESS: bronze=28,816, silver=27,402.
  • Three Free-Edition lessons learned live: ship a lean wheel; pin the compute Python version (3.12); branch dbt dialect on target.type.
2026-06-29

dbt semantic layer + real pgvector RAG ✅

  • MetricFlow semantic layer: 7 composable metrics declared in YAML over a new fct_patient_metrics base. mf query results parity-proven against the marts.
  • pgvector replaces the TF-IDF placeholder: Docker (pgvector/pgvector:pg16), fastembed bge-small-en-v1.5 (384-d ONNX/CPU, no API keys), HNSW cosine, idempotent upsert. TF-IDF remains the fallback.
2026-06-26

Databricks deploy path + ops hardening ✅

  • Asset Bundle job (databricks.yml) ships gold as a scheduled serverless job, verified TERMINATED SUCCESS. Two modes, one codebase behind a target switch.
  • Failure alerts (on_failure email) and drift monitoring as a job task (spark_python_task downstream of gold_dbt).
  • Hermetic CI gate: ruff + unit tests + full local pipeline, on every push.
2026-06-23

Phase 4: governance & polish ✅

  • Drift monitoring (monitoring.py): PSI per feature. Stable on a natural split; correctly flags an injected population shift.
  • Auto-generated data dictionary + lineage (catalog.py) from dbt's manifest/catalog.
  • Governance page + ADRs for the four non-obvious decisions.
2026-06-23

Earlier phases — MVP → OMOP → multi-source → streaming ✅

  • Phase 1 MVP: generate → bronze → silver → dbt gold → serve, one command (make run). Surgery-risk model ROC-AUC 0.825.
  • Phase 2: OMOP CDM (omop_person / condition_occurrence / measurement); claims, PRO, wearables added; 20-feature store across 4 sources.
  • Phase 3: Spark Structured Streaming (15,169 events, 448 outliers nulled) + PySpark window function.
  • Kickoff: locked the architecture (medallion + healthcare layer), scaffolded the repo and this documentation site.
← PREVIOUS
Roadmap