CraneDev

Data2025

Atlas Health

Backfills went from 9 hours to 22 minutes.

Placeholder cover photograph for the Atlas Health project
Stack
Python · dbt · BigQuery
Duration
9 weeks
Year
2025
Sector
Data

The problem

Every ingest reprocessed the full window, so a one-row fix cost a night.

What we built

Incremental ingestion with watermarking and idempotent upserts.

What it measured

Backfills went from 9 hours to 22 minutes.

Talk to the people who would do the work.

Bring the system you already have. We will tell you where we would start, what we would leave alone, and how long the first release takes.

Book a call