If you've shipped CI/CD for application code, you know the rhythm: commit, run unit tests, build, deploy to staging, smoke test, promote. Data teams reach for the same shape — and then hit walls that don't exist in the application world.
State changes everything
Application deploys are mostly stateless. You ship new code; the runtime starts using it. Data deploys are almost never stateless. Schema changes, backfills, and migrations all have to coexist with running workloads — and the production data is the asset, not the artifact.
Test data is its own engineering problem
You can't just spin up a unit test database with three rows. Real data has skew, real production has nulls in unexpected places, and real pipelines fail on edge cases that no synthetic dataset will catch. We default to running transformation tests against frozen snapshots of real data — sampled, masked, and version-pinned.
Feedback loops are slow
An app's CI run is minutes. A data pipeline's full integration test might take an hour. This shapes everything about how you structure the pipeline: fast unit tests on transformation logic, slower integration tests on end-to-end flow, and acceptance tests that run on a cadence rather than per-commit.
What we ship by default
- Per-PR runs of unit and contract tests against representative data
- Schema diffs surfaced in the PR with an "approve schema change" gate
- Lineage-aware deploys that detect downstream impact
- Environment promotion with rollback windows, not just rollback hooks
None of this is novel. The thing that derails teams is treating data CI/CD as "the same thing, just for SQL." It isn't.