Why I Started Versioning Data Like Software: The pip install for Healthcare Pipelines

Here is a look back at the journey my team and I took to fundamentally change how we share data. After spending months building a robust Medallion architecture to clean up chaotic government healthcare exports, we had finally secured our internal database. Our data was clean, validated, and safely stored in our Gold layer.

But cleaning the data was only half the battle. We regularly export this data to hundreds of external collaborators and researchers. When you are supplying data for medical research, the stakes are incredibly high. If we realize today that a batch of data from six months ago contained an error, the fallout isn't just a broken dashboard—it could invalidate a published scientific paper.

Things can often be repurposed into something new and robust. I realized that the software engineering world solved the problem of distributing constantly changing code decades ago. It was time we applied those exact same rules to our data.

1. The First Iteration: Shared Drives and Lying Markdown

Before I landed on the "Data as Code" design, I started the project the way most data teams distribute information: we dropped CSV extracts into secure shared folders for our researchers, accompanied by a beautifully written Markdown file explaining the data dictionary.

The environment, however, quickly became a massive hurdle.

If a researcher wrote a Python model expecting a column named blood_pressure_sys, and we quietly updated our internal systems to rename it to bp_systolic, their code would violently crash the next time they downloaded our CSV. Even worse was the "silent failure." If we discovered a historical error—like the time we realized "NA" stood for Natrium (Sodium) and not a null value—we had to send out mass emails with the subject line "URGENT: Data Correction" and pray that the researchers actually read it before publishing their findings.

I had to weigh our approaches. Writing data documentation in a Wiki or Markdown is essentially writing a code comment: it cannot be trusted, and it will inevitably drift from reality. I realized we needed to stop treating data like a passive, hostile file, and start treating it like an active software dependency.

2. The Architecture: Executable Contracts and pip

Simplicity won out, and I decided to package our data pipeline exactly like a Python library. Every data scientist knows the joy of typing from sklearn.datasets import load_wine and instantly getting a perfect Pandas DataFrame without worrying about file paths. We decided to replicate that exact experience for our healthcare data.

We didn't package the raw data itself—that would create a massive, unscalable Python package. Instead, the raw data stays safely in our cloud storage, flowing through our internal Medallion system (from raw Bronze, to quarantined Silver, to clean Gold). What we distribute to the researchers is an internal Python library containing the Data Contract and the load function.

Here is how the mechanics work:

Executable Schemas

We built our Data Contracts using Pandera (an executable schema library). Before our internal Python package is even built and distributed, a CI/CD process runs a strict validation against the current Gold layer data. If the data violates the contract, the Python package is never built. The contract is not a Markdown suggestion; it is a compile-time guarantee.

Semantic Versioning (SemVer) for Data

We began versioning our data packages using standard software Semantic Versioning, allowing us to communicate exactly what was happening to the data:

• Major (v2.0.0): A breaking schema change (e.g., dropping a column or renaming blood_pressure_sys).
• Minor (v1.1.0): A new batch of weekly incremental data is available, but the schema is identical.
• Patch (v1.0.1): A fix to historical data (e.g., correcting the "NA" Natrium error from a previous dataset).

pip as the Gatekeeper

If a researcher builds a predictive model that requires hospital_data==1.0.1, their Python environment (pip) will physically refuse to let them install v2.0.0 (the breaking schema change) by accident. The code actively protects itself from unwanted or breaking data.

3. The Control Tower: Lineage and the Blast Radius

The most profound relief came from tying this versioning system into our Data Lineage.

When you export data to hundreds of collaborators, tracking the "blast radius" of a data bug is terrifying. Because our data is now versioned like software, our lineage graph doesn't just show internal table joins; it shows external consumption.

If we discover an anomaly today and release a Patch (v1.0.2), our lineage system tells us exactly which of our 100 collaborators originally downloaded the flawed v1.0.0 package. We no longer send mass "URGENT" emails to everyone. We can look at the lineage, see exactly which researchers are utilizing the affected data for their upcoming papers, and proactively alert them to bump their package version to 1.0.2 to receive the fix.

4. Lessons Learned

After months of wrestling with email chains and broken research models, we finally built a distribution system we can trust.

• Software compatibility is a solved problem. Stop trying to invent complex alerting rules for schema changes. Wrap your data in a Python package and let standard dependency management (pip) handle the version conflicts.
• Markdown is a lie. If your schema or data dictionary cannot fail a CI/CD build, it is just a suggestion. Schemas must be executable to be trusted.
• Lineage protects science. Data engineering isn't just about moving bytes from left to right; it is about preserving the integrity of the downstream systems. Knowing exactly who is using what version of your data is the only way to safely correct the historical record.

---