Deploying a Version Is Not the Same as Applying a Change

The publisher thought that their chapter update would be easy. It should be easy.

Chapter 3 of a large reference book had been rewritten, reviewed, and tested in a separate copy of the book.

The library kept its authoritative copy in a large binder, so replacing the chapter seemed simple: open the rings, remove the old Chapter 3, insert the new one, and close the book.

But that is when the problems started.

The new Chapter 3 referred readers to the case studies in Chapter 9. In the library’s copy, Chapter 9 was about documentation and didn’t have any case studies at all.

Another new section in Chapter 3 pointed readers to Chapter 15—the definitive guide to parakeets.

There was no Chapter 15 and in fact no chapter at all on parakeets. It had been an experimental chapter in the test copy of the reference book.

Then someone happened to notice that the new Chapter 3 was longer than the old one, so every chapter after it had moved several pages.

The table of contents was wrong, as were references elsewhere in the book that pointed to pages after Chapter 3.

Nothing was wrong-wrong with the new chapter. But it had been tested in a different version of the surrounding book.

That distinction matters in software too. In fact, I’m not sure if any experienced publishers would make the above mistake, but updating data with dependencies and running head first into problems, that happens ALL the time in software.

Deploying a Version

A typical software deployment often works something like this:

Build Version 12 → Test Version 12 → Deploy Version 12

We create a known but slightly edited version, test it, and move that version into production.

If our publisher were releasing an entirely new edition, they could update Chapter 3, add Chapter 15, change Chapter 9, regenerate the table of contents, fix every reference, test the finished book, and publish that edition. The old book would be moved to archives rather than staying in place.

Thus the thing being tested is essentially the thing being deployed—mostly—this is how we tend to think about CI/CD pipelines. We are testing a full version of the software and deploying the whole codebase out to production once it has passed its tests.

Data changes often work differently. Instead of replacing the whole thing, we may be applying one change, or a handful of changes to data that already exists:

Existing production state + Change X → New production state

Now the starting state matters.

A Small Data Example

Imagine a database of pets.

A pet record might look conceptually like this:

Bailey → Handler 42 → Care Plan 17

Suppose Bailey’s record is moved from a test database into production. In test, Handler 42 is Maria.

But in production, what if Handler 42 is James who has no experience at all with parrots. James is actually a cat specialist.

Or what if Handler 42 doesn’t exist.

The pet record itself can be perfectly valid while the relationships around it are not. Worse, the operation may succeed.

If Handler 42 exists in production, the database may happily accept the row and assign Bailey to James. Nothing technically failed, but the result is wrong.

That is the part that makes state-dependent data changes different from a software pipeline for deploying a tested version.

What Did We Actually Test?

When a data change succeeds in a test environment, it is tempting to summarize the result as:

The change works.

More precisely, what we demonstrated is:

The change works against this state.

If the production state is equivalent in all the ways that matter, that distinction may be unimportant.

If it is not, the same change may fail, behave differently, or succeed while producing an unintended result.

That is why a data-promotion process should not automatically be treated as another software delivery pipeline.

The question is not only: Was the change tested?

It is also: How much does that change depend on the state in which it was tested?

A useful shorthand is:

Test State A + Change X → Expected Result
Production State B + Change X → ?

The question mark is the important part. We are not always deploying the thing we tested.

Sometimes we are applying a change we tested to a state we did not.