A Migration Has Two Halves

The change that looks like a rename

Some renames aren’t renames. Changing a variable is a rename. Changing an identifier that live resources carry — an annotation prefix, a label key, a naming convention, a database column, an API field — is a data migration wearing a rename’s clothing. The tell is simple: if existing objects reference the old name, and something in production keys off that reference, you cannot just change the default and redeploy.

I hit this moving a Kubernetes controller’s annotation prefix from one domain to another. The controller (Replikate) replicates ConfigMaps and Secrets across namespaces; every managed copy is stamped with labels like <domain>/managed-by, and every source is marked <domain>/sync. In the live cluster that was 93 managed copies — including production TLS secrets behind ingresses — and 13 sources, all carrying the old prefix.

Flipping the default prefix in one release would have been catastrophic. The controller would stop recognizing every object it already managed: 93 copies would fail the “do I own this?” check, silently stop being updated — including on certificate renewal — and expire weeks later. That’s not a rename. That’s manufacturing an outage on a delay.

The pattern: honor both, write the new, relabel in place

The safe shape of this migration is well-worn, and worth stating plainly because it’s the same shape whether you’re renaming a label prefix, a column, or a field:

1. Honor both names for reads and matching. The new name is primary; the old name stays recognized. Nothing you already own becomes invisible.
2. Write only the new name. Every time you touch a resource, stamp it with the new identifier and drop the old.
3. Let normal operation migrate the data. Each reconcile (or write, or read-through) converts one more resource. No big-bang mutation of production state.
4. Relabel in place — never delete-and-recreate. These were live TLS secrets serving traffic. A migration that deletes and recreates a live secret has a window where the secret doesn’t exist. In place, or not at all.
5. Drop the old name last, in a separate step, once nothing carries it.

None of that is clever. It’s just the discipline of treating a rename as a migration. The interesting part is step 5 — knowing when nothing carries the old name — because that’s where I almost got it wrong.

The two halves

Here’s the trap. After deploying the dual-name version, I watched the copies migrate. A metric counted managed copies by domain, and I could watch the old-domain count fall as reconciles re-stamped each one. Deploy, restart, watch it go: 93 → 0. Done, right?

No. The controller re-stamps copies automatically, because it writes them. It does not rewrite sources — it only reads those. The 93 copies moved on their own; the 13 sources still carried the old prefix, untouched, because nothing in the normal write path ever touches a source’s own annotation.

So the migration had two halves:

• The visible half — the copies — migrated by itself and gave me a satisfying metric going to zero.
• The invisible half — the sources — required a completely separate, manual action (re-annotating them at their origin, which for these lived in cert-manager Certificate templates).

And the two halves were coupled by a landmine: if I had dropped the old prefix when the copy count hit zero, the controller would have stopped recognizing the 13 still-old-named sources, concluded they were no longer sources, and run cleanup — deleting all 93 copies I had just carefully migrated. The metric that said “done” was measuring the half that couldn’t hurt me.

The principle

Completion is when every half is off the old contract — not when the half you can see is.

The dangerous property of these migrations is that the parts don’t move in lockstep, and the part that’s easy to observe is usually the part that migrates itself. The write path fixes what it writes. What it only reads — sources, upstream config, the other service’s copy of the enum — sits there looking fine and quietly holds the old contract, and it’s often exactly the thing whose disappearance is catastrophic.

Three habits fall out of this:

• Enumerate the halves before you start. For every place the old name appears, ask: does normal operation rewrite this, or only read it? The read-only occurrences are your manual migration list. Write them down; they will not announce themselves.
• Don’t trust a convergence metric you didn’t design to cover every half. “Copies by domain → 0” was a real signal, but it answered a narrower question than the one I was about to act on. Make the completion check span all the halves, or don’t treat it as a completion check.
• Make the last step — dropping the old name — impossible to take early. Gate it on the total count across all halves, and treat “the old name still appears anywhere” as a hard stop. The cost of leaving both names honored for another week is nothing. The cost of dropping the old one a day early is your data.

Beyond Kubernetes

This isn’t a Kubernetes lesson; that was just where I met it. Rename a database column and you’ll dual-write to old and new while backfilled rows migrate on read — but the reports, the ETL job, and the other team’s query still SELECT old_column. Rename an API field and your service emits both while clients cut over — but the partner integration you don’t control is a source you only read. Change an ID scheme and the records you write get the new one — but the foreign keys pointing in from elsewhere don’t.

Same shape every time: honor both, write the new, relabel in place, and remember that the half doing the migrating for free is not the half that decides when you’re done.