Caches Outlive the Facts They Hold

The caches you don’t call caches

Nobody forgets that Redis is a cache. The caches that bite are the ones that don’t look like caches at all: a compiled PHP file, a modulo over a server list, a Docker build layer, an environment variable read at startup. Each one captures a fact at some moment and keeps serving it long after the fact has changed — and because nobody thinks of it as a cache, nobody asks when it expires.

Three of these surfaced within a few weeks on the same platform. Same root cause every time.

1. opcache froze a file that changes every five seconds

A routing service resolved which backend server owned a given live session. It got that mapping by require-ing a flat PHP file that an admin process rewrote every ~5 seconds.

Production ran with opcache.validate_timestamps=0 — the standard production setting, which tells PHP never to check whether a file has changed since it was compiled. So each worker compiled the routing file once, at boot, and served that snapshot for its entire life. Any session that had moved to a different server since the worker started was routed to the old one. Requests went to a server that said “not found,” while the server actually hosting the session got nothing.

The resolver logic was correct. The data file was correct. The cache key — the file path — didn’t include the thing that changed: the file’s contents.

Fix: when opcache won’t revalidate on its own, explicitly opcache_invalidate() that one file before require-ing it. A file that’s rewritten as data shouldn’t be loaded as code.

2. A hash over the live server count moved a queue

A legacy producer pushed jobs onto a queue, choosing the Redis server with crc32(queue_name) % count_of_live_servers. With all 12 servers up, that resolved to server 3, and the consumer was configured to drain server 3. It worked for about a month.

Then one server dropped out of the live pool. % 11 resolved to server 8. The producer followed the hash; the consumer’s static configuration did not. The queue sat at its 5,000-job trim cap on server 8 while a perfectly healthy consumer drained an empty queue on server 3, reporting queue_depth: 0.

A modulo over a dynamic list is a cache — it caches the assumption that the list won’t change. Every consumer that hard-codes the result has a copy of that assumption with no way to invalidate it.

Fix: pin both sides to one explicit server so the hash can’t relocate the key. (The right long-term fix is consistent hashing or a real service address, but pinning made the coupling explicit immediately.)

3. Docker cached a fetch on its instruction text

A Dockerfile pulled a model’s training examples from an internal mirror in a RUN step and baked them into the image. The team updated the mirror, rebuilt, redeployed — and production kept the old examples.

The build cache keys a RUN step on the text of the instruction, not on what the command would fetch. An unchanged curl line is a cache hit, even on a “fresh” build, even when the remote content has changed. The rebuild faithfully reused the stale layer.

Fix: a cache-busting build argument (ARG EXAMPLES_CACHEBUST) set to something that changes when the content does — a content hash or a mirror version — so the instruction text changes when the fact does.

And the quieter ones

• Credentials read once at container start. Moving secrets into a vault and injecting them at boot is good practice — until you rotate a credential and discover every running pod holds the old one until it restarts. “Rotate” now means “rotate, then roll.”
• Serialized sessions across two client libraries. The newer PHP memcached extension compresses values over about 2 KB by default; the legacy memcache extension can’t read compressed values. Small sessions worked across both stacks. Sessions broke exactly when they got big enough to matter.

The question to ask

For every place your system captures a value and reuses it later, ask one question: what does the key include, and what does the fact depend on? If the fact depends on something the key doesn’t include, you have a staleness bug with a timer on it.

Cache	Key	Fact actually depends on
opcache	file path	file contents
crc32 % n	queue name	server count
Docker RUN	instruction text	remote content
env at boot	pod lifetime	secret version

Then for each mismatch, pick one: put the dependency in the key, invalidate explicitly when the fact changes, or make the lifetime short enough that staleness doesn’t matter. The one option that doesn’t work is assuming the fact won’t change — because it always does, and usually a month after you’ve stopped thinking about it.

The principle

A cache is a promise that a fact will hold for the life of the entry. The dangerous caches are the ones nobody recognizes as caches, because nobody ever wrote down what they promised. Find them by looking for anything computed once and reused — and ask what it would take for that computation to be wrong.