A copy of a password somebody else rotates

The shape of the incident#

Three things have to line up, and all three are ordinary. A managed database owns its own master credential and rotates it on a schedule nobody watches. A copy of that credential sits in a deployment environment file because it was needed once. And a long-running process reads its configuration exactly once, at start.

The consequence is a fault with a delay built into it. The rotation lands while the application is up and connected, so nothing breaks: existing connections keep working and no new authentication happens. The stale copy becomes visible only when the process next starts — which might be hours or days later, and is almost always just after somebody deployed something.

Cause

A credential was copied, and the authority for that credential changed it

Symptom

Authentication refused for a value nobody edited

Delay

Between the rotation and the next restart — unbounded

Prime suspect

Whichever change happened to be deployed inside that window

Why it reads as a bad deploy, and how to tell#

Everything about the timing points at the release. The application was healthy before, a change went out, and now it cannot reach its database. The natural next move — roll back — does not help, because the previous revision reads the same stale copy and fails identically. That second failure is the useful signal, and it usually arrives after the rollback has already spent the window it was meant to save.

1. Read the error rather than the timeline. An authentication refusal names credentials; a bad deploy usually names an import, a migration or a port.
2. Ask whether the rolled-back revision fails the same way. If it does, the cause is older than both revisions.
3. Ask what else changed that nobody deployed — a rotation, an expiry, a revoked key, a certificate. Each has its own schedule, and none of them appear in a change log.
4. Only then compare the two revisions.

Reading the value instead of holding a copy#

The repair is not a better copy — it is not holding one. .env names the secret rather than carrying its contents: MAYA_MAIL_DB_SECRET_ARN is an identifier, and boot resolves it against Secrets Manager to get whatever is current at that moment. A rotation then costs nothing, because the next boot asks again.

The same principle runs through the rest of the product's handling of credentials. Provider credentials are sealed by settings_store on every save and never echoed back to a screen, so there is no route by which a second copy gets made at all. And there is no password sign-in anywhere in Connect — Google OAuth is the only way in — which leaves the database's own credential very nearly the only one of its kind in the system.

What this does not solve#

• A rotation that lands *between* boots is still invisible until the next one. Reading at boot removes the stale copy; it does not make a running process notice a change.
• The secret store becomes a boot dependency. If it is unreachable, or the instance's permission is wrong, the application does not start — the correct failure, but a new one to recognise.
• The identifier in .env is configuration, not a secret, and it can still be wrong. Pointing at the wrong secret fails in a way that looks exactly like a bad credential.
• None of this is a certification of anything. It is one decision about where a value is read from.

Questions#

Related#

• Security and isolation
• Three independent layers of tenant isolation
• A probe that fires on a correct release is worse than no probe
• Migrations that run on every boot
• Account and access