Responding to a suspected access problem, end to end
Work in one order: stop the access, establish what was actually reachable, then rotate what could have been observed. The evidence is the decision record, the health verdicts and the record of permission changes — all of which survive the incident. Guessing at scope before reading them is how a small problem gets treated as a large one and a large one as small.
The flow#
- Trigger — something about access looks wrong: an action nobody claims, a sign-in that surprises somebody, a device that was lost, a colleague who has left.
- User or external event — a person reports it. The report is a question about a workspace, not yet a conclusion about one.
- Authentication and workspace resolution — establish which workspace and which audience is actually involved before anything else, because the answer changes every later step.
- Ingest — gather what exists rather than what is remembered: the decision record, permission changes, health verdicts, the timeline of the records concerned.
- Canonical record — write down the incident's own facts: when it starts, when it was noticed, what the reporter saw. The gap between the first two is usually the interesting number.
- Reasoning — separate three things that look identical from a screen: a refusal, an absence of data, and an actual unauthorised action.
- Knowledge, memory and rules — check whether behaviour changed because instruction changed. A standing instruction or memory edited by somebody explains a surprising action without any breach at all.
- Autonomy and approval — check whether the autonomy rule or a permission was changed, and by whom. That change leaves its own record.
- Action through a provider — contain: end the access at the identity provider, remove the person's permission, and stop the channel if a channel is involved.
- Result — confirm containment behaviourally. An access you believe is ended and have not tested is an assumption.
- Relationship, timeline and memory — check what the people on the other side saw. A message that went out is the part a customer experienced, and it does not un-send.
- Audit, usage and Needs You — record what was found, what was completed and what was not. The last of those is the part people omit and later need.
| Stage | What you see | What changes | What can fail |
|---|---|---|---|
| Contain | Access stops working | Sessions and permissions | Containing the wrong thing because scope was assumed rather than read |
| Establish scope | A timeline built from records, not recollection | Nothing | Reading only successes, so refusals — the cheapest evidence — are missed |
| Rotate | Channels reconnect under new values | Credentials | Rotating before containment, so the new value is observed too |
| Record | A written account with both halves | Nothing | Recording what was fixed and not what remains open |
Containing access, given how sign-in works#
There is no password to change, because Connect has no password sign-in: Google is the only way in. That makes containment simpler than it usually is. Access is ended at the identity provider and by removing the person's permission in the workspace, not by resetting anything inside Connect.
One behaviour is worth knowing before you rely on it: signing in as somebody else ends the session it replaced on that browser, and only that one. It is not a way to end that person's sessions on their other devices, and treating it as one leaves access open that you have recorded as closed.
What counts as evidence here#
- The decision record
- What was decided, by what, under which rule, and what happened. Refusals are in it too, which matters: a refusal explains an absence that otherwise looks like an incident.
- Permission and autonomy changes
- A behaviour that changed on a date usually has a change on that date behind it. This is the first place to look, before anything more dramatic.
- Health verdicts
- A channel that stopped working, and when. Useful for bounding a period even when the incident is not about a channel.
- The people's timelines
- What actually reached a customer. This is the only part that cannot be corrected afterwards, so it belongs in the account.
Three things look the same on a screen and are not the same event: a refusal by a boundary, records that are present but not visible, and an action taken by somebody who should not have taken it. A screen returned 403 and Data that should be here is not separate the first two, and separating them early is usually the whole investigation.
Rotating afterwards#
Rotate what could have been observed, not everything. A credential is worth rotating when the incident plausibly exposed it; rotating channels that were never involved adds an outage risk to an incident that already has enough moving parts. Order matters — containment first, rotation second, or the new values are exposed to the same access you have not yet ended. Rotating a provider credential, end to end is the procedure.
Writing it down honestly#
- What was established, with the record it came from.
- What was contained, and the test that proved it.
- What was not established — the gaps. An account with no unknowns in it is usually an account somebody stopped writing early.
- What reached a customer, since that is the part no fix reverses.
- What remains open, and who holds it.
Questions#
Is a 403 an incident?
Usually the opposite: it is a boundary working. It becomes worth investigating when the same screen answers for one audience and refuses for the other, which is a routing or allowlist gap rather than an access problem.
Should I remove the person's permission or wait for a decision?
Remove it. Permission is easy to restore and impossible to un-use, and the decision you are waiting for will be better informed once nothing further is happening while you make it.
Does an incident affect other workspaces?
A session resolves to exactly one workspace, and the three isolation layers apply to every request it makes. That is the reasoning; Proving a workspace is isolated, end to end is how to check it rather than assume it.