Findings a setting cannot fix
Some faults in a spoken conversation cannot be reached by any control an operator holds. Three were measured here and rolled back rather than left in: a respect-word cadence rule, murmur handling, and a host-spoken acknowledgement that cut the caller off and stalled the model for 22 seconds. Each is still counted and marked model-limited, so the same prompt characters are not spent twice.
Why a negative result earns a page#
An idea that was tried, measured and abandoned is indistinguishable from an idea nobody has tried, unless somebody writes down which it was. Every behaviour below is obvious enough that it returns every few weeks as a suggestion. Keeping the measurement is cheaper than repeating it.
All three share a shape: the fault is real and audible, the instruction that addresses it is easy to write, and the model does not act on it. What separates them from ordinary tuning problems is that the counters kept measuring after the change and reported the same numbers.
One — a respect-word cadence rule#
Indian-language politeness markers such as *andi* carry real social weight, and their placement is part of sounding like a person rather than a system. A cadence rule describing where the marker belongs was rendered into the instructions. The outcome did not move: 6 of 8 replies landed the same way as before, and one English sentence came back answered in Telugu.
The second half of that result is the important one. The instruction did not merely fail to take; it perturbed language selection, which is a worse fault than the one being fixed. Speech style stayed with the profile and the regional layer, where it is a description of a person rather than a rule about a token.
Two — murmur handling#
A caller mumbles, coughs mid-sentence, or says something a transcriber cannot resolve. The model writes that mumble down as a word in some language and answers it, producing a reply to something nobody said. A prompt sentence telling it to ask again instead changed nothing measurable.
The behaviour is instrumented rather than assumed away: conversation.is_murmur classifies, murmur_reactions counts the replies that answered one, and murmurs_accepted counts the ones taken as real input. That means the day a model version behaves differently, the change is visible in existing numbers instead of requiring a fresh experiment.
Three — a host-spoken acknowledgement#
The most tempting of the three. A caller finishing a long sentence hears nothing for a beat, so the host speaks a short *Mm-hm.* to signal it is listening — rt_ack_after_ms. On a Gemini 3.1 Flash Live call it cut the caller off and stalled the model for 22 s.
The mechanism explains the size of the failure. Anything the host says into the line arrives at the model as speech it did not produce, on a session it cannot be told about — 3.1's session is marked immutable by the SDK. On that model no check-in and no goodbye is spoken at all for the same reason, and the fallback text-to-speech carries the live voice's name rather than its sound. True abandonment is ended by the hang-up watcher without a word.
How the score treats a fault with no control#
voice_quality.review produces deterministic findings and attaches to each one the setting that addresses it. A finding with no such setting would be noise, so those are marked MODEL_LIMITED instead: listed, counted, explained — and excluded from score, while score_all keeps the old number so the two are comparable.
| Situation | Classification | What the review offers |
|---|---|---|
| Reply over 4 s, model first token under 60% of the wait | Ordinary finding | The specific setting to change |
| Reply over 4 s, model first token at or over 60% | MODEL_LIMITED | An explanation and no control |
| Behaviour unchanged by instruction, measured twice | Rolled back | Nothing — the knob is removed |
A score that falls because of something nobody can change teaches operators to ignore the score. Counting the fault without charging it keeps both halves honest.
What would change these answers#
- A model version. All three results are specific to the models measured, and the instrumentation is what makes a retest cheap.
- A mutable session. Two of the three failures trace to 3.1 refusing mid-call changes; 2.5 native audio accepts notes and supervisor guidance, at 1.2–2.2 s to first audio against 0.6 s.
- A different signal for listening. The acknowledgement failed as *audio into the line*; whether some non-audio cue would work is UNKNOWN here.
- None of the three should be retried by writing a longer instruction. That was the experiment.
Questions#
How do you know the instruction was not simply badly worded?
Wording was not the variable that decided it. The measured rate was the same with and without the rule, and in the cadence case the instruction produced a side effect — a reply in the wrong language — that a better-worded version of the same idea would have been just as likely to produce. At some point the honest conclusion is that the model is not acting on that class of direction.
Why keep counters for behaviour nobody can fix?
Because the counters are the retest. Model versions change under a running product, and a fault that becomes fixable should be noticed by the existing numbers moving rather than by somebody deciding to look again.
Does a model-limited finding still appear to the operator?
Yes, with the reason. Hiding it would leave a real fault invisible; charging it to the score would make the score unusable. It appears as a finding that names the model rather than a setting, and score_all shows what the score would have been if it were charged.