VoiceDesk
A voice support agent that checks every sentence for grounding before it is spoken
A real-time phone-support agent that answers from a policy knowledge base, takes actions through tools, and hands off to a human with a written summary when it is not confident. The engine is domain-agnostic; the reference deployment is tier-one support for a fictional ticketing company. What distinguishes it is the measurement: every number it reports comes from a dated run directory, the improvements are paired significance tests on identical seeds, and the figures produced without a live model are labelled as such everywhere rather than left to imply more than they show.
Verified: each of these is checkable in the repository
- p = 1.5e-5
- McNemar exact, containment 0.737 to 0.793results/2026-10-15-feedback-cycle/metrics.json: 300 calls, identical seeds, paired +17 and -0
- 0
- duplicate side effects across 34 fault testseval/fault_injection: 34 tests counted, including commit-then-timeout and concurrent duplicates
- 0.31 → 0.86
- handoff recall after calibrationresults/2026-10-15-handoff-calibration/metrics.json: 95% CI 0.72 to 0.93, n = 94
- 2 of 46
- checker false positives on known-good answersresults/checker-catch-rate: invented numbers, dates and entities caught 23/23, 5/5, 4/4
A support agent that speaks has no undo. Text gives a reader a moment to notice that an answer looks wrong, and a caller gets none: a fabricated refund window or an invented date is simply said, out loud, and acted upon. Two consequences follow. The sentence has to be checked before it reaches the speaker rather than after, and any claim that a change made the agent better has to survive a paired test rather than a demo, because a voice system has far too many ways to look improved on the call you chose to replay.
A turn flows from endpointing through transcription into a conversation loop that imports nothing from the audio layer, which is what makes a turn testable without a microphone. The loop retrieves over a hybrid of BM25 and vector search fused with reciprocal rank, calls the model, and passes each generated sentence to a grounding checker before anything is spoken. A sentence that cannot be traced to a retrieved passage does not get said; hard rules and a calibrated soft score decide between answering and handing off with a written summary. Tool calls go through an idempotency ledger backed by a partial unique index, so a commit followed by a timeout does not produce a second transfer. Improvements are measured by running a simulated caller population, generating a gap report, making exactly one change, and re-running on identical seeds.
Why it is built this way
01Check grounding before the speaker, not after
The checker runs per sentence at a p95 of 0.07 milliseconds, which is what makes it affordable in the speaking path at all. It catches invented numbers, dates and entities at 23 of 23, 5 of 5 and 4 of 4, and fabricated policy claims at 7 of 8, against 2 false positives in 46 known-good answers. The 7 of 8 is the honest number: it is the category a template cannot fully cover, and the repo does not round it up.
02One change per cycle, on identical seeds
The feedback loop runs a baseline, produces a gap report, applies a single corpus change, and re-runs the same 300 simulated calls with the same seed. The run directory records the corpus hash before and after and names the higher-ranked action that was deliberately skipped, so the result is attributable to one change rather than to a batch of them.
03Report the paired test, not the difference
Containment moving from 0.737 to 0.793 is two overlapping confidence intervals and proves little on its own. The pairing is what carries it: the same calls, +17 improved and -0 regressed, McNemar exact p = 1.5e-5. The same cycle rerun on the live model moved the target intent +16 and -0, while other intents moved +14 and -22 at p = 0.24, which the repo calls sampling noise instead of a regression.
04Make duplicate side effects structurally impossible
A partial unique index in Postgres is the ledger, so a retried transfer collides at the database rather than relying on the agent remembering. The fault suite drove 34 cases at it, including commit-then-timeout and five concurrent identical transfers, and the concurrent case resolved to one committed row. The race that test exposed was real and is written up rather than quietly fixed.
05Label the agent behind every number
No API key was available while the measurement harness was built, so most behavioural figures come from an offline extractive agent, and every result says which agent produced it. A README that implied a live model would have been easier to read and worth nothing, and the list of what is still unmeasured, including voice latency over a real sample, sits beside the results rather than at the end.
I set out to build the agent and ended up building the instrument. The measurement harness is most of this repository, and the part that changed how I work is the convention of keeping superseded run directories instead of deleting them: two of the fourteen are labelled superseded and remain, so the history of what I believed and when is recoverable. It is also where the uncomfortable results live. The retrieval sufficiency signal showed dense cosine at 0.93 AUC against the fused score's 0.51, which means the score I had been ranking with was the worse predictor of whether retrieval had actually found the answer.
Next project
AdSight
Ad campaign dashboard over 10,000 rows, measured against a naive build