GlassBox
One AI endpoint, instrumented until nothing is hidden
A deliberately small FastAPI service with a single question-answering endpoint backed by Gemini on Vertex AI, instrumented so every request produces a trace, eight metrics and a structured log correlated by trace ID. The centrepiece is a comparison: the same image reports as local or production via a resource attribute, so identical code can be compared side by side in one SigNoz instance.
Verified: each of these is checkable in the repository
- 8
- custom metric instrumentsmetrics.py, rate, latency, tokens, cost, errors, SLO
- 11
- dashboard panels, 3 comparing local to productiondashboards/glassbox-dashboard.json
- 3
- alerts across metrics and logsp95 latency, error rate, per-session error clustering
- 80
- characters of a question ever loggedquestion_preview truncation
AI services have three blind spots that generic instrumentation misses: you cannot see why a request was slow, what it cost, or why it failed. Token spend in particular is invisible in ordinary request metrics, two requests with identical latency can differ tenfold in cost.
One endpoint, three signals. An auto-instrumented server span wraps a manual span around the model call carrying token counts and an estimated cost. Eight metric instruments record rate, a latency histogram, input and output tokens, cumulative spend, errors, cost attributed per session, and how many requests fell inside the latency objective. One structured log per request carries the trace ID and a truncated preview of the question. Three scenarios (normal, slow and failing) make each signal do something visible.
Why it is built this way
01Record metrics in a finally block
Failures are recorded exactly like successes. Instrumentation that only runs on the happy path hides the requests you most need to see.
02Cost as a first-class metric
Token counts convert to dollars and accumulate per session, so spend is attributable to a user rather than being one line on a monthly bill.
03Alert on per-session error clustering
Three or more errors for one session in five minutes is a separate alert from overall error rate, because one user failing repeatedly is a different incident from scattered failures.
04Environment as a resource attribute
Local and production emit the same signals tagged differently, which turns the comparison into a group-by rather than two dashboards to eyeball.
05Truncate the prompt in logs
Only the first eighty characters of a question are ever logged. Enough to debug, not enough to be a data-retention problem.
Making cost a metric rather than a line item changed how I read the dashboard. Latency and error rate are the reflexive things to graph, but on an AI service spend is the one that surprises you, and attributing it per session immediately showed that a small number of sessions accounted for most of it.
Next project
MealMind
Snowflake-native multi-agent nutrition planning