ObserveAI
A distributed system built to be observed while it fails
Nine instrumented FastAPI services around a checkout flow, wired through Redpanda, Postgres and Redis, with OpenTelemetry traces, metrics and logs flowing into self-hosted SigNoz. Twelve scripted failure scenarios inject real latency, errors and poison messages so the telemetry has something to reveal. Trace context propagates across the Kafka boundary, so a consumer span is a child of the checkout trace that caused it.
Verified: each of these is checkable in the repository
- 12
- scripted failure scenariosall confirmed in services/traffic and services/ui
- 14
- alert definitions with thresholdsalerts/. ClickHouse queries against signoz_traces
- 22
- dashboard queriesdashboards/, 6 groups
- 3
- Kafka topics, including a dead-letter queuefraud.check.requested / .completed / .dlq
Observability is easy to read about and hard to feel. Tutorials show you a trace waterfall that is already broken, which teaches you what the UI looks like but not how to get from an alert to a cause. I wanted a system where I could break a specific thing on purpose and then find it from the telemetry alone.
A synthetic e-commerce checkout: a checkout service orchestrates cart, inventory and payment over HTTP, writes orders to Postgres, and publishes a fraud-check request to Redpanda. A fraud service consumes it, scores it with deterministic rules, and publishes a result that two independent consumer groups (notification and analytics) both read. Failed messages retry three times and then go to a dead-letter topic that analytics also consumes. Everything exports OTLP to a collector and on into SigNoz, with twelve scenarios that inject specific pathologies on demand.
Twelve ways to break it on purpose
Each is a named, triggerable code path rather than something you cause by hand. The point of the system is that the telemetry explains what happened.
| Scenario | Injects | Expects |
|---|---|---|
| Normal | 200 | |
| Slow payment | sleep 1.2s | 200 |
| Payment declined | HTTP 502 | 502 |
| Provider timeout | HTTP 502 | 502 |
| Out of stock | HTTP 409 | 409 |
| Slow fraud inference | sleep 1.5s | 200 |
| Consumer lag | sleep 5.0s | 200 |
| Poison message → DLQ | 3 retries then DLQ | 200 |
| Slow notification | sleep 2.0s | 200 |
| Notification failure | RuntimeError | 200 |
| Slow analytics | sleep 1.5s | 200 |
| Slow query | pg_sleep(1) | 200 |
Why it is built this way
01Trace context that survives the broker
A header carrier injects trace context on publish and extracts it on consume, so an async consumer span is a genuine child of the checkout trace. Without this the async half of the system is a separate, uncorrelated island, which is exactly where real incidents hide.
02Rules-based fraud scoring, deliberately
The fraud service uses thresholds on order value and item count, not a model. The subject of this project is the telemetry, and a deterministic scorer means a given input always produces the same span. Adding a model would have made the traces harder to reason about, not more impressive.
03Failure scenarios as first-class code
Each of the twelve scenarios is a named, triggerable path rather than something you cause by hand: slow payment, provider timeout, out-of-stock, slow fraud inference, consumer lag, poison message to DLQ, slow and failing notifications, slow analytics, and a Postgres slow query.
04Alerting on the shape of failure, not just its volume
One alert fires on three or more errors for the same session within five minutes. Five users failing once is a blip; one user failing five times is someone stuck. Aggregate error rate cannot tell those apart.
05Ports offset to coexist with SigNoz
SigNoz runs separately and owns the conventional ports, so every service here is deliberately offset. Small detail, but it is why the two stacks can run on one laptop at all.
Building the failure scenarios taught me more than building the services. Writing the poison-message path forced me to decide what a retry means, where the message goes when retries are exhausted, and who is responsible for noticing, and those are the questions that separate a system that logs from one you can operate.
Next project
SecSignal
Agentic RAG over SEC filings, deployed and evaluated