Gatekeeper Enclave
The same assistant, sealed into a GCP network with no route to the internet
A companion deployment that puts Gatekeeper into a private GCP project with no NAT, no internet route and no external IP, where the only way out is Google's restricted API range and the only way in is an IAP tunnel to a bastion. Releases arrive as one checksummed offline bundle because nothing inside can download anything. Every guarantee it claims was drilled on the running system and written up with timings, including the two drills that failed on the first attempt.
Verified: each of these is checkable in the repository
- 0
- failed requests through upgrade, rollback and failoverdocs/drills: upgrade and HA drills under k6 load, each passed on a second attempt after a documented failure
- 24 min
- to recover from a deletion, against a 60-minute targetdocs/drills/2026-10-23-restore.md: RTO 24 minutes, RPO about 5 seconds
- ~3.5 min
- from an egress attempt to the alert firingdocs/drills/2026-10-23-egress-alert.md
- 10
- Terraform modules, with a CI guard against drifting openproof/check_terraform_plan.py rejects NAT, 0.0.0.0/0 routes, external IPs and public endpoints
A customer whose security team will not allow outbound internet cannot run most AI applications at all, because the usual deployment assumes a model behind a public API, images pulled at deploy time, and packages installed on the fly. Saying a system is air-gapped is easy. The harder question is whether the claim survives contact with an operator: whether a release can actually get in, whether a failover drops requests, whether a restore meets its target, and whether anyone has ever checked rather than assumed.
The network is built with no NAT, no default route and no external IPs, so its only egress is Google's restricted API range, which is also how it reaches Gemini on Vertex. That constraint is what swapped the model: the application repo calls Claude over the public internet, which here would be an egress violation. A deny-all egress firewall logs every attempt and raises an alert. Releases are built on a separate machine into one checksummed bundle carrying images, chart, SBOMs and signatures, and the installer on the bastion verifies each pushed digest against the signed one before Binary Authorization will admit it. Pods use Workload Identity with no service-account keys, and secrets arrive from Secret Manager as mounted files rather than Kubernetes Secrets. Ten Terraform modules describe the whole thing, and a plan guard rejects any change that would reintroduce a route out.
Why it is built this way
01Make the network unable to drift open
A guard reads every Terraform plan and fails on a Cloud NAT, a route to anywhere, an external IP, a public GKE control plane or a Cloud SQL public address. The point is that zero egress is a property someone can reinstate by accident in one line, so the check belongs in the pipeline rather than in a reviewer's memory. The guard has its own unit tests, which run in CI.
02Ship releases as a bundle, because nothing can download
The build machine records each image digest, generates an SBOM and signs every digest. The bastion verifies checksums, pushes, confirms each pushed digest matches the signed one, and installs the chart by digest. Binary Authorization then refuses anything unsigned, which the supply-chain drill proved by pushing a tampered image under the release's own tag and watching it get refused and logged.
03Drill the claims, and keep the failures
Each guarantee has a script in proof/ and a dated write-up in docs/drills/. The upgrade drill and the node-drain drill both failed the first time and the write-ups say so, along with what changed. A drill that has never failed has mostly tested the person writing it, so the first attempts are the most useful artifacts in the repository.
04Separate the recovery target from the recovery time
The restore drill set an RPO of 15 minutes and an RTO of an hour, then measured about 5 seconds and 24 minutes. Nearly all the 24 went to Cloud SQL cloning, and the drill found that the clone command gives up after 10 minutes while this clone took 24, so the script now starts it and polls instead of waiting on the call.
05State what is not hardened
Keycloak runs as a single replica, VPC Service Controls are absent because they need a Google Cloud Organization this project does not have, and the private CA is a Terraform-managed key where production would use Certificate Authority Service. None of those undo zero egress, and listing them is cheaper than having a security reviewer find them.
The game day was worth more than any individual drill. Three failures at once, fixed using nothing but the runbook, took 11 minutes 54 seconds, and about 9 of those went to a gap in the runbook rather than to the failures themselves. Fixing the document brought the expected time to roughly 3 minutes. I had been treating the runbook as documentation of the system, and it is better understood as part of the system, with its own failure modes and its own need to be exercised.
Next project
VoiceDesk
A voice support agent that checks every sentence for grounding before it is spoken