Operations & observability
Check health, set capacity limits, read logs and OpenTelemetry signals, and run Brain in a container.
This guide covers what you need to run Brain as a service: health checks, capacity limits, logs, OpenTelemetry traces and metrics, the container image, and request routing when you run more than one instance.
Check health
Health endpoints need no token and never call a model.
| Endpoint | Response | Use it for |
|---|---|---|
GET /health/live | 200 with {"status":"ok"} while the HTTP server is running. | Liveness probes. |
GET /health/ready | 200 with {"status":"ready"} when Brain can accept new work, otherwise 503 with {"status":"not_ready"}. | Readiness probes and load balancer health checks. |
/health/ready returns 503 while Brain is shutting down. With external persistence configured, it also returns 503 until Brain has checked which local sessions still need uploading after startup, and whenever the upload backlog is full.
Set capacity limits
Every queue and buffer in Brain has a limit. Set them in the limits object of the host config:
| Limit | Default | What it bounds | When the limit is reached |
|---|---|---|---|
active | 16 | Turns that run at the same time, and harness processes kept running between turns. | New turns wait for a slot. Brain stops an idle harness process to start a new one. |
loaded | 1024 | Sessions held in memory. Must be at least active. | Brain unloads an idle session. If none is idle, the request gets 429. |
queue | 16 | Messages waiting in one session, from 1 to 1024. | Sending returns 429. |
input_bytes | 64 KiB | Message text and session instructions, up to 1 MiB. Request bodies may be 16 KiB larger. | Oversized text returns 400; an oversized body returns 413. |
session_bytes | 1 GiB | One session's event log, at least 16 MiB. | Sending returns 429 near the limit. |
subscribers | 128 | Open SSE streams on the server. | A new stream returns 429. |
subscriber_events | 128 | Events buffered for one SSE stream. | Brain disconnects the slow client. |
subscriber_bytes | 2 MiB | Bytes buffered for one SSE stream, at least 4096. | Brain disconnects the slow client. |
When Brain is out of capacity, it returns 429 with code overloaded and Retry-After: 1 instead of slowing down turns that are already running. Retry with backoff. Brain also returns 429 while it shuts down and, with external persistence, while the upload backlog is full.
Read logs
Brain writes structured JSON logs, one object per line, to stderr. Logs don't include request bodies or credentials.
Every HTTP response has an X-Request-ID header, and every error response body includes the same value as request_id:
{ "error": { "code": "not_found", "message": "session not found" }, "request_id": "5f0c1e7a9b2d4c6e8f1a3b5c7d9e0f2a" }Brain logs each 500 response with its request_id, and each stopped turn with its session ID and error. Quote the request ID when you report a problem.
Export traces and metrics
Set OTEL_EXPORTER_OTLP_ENDPOINT to export OpenTelemetry traces and metrics over OTLP/HTTP. Brain reports as service brain and exports metrics every 15 seconds. Without the variable, Brain exports nothing.
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"| Metric | Measures |
|---|---|
brain.http.requests, brain.http.duration | Requests and their latency, by route and method. SSE streams are counted but not timed. |
brain.http.rejected | Error responses, by error code. |
brain.executions.active | Number of running turns. |
brain.input.queue_delay | Time a message waited before its turn started. |
brain.assistant.first_output | Time from accepting a message to the first assistant output for it. |
brain.runner.startup | Time to start a harness process, by harness. |
brain.tool.calls, brain.tool.duration | Tool calls and their latency, by tool and outcome (success or tool_error). |
brain.transcript.bytes, brain.native.appended_bytes | Growth of event logs and native harness state. |
brain.sync.pending, brain.sync.duration, brain.sync.failures | External storage upload backlog, upload time, and failures. |
Metrics carry no tenant or session labels, so their cardinality stays bounded as sessions grow.
Brain records a session.execute span for each turn, with the session ID and harness name, and a tool.call span for each tool call, with the tool name. Brain propagates W3C trace context: if the request that sends a message has a traceparent header, the turn joins that trace, and Brain forwards traceparent to HTTP tools so their spans join it too.
Run Brain in a container
The repository's Dockerfile builds an image with the Brain server at /app/brain, the Pi runner at /app/runners/pi, and the resource tool service at /app/resource-tools. The image runs as the unprivileged node user (uid 1000), exposes port 8080, and starts Brain with -config /config/brain.json.
To run the image:
- Mount your host config at
/config/brain.json. - Listen on all interfaces: Set
"listen": "0.0.0.0:8080"in the config. - Store state on a volume: Set
"data_dir": "/data"and mount a volume at/data. - Pass secrets as environment variables: For example
BRAIN_API_KEYand your provider keys.
docker build --tag brain:local-dev .
docker run --rm -p 127.0.0.1:8080:8080 \
--mount type=bind,source="$PWD/brain.json",target=/config/brain.json,readonly \
--mount type=volume,source=brain-data,target=/data \
-e BRAIN_API_KEY -e OPENAI_API_KEY \
brain:local-devTo check a locally built image, run node scripts/container-smoke.mjs. It starts brain:local-dev with a recording model provider, so it needs no model credentials.
CI publishes tested images for every commit to main as ghcr.io/aexhq/brain and ghcr.io/aexhq/brain-sandbox-tools, tagged sha-<commit> and main, with provenance attestations. Pin production images by digest, and use the SDK version that matches your server commit.
Route sessions across instances
Each session must have exactly one owning Brain instance at a time. Your routing layer sends every request for a session to its owner and handles failover.
On SIGINT or SIGTERM, Brain stops accepting new work, interrupts running turns, ends open event streams, and exits within about 20 seconds. Sessions whose turns were interrupted need a resume before they accept new messages. To move sessions to another instance, upload their checkpoints first; see Persistence & recovery for the handoff steps.
Size your deployment
Load-test with your own workload before you size a fleet.