Persistence adapters
Object storage and metadata indexing interfaces for background checkpoints.
Persistence adapters implement two small Go interfaces in internal/persistence: Objects for immutable checkpoint blocks and manifests, and Metadata for optional cross-instance session discovery (see Go interfaces).
Authoritative local database
store.Metadata is separate from the eventual persistence.Metadata index. Its
Create, Get, Save, List, Scan, and Import operations must preserve durable
writes and tenant ownership. List and host-only Scan page by ascending ID.
Save cannot create a missing record; Import cannot transfer a session to another
tenant. All operations honor context cancellation.
store.SQLite is the built-in implementation. store.New(root, metadata) combines
an adapter with local session files and takes ownership of its lifecycle. It
prepares native directories before creating metadata; a database failure propagates
to the caller. Run internal/store/metadata_test.go against each implementation,
plus the restart/recovery journeys. A remote SQL database alone does not provide
session placement, distributed ownership, or remote native files.
Built-in implementations
| Interface | Purpose | Built-in implementations |
|---|---|---|
persistence.Objects | Stores 64 KiB immutable file blocks, bounded tail objects, checkpoint manifests, and per-session latest pointers. | FileObjects (filesystem), S3Objects (s3 and r2 in internal/persistence/objects.go), GCSObjects (gcs in internal/persistence/gcs.go). |
persistence.Metadata | Maintains an eventually consistent tenant-scoped index of sessions and their latest checkpoint IDs. | Postgres (postgres_env in internal/persistence/metadata.go). |
Objects interface
Objects provides Put(ctx, key, bytes, immutable) and Get(ctx, key):
- Immutable writes: When
immutableistrue,Putmust never overwrite an existing object with different content (FileObjectsuses hard links;S3ObjectssendsIf-None-Match: "*";GCSObjectssetsDoesNotExist: true). If the key already exists from an earlier upload attempt, returningnilis safe because block and manifest keys are generation- and checkpoint-scoped. - Error distinction: Missing objects on
Getmust wrapos.ErrNotExistsopersistence.Archivecan distinguish missing checkpoints (404 not_found) from transient network or storage failures (500 brain_error). - Scoping and bounds: Implementations must validate keys with
validKey(rejecting empty keys, leading slashes, backslashes, or path traversal), prepend the configuredprefix, and cap individual object sizes at2 MiB(MaxObjectBytes = 2 << 20).
How Archive publishes and restores checkpoints
persistence.Archive (internal/persistence/archive.go) coordinates incremental uploads of events.jsonl and append-only native/ files (up to 128 files per checkpoint):
Complete 64 KiB (BlockBytes = 64 << 10) blocks are uploaded once under sessions/<id>/blocks/<generation>/<index> (immutable: true). Subsequent checkpoints reuse the same generation ID and upload only newly appended full blocks plus the bounded remainder tail under sessions/<id>/tails/<checkpoint_id>/<generation> (immutable: true).
Only after every referenced block and tail object is stored does Archive.Publish write sessions/<id>/checkpoints/<checkpoint_id> (immutable: true). A failed upload can leave unreachable blocks, never a partially readable checkpoint.
Archive.Publish updates the mutable pointer sessions/<id>/latest (immutable: false), updates Metadata.Put if postgres_env is configured, and saves uploaded.json in the local session directory.
Metadata interface
Metadata provides Put(ctx, session, checkpointID), Get(ctx, tenant, id), List(ctx, tenant, after, limit), and Close():
GetandListare strictly scoped bytenant, andListreturns sessions ordered byidascending (id > after,limit1–1000).Putupserts(id, tenant, data, checkpoint)and rejects cross-tenant collisions with403 forbidden(metadata tenant mismatch).- External metadata is an eventually consistent discovery index updated after checkpoint publication; it is never used for locking or leader election.
Testing an adapter
Every storage adapter should pass the contract tests in internal/persistence/:
- Immutable block writes (
immutable: true) and idempotent retries when a block or manifest already exists - Interrupted manifest publication (unreachable blocks must never expose a half-written checkpoint)
- Missing object detection (
errors.Is(err, os.ErrNotExist)) and tenant/path isolation - Restoring into an empty local cache and verifying the next Pi model turn preserves conversation history