Skip to content
GitHub
Infrastructure

Deployment Topology

The deployment topology is single-color rolling with deployment-generation stamping — gen-N events are processed by gen-N code as a structural guarantee, not a discipline. Decision lineage in ADR-048; CD orchestration in ADR-053; operational runbooks at docs/runbooks/deployment.md and docs/runbooks/legacy-drain.md.

Eight deployables per environment: six runtime services (web × 3, background worker, two crons) and two static-site projects. Concrete service names live in render.yaml and the deployment runbook; this page covers the primitives.

Single workers, single outbox, single event_handled

Section titled “Single workers, single outbox, single event_handled”

All three Spectral agents (Spectral / Ops / World) run in workers. Workers is the framework-layer composition seam where dependencies that span worlds and platform wire up at startup (per agent tool invocation).

The event substrate is a single core.outbox table plus a single core.event_handled table keyed on (handler_name, idempotency_key). handler_name must be scope-qualified.

SPECTRAL_GENERATION is a per-service env var, set at deploy time via the platform API — atomic with the image, rolling-restart safe. Env-group changes never bump generation.

  • Publishers stamp every outbox row with the current SPECTRAL_GENERATION.
  • Workers LISTEN only on outbox_gen_<N> for their own generation.
  • Worker claim filter: WHERE generation = $MY_GENERATION AND status='pending'.
  • A V2 worker is structurally incapable of receiving V1 NOTIFY.

core.deployments tracks generations. The CD pipeline allocates a fresh generation via INSERT INTO core.deployments RETURNING generation — atomic, single round-trip.

The reaper re-PENDs stuck IN_FLIGHT rows within its own generation (crash recovery). Cross-generation orphan-sweep is dropped — it would violate the structural guarantee.

When stranded gen-N rows exist (e.g. after rollback), drain-legacy-generation.yml reads core.deployments for the code reference at the target generation, deploys a temporary worker with SPECTRAL_GENERATION=<N> and SPECTRAL_DRAIN_AND_EXIT=true, and deletes the service when drain completes. See docs/runbooks/legacy-drain.md.

Persistent preview branch for staging (one project, two branches). Orchestration through the database management API from CD, enabling tag-based trunk dev.

Migration discipline:

  • AST-level compat lint rejects DROP COLUMN / DROP TABLE / ALTER COLUMN TYPE to incompatible target / ADD COLUMN NOT NULL without DEFAULT / ADD UNIQUE on populated column — without an explicit -- compat: breaking (reason: ...) marker.
  • Schema-version gate in cutover (green must report the expected migration head).
  • Pre-merge dry-run on a throwaway branch with --with-data.
  • Maintenance-window pattern documented for truly breaking migrations.

True DB-layer blue/green is not achievable on managed Postgres. Hardened expand/contract is the realistic path.

.github/deploy-manifest.yml declares path → service mapping. The CD pipeline diffs the current commit against the previous-deployed commit, maps changed paths to an affected target set, and deploys only those.

  • api + workers are a coupled-deploy pair (generation alignment).
  • Force-full-redeploy paths: render.yaml variants, .github/deploy-manifest.yml, infra/**.
  • Coverage check: tools/quality/check_deploy_manifest_coverage.py asserts every directory under apps/ and src/spectral/ is mapped or declared non_deployed:.

Every web service exposes three endpoints:

  • /health — public, binary: 200 ok or 503 degraded. No JSON, no check names. Probes: database + auth.
  • /version — public, minimal: { commit_sha, schema_version, generation, deployed_at }.
  • /version/detail — auth-gated via dual-path key-exchange middleware. Adds runtime/framework/os, deps_lock_hash, build_time, start_time, per-check status with latency.

/version/detail auth: the key-exchange middleware extracts a key from Authorization: Bearer or X-API-Key, validates against an env-var-sourced registry, and mints an internal JWT with the scope/issuer taxonomy. Auth middleware downstream is a no-op multi-issuer validator. Secret rotation = deploy side-effect.

core.workers carries the heartbeat / diagnostic table for the worker equivalent (no HTTP surface).

  • ADR-048 — decision lineage
  • ADR-053 — CD pipeline
  • Event substrate — outbox + generation routing
  • ADR-049 — container strategy + image inventory
  • Hosting
  • Agent tool invocation — workers as composition seam
  • docs/runbooks/deployment.md, docs/runbooks/deployment-topology.md, docs/runbooks/legacy-drain.md, docs/runbooks/rollback.md