Secrets management runbook

Operational runbook for Spectral’s provisioning, rotation, and audit flows. The runtime backend is Render Environment Groups (per ADR-046 D11/D12); GCP Secret Manager via Workload Identity Federation backs the off-platform backup credential isolation (per ADR-040 D7). The provisioning script + target-swap discipline come from ADR-037 (D1/D2 superseded by ADR-046 D11/D12; remaining Ds — operator interface, env-var contract, target-swap precedence — remain authoritative). CI secrets handling per ADR-062.

Secret classes

1. Platform operational secrets — provider API keys, observability vendor keys, Supabase keys, DB connection strings. Provisioned via tools/provision/setup.sh → runtime storage per environment.
2. Customer BYO credentials — customer-provided provider keys (per ADR-035 D4 reservation). Post-alpha. Not handled by the provisioning script; lands in Supabase Vault under core.workspace_secrets with migration path to envelope encryption using GCP KMS master (ADR-037 D6 / D12).
3. CI secrets — scoped to GitHub Environments (staging, production, test-live) per ADR-062; live-secret runs gated to non-PR triggers. Render API key + Supabase Management PAT + Cloudflare API token live as Environment-scoped GitHub secrets with quarterly rotation. See docs/runbooks/ci-secrets.md.
4. Local dev secrets — developer laptops use .env.local (gitignored) with developer-provided dev-tier credentials. Dev laptops do not use the provisioning script.

Standard rotation

Example: rotating ANTHROPIC_API_KEY in production.

1. Mint a new key at the vendor (https://console.anthropic.com/settings/keys).
2. Dry-run the rotation to verify state:

   tools/provision/setup.sh --env=production --mode=rotate --secret=ANTHROPIC_API_KEY --dry-run

3. Execute the rotation:

   tools/provision/setup.sh --env=production --mode=rotate --secret=ANTHROPIC_API_KEY

   The script: prompts for the new value → writes the appropriate .env.provision cache entry → pushes to all targets → triggers rolling restart.
4. Back up the updated cache to the team’s shared secure-vault entry.
5. Revoke the old key at the vendor console.
6. Verify new key is in use: tools/provision/setup.sh --env=production --mode=verify --dry-run.

Target: < 15 minutes end-to-end for vendor-key rotations.

Rotating shared-scope secrets

Shared secrets have one cached value but push to both env targets.

tools/provision/setup.sh --env=staging    --mode=rotate --secret=<NAME>
# (cache updated + staging targets pushed)
tools/provision/setup.sh --env=production --mode=rotate --secret=<NAME>
# (cache already current — dispatch-only flow; production targets pushed)

Two invocations; the second does not re-prompt.

Emergency rotation

Triggered by: suspected compromise, leaked cache file, suspected insider access, vendor-reported anomalous usage.

1. Identify blast radius — which secrets might be exposed? Default to rotating every secret the suspected-compromised identity could reach.
2. Run rotation for each affected secret using the standard flow above. For bulk rotation in one session, iterate through the .env.example list.
3. Revoke old values at vendors immediately after the new values are provisioned. Do not wait for rolling-restart confirmation — vendor revocation is idempotent with rotation.
4. Log the incident in Linear (create an incident issue) with the list of secrets rotated and the commit SHA of the provisioning run.
5. Post-incident: audit all targets for stale references to the old values; review access logs (see below) for anomalous reads during the suspected-compromise window.

.env.provision backup discipline

The provisioning script caches collected values in a gitignored .env.provision file at the repo root, using scope-prefixed entries (shared:<NAME>, staging:<NAME>, production:<NAME>). This file contains plaintext secrets on the deploy engineer’s laptop — handle accordingly.

Required after every provisioning run:

1. Copy the updated .env.provision to the team’s shared secure-vault entry as an encrypted attachment.
2. Ensure the deploy laptop has full-disk encryption enabled.
3. Ensure the laptop auto-locks on sleep / inactivity.

On deploy-engineer role change:

1. The outgoing engineer deletes their local .env.provision.
2. The incoming engineer pulls a fresh copy from the shared vault.
3. Consider preemptive rotation of the highest-blast-radius secrets (service_role keys, provider API keys) if the transition is unplanned.

Audit trail

Composite audit across five surfaces at alpha:

Surface	What it records	Where it lives
Git history	tools/provision/setup.sh changes, .env.example additions	GitHub commits
Shared-vault version history	.env.provision snapshots per run	Team secure vault
Render audit + GCP Cloud Audit Logs	Env-group changes (Render), and Secret Manager API calls when KMS lights up post-alpha	Render dashboard + GCP Cloud Logging
GitHub audit log	Actions Secrets mutations, workflow-triggered secret reads (partial), Environment access	GitHub org admin
Supabase PGAudit	Supabase dashboard config changes + DB-side access events	Supabase dashboard

Enable GCP Data Access logs for Secret Manager: Cloud Console → IAM & Admin → Audit Logs → search “Secret Manager API” → enable “Data Read” and “Data Write.” Required to capture per-secret access events.

Post-alpha (ADR-037 D8): BigQuery export target for unified cross-surface querying once SOC2 readiness requires it.

Secret ownership

The deploy engineer running tools/provision/setup.sh is the sole operator for platform operational secrets at alpha. Expand to a named rotation roster (two engineers minimum) when team grows to ≥ 4.

BYO customer credentials (post-alpha)

Not handled by this runbook or tools/provision/setup.sh.

Customer-provided provider API keys are tenant data, not operational configuration. When BYOK ships as a customer feature:

1. Encrypted blobs in a platform-scoped core.workspace_secrets table.
2. Access via SECURITY DEFINER function checking auth.uid() against workspace_members.
3. AEAD via Supabase Vault’s libsodium backing, with workspace_id as AAD.
4. Migration path (when customer UX or compliance requires it): move to application-layer envelope encryption with GCP KMS as master key holder (ADR-037 D12).

Render Environment Groups as runtime substrate

Runtime secrets reach services via per-environment Render Environment Groups:

• spectral-staging-runtime — consumed by every Render service in the staging environment.
• spectral-production-runtime — consumed by every Render service in the production environment.

CI deploy authentication uses a Render API key held in GitHub Actions secrets, scoped per-workspace and per-environment, with quarterly rotation (see ADR-046 D12 + ADR-062 D5 for the discipline calibration).

Deploy-as-rotation pattern

Machine-identity keys managed by the deploy pipeline — deploy-bot API keys, the internal JWT signing keypair, Render / Cloudflare / Supabase PATs — rotate with every deploy. Every deploy regenerates env group contents; a suspected leak is remediated by triggering a deploy. No standalone rotation runbook applies to these secrets.

Human-visible runtime secrets (provider API keys, observability vendor keys, DB credentials) follow the standard rotation flow above.

Env group composition

Each env group carries:

• Supabase keys for that environment (anon key, service role key, Management API PAT scoped to the env’s Supabase branch)
• Provider API keys (Anthropic, OpenAI, Google, observability vendors)
• Render API key (scoped to the env’s services)
• Cloudflare API token (scoped to Pages + DNS for the env)
• Deploy-bot API key material (sk_deploy_<32chars> format, ADR-048 D7)
• Internal JWT signer keypair (ADR-048 D7 key-exchange middleware)
• SPECTRAL_GENERATION — current deployment generation (ADR-048 D5)

Provider-swap seam

The push_<target> target-function convention in tools/provision/setup.sh is the swap seam for the hosting provider. The active target is push_render_env_group. .env.example, .env.provision, and the prompting / rotation / verify flow are provider-independent. See tools/provision/README.md for the target-function specification.

Customer-BYOK envelope encryption uses a separate seam with GCP KMS as the reserved master key holder (ADR-037 D6 / D12).

Related

• ADR-037 — Secrets management (provisioning script + target-swap discipline).
• ADR-046 — Alpha hosting (Render Environment Groups + Render API key in GitHub secrets; supersedes ADR-037 D1/D2).
• ADR-048 — Deployment topology (SPECTRAL_GENERATION per-service env-var, key-exchange middleware, deploy-as-rotation pattern).
• ADR-062 — CI secrets handling (GitHub Environment scoping; test-live Environment; quarterly rotation cadence per D5).
• ADR-035 — LLM stack (provider key context; D4 customer-BYO reservation).
• ADR-036 — Observability stack (D10 vendor key binding).
• ADR-033 — Tenancy enforcement (RLS posture backing the BYOK Vault pattern).
• docs/runbooks/hosting.md — per-deployable hosting map + Render-specific operational content.
• tools/provision/README.md — provisioning-script operator documentation.