Skip to content
Spectral Codex
GitHub
Decisions

ADR-066: Snapshot-based contract testing with syrupy

Status: Accepted (2026-04-29)

Context

ADR-065 establishes the contract surface between contexts: producer-owned typed event payloads in <context>.contracts.events.* (single source of truth for the wire shape), consumer-implements-from-Codex-docs (no typed-model imports between contexts), and bilateral contract tests in tests/contracts/ that verify producer/consumer round-trip + detect schema drift via snapshot tests on <EventName>Payload.model_json_schema().

The HTTP API surface (FastAPI in apps/api) needs comparable drift detection: any unintentional change to the API contract should surface at PR time as a diff in a snapshot. The OpenAPI document and representative endpoint responses are the contract surfaces.

A specific snapshot-testing tool needs to be selected before any new inter-context epic refines events under ADR-065’s doctrine and before alpha API work proceeds. Phase 4 ref-impl pressure-tested syrupy end-to-end against the bilateral contract test pattern; the tool worked as designed, with one developer-onboarding gotcha (first-run baseline creation) noted for the testing-guide page.

Decision

Adopt syrupy as the snapshot-testing tool across both surfaces — events between contexts and HTTP API.

D1. Event schema snapshots between contexts

Each event between contexts under <context>.contracts.events.* is covered by a tests/contracts/<event>_test.py file containing:

  • A round-trip test verifying the consumer’s local <EventName>Event model parses the producer’s model_dump(mode="json") output (catches structural mismatch at PR time).
  • A schema-drift snapshot test capturing <EventName>Payload.model_json_schema() as a syrupy snapshot. Producer schema changes surface as snapshot diffs in PR; intentional changes update the snapshot in the same commit; unintentional drift fails CI.

Per ADR-065 D6, tests/contracts/ is exempt from the inter-context import rule (validator rule 6) so contract tests may import both producer and consumer models.

D2. OpenAPI document snapshot

apps/api/tests/ includes a snapshot test that captures FastAPI’s auto-generated openapi.json as a syrupy snapshot. Any API surface change (new endpoint, parameter shape change, response shape change, status-code change) surfaces as a diff. Intentional changes update the snapshot in the same commit; unintentional drift fails CI.

D3. Representative endpoint response snapshots

For each significant endpoint, apps/api/tests/ includes a TestClient-driven test that hits the endpoint with canonical inputs and snapshots the response. This catches subtle response-shape changes the OpenAPI snapshot might miss (e.g. nested structure changes within a stable top-level type).

D4. Out of scope

  • Property-based API behavior testing (Schemathesis territory). Generating test cases from the OpenAPI spec to verify endpoint behavior across input ranges is covered by Spectral’s existing per-layer coverage floors and integration-test AC; if property-based testing becomes valuable later, Schemathesis layers in alongside syrupy.
  • Frontend / backend type alignment. Generating TypeScript types from the OpenAPI spec for the dashboard is a codegen concern handled in dashboard tooling, not contract-test scope.

D5. Developer onboarding

First run of a new contract test creates the syrupy snapshot baseline (pytest --snapshot-update); the author commits both the test file and the __snapshots__/ directory. Subsequent runs verify against the committed baseline. This onboarding step is documented on the Codex developer-guide/testing.mdx page.

Alternatives considered

  • Pact / consumer-driven contracts. Designed for distributed systems with separate consumer and producer teams. Overkill for an in-process single-team monolith; the operational ceremony exceeds the value of the discipline at this stage.
  • pytest-snapshot. Older, less polished, weaker diff output than syrupy. The diff quality matters because reviewer judgment of intent depends on a readable diff.
  • Hand-rolled JSON file comparisons. Reinvents diffing without producing better output than the existing tool.
  • inline-snapshot. Auto-updating snapshots embedded in source code; interesting ergonomically but carries accidental-commit risk (an unintended mutation lands in source rather than as a separate diff in __snapshots__/) and is less proven at production scale.

Consequences

  • One added dev-dependency: syrupy in pyproject.toml.
  • CI gates on snapshot drift in tests/contracts/ and the API-surface tests under apps/api/tests/.
  • Intentional contract evolution requires updating snapshots in the same commit; reviewer sees the snapshot diff and validates intent.
  • The Codex doc generator that walks <context>.contracts.events.* and <context>.contracts.protocols.* (per ADR-065 Consequences) and the bilateral contract test pattern (per ADR-065 D6) both rely on this tool as the snapshot mechanism.
  • Pre-push hook gains a fast-feedback subset of the snapshot tests per ADR-012’s tiered-hooks discipline; the full snapshot suite runs in CI.
  • When the first async contract test lands, pytest-asyncio configuration ([tool.pytest.ini_options] asyncio_mode = "auto") is added to pyproject.toml. Phase 4 ref-impl flagged this as a known first-async-test concern, not a blocker.