Skip to content
GitHub
Integration Issues

Outbox row claims are exclusive — a second consumer on the same event type needs co-handler fan-out, not a second listener

Outbox row claims are exclusive — a second consumer on one event type needs co-handler fan-out

Problem

SPEC-645 added a second consumer of decision.recorded (the worlds rule-health handler) alongside the existing platform override-pattern aggregator. The outbox listener’s dispatch claims a row (pending → in-flight → delivered) for exactly ONE handler: whichever listener claims the row consumes it, and the other consumer never sees it. Booting two listeners on the same event type makes each starve the other nondeterministically.

Root Cause

The claim is row-exclusive by design (at-least-once delivery to a handler). Delivery state lives on the row itself, so “delivered” means delivered to whoever claimed it — the schema has no per-handler delivery tracking on the outbox row.

Solution

Fan out inside one listener instead of adding a second listener (src/spectral/core/events/infrastructure/outbox_listener.py):

  • The listener for an event type takes a primary handler plus registered co_handlers (RegisteredHandler).
  • Each claimed row is dispatched to every handler in the one row transaction.
  • Idempotency is per handler: each handler records its own (handler_name, idempotency_key) row in core.event_handled, so a re-run skips handlers that already committed.
  • Partial failure rolls back the whole row (re-pended); committed dedup rows make the retry a no-op for the handlers that already succeeded.
  • Composition: build_decision_recorded_consumer wraps the previous single-handler builder (spectral_workers/event_consumers.py; boot list in runtime.py _CONSUMER_BUILDERS).

Coupled-fate caveat (by design, documented in the listener docstring): a persistently-failing co-handler re-pends the shared row and blocks every handler riding it until the failure clears. Watch worker logs when adding a co-handler.

Prevention

  • Adding a consumer for an event type that already has one? Register a co-handler on the existing listener — never boot a second listener for the same type.
  • Give every handler its own HANDLER_NAME and dedup row; never share one.
  • Integration-test the fan-out: drain a real row, assert BOTH handler names in core.event_handled, then redeliver and assert no drift (tests/integration/worlds/test_rule_health_consumer.py is the template).

References

  • SPEC-645 merge 9b8b55ed (Task 0 substrate check → fix)
  • ADR-044 (event substrate), ADR-065 D4 (consumer ACL discipline)