Outbox row claims are exclusive — a second consumer on the same event type needs co-handler fan-out, not a second listener
Outbox row claims are exclusive — a second consumer on one event type needs co-handler fan-out
Problem
SPEC-645 added a second consumer of decision.recorded (the worlds rule-health
handler) alongside the existing platform override-pattern aggregator. The outbox
listener’s dispatch claims a row (pending → in-flight → delivered) for
exactly ONE handler: whichever listener claims the row consumes it, and the
other consumer never sees it. Booting two listeners on the same event type
makes each starve the other nondeterministically.
Root Cause
The claim is row-exclusive by design (at-least-once delivery to a handler). Delivery state lives on the row itself, so “delivered” means delivered to whoever claimed it — the schema has no per-handler delivery tracking on the outbox row.
Solution
Fan out inside one listener instead of adding a second listener
(src/spectral/core/events/infrastructure/outbox_listener.py):
- The listener for an event type takes a primary handler plus registered
co_handlers(RegisteredHandler). - Each claimed row is dispatched to every handler in the one row transaction.
- Idempotency is per handler: each handler records its own
(handler_name, idempotency_key)row incore.event_handled, so a re-run skips handlers that already committed. - Partial failure rolls back the whole row (re-pended); committed dedup rows make the retry a no-op for the handlers that already succeeded.
- Composition:
build_decision_recorded_consumerwraps the previous single-handler builder (spectral_workers/event_consumers.py; boot list inruntime.py_CONSUMER_BUILDERS).
Coupled-fate caveat (by design, documented in the listener docstring): a persistently-failing co-handler re-pends the shared row and blocks every handler riding it until the failure clears. Watch worker logs when adding a co-handler.
Prevention
- Adding a consumer for an event type that already has one? Register a co-handler on the existing listener — never boot a second listener for the same type.
- Give every handler its own
HANDLER_NAMEand dedup row; never share one. - Integration-test the fan-out: drain a real row, assert BOTH handler names in
core.event_handled, then redeliver and assert no drift (tests/integration/worlds/test_rule_health_consumer.pyis the template).
References
- SPEC-645 merge
9b8b55ed(Task 0 substrate check → fix) - ADR-044 (event substrate), ADR-065 D4 (consumer ACL discipline)