Repository navigation
fix(kafka): make the Kafka event broker deliver events; add competing-consumer subscriptions; harden it - #1
Open
xaviercallens wants to merge 1 commit into
Conversation
…competing-consumer subscriptions; fix 7 robustness defects The Kafka broker never delivered an event (4 of 5 new integration tests failed on the old code): publish and subscribe used different topic names, a second subscribe replaced the first, wildcards were passed as literal names, the consumer was used from two threads, and every received record lost its topic. Rewritten with one poll thread per consumer, regex subscriptions, CloudEvents structured JSON, fan-out (per-instance group) and subscribeQueue (shared group) semantics, manual offset commit after handlers complete (at-least-once; handlers must be idempotent), failure surfacing instead of silent thread death, retried subscription updates, a warning for queue records with no local match, matching rules between Event.matchesTopic and the Kafka patterns, and InMemoryEventBroker restart after stop. 9 amcp-core and 23 amcp-broker-kafka tests (9 against a real broker, skipped unless AMCP_KAFKA_BOOTSTRAP is set). See amcp-broker-kafka/README.md and CHANGELOG.md. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JWcvy9P3NMPXiV6BycKYyY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make the Kafka event broker deliver events (and harden it)
The problem
KafkaEventBrokernever delivered an event. I wrote integration tests against a real Kafka 3.8 broker (KRaft, single node) and 4 of 5 failed on the current code, with zero events ever delivered. Causes found by reading the code and confirmed by the tests:publish()sent to the AMCP topic verbatim (a.b.c) whilesubscribe()subscribed toa-b-c: they never met.consumer.subscribe(singletonList(x))replaced earlier subscriptions: only the lastsubscribe()worked.KafkaConsumerwas touched from the caller thread and the poll thread (not thread-safe); with no subscription the loop spun logging an error thousands of times a second.Event.create("amcp.event", json): the topic was lost, so no subscription matched.CloudEvent.toString()was used as the wire format.Event.matchesTopic("p.**")usedstartsWith, so it also matchedphive....What changed
p.*,p.**) become one union regex subscription, re-applied when the set changes (TopicPatterns, unit-tested).subscribeis fan-out (every instance receives every matching event; per-instance group; events older than the subscription are skipped), and the newsubscribeQueue(pattern, queueName, handler)is competing consumers (a shared group). It is a default method onEventBroker(falls back tosubscribe) and is round-robin inInMemoryEventBroker.publish()completes on broker acknowledgement and fails fast if the broker is not running.start()/subscribeQueueinstead of dying silently; a failed subscription update is logged once and retried with backoff; offsets are committed manually after the handlers of a batch complete (at-least-once; a poison record is given up after 3 attempts); a queue record with no local match logs a warning once per topic and queue;Event.matchesTopicand the Kafka patterns agree on trailing-dot topics (22-row table test through both); the README timing statements match the code;InMemoryEventBrokercan be restarted afterstop().Behaviour changes to be aware of
subscribeQueuecan throwIllegalStateExceptionif the consumer cannot be built.p.andp.x.match only an identical pattern)..versus_collide (documented).Tests
amcp-core: 9 tests.amcp-broker-kafka: 23 tests (7 pattern, 7 unit tests with a replaying in-memory consumer, 9 integration tests against a real broker). 0 failures, repeated three times with no flakiness.AMCP_KAFKA_BOOTSTRAPis set (for example127.0.0.1:9092); otherwise that class is skipped (surefire reports it as 0 tests run). Each test uses its own topic prefix.CHANGELOG.md).Not done / limits
subscribe()returns, so an event consumed in that window can be missed (documented).queueNamesubscribes the same patterns (Kafka cannot hand a record back to a chosen group member); a warning is logged when this is violated.amcp-corewere touched. NATS, A2A, MCP and the Quarkus extension were not reviewed.🤖 Generated with Claude Code
https://claude.ai/code/session_01JWcvy9P3NMPXiV6BycKYyY