Skip to content

fix(kafka): make the Kafka event broker deliver events; add competing-consumer subscriptions; harden it - #1

Open
xaviercallens wants to merge 1 commit into
agentmeshcommunicationprotocol:mainfrom
xaviercallens:fix/kafka-broker-delivery
Open

xaviercallens wants to merge 1 commit into
agentmeshcommunicationprotocol:mainfrom
xaviercallens:fix/kafka-broker-delivery

Conversation

@xaviercallens

Copy link
Copy Markdown
Contributor

Make the Kafka event broker deliver events (and harden it)

The problem

KafkaEventBroker never delivered an event. I wrote integration tests against a real Kafka 3.8 broker (KRaft, single node) and 4 of 5 failed on the current code, with zero events ever delivered. Causes found by reading the code and confirmed by the tests:

  1. publish() sent to the AMCP topic verbatim (a.b.c) while subscribe() subscribed to a-b-c: they never met.
  2. consumer.subscribe(singletonList(x)) replaced earlier subscriptions: only the last subscribe() worked.
  3. Wildcard patterns were passed to Kafka as literal topic names.
  4. One KafkaConsumer was touched from the caller thread and the poll thread (not thread-safe); with no subscription the loop spun logging an error thousands of times a second.
  5. Every received record was rebuilt as Event.create("amcp.event", json): the topic was lost, so no subscription matched. CloudEvent.toString() was used as the wire format.
  6. Event.matchesTopic("p.**") used startsWith, so it also matched phive....

What changed

  • One dedicated poll thread per consumer; subscribe/unsubscribe are queued and applied on that thread; handlers run on virtual threads.
  • Kafka topic = AMCP topic, verbatim. Patterns (p.*, p.**) become one union regex subscription, re-applied when the set changes (TopicPatterns, unit-tested).
  • CloudEvents structured JSON on the wire; the topic, id, time, source and payload survive the round trip.
  • Two delivery modes: subscribe is fan-out (every instance receives every matching event; per-instance group; events older than the subscription are skipped), and the new subscribeQueue(pattern, queueName, handler) is competing consumers (a shared group). It is a default method on EventBroker (falls back to subscribe) and is round-robin in InMemoryEventBroker.
  • publish() completes on broker acknowledgement and fails fast if the broker is not running.
  • Seven robustness fixes found by an independent review of the first version: a consumer that cannot be built now fails start()/subscribeQueue instead of dying silently; a failed subscription update is logged once and retried with backoff; offsets are committed manually after the handlers of a batch complete (at-least-once; a poison record is given up after 3 attempts); a queue record with no local match logs a warning once per topic and queue; Event.matchesTopic and the Kafka patterns agree on trailing-dot topics (22-row table test through both); the README timing statements match the code; InMemoryEventBroker can be restarted after stop().

Behaviour changes to be aware of

  • Delivery is now at-least-once, not at-most-once: handlers must be idempotent. After a rewind, fan-out handlers that already succeeded for a record can see it again.
  • A slow handler can delay later records of the same consumer, up to 30 s per batch.
  • subscribeQueue can throw IllegalStateException if the consumer cannot be built.
  • Wildcards never match an empty segment (p. and p.x. match only an identical pattern).
  • Kafka topic names equal AMCP topic strings, so names differing only by . versus _ collide (documented).
  • The public API is otherwise unchanged. New test seams are package-private.

Tests

  • amcp-core: 9 tests. amcp-broker-kafka: 23 tests (7 pattern, 7 unit tests with a replaying in-memory consumer, 9 integration tests against a real broker). 0 failures, repeated three times with no flakiness.
  • The integration tests run only when AMCP_KAFKA_BOOTSTRAP is set (for example 127.0.0.1:9092); otherwise that class is skipped (surefire reports it as 0 tests run). Each test uses its own topic prefix.
  • Eight of the new tests fail on the code before this change and pass after (listed in the commit and in CHANGELOG.md).
  • Used downstream: a Java 21 "bee mesh" (event-driven agents, hosts and users) runs 1000 simulated agents on this broker in 4 small JVMs (21 tests pass against this version).

Not done / limits

  • No performance claims. A subscription takes effect shortly after subscribe() returns, so an event consumed in that window can be missed (documented).
  • Queue semantics hold only if every instance sharing a queueName subscribes the same patterns (Kafka cannot hand a record back to a chosen group member); a warning is logged when this is violated.
  • Only the Kafka module and amcp-core were touched. NATS, A2A, MCP and the Quarkus extension were not reviewed.
  • Tested with Kafka 3.8.0 and Java 21 only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JWcvy9P3NMPXiV6BycKYyY

…competing-consumer subscriptions; fix 7 robustness defects

The Kafka broker never delivered an event (4 of 5 new integration tests failed on the old code): publish and subscribe used different topic names, a second subscribe replaced the first, wildcards were passed as literal names, the consumer was used from two threads, and every received record lost its topic. Rewritten with one poll thread per consumer, regex subscriptions, CloudEvents structured JSON, fan-out (per-instance group) and subscribeQueue (shared group) semantics, manual offset commit after handlers complete (at-least-once; handlers must be idempotent), failure surfacing instead of silent thread death, retried subscription updates, a warning for queue records with no local match, matching rules between Event.matchesTopic and the Kafka patterns, and InMemoryEventBroker restart after stop. 9 amcp-core and 23 amcp-broker-kafka tests (9 against a real broker, skipped unless AMCP_KAFKA_BOOTSTRAP is set). See amcp-broker-kafka/README.md and CHANGELOG.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JWcvy9P3NMPXiV6BycKYyY
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant