Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,58 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

- `EventBroker.subscribeQueue(topicPattern, queueName, handler)`: competing-consumer subscriptions. Each event is
delivered to one subscriber per queue name. The default implementation falls back to `subscribe`, so existing
brokers keep working. `InMemoryEventBroker` implements it round-robin.
- `KafkaEventBroker` constructor `(bootstrap, groupId, instanceId, java.util.Properties extra)`. The extra properties
apply to the producer and to every consumer. The 3-argument constructor is unchanged.
- `KafkaEventBroker` now uses the structured-mode CloudEvents JSON wire format, so topic, id, time, source and
payload survive the round trip. A malformed record is logged and skipped.

### Fixed

- Kafka topic names are now the AMCP topic strings verbatim, for both publish and subscribe. Previously publish used
`a.b.c` and subscribe used `a-b-c`, so the two never met.
- A `KafkaEventBroker` consumer now keeps all its subscriptions. A new subscribe no longer replaces earlier ones.
- Wildcard subscriptions use a Kafka `Pattern` (the union of all active patterns) instead of literal topic names.
- Each `KafkaConsumer` is created, polled and closed by one dedicated thread. Subscribe and unsubscribe enqueue
changes and wake that thread. Handlers run on a separate virtual-thread executor. With no subscriptions the poll
loop idles quietly, with no "Consumer is not subscribed" error spam.
- Received events keep their topic. Previously every record was rebuilt as `amcp.event`, so no subscription matched.
- `Event.matchesTopic("p.**")` matches `p` and `p.<anything>` only. It no longer matches `phive...`.
- `KafkaEventBroker.publish` completes only after Kafka acknowledges the record, and fails when the broker is not
running.

- Review follow-ups on the Kafka transport (seven fixes):
1. A `KafkaConsumer` that cannot be built (for example `max.poll.records=abc` through the extra properties) now makes
`start()` return a failed future, or `subscribeQueue` throw `IllegalStateException` with the cause. Before, the
poll thread died silently and `start()` reported success. A dead worker is never left in the worker map.
2. A failed `consumer.subscribe/unsubscribe` no longer marks the pattern set as applied: the cause is logged once and
the update is retried with bounded backoff.
3. Delivery is now at-least-once: manual commit after the handlers of the polled batch finished (bounded wait of
30 s), rewind and redelivery of failed records, at most 3 attempts per record. Auto-commit used to commit on poll,
before the asynchronous handler ran, which was at-most-once. Handlers must be idempotent.
4. Queue mode: a record on an owned partition with no matching local subscription is still committed for the group
(a Kafka limitation), but a WARN is logged once per (topic, queue) pair, and the README requires every instance
sharing a queue name to subscribe the same patterns.
5. `Event.matchesTopic` and `TopicPatterns.toRegex` now agree on topics with empty segments: a wildcard never
matches a trailing dot or doubled dot. A table-driven test runs the same cases through both.
6. README: the topic-pickup time is stated in terms of `metadata.max.age.ms` (1 s here, plus a rebalance), and a
subscription takes effect shortly after `subscribe()` returns.
7. `InMemoryEventBroker.stop()` followed by `start()` no longer makes `publish` throw `RejectedExecutionException`.
Not done: the queue-mode drop in item 4 is documented and logged, not prevented.

### Changed

- `KafkaEventBroker.subscribe` (fan-out) skips events older than the subscription's creation time minus 5 seconds,
so a new broker instance does not replay history.
- `KafkaEventBroker.stop` wakes and joins every poll thread, closes consumers and producer, and shuts the executor
down. Calling `start` or `stop` more than once is safe.

## [1.6.0] - 2024-11-10

### Added
Expand Down
126 changes: 126 additions & 0 deletions amcp-broker-kafka/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# AMCP Kafka Broker

`KafkaEventBroker` is an `io.amcp.core.EventBroker` backed by Apache Kafka. Agents on different JVMs publish and
subscribe through the same topics.

```java
KafkaEventBroker broker = new KafkaEventBroker("127.0.0.1:9092", "mesh", "instance-1");
broker.start().get();

broker.subscribe("bee.*", event -> System.out.println(event.getTopic())); // fan-out
broker.subscribeQueue("jobs.run", "workers", event -> runJob(event)); // competing consumers
broker.publish(Event.create("bee.heartbeat", Map.of("bee", "b1"))).get();

broker.stop().get();
```

The 4-argument constructor `(bootstrap, groupId, instanceId, java.util.Properties extra)` passes extra Kafka client
properties to the producer and to every consumer. The bootstrap servers, group ids and serializers are always set by
the broker and override any conflicting extra property.

## Delivery semantics

**Correction:** an earlier revision of this broker used Kafka auto-commit while running handlers asynchronously.
Offsets were then committed on poll, before the handler ran, which is at-most-once: a crash or a failing handler
lost the event. That was wrong and is fixed. The broker now commits manually.

- **At-least-once.** The poll thread hands a polled batch to the handlers, waits for all of them (at most 30 seconds),
and only then commits the offsets of the records whose handlers returned normally. If a handler throws, or does not
finish in time, that record is not committed and the partition is rewound to it, so the record (and the records after
it in the same partition and batch) is delivered again. After 3 failed deliveries of the same record it is logged at
ERROR and skipped, so one poison record cannot block a partition forever.
- **Handlers must be idempotent.** A record can be delivered more than once (a retry after a failure, a rebalance, a
crash between handler completion and commit, a handler that finished after the 30-second wait). In fan-out mode a
retry also repeats the handlers that already succeeded for that record.
- `enable.auto.commit` is always `false`; an extra property cannot turn it back on.
- Because the poll thread waits for the batch, a handler that blocks for a long time delays all later records of that
consumer (the wait is bounded, and Kafka's `max.poll.interval.ms` default of 5 minutes is not at risk).

The broker offers two delivery modes. Choose the one that matches what the subscriber should see.

### Fan-out: `subscribe(topicPattern, handler)`

Every broker instance receives every matching event, which is the same meaning as `InMemoryEventBroker`.

- Each instance runs one consumer, in consumer group `groupId + ".fanout." + instanceId`.
- The consumer starts at the earliest offset, but it skips any event whose CloudEvent `time` is older than the
subscription's creation time minus 5 seconds. A newly started instance therefore does not replay history.
Events without a `time` are not filtered.
- The 5-second margin is a semantic rule about clocks, not a performance claim. If a producer's clock is more than
about 5 seconds behind the subscriber's clock, its fresh events are skipped by this filter.

### Competing consumers: `subscribeQueue(topicPattern, queueName, handler)`

Each event is delivered to exactly one subscriber across all broker instances that share `groupId` and `queueName`.

- There is one consumer per queue name, in consumer group `groupId + ".queue." + queueName`. Different queue names
each receive a full copy of the stream.
- The consumer starts at the earliest offset and has no history filter. A queue that is new therefore processes
events still retained in the topic.
- Within one instance, if several subscriptions share the queue name, each event goes to one of them, round-robin.
- **Every instance that shares a `queueName` MUST subscribe the same topic patterns.** Kafka assigns partitions of
the union of the group's topics to its members. If a record lands on an instance that has no local subscription
matching its topic, that instance cannot handle it, and its offset is committed for the whole group, so no other
instance sees it: the record is lost for that queue. The broker cannot repair this (Kafka groups do not allow
handing a record back to a chosen member). It logs one WARN per (topic, queueName) pair on the first such record so
the misconfiguration is visible. Use the same patterns everywhere, or use different queue names.
- Calling `subscribeQueue` on an `EventBroker` that does not override it (for example, a custom broker) falls back
to fan-out `subscribe`.

### Other rules

- Handlers run on a virtual-thread executor. A slow handler does not block polling. Handler order across events is
not guaranteed.
- `publish(event)` returns a future that completes when Kafka acknowledges the record (`acks=all`). It completes
exceptionally on a send error, and it fails at once with `IllegalStateException` when the broker is not running.
- A subscription takes effect shortly after `subscribe` or `subscribeQueue` returns, not before: the poll thread
applies it, and a queue subscription also waits for the group rebalance. Fan-out consumers start at the earliest
offset and filter by event time, so recent events are still seen. Do not rely on delivery of an event that was
published immediately before or after the call; wait for the assignment first.
- If the Kafka consumer of a worker cannot be created (for example an invalid consumer property in the 4-argument
constructor), `start()` returns a failed future, or `subscribeQueue` throws `IllegalStateException` with the cause.
If a subscription update fails (`subscribe` on the Kafka consumer throws), the broker logs the cause once and retries
with bounded backoff (1 s doubling to 30 s) until it succeeds.

## Topic naming and patterns

- The Kafka topic name is the AMCP topic string, verbatim. `publish` sends `bee.heartbeat` to the topic
`bee.heartbeat`. `subscribe("bee.heartbeat", ...)` subscribes to that same topic.
- Kafka topic names may contain only `a-z`, `A-Z`, `0-9`, `.`, `_` and `-`, up to 249 characters. Names that differ
only by `.` versus `_` collide, and Kafka rejects the second topic created.
- Patterns are translated to regular expressions, and the broker subscribes with one `Pattern` that is the union of
all active patterns of the consumer:

| AMCP pattern | Matches | Does not match |
|---|---|---|
| `a.b.c` | `a.b.c` | `a.b.cd`, `aXb.c` |
| `p.*` | `p.x` (exactly one segment) | `p`, `p.x.y`, `px.x` |
| `p.**` | `p`, `p.x`, `p.x.y` | `px`, `phive.x` |

- A wildcard never matches an empty segment. A topic with a trailing dot (`p.x.`) or a doubled dot (`p..x`) is matched
only by a pattern equal to the topic, never by `p.*` or `p.**`. `Event.matchesTopic` and the Kafka regex follow the
same rule (a table-driven test runs the same cases through both).
- Kafka subscriptions find topics by metadata refresh. The broker sets `metadata.max.age.ms=1000` (override it through
the extra properties), so a topic created after a pattern subscription is discovered at most about 1 second later;
the group rebalance that follows can add a second or more. Allow 2 to 5 seconds before expecting events from a
brand-new topic. Publishing to a topic that does not exist creates it when the cluster has
`auto.create.topics.enable=true`.

## Wire format

Records use the CloudEvents structured JSON format (`application/cloudevents+json`), through
`io.cloudevents.jackson.JsonFormat`. The record key is the CloudEvent extension `partitionkey` when present, and
otherwise the event id. A record that cannot be parsed is logged once by the consumer and skipped.

## Lifecycle

- `start()` and `stop()` are idempotent.
- `stop()` wakes and joins each poll thread, closes the consumers and the producer, and shuts the executor down. Each
consumer is created, polled and closed by one dedicated platform thread.

## Tests

`KafkaEventBrokerIT` runs against a real broker when `AMCP_KAFKA_BOOTSTRAP` is set, for example
`AMCP_KAFKA_BOOTSTRAP=127.0.0.1:9092 mvn -pl amcp-broker-kafka verify`. Without it, the integration tests are skipped
and the build stays green. `TopicPatternsTest` and `KafkaEventBrokerUnitTest` are plain unit tests that need no broker (the latter drives the
poll logic with an in-memory consumer: redelivery, commit timing, retry, queue WARN, start failure).
22 changes: 22 additions & 0 deletions amcp-broker-kafka/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@
<groupId>io.cloudevents</groupId>
<artifactId>cloudevents-kafka</artifactId>
</dependency>
<!-- CloudEvents structured JSON wire format -->
<dependency>
<groupId>io.cloudevents</groupId>
<artifactId>cloudevents-json-jackson</artifactId>
</dependency>

<!-- Logging -->
<dependency>
Expand All @@ -57,4 +62,21 @@
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!-- Run unit tests and Kafka integration tests (*IT); the default includes skip *IT. -->
<includes>
<include>**/*Test.java</include>
<include>**/*IT.java</include>
</includes>
<argLine>-Xmx512m</argLine>
</configuration>
</plugin>
</plugins>
</build>
</project>
Loading
Loading