From 4dc7cf2d3b4993eaac40965cf57a0a38d3c275d7 Mon Sep 17 00:00:00 2001 From: Claude Haiku Date: Tue, 27 Oct 2026 20:41:28 +0000 Subject: [PATCH] fix(kafka): make the Kafka event broker actually deliver events; add competing-consumer subscriptions; fix 7 robustness defects The Kafka broker never delivered an event (4 of 5 new integration tests failed on the old code): publish and subscribe used different topic names, a second subscribe replaced the first, wildcards were passed as literal names, the consumer was used from two threads, and every received record lost its topic. Rewritten with one poll thread per consumer, regex subscriptions, CloudEvents structured JSON, fan-out (per-instance group) and subscribeQueue (shared group) semantics, manual offset commit after handlers complete (at-least-once; handlers must be idempotent), failure surfacing instead of silent thread death, retried subscription updates, a warning for queue records with no local match, matching rules between Event.matchesTopic and the Kafka patterns, and InMemoryEventBroker restart after stop. 9 amcp-core and 23 amcp-broker-kafka tests (9 against a real broker, skipped unless AMCP_KAFKA_BOOTSTRAP is set). See amcp-broker-kafka/README.md and CHANGELOG.md. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01JWcvy9P3NMPXiV6BycKYyY --- CHANGELOG.md | 52 + amcp-broker-kafka/README.md | 126 +++ amcp-broker-kafka/pom.xml | 22 + .../amcp/broker/kafka/KafkaEventBroker.java | 902 ++++++++++++++---- .../io/amcp/broker/kafka/TopicPatterns.java | 56 ++ .../io/amcp/broker/kafka/FakeConsumer.java | 114 +++ .../amcp/broker/kafka/KafkaEventBrokerIT.java | 240 +++++ .../kafka/KafkaEventBrokerUnitTest.java | 225 +++++ .../amcp/broker/kafka/TopicPatternsTest.java | 111 +++ amcp-core/pom.xml | 3 + .../src/main/java/io/amcp/core/Event.java | 34 +- .../main/java/io/amcp/core/EventBroker.java | 18 + .../io/amcp/core/InMemoryEventBroker.java | 78 +- .../src/test/java/io/amcp/core/EventTest.java | 10 + .../io/amcp/core/InMemoryEventBrokerTest.java | 76 ++ 15 files changed, 1876 insertions(+), 191 deletions(-) create mode 100644 amcp-broker-kafka/README.md create mode 100644 amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/TopicPatterns.java create mode 100644 amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/FakeConsumer.java create mode 100644 amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerIT.java create mode 100644 amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerUnitTest.java create mode 100644 amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/TopicPatternsTest.java create mode 100644 amcp-core/src/test/java/io/amcp/core/InMemoryEventBrokerTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 6396a9c..7c464f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,58 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- `EventBroker.subscribeQueue(topicPattern, queueName, handler)`: competing-consumer subscriptions. Each event is + delivered to one subscriber per queue name. The default implementation falls back to `subscribe`, so existing + brokers keep working. `InMemoryEventBroker` implements it round-robin. +- `KafkaEventBroker` constructor `(bootstrap, groupId, instanceId, java.util.Properties extra)`. The extra properties + apply to the producer and to every consumer. The 3-argument constructor is unchanged. +- `KafkaEventBroker` now uses the structured-mode CloudEvents JSON wire format, so topic, id, time, source and + payload survive the round trip. A malformed record is logged and skipped. + +### Fixed + +- Kafka topic names are now the AMCP topic strings verbatim, for both publish and subscribe. Previously publish used + `a.b.c` and subscribe used `a-b-c`, so the two never met. +- A `KafkaEventBroker` consumer now keeps all its subscriptions. A new subscribe no longer replaces earlier ones. +- Wildcard subscriptions use a Kafka `Pattern` (the union of all active patterns) instead of literal topic names. +- Each `KafkaConsumer` is created, polled and closed by one dedicated thread. Subscribe and unsubscribe enqueue + changes and wake that thread. Handlers run on a separate virtual-thread executor. With no subscriptions the poll + loop idles quietly, with no "Consumer is not subscribed" error spam. +- Received events keep their topic. Previously every record was rebuilt as `amcp.event`, so no subscription matched. +- `Event.matchesTopic("p.**")` matches `p` and `p.` only. It no longer matches `phive...`. +- `KafkaEventBroker.publish` completes only after Kafka acknowledges the record, and fails when the broker is not + running. + +- Review follow-ups on the Kafka transport (seven fixes): + 1. A `KafkaConsumer` that cannot be built (for example `max.poll.records=abc` through the extra properties) now makes + `start()` return a failed future, or `subscribeQueue` throw `IllegalStateException` with the cause. Before, the + poll thread died silently and `start()` reported success. A dead worker is never left in the worker map. + 2. A failed `consumer.subscribe/unsubscribe` no longer marks the pattern set as applied: the cause is logged once and + the update is retried with bounded backoff. + 3. Delivery is now at-least-once: manual commit after the handlers of the polled batch finished (bounded wait of + 30 s), rewind and redelivery of failed records, at most 3 attempts per record. Auto-commit used to commit on poll, + before the asynchronous handler ran, which was at-most-once. Handlers must be idempotent. + 4. Queue mode: a record on an owned partition with no matching local subscription is still committed for the group + (a Kafka limitation), but a WARN is logged once per (topic, queue) pair, and the README requires every instance + sharing a queue name to subscribe the same patterns. + 5. `Event.matchesTopic` and `TopicPatterns.toRegex` now agree on topics with empty segments: a wildcard never + matches a trailing dot or doubled dot. A table-driven test runs the same cases through both. + 6. README: the topic-pickup time is stated in terms of `metadata.max.age.ms` (1 s here, plus a rebalance), and a + subscription takes effect shortly after `subscribe()` returns. + 7. `InMemoryEventBroker.stop()` followed by `start()` no longer makes `publish` throw `RejectedExecutionException`. + Not done: the queue-mode drop in item 4 is documented and logged, not prevented. + +### Changed + +- `KafkaEventBroker.subscribe` (fan-out) skips events older than the subscription's creation time minus 5 seconds, + so a new broker instance does not replay history. +- `KafkaEventBroker.stop` wakes and joins every poll thread, closes consumers and producer, and shuts the executor + down. Calling `start` or `stop` more than once is safe. + ## [1.6.0] - 2024-11-10 ### Added diff --git a/amcp-broker-kafka/README.md b/amcp-broker-kafka/README.md new file mode 100644 index 0000000..b2d78e1 --- /dev/null +++ b/amcp-broker-kafka/README.md @@ -0,0 +1,126 @@ +# AMCP Kafka Broker + +`KafkaEventBroker` is an `io.amcp.core.EventBroker` backed by Apache Kafka. Agents on different JVMs publish and +subscribe through the same topics. + +```java +KafkaEventBroker broker = new KafkaEventBroker("127.0.0.1:9092", "mesh", "instance-1"); +broker.start().get(); + +broker.subscribe("bee.*", event -> System.out.println(event.getTopic())); // fan-out +broker.subscribeQueue("jobs.run", "workers", event -> runJob(event)); // competing consumers +broker.publish(Event.create("bee.heartbeat", Map.of("bee", "b1"))).get(); + +broker.stop().get(); +``` + +The 4-argument constructor `(bootstrap, groupId, instanceId, java.util.Properties extra)` passes extra Kafka client +properties to the producer and to every consumer. The bootstrap servers, group ids and serializers are always set by +the broker and override any conflicting extra property. + +## Delivery semantics + +**Correction:** an earlier revision of this broker used Kafka auto-commit while running handlers asynchronously. +Offsets were then committed on poll, before the handler ran, which is at-most-once: a crash or a failing handler +lost the event. That was wrong and is fixed. The broker now commits manually. + +- **At-least-once.** The poll thread hands a polled batch to the handlers, waits for all of them (at most 30 seconds), + and only then commits the offsets of the records whose handlers returned normally. If a handler throws, or does not + finish in time, that record is not committed and the partition is rewound to it, so the record (and the records after + it in the same partition and batch) is delivered again. After 3 failed deliveries of the same record it is logged at + ERROR and skipped, so one poison record cannot block a partition forever. +- **Handlers must be idempotent.** A record can be delivered more than once (a retry after a failure, a rebalance, a + crash between handler completion and commit, a handler that finished after the 30-second wait). In fan-out mode a + retry also repeats the handlers that already succeeded for that record. +- `enable.auto.commit` is always `false`; an extra property cannot turn it back on. +- Because the poll thread waits for the batch, a handler that blocks for a long time delays all later records of that + consumer (the wait is bounded, and Kafka's `max.poll.interval.ms` default of 5 minutes is not at risk). + +The broker offers two delivery modes. Choose the one that matches what the subscriber should see. + +### Fan-out: `subscribe(topicPattern, handler)` + +Every broker instance receives every matching event, which is the same meaning as `InMemoryEventBroker`. + +- Each instance runs one consumer, in consumer group `groupId + ".fanout." + instanceId`. +- The consumer starts at the earliest offset, but it skips any event whose CloudEvent `time` is older than the + subscription's creation time minus 5 seconds. A newly started instance therefore does not replay history. + Events without a `time` are not filtered. +- The 5-second margin is a semantic rule about clocks, not a performance claim. If a producer's clock is more than + about 5 seconds behind the subscriber's clock, its fresh events are skipped by this filter. + +### Competing consumers: `subscribeQueue(topicPattern, queueName, handler)` + +Each event is delivered to exactly one subscriber across all broker instances that share `groupId` and `queueName`. + +- There is one consumer per queue name, in consumer group `groupId + ".queue." + queueName`. Different queue names + each receive a full copy of the stream. +- The consumer starts at the earliest offset and has no history filter. A queue that is new therefore processes + events still retained in the topic. +- Within one instance, if several subscriptions share the queue name, each event goes to one of them, round-robin. +- **Every instance that shares a `queueName` MUST subscribe the same topic patterns.** Kafka assigns partitions of + the union of the group's topics to its members. If a record lands on an instance that has no local subscription + matching its topic, that instance cannot handle it, and its offset is committed for the whole group, so no other + instance sees it: the record is lost for that queue. The broker cannot repair this (Kafka groups do not allow + handing a record back to a chosen member). It logs one WARN per (topic, queueName) pair on the first such record so + the misconfiguration is visible. Use the same patterns everywhere, or use different queue names. +- Calling `subscribeQueue` on an `EventBroker` that does not override it (for example, a custom broker) falls back + to fan-out `subscribe`. + +### Other rules + +- Handlers run on a virtual-thread executor. A slow handler does not block polling. Handler order across events is + not guaranteed. +- `publish(event)` returns a future that completes when Kafka acknowledges the record (`acks=all`). It completes + exceptionally on a send error, and it fails at once with `IllegalStateException` when the broker is not running. +- A subscription takes effect shortly after `subscribe` or `subscribeQueue` returns, not before: the poll thread + applies it, and a queue subscription also waits for the group rebalance. Fan-out consumers start at the earliest + offset and filter by event time, so recent events are still seen. Do not rely on delivery of an event that was + published immediately before or after the call; wait for the assignment first. +- If the Kafka consumer of a worker cannot be created (for example an invalid consumer property in the 4-argument + constructor), `start()` returns a failed future, or `subscribeQueue` throws `IllegalStateException` with the cause. + If a subscription update fails (`subscribe` on the Kafka consumer throws), the broker logs the cause once and retries + with bounded backoff (1 s doubling to 30 s) until it succeeds. + +## Topic naming and patterns + +- The Kafka topic name is the AMCP topic string, verbatim. `publish` sends `bee.heartbeat` to the topic + `bee.heartbeat`. `subscribe("bee.heartbeat", ...)` subscribes to that same topic. +- Kafka topic names may contain only `a-z`, `A-Z`, `0-9`, `.`, `_` and `-`, up to 249 characters. Names that differ + only by `.` versus `_` collide, and Kafka rejects the second topic created. +- Patterns are translated to regular expressions, and the broker subscribes with one `Pattern` that is the union of + all active patterns of the consumer: + +| AMCP pattern | Matches | Does not match | +|---|---|---| +| `a.b.c` | `a.b.c` | `a.b.cd`, `aXb.c` | +| `p.*` | `p.x` (exactly one segment) | `p`, `p.x.y`, `px.x` | +| `p.**` | `p`, `p.x`, `p.x.y` | `px`, `phive.x` | + +- A wildcard never matches an empty segment. A topic with a trailing dot (`p.x.`) or a doubled dot (`p..x`) is matched + only by a pattern equal to the topic, never by `p.*` or `p.**`. `Event.matchesTopic` and the Kafka regex follow the + same rule (a table-driven test runs the same cases through both). +- Kafka subscriptions find topics by metadata refresh. The broker sets `metadata.max.age.ms=1000` (override it through + the extra properties), so a topic created after a pattern subscription is discovered at most about 1 second later; + the group rebalance that follows can add a second or more. Allow 2 to 5 seconds before expecting events from a + brand-new topic. Publishing to a topic that does not exist creates it when the cluster has + `auto.create.topics.enable=true`. + +## Wire format + +Records use the CloudEvents structured JSON format (`application/cloudevents+json`), through +`io.cloudevents.jackson.JsonFormat`. The record key is the CloudEvent extension `partitionkey` when present, and +otherwise the event id. A record that cannot be parsed is logged once by the consumer and skipped. + +## Lifecycle + +- `start()` and `stop()` are idempotent. +- `stop()` wakes and joins each poll thread, closes the consumers and the producer, and shuts the executor down. Each + consumer is created, polled and closed by one dedicated platform thread. + +## Tests + +`KafkaEventBrokerIT` runs against a real broker when `AMCP_KAFKA_BOOTSTRAP` is set, for example +`AMCP_KAFKA_BOOTSTRAP=127.0.0.1:9092 mvn -pl amcp-broker-kafka verify`. Without it, the integration tests are skipped +and the build stays green. `TopicPatternsTest` and `KafkaEventBrokerUnitTest` are plain unit tests that need no broker (the latter drives the +poll logic with an in-memory consumer: redelivery, commit timing, retry, queue WARN, start failure). diff --git a/amcp-broker-kafka/pom.xml b/amcp-broker-kafka/pom.xml index 9e2f78c..758ae55 100644 --- a/amcp-broker-kafka/pom.xml +++ b/amcp-broker-kafka/pom.xml @@ -33,6 +33,11 @@ io.cloudevents cloudevents-kafka + + + io.cloudevents + cloudevents-json-jackson + @@ -57,4 +62,21 @@ test + + + + + org.apache.maven.plugins + maven-surefire-plugin + + + + **/*Test.java + **/*IT.java + + -Xmx512m + + + + diff --git a/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/KafkaEventBroker.java b/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/KafkaEventBroker.java index bfb3e22..9cfa1dd 100644 --- a/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/KafkaEventBroker.java +++ b/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/KafkaEventBroker.java @@ -3,52 +3,535 @@ import io.amcp.core.Event; import io.amcp.core.EventBroker; import io.cloudevents.CloudEvent; +import io.cloudevents.core.format.EventFormat; +import io.cloudevents.core.provider.EventFormatProvider; import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.consumer.OffsetAndMetadata; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.common.errors.WakeupException; import org.apache.kafka.common.serialization.StringDeserializer; import org.apache.kafka.common.serialization.StringSerializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.Instant; import java.time.OffsetDateTime; -import java.util.*; -import java.util.concurrent.*; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Properties; +import java.util.Set; +import java.util.TreeSet; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Consumer; /** - * Kafka-based EventBroker implementation for distributed AMCP mesh. - * - * Features: - * - Distributed message routing via Kafka - * - Topic-based pub/sub - * - Consumer groups for scalability - * - CloudEvents serialization - * - Multi-instance support + * Kafka-based EventBroker for a distributed AMCP mesh. + * + *

Topic names are the AMCP topic strings verbatim. Two delivery semantics are offered: + *

    + *
  • {@link #subscribe} is fan-out: every broker instance receives every matching event. + * Each instance runs one consumer with group {@code groupId + ".fanout." + instanceId}. + * Events older than the subscription's creation time minus 5 seconds are skipped, so a new + * instance does not replay history.
  • + *
  • {@link #subscribeQueue} is competing consumers: each event goes to one subscriber across + * all instances that share {@code groupId} and {@code queueName}. The consumer group is + * {@code groupId + ".queue." + queueName}; there is no history filter.
  • + *
+ * + *

Delivery is at-least-once: offsets are committed manually, after the handlers of a polled batch have + * completed (bounded wait). A failed or unfinished record is rewound and redelivered, at most + * {@link #MAX_DELIVERY_ATTEMPTS} times. Handlers must be idempotent. In queue mode every instance sharing a + * {@code queueName} must subscribe the same patterns; otherwise a record can be committed without any handler. + * + *

Threading: every {@link KafkaConsumer} is created, polled and closed by one dedicated platform + * thread. Subscribe and unsubscribe calls enqueue changes and wake that thread. Handlers run on a + * virtual-thread executor, so a slow handler never blocks polling. */ public class KafkaEventBroker implements EventBroker { private static final Logger logger = LoggerFactory.getLogger(KafkaEventBroker.class); - + + /** Lookback applied to fan-out subscriptions: events older than (creation - 5 s) are skipped. */ + static final Duration FANOUT_HISTORY_GRACE = Duration.ofSeconds(5); + + /** A record whose handler keeps failing is delivered at most this many times, then skipped with an ERROR log. */ + static final int MAX_DELIVERY_ATTEMPTS = 3; + + private static final String FANOUT_KEY = "fanout"; + private static final String QUEUE_KEY_PREFIX = "queue:"; + private static final Duration POLL_TIMEOUT = Duration.ofMillis(500); + private static final long IDLE_WAIT_MS = 500; + private static final long STOP_JOIN_MS = 10_000; + private static final long ERROR_BACKOFF_MS = 1_000; + private static final long MAX_RETRY_BACKOFF_MS = 30_000; + private static final long CONSUMER_CREATE_TIMEOUT_S = 30; + /** metadata.max.age.ms set on every consumer (extra properties may override it). */ + static final int METADATA_MAX_AGE_MS = 1000; + private static final String CLOUDEVENTS_JSON = "application/cloudevents+json"; + + private static final EventFormat JSON_FORMAT = Objects.requireNonNull( + EventFormatProvider.getInstance().resolveFormat(CLOUDEVENTS_JSON), + "CloudEvents JSON format not found on the classpath (cloudevents-json-jackson)"); + private final String bootstrapServers; private final String groupId; private final String instanceId; - - private KafkaProducer producer; - private KafkaConsumer consumer; - private ExecutorService executor; - private volatile boolean running = false; + private final Properties extraProperties; + + /** Guards start, stop, subscribe, unsubscribe and worker creation. Never taken by publish(). */ + private final Object lifecycle = new Object(); private final Map subscriptions = new ConcurrentHashMap<>(); + private final Map workers = new ConcurrentHashMap<>(); + private volatile KafkaProducer producer; + private volatile ExecutorService executor; + private volatile boolean running = false; + + // Test seams (package-private): replaced in unit tests, never by production code. + volatile java.util.function.Function> consumerFactory = + props -> new KafkaConsumer<>(props); + volatile Duration batchTimeout = Duration.ofSeconds(30); + volatile long baseRetryBackoffMs = 1_000; - private static class Subscription { + /** One subscription as registered by the caller. Immutable apart from the poll thread's own copy. */ + private static final class Subscription { + final String id; final String topicPattern; final Consumer handler; + final String queueName; // null for fan-out + final Instant notBefore; // null for queue subscriptions (no history filter) - Subscription(String topicPattern, Consumer handler) { + Subscription(String id, String topicPattern, Consumer handler, String queueName, Instant notBefore) { + this.id = id; this.topicPattern = topicPattern; this.handler = handler; + this.queueName = queueName; + this.notBefore = notBefore; + } + + String workerKey() { + return queueName == null ? FANOUT_KEY : QUEUE_KEY_PREFIX + queueName; + } + + boolean accepts(Event event) { + if (!event.matchesTopic(topicPattern)) { + return false; + } + if (notBefore == null) { + return true; + } + OffsetDateTime time = event.getTime(); + return time == null || !time.toInstant().isBefore(notBefore); + } + } + + private record Change(Subscription subscription, boolean add) { + } + + /** + * Owns one KafkaConsumer and the thread that drives it. Fields marked "poll thread only" are + * touched only by that thread. + */ + private final class PollWorker implements Runnable { + private final String name; + private final String consumerGroup; + private final boolean queue; + private final Thread thread; + /** Completed once the consumer exists, or completed exceptionally when it could not be created. */ + private final CompletableFuture ready = new CompletableFuture<>(); + private final ConcurrentLinkedQueue changes = new ConcurrentLinkedQueue<>(); + private final Object monitor = new Object(); + private final Map active = new LinkedHashMap<>(); // poll thread only + private final AtomicInteger roundRobin = new AtomicInteger(); + private final Set warnedDrops = new HashSet<>(); // poll thread only + private final Map failedAttempts = new HashMap<>(); // poll thread only + private Set appliedPatterns = new TreeSet<>(); // poll thread only + private int applyFailures = 0; // poll thread only + private long applyRetryAtNanos = 0; // poll thread only + private boolean applyFailureLogged = false; // poll thread only + private volatile org.apache.kafka.clients.consumer.Consumer consumer; + private volatile boolean stopping = false; + private volatile int assignedPartitions = 0; + + PollWorker(String name, String consumerGroup, boolean queue) { + this.name = name; + this.consumerGroup = consumerGroup; + this.queue = queue; + this.thread = new Thread(this, "amcp-kafka-" + name); + this.thread.setDaemon(true); + } + + void enqueue(Change change) { + changes.add(change); + synchronized (monitor) { + monitor.notifyAll(); + } + wakeConsumer(); + } + + void requestStop() { + stopping = true; + synchronized (monitor) { + monitor.notifyAll(); + } + wakeConsumer(); + } + + private void wakeConsumer() { + org.apache.kafka.clients.consumer.Consumer c = consumer; + if (c != null) { + try { + c.wakeup(); + } catch (RuntimeException e) { + logger.debug("wakeup on consumer {} ignored: {}", name, e.getMessage()); + } + } + } + + void awaitTermination(long timeoutMs) { + try { + thread.join(timeoutMs); + if (thread.isAlive()) { + logger.warn("Kafka poll thread {} did not stop within {} ms", name, timeoutMs); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + + @Override + public void run() { + org.apache.kafka.clients.consumer.Consumer c; + try { + c = consumerFactory.apply(consumerProperties(consumerGroup)); + } catch (RuntimeException | Error e) { + logger.error("Could not create Kafka consumer {}: {}", name, e.getMessage(), e); + workers.remove(name, this); + ready.completeExceptionally(e); + return; + } + consumer = c; + logger.info("Kafka consumer {} created in group {}", name, consumerGroup); + ready.complete(null); + try { + while (!stopping) { + try { + applyChanges(c); + if (active.isEmpty() || appliedPatterns.isEmpty()) { + // Nothing is subscribed (or the last update failed and is waiting for its retry): + // never call poll() on a consumer without a subscription. + idle(); + continue; + } + ConsumerRecords records = c.poll(POLL_TIMEOUT); + assignedPartitions = c.assignment().size(); + if (!records.isEmpty()) { + processBatch(c, records); + } + } catch (WakeupException expected) { + // Raised on purpose by wakeup(): a change is queued or stop was requested. + } catch (RuntimeException e) { + if (stopping) { + break; + } + logger.error("Kafka poll loop {} error: {}", name, e.getMessage()); + sleepMs(ERROR_BACKOFF_MS); + } + } + } finally { + // Never leave a dead worker in the map: a later subscribe must create a fresh one. + workers.remove(name, this); + consumer = null; + try { + c.close(Duration.ofSeconds(5)); + } catch (RuntimeException e) { + logger.warn("Error closing Kafka consumer {}: {}", name, e.getMessage()); + } + logger.info("Kafka consumer {} closed", name); + } + } + + /** + * Drains queued changes and resubscribes only when the set of patterns actually changed. The applied set + * is updated only after the Kafka call succeeded; on failure the update is retried with bounded backoff. + */ + private void applyChanges(org.apache.kafka.clients.consumer.Consumer c) { + Change change; + while ((change = changes.poll()) != null) { + if (change.add()) { + active.put(change.subscription().id, change.subscription()); + } else { + active.remove(change.subscription().id); + } + } + Set wanted = new TreeSet<>(); + for (Subscription s : active.values()) { + wanted.add(s.topicPattern); + } + if (wanted.equals(appliedPatterns)) { + return; + } + if (applyFailures > 0 && System.nanoTime() - applyRetryAtNanos < 0) { + return; // still backing off after a failed update + } + try { + if (wanted.isEmpty()) { + c.unsubscribe(); + logger.debug("Consumer {} unsubscribed (no active subscriptions)", name); + } else { + c.subscribe(TopicPatterns.union(wanted)); + logger.debug("Consumer {} subscribed to {}", name, wanted); + } + appliedPatterns = wanted; + applyFailures = 0; + applyFailureLogged = false; + } catch (RuntimeException e) { + applyFailures++; + long backoffMs = Math.min(MAX_RETRY_BACKOFF_MS, + baseRetryBackoffMs << Math.min(applyFailures - 1, 10)); + applyRetryAtNanos = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(backoffMs); + if (!applyFailureLogged) { + applyFailureLogged = true; + logger.error("Consumer {} could not update its subscription to {}; the subscription stays " + + "inactive and the update is retried with backoff: {}", name, wanted, e.toString(), e); + } + } + } + + /** Waits without touching the consumer; used when nothing is applied, so poll() is never called. */ + private void idle() { + long waitMs = IDLE_WAIT_MS; + if (applyFailures > 0) { + long untilRetry = TimeUnit.NANOSECONDS.toMillis(applyRetryAtNanos - System.nanoTime()); + waitMs = Math.max(1, Math.min(waitMs, untilRetry)); + } + synchronized (monitor) { + if (changes.isEmpty() && !stopping) { + try { + monitor.wait(waitMs); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + stopping = true; + } + } + } + } + + private void sleepMs(long ms) { + try { + Thread.sleep(ms); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + stopping = true; + } + } + + private record Dispatched(ConsumerRecord record, CompletableFuture done) { + } + + /** + * Hands every record of the polled batch to the handlers, waits (bounded) for all of them, and commits + * only the offsets whose handlers completed. A failed or unfinished record is not committed and the + * partition is rewound to it, so it is redelivered: at-least-once. After {@link #MAX_DELIVERY_ATTEMPTS} + * failed deliveries a record is logged and skipped so one poison record cannot block a partition. + */ + private void processBatch(org.apache.kafka.clients.consumer.Consumer c, + ConsumerRecords records) { + Map> byPartition = new LinkedHashMap<>(); + List> all = new ArrayList<>(); + for (TopicPartition tp : records.partitions()) { + List list = new ArrayList<>(); + for (ConsumerRecord record : records.records(tp)) { + CompletableFuture done = dispatch(record); + list.add(new Dispatched(record, done)); + all.add(done); + } + byPartition.put(tp, list); + } + if (!awaitAll(all)) { + return; // stopping: commit nothing, the batch is redelivered on the next start + } + + Map commit = new LinkedHashMap<>(); + Map rewind = new LinkedHashMap<>(); + for (Map.Entry> entry : byPartition.entrySet()) { + TopicPartition tp = entry.getKey(); + boolean blocked = false; + for (Dispatched d : entry.getValue()) { + ConsumerRecord r = d.record(); + String key = r.topic() + "/" + r.partition() + "/" + r.offset(); + boolean ok = d.done().isDone() && !d.done().isCompletedExceptionally(); + if (ok) { + failedAttempts.remove(key); + } else { + int attempts = failedAttempts.merge(key, 1, Integer::sum); + if (attempts >= MAX_DELIVERY_ATTEMPTS) { + logger.error("Giving up on record {} after {} failed delivery attempts; it is skipped " + + "and its offset is committed", key, attempts); + failedAttempts.remove(key); + ok = true; + } + } + if (blocked) { + continue; + } + if (ok) { + commit.put(tp, new OffsetAndMetadata(r.offset() + 1)); + } else { + blocked = true; + rewind.put(tp, r.offset()); + } + } + } + commitOffsets(c, commit); + if (!rewind.isEmpty()) { + for (Map.Entry e : rewind.entrySet()) { + c.seek(e.getKey(), e.getValue()); + } + logger.warn("Consumer {} will redeliver unfinished or failed records from {}", name, rewind); + sleepMs(baseRetryBackoffMs); + } + } + + private void commitOffsets(org.apache.kafka.clients.consumer.Consumer c, + Map commit) { + if (commit.isEmpty()) { + return; + } + for (int attempt = 0; attempt < 3; attempt++) { + try { + c.commitSync(commit); + return; + } catch (WakeupException wakeup) { + // A subscribe/unsubscribe woke the consumer while we were committing: try again. + } catch (RuntimeException e) { + logger.warn("Consumer {} could not commit offsets {} (the records may be redelivered): {}", + name, commit, e.toString()); + return; + } + } + logger.warn("Consumer {} could not commit offsets {} after repeated wakeups", name, commit); + } + + /** Waits for all handler futures, at most {@code batchTimeout}. Returns false when the worker is stopping. */ + private boolean awaitAll(List> futures) { + CompletableFuture all = CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); + long deadline = System.nanoTime() + batchTimeout.toNanos(); + while (!all.isDone()) { + if (stopping) { + return false; + } + long remaining = deadline - System.nanoTime(); + if (remaining <= 0) { + logger.warn("Consumer {}: handlers did not finish within {} ms; unfinished records are not " + + "committed and will be redelivered", name, batchTimeout.toMillis()); + return true; + } + try { + all.get(Math.min(remaining, TimeUnit.MILLISECONDS.toNanos(100)), TimeUnit.NANOSECONDS); + } catch (TimeoutException | ExecutionException ignored) { + // re-check state; failures are read per record afterwards + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + stopping = true; + return false; + } + } + return true; + } + + /** + * Routes one record. Runs on the poll thread and only hands work to the executor. The returned future + * completes normally when every handler it was given returned normally (or nobody had to handle it). + */ + private CompletableFuture dispatch(ConsumerRecord record) { + Event event; + try { + event = decode(record.value()); + } catch (RuntimeException e) { + logger.warn("Skipping malformed record on topic {} at offset {}: {}", + record.topic(), record.offset(), e.getMessage()); + return CompletableFuture.completedFuture(null); + } + List matches = new ArrayList<>(); + for (Subscription s : active.values()) { + if (s.accepts(event)) { + matches.add(s); + } + } + if (!queue) { + List> futures = new ArrayList<>(); + for (Subscription s : matches) { + futures.add(deliver(s, event)); + } + return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); + } + if (matches.isEmpty()) { + // The group assigned us a partition, but no local subscription matches. The offset is committed + // with the batch, so no other instance will see this record: all instances sharing a queue name + // must subscribe the same patterns. + if (warnedDrops.add(record.topic())) { + logger.warn("Queue '{}' dropped a record on topic '{}': this instance has no local subscription " + + "that matches it, and the offset is committed for the whole queue group. Every " + + "instance sharing a queue name must subscribe the same topic patterns.", + queueNameOf(), record.topic()); + } + return CompletableFuture.completedFuture(null); + } + return deliver(matches.get(Math.floorMod(roundRobin.getAndIncrement(), matches.size())), event); + } + + private String queueNameOf() { + return name.startsWith(QUEUE_KEY_PREFIX) ? name.substring(QUEUE_KEY_PREFIX.length()) : name; + } + + private CompletableFuture deliver(Subscription s, Event event) { + CompletableFuture done = new CompletableFuture<>(); + ExecutorService ex = executor; + if (ex == null) { + done.completeExceptionally(new IllegalStateException("broker is stopping")); + return done; + } + try { + ex.execute(() -> { + try { + s.handler.accept(event); + done.complete(null); + } catch (Throwable t) { + logger.error("Handler for {} failed on event {}: {}", + event.getTopic(), event.getId(), t.getMessage(), t); + done.completeExceptionally(t); + if (t instanceof Error err) { + throw err; + } + } + }); + } catch (RejectedExecutionException e) { + logger.warn("Not delivering event {} because the broker is stopping", event.getId()); + done.completeExceptionally(e); + } + return done; } } @@ -56,143 +539,234 @@ private static class Subscription { * Creates a new KafkaEventBroker. * * @param bootstrapServers Kafka bootstrap servers (e.g., "localhost:9092") - * @param groupId Consumer group ID for this instance + * @param groupId Consumer group base ID shared by the mesh * @param instanceId Unique instance identifier */ public KafkaEventBroker(String bootstrapServers, String groupId, String instanceId) { - this.bootstrapServers = bootstrapServers; - this.groupId = groupId; - this.instanceId = instanceId; + this(bootstrapServers, groupId, instanceId, new Properties()); + } + + /** + * Creates a new KafkaEventBroker with extra Kafka client properties. + * + *

The extra properties apply to the producer and to every consumer. Connection and identity + * settings (bootstrap servers, group id, serializers) are always set by the broker and win over + * the extra properties. + * + * @param bootstrapServers Kafka bootstrap servers + * @param groupId Consumer group base ID shared by the mesh + * @param instanceId Unique instance identifier + * @param extraProperties additional Kafka client properties (may be null) + */ + public KafkaEventBroker(String bootstrapServers, String groupId, String instanceId, Properties extraProperties) { + this.bootstrapServers = Objects.requireNonNull(bootstrapServers, "bootstrapServers"); + this.groupId = Objects.requireNonNull(groupId, "groupId"); + this.instanceId = Objects.requireNonNull(instanceId, "instanceId"); + this.extraProperties = new Properties(); + if (extraProperties != null) { + this.extraProperties.putAll(extraProperties); + } } @Override public CompletableFuture publish(Event event) { - if (!running) { - return CompletableFuture.failedFuture( - new IllegalStateException("Broker not running")); + KafkaProducer p = producer; + ExecutorService ex = executor; + if (!running || p == null || ex == null) { + return CompletableFuture.failedFuture(new IllegalStateException("Broker not running")); } + CompletableFuture acknowledged = new CompletableFuture<>(); + try { + ex.execute(() -> send(p, event, acknowledged)); + } catch (RejectedExecutionException e) { + acknowledged.completeExceptionally(new IllegalStateException("Broker not running", e)); + } + return acknowledged; + } - return CompletableFuture.runAsync(() -> { - try { - String topic = event.getTopic(); - String key = instanceId + "-" + UUID.randomUUID(); - String value = serializeEvent(event); - - ProducerRecord record = new ProducerRecord<>(topic, key, value); - producer.send(record, (metadata, exception) -> { - if (exception != null) { - logger.error("Error publishing to topic {}: {}", topic, exception.getMessage()); - } else { - logger.debug("Published to topic {} partition {}", topic, metadata.partition()); - } - }); - - } catch (Exception e) { - logger.error("Error serializing event: {}", e.getMessage(), e); - } - }, executor); + private void send(KafkaProducer p, Event event, CompletableFuture acknowledged) { + try { + String topic = event.getTopic(); + ProducerRecord record = new ProducerRecord<>(topic, recordKey(event), encode(event)); + p.send(record, (metadata, error) -> { + if (error != null) { + logger.error("Error publishing to topic {}: {}", topic, error.getMessage()); + acknowledged.completeExceptionally(error); + } else { + logger.debug("Published to topic {} partition {} offset {}", + topic, metadata.partition(), metadata.offset()); + acknowledged.complete(null); + } + }); + } catch (RuntimeException e) { + logger.error("Error publishing event {}: {}", event.getId(), e.getMessage(), e); + acknowledged.completeExceptionally(e); + } + } + + /** Record key: the CloudEvent "partitionkey" extension when present, otherwise the event id. */ + private static String recordKey(Event event) { + Object partitionKey = event.getCloudEvent().getExtension("partitionkey"); + return partitionKey != null ? partitionKey.toString() : event.getId(); } @Override public String subscribe(String topicPattern, Consumer handler) { - String subscriptionId = UUID.randomUUID().toString(); - subscriptions.put(subscriptionId, new Subscription(topicPattern, handler)); - - // Convert pattern to Kafka topic - String kafkaTopic = topicPattern.replace("**", "*").replace(".", "-"); - - try { - consumer.subscribe(Collections.singletonList(kafkaTopic)); - logger.debug("Subscribed to topic pattern: {}", topicPattern); - } catch (Exception e) { - logger.error("Error subscribing to topic {}: {}", topicPattern, e.getMessage()); + Instant notBefore = Instant.now().minus(FANOUT_HISTORY_GRACE); + return register(new Subscription(UUID.randomUUID().toString(), topicPattern, handler, null, notBefore)); + } + + @Override + public String subscribeQueue(String topicPattern, String queueName, Consumer handler) { + if (queueName == null || queueName.isBlank()) { + throw new IllegalArgumentException("queueName must not be blank"); } - - return subscriptionId; + return register(new Subscription(UUID.randomUUID().toString(), topicPattern, handler, queueName, null)); + } + + private String register(Subscription subscription) { + synchronized (lifecycle) { + // Create the worker first: if its consumer cannot be built this throws and nothing is registered. + PollWorker worker = running ? workerFor(subscription) : null; + subscriptions.put(subscription.id, subscription); + if (worker != null) { + worker.enqueue(new Change(subscription, true)); + } + } + logger.debug("Registered subscription {} for pattern {} (queue {})", + subscription.id, subscription.topicPattern, subscription.queueName); + return subscription.id; } @Override public void unsubscribe(String subscriptionId) { - subscriptions.remove(subscriptionId); + synchronized (lifecycle) { + Subscription subscription = subscriptions.remove(subscriptionId); + if (subscription != null && running) { + PollWorker worker = workers.get(subscription.workerKey()); + if (worker != null) { + worker.enqueue(new Change(subscription, false)); + } + } + } logger.debug("Removed subscription: {}", subscriptionId); } - @Override - public CompletableFuture start() { - if (running) { - return CompletableFuture.completedFuture(null); + /** Must be called while holding {@code lifecycle} and while the broker is running. */ + private PollWorker workerFor(Subscription subscription) { + return worker(subscription.workerKey(), + subscription.queueName == null + ? groupId + ".fanout." + instanceId + : groupId + ".queue." + subscription.queueName, + subscription.queueName != null); + } + + /** Returns the live worker for {@code key}, creating it and waiting for its consumer. Holds {@code lifecycle}. */ + private PollWorker worker(String key, String consumerGroup, boolean queue) { + PollWorker existing = workers.get(key); + if (existing != null) { + return existing; } + PollWorker w = new PollWorker(key, consumerGroup, queue); + workers.put(key, w); + w.thread.start(); + try { + w.ready.get(CONSUMER_CREATE_TIMEOUT_S, TimeUnit.SECONDS); + } catch (ExecutionException e) { + workers.remove(key, w); + throw new IllegalStateException("Could not create the Kafka consumer for " + key + ": " + + e.getCause().getMessage(), e.getCause()); + } catch (TimeoutException e) { + w.requestStop(); + workers.remove(key, w); + throw new IllegalStateException("Timed out creating the Kafka consumer for " + key, e); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + w.requestStop(); + workers.remove(key, w); + throw new IllegalStateException("Interrupted while creating the Kafka consumer for " + key, e); + } + return w; + } - return CompletableFuture.runAsync(() -> { + @Override + public CompletableFuture start() { + synchronized (lifecycle) { + if (running) { + return CompletableFuture.completedFuture(null); + } try { - // Create producer - Properties producerProps = new Properties(); - producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); - producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); - producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); - producerProps.put(ProducerConfig.ACKS_CONFIG, "all"); - producerProps.put(ProducerConfig.RETRIES_CONFIG, 3); - - producer = new KafkaProducer<>(producerProps); - logger.info("Kafka producer created"); - - // Create consumer - Properties consumerProps = new Properties(); - consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); - consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, groupId); - consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); - consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); - consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); - consumerProps.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true); - - consumer = new KafkaConsumer<>(consumerProps); - logger.info("Kafka consumer created with group: {}", groupId); - - // Start consumer thread + producer = new KafkaProducer<>(producerProperties()); executor = Executors.newVirtualThreadPerTaskExecutor(); + worker(FANOUT_KEY, groupId + ".fanout." + instanceId, false); + for (Subscription s : subscriptions.values()) { + workerFor(s).enqueue(new Change(s, true)); + } running = true; - - executor.submit(this::consumerLoop); - logger.info("KafkaEventBroker started for instance: {}", instanceId); - - } catch (Exception e) { + logger.info("KafkaEventBroker started for instance {} ({} subscriptions)", + instanceId, subscriptions.size()); + return CompletableFuture.completedFuture(null); + } catch (RuntimeException e) { + running = false; + shutdownLocked(); logger.error("Failed to start KafkaEventBroker: {}", e.getMessage(), e); - throw new RuntimeException("Failed to start Kafka broker", e); + return CompletableFuture.failedFuture(new RuntimeException("Failed to start Kafka broker", e)); } - }); + } } @Override public CompletableFuture stop() { - if (!running) { - return CompletableFuture.completedFuture(null); + synchronized (lifecycle) { + if (!running) { + return CompletableFuture.completedFuture(null); + } + running = false; + shutdownLocked(); + logger.info("KafkaEventBroker stopped for instance {}", instanceId); + } + return CompletableFuture.completedFuture(null); + } + + /** Stops every worker, the executor and the producer. Must hold {@code lifecycle}. Safe on a half-started broker. */ + private void shutdownLocked() { + List stopped = new ArrayList<>(workers.values()); + workers.clear(); + for (PollWorker w : stopped) { + w.requestStop(); + } + for (PollWorker w : stopped) { + w.awaitTermination(STOP_JOIN_MS); } - return CompletableFuture.runAsync(() -> { + ExecutorService ex = executor; + executor = null; + if (ex != null) { + ex.shutdown(); try { - running = false; - - if (consumer != null) { - consumer.close(); + if (!ex.awaitTermination(5, TimeUnit.SECONDS)) { + ex.shutdownNow(); } - - if (producer != null) { - producer.close(); - } - - if (executor != null) { - executor.shutdown(); - if (!executor.awaitTermination(5, TimeUnit.SECONDS)) { - executor.shutdownNow(); - } - } - - logger.info("KafkaEventBroker stopped"); - - } catch (Exception e) { - logger.error("Error stopping KafkaEventBroker: {}", e.getMessage(), e); + } catch (InterruptedException e) { + ex.shutdownNow(); + Thread.currentThread().interrupt(); + } + } + + KafkaProducer p = producer; + producer = null; + if (p != null) { + try { + p.close(Duration.ofSeconds(5)); + } catch (RuntimeException e) { + logger.warn("Error closing Kafka producer: {}", e.getMessage()); } - }); + } + } + + /** Test hook: number of live poll workers. */ + int workerCount() { + return workers.size(); } @Override @@ -206,53 +780,61 @@ public String getBrokerType() { } /** - * Consumer loop for processing messages. + * Test hook: waits until the queue consumer for {@code queueName} on this instance has at least one + * partition assigned. Returns false if that does not happen within the timeout. */ - private void consumerLoop() { - while (running) { - try { - var records = consumer.poll(java.time.Duration.ofSeconds(1)); - - records.forEach(record -> { - try { - Event event = deserializeEvent(record.value()); - - // Route to matching subscribers - for (Subscription sub : subscriptions.values()) { - if (event.matchesTopic(sub.topicPattern)) { - try { - executor.submit(() -> sub.handler.accept(event)); - } catch (Exception e) { - logger.error("Error delivering event: {}", e.getMessage()); - } - } - } - } catch (Exception e) { - logger.error("Error processing message: {}", e.getMessage(), e); - } - }); - - } catch (Exception e) { - logger.error("Error in consumer loop: {}", e.getMessage()); + boolean awaitQueueAssignment(String queueName, Duration timeout) throws InterruptedException { + long deadline = System.nanoTime() + timeout.toNanos(); + while (System.nanoTime() < deadline) { + PollWorker w = workers.get(QUEUE_KEY_PREFIX + queueName); + if (w != null && w.assignedPartitions > 0) { + return true; } + Thread.sleep(50); } + return false; } - /** - * Serializes an Event to JSON string. - */ - private String serializeEvent(Event event) { - // Use CloudEvents JSON format - CloudEvent ce = event.getCloudEvent(); - return ce.toString(); + private Properties producerProperties() { + Properties props = new Properties(); + props.put(ProducerConfig.ACKS_CONFIG, "all"); + props.put(ProducerConfig.RETRIES_CONFIG, 3); + props.putAll(extraProperties); + props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); + props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + return props; } - /** - * Deserializes a JSON string to an Event. - */ - private Event deserializeEvent(String json) { - // Parse CloudEvents JSON and reconstruct Event - // This is a simplified version - full implementation would use proper JSON parsing - return Event.create("amcp.event", json); + private Properties consumerProperties(String consumerGroup) { + Properties props = new Properties(); + props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); + // Topics created after a pattern subscription are discovered on the next metadata refresh, which happens + // at most every metadata.max.age.ms; the group rebalance that follows adds a little more. + props.put(ConsumerConfig.METADATA_MAX_AGE_CONFIG, METADATA_MAX_AGE_MS); + props.putAll(extraProperties); + // Offsets are committed by the poll thread after the handlers of a batch finished (at-least-once). + // Auto-commit would commit on poll, before the asynchronous handlers run, so it is never allowed. + props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); + props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); + props.put(ConsumerConfig.GROUP_ID_CONFIG, consumerGroup); + props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); + props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); + return props; + } + + /** Serializes an Event as a CloudEvents structured JSON document. */ + static String encode(Event event) { + byte[] bytes = JSON_FORMAT.serialize(event.getCloudEvent()); + return new String(bytes, StandardCharsets.UTF_8); + } + + /** Parses a CloudEvents structured JSON document into an Event. Throws on malformed input. */ + static Event decode(String json) { + if (json == null) { + throw new IllegalArgumentException("null record value"); + } + CloudEvent ce = JSON_FORMAT.deserialize(json.getBytes(StandardCharsets.UTF_8)); + return new Event(ce); } } diff --git a/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/TopicPatterns.java b/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/TopicPatterns.java new file mode 100644 index 0000000..7afc22c --- /dev/null +++ b/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/TopicPatterns.java @@ -0,0 +1,56 @@ +package io.amcp.broker.kafka; + +import java.util.Collection; +import java.util.StringJoiner; +import java.util.regex.Pattern; + +/** + * Translates AMCP topic patterns into Kafka topic regexes. + * + *

    + *
  • {@code a.b.c} (no wildcard) matches exactly the topic {@code a.b.c}.
  • + *
  • {@code p.*} matches {@code p.X} where X is exactly one non-empty segment.
  • + *
  • {@code p.**} matches {@code p} itself and {@code p.}, never {@code px}.
  • + *
+ * + *

A wildcard never matches an empty segment: {@code p.x.} and {@code p..x} match {@code p.*} / {@code p.**} not at + * all (they match only a pattern equal to the topic). {@code io.amcp.core.Event#matchesTopic} follows the same rule. + * + *

Kafka matches a subscription pattern against the whole topic name, so the regexes produced + * here are anchored by the caller's {@code matches()} call. + */ +final class TopicPatterns { + + private TopicPatterns() { + } + + /** + * Returns the regex (not compiled) for one AMCP topic pattern. + */ + static String toRegex(String amcpPattern) { + if (amcpPattern.endsWith(".**")) { + String prefix = amcpPattern.substring(0, amcpPattern.length() - 3); + return Pattern.quote(prefix) + "(\\.[^.]+(\\.[^.]+)*)?"; + } + if (amcpPattern.endsWith(".*")) { + String prefix = amcpPattern.substring(0, amcpPattern.length() - 2); + return Pattern.quote(prefix) + "\\.[^.]+"; + } + return Pattern.quote(amcpPattern); + } + + /** + * Returns one compiled pattern that matches any topic matched by any of the given AMCP patterns, + * or {@code null} when the collection is empty. + */ + static Pattern union(Collection amcpPatterns) { + if (amcpPatterns.isEmpty()) { + return null; + } + StringJoiner joiner = new StringJoiner("|", "(?:", ")"); + for (String p : amcpPatterns) { + joiner.add(toRegex(p)); + } + return Pattern.compile(joiner.toString()); + } +} diff --git a/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/FakeConsumer.java b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/FakeConsumer.java new file mode 100644 index 0000000..d0acd76 --- /dev/null +++ b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/FakeConsumer.java @@ -0,0 +1,114 @@ +package io.amcp.broker.kafka; + +import org.apache.kafka.clients.consumer.ConsumerRebalanceListener; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.clients.consumer.MockConsumer; +import org.apache.kafka.clients.consumer.OffsetAndMetadata; +import org.apache.kafka.clients.consumer.OffsetResetStrategy; +import org.apache.kafka.common.TopicPartition; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.regex.Pattern; + +/** + * A deterministic in-memory consumer for unit tests. It behaves like a partitioned log: records stay in the log, a + * seek replays them, and every commit is recorded. subscribe(Pattern) assigns every known partition without checking + * the pattern, which is how a group member can own a partition for a topic it never subscribed to. + */ +class FakeConsumer extends MockConsumer { + final List> log = Collections.synchronizedList(new ArrayList<>()); + final List> commits = Collections.synchronizedList(new ArrayList<>()); + final AtomicInteger subscribeAttempts = new AtomicInteger(); + volatile int failSubscribeTimes = 0; + private final Map nextOffset = new HashMap<>(); + private final Set known = new HashSet<>(); + private boolean subscribed = false; + + FakeConsumer() { + super(OffsetResetStrategy.EARLIEST); + } + + synchronized void append(String topic, String value) { + long offset = nextOffset.merge(topic, 1L, Long::sum) - 1; + ConsumerRecord r = new ConsumerRecord<>(topic, 0, offset, "k", value); + log.add(r); + known.add(new TopicPartition(topic, 0)); + if (subscribed) { + assignKnown(); + addRecord(r); + } + } + + private void assignKnown() { + assign(new HashSet<>(known)); + Map begin = new HashMap<>(); + for (TopicPartition tp : known) { + begin.put(tp, 0L); + } + updateBeginningOffsets(begin); + } + + @Override + public synchronized void subscribe(Pattern pattern, ConsumerRebalanceListener listener) { + subscribeAttempts.incrementAndGet(); + if (failSubscribeTimes > 0) { + failSubscribeTimes--; + throw new IllegalStateException("simulated subscribe failure"); + } + if (!subscribed) { + subscribed = true; + assignKnown(); + for (ConsumerRecord r : new ArrayList<>(log)) { + addRecord(r); + } + } + } + + @Override + public synchronized void subscribe(Pattern pattern) { + subscribe(pattern, new ConsumerRebalanceListener() { + @Override + public void onPartitionsRevoked(java.util.Collection partitions) { + } + + @Override + public void onPartitionsAssigned(java.util.Collection partitions) { + } + }); + } + + @Override + public synchronized ConsumerRecords poll(Duration timeout) { + try { + Thread.sleep(10); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return super.poll(timeout); + } + + @Override + public synchronized void seek(TopicPartition partition, long offset) { + super.seek(partition, offset); + for (ConsumerRecord r : new ArrayList<>(log)) { + if (r.topic().equals(partition.topic()) && r.offset() >= offset) { + addRecord(r); + } + } + } + + @Override + public synchronized void commitSync(Map offsets) { + commits.add(new HashMap<>(offsets)); + super.commitSync(offsets); + } +} diff --git a/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerIT.java b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerIT.java new file mode 100644 index 0000000..147bd2b --- /dev/null +++ b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerIT.java @@ -0,0 +1,240 @@ +package io.amcp.broker.kafka; + +import io.amcp.core.Event; +import org.apache.kafka.clients.admin.Admin; +import org.apache.kafka.clients.admin.NewTopic; +import org.apache.kafka.common.errors.TopicExistsException; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; + +import java.net.InetSocketAddress; +import java.net.Socket; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assumptions.assumeTrue; + +/** + * Integration tests against a real Kafka broker. + * + *

Set {@code AMCP_KAFKA_BOOTSTRAP} (for example {@code 127.0.0.1:9092}) to run them; without it they are skipped, + * so a plain {@code mvn test} stays green on machines without Kafka. Every test uses its own random topic prefix + * and consumer groups, so tests do not see each other's events. + */ +class KafkaEventBrokerIT { + + private static String bootstrap; + private final List brokers = new ArrayList<>(); + private final String prefix = "amcpit" + UUID.randomUUID().toString().replace("-", "").substring(0, 10); + + @BeforeAll + static void requireKafka() { + bootstrap = System.getenv("AMCP_KAFKA_BOOTSTRAP"); + assumeTrue(bootstrap != null && !bootstrap.isBlank(), "AMCP_KAFKA_BOOTSTRAP not set: Kafka integration tests skipped"); + String[] hp = bootstrap.split(",")[0].split(":"); + try (Socket s = new Socket()) { + s.connect(new InetSocketAddress(hp[0], Integer.parseInt(hp[1])), 2000); + } catch (Exception e) { + assumeTrue(false, "Kafka not reachable at " + bootstrap); + } + } + + @AfterEach + void stopBrokers() throws Exception { + for (KafkaEventBroker b : brokers) { + b.stop().get(10, TimeUnit.SECONDS); + } + } + + private KafkaEventBroker broker(String group, String instance) throws Exception { + KafkaEventBroker b = new KafkaEventBroker(bootstrap, prefix + "-" + group, instance); + b.start().get(20, TimeUnit.SECONDS); + brokers.add(b); + return b; + } + + private static List sink() { + return Collections.synchronizedList(new ArrayList<>()); + } + + private static void await(List got, int n, long seconds) throws InterruptedException { + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(seconds); + while (got.size() < n && System.nanoTime() < end) { + Thread.sleep(50); + } + } + + private static void createTopic(String bootstrapServers, String topic, int partitions) throws Exception { + Map config = Map.of("bootstrap.servers", bootstrapServers); + try (Admin admin = Admin.create(config)) { + admin.createTopics(List.of(new NewTopic(topic, partitions, (short) 1))).all().get(30, TimeUnit.SECONDS); + } catch (ExecutionException e) { + if (!(e.getCause() instanceof TopicExistsException)) { + throw e; + } + } + } + + @Test + void roundTripKeepsTopicAndPayload() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List got = sink(); + String topic = prefix + ".bee.heartbeat"; + b.subscribe(topic, got::add); + b.publish(Event.create(topic, Map.of("bee", "b1", "load", 2))).get(10, TimeUnit.SECONDS); + await(got, 1, 30); + assertEquals(1, got.size(), "event delivered once"); + assertEquals(topic, got.get(0).getTopic(), "topic survives the round trip"); + @SuppressWarnings("unchecked") + Map payload = got.get(0).getPayload(Map.class); + assertEquals("b1", payload.get("bee")); + assertEquals(2, ((Number) payload.get("load")).intValue()); + } + + @Test + void severalSubscriptionsOnOneBrokerAllReceive() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List a = sink(); + List c = sink(); + b.subscribe(prefix + ".task.request", a::add); + b.subscribe(prefix + ".task.result", c::add); + b.publish(Event.create(prefix + ".task.request", Map.of("id", 1))).get(10, TimeUnit.SECONDS); + b.publish(Event.create(prefix + ".task.result", Map.of("id", 1))).get(10, TimeUnit.SECONDS); + await(a, 1, 30); + await(c, 1, 30); + assertEquals(1, a.size(), "first subscription still receives after a second subscribe"); + assertEquals(1, c.size(), "second subscription receives"); + } + + @Test + void singleLevelWildcardMatchesOneLevelOnly() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List got = sink(); + b.subscribe(prefix + ".bee.*", got::add); + b.publish(Event.create(prefix + ".bee.alpha", Map.of("n", 1))).get(10, TimeUnit.SECONDS); + b.publish(Event.create(prefix + ".bee.beta", Map.of("n", 2))).get(10, TimeUnit.SECONDS); + b.publish(Event.create(prefix + ".bee.alpha.deep", Map.of("n", 3))).get(10, TimeUnit.SECONDS); + await(got, 2, 30); + Thread.sleep(2000); // give a wrongly matched third event time to arrive + List topics = got.stream().map(Event::getTopic).sorted().toList(); + assertEquals(List.of(prefix + ".bee.alpha", prefix + ".bee.beta"), topics); + } + + @Test + void fanOutReachesEveryBrokerInstance() throws Exception { + KafkaEventBroker b1 = broker("shared", "i1"); + KafkaEventBroker b2 = broker("shared", "i2"); + List g1 = sink(); + List g2 = sink(); + String topic = prefix + ".host.announce"; + b1.subscribe(topic, g1::add); + b2.subscribe(topic, g2::add); + b1.publish(Event.create(topic, Map.of("hello", "mesh"))).get(10, TimeUnit.SECONDS); + await(g1, 1, 30); + await(g2, 1, 30); + assertEquals(1, g1.size(), "instance 1 receives a broadcast"); + assertEquals(1, g2.size(), "instance 2 receives the same broadcast"); + } + + @Test + void publishCompletesOnlyAfterTheBrokerAcknowledges() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + var f = b.publish(Event.create(prefix + ".ack", Map.of("x", 1))); + f.get(10, TimeUnit.SECONDS); + assertTrue(f.isDone() && !f.isCompletedExceptionally()); + } + + @Test + void queueDeliversEachEventOnceAcrossInstances() throws Exception { + String topic = prefix + ".jobs.run"; + createTopic(bootstrap, topic, 2); + KafkaEventBroker b1 = broker("qg", "i1"); + KafkaEventBroker b2 = broker("qg", "i2"); + List ids1 = Collections.synchronizedList(new ArrayList<>()); + List ids2 = Collections.synchronizedList(new ArrayList<>()); + b1.subscribeQueue(topic, "workers", e -> ids1.add(e.getId())); + b2.subscribeQueue(topic, "workers", e -> ids2.add(e.getId())); + assertTrue(b1.awaitQueueAssignment("workers", Duration.ofSeconds(60)), "instance 1 gets a partition"); + assertTrue(b2.awaitQueueAssignment("workers", Duration.ofSeconds(60)), "instance 2 gets a partition"); + + List> sent = new ArrayList<>(); + for (int i = 0; i < 50; i++) { + sent.add(b1.publish(Event.create(topic, Map.of("n", i)))); + } + CompletableFuture.allOf(sent.toArray(new CompletableFuture[0])).get(60, TimeUnit.SECONDS); + + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(60); + while (ids1.size() + ids2.size() < 50 && System.nanoTime() < end) { + Thread.sleep(50); + } + Thread.sleep(1000); // allow a duplicate delivery to show up before counting + Set union = new HashSet<>(ids1); + union.addAll(ids2); + assertEquals(50, ids1.size() + ids2.size(), "every event delivered exactly once in total"); + assertEquals(50, union.size(), "no event id delivered twice"); + assertTrue(!ids1.isEmpty(), "instance 1 received at least one event"); + assertTrue(!ids2.isEmpty(), "instance 2 received at least one event"); + } + + @Test + void topicCreatedAfterSubscribeIsDelivered() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List got = sink(); + String topic = prefix + ".late.topic"; + b.subscribe(prefix + ".late.*", got::add); + Thread.sleep(3000); // the pattern subscription is active before the topic exists + createTopic(bootstrap, topic, 1); + Thread.sleep(3000); // metadata refresh runs every 1 s + b.publish(Event.create(topic, Map.of("late", true))).get(20, TimeUnit.SECONDS); + await(got, 1, 60); + assertEquals(1, got.size(), "event on a topic created after subscribe is delivered"); + assertEquals(topic, got.get(0).getTopic()); + } + + @Test + void newInstanceDoesNotReplayOldHistory() throws Exception { + String topic = prefix + ".hist.event"; + KafkaEventBroker publisher = broker("hg", "old"); + Event old = Event.create(topic, Map.of("when", "old")); + publisher.publish(old).get(20, TimeUnit.SECONDS); + Thread.sleep(7000); + + KafkaEventBroker fresh = broker("hg", "new"); + List got = sink(); + fresh.subscribe(topic, got::add); + Event recent = Event.create(topic, Map.of("when", "new")); + fresh.publish(recent).get(20, TimeUnit.SECONDS); + await(got, 1, 60); + Thread.sleep(2000); // the old event would have arrived before the new one if it were replayed + + List ids = got.stream().map(Event::getId).toList(); + assertEquals(List.of(recent.getId()), ids, "history older than 5 s before subscribe is not replayed"); + } + + @Test + void doubleStarRespectsSegmentBoundary() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List got = sink(); + b.subscribe(prefix + ".p.**", got::add); + for (String t : List.of(prefix + ".p", prefix + ".p.x", prefix + ".px", prefix + ".p.x.y")) { + b.publish(Event.create(t, Map.of("t", t))).get(10, TimeUnit.SECONDS); + } + await(got, 3, 60); + Thread.sleep(2000); // give a wrongly matched "px" event time to arrive + List topics = got.stream().map(Event::getTopic).sorted().toList(); + assertEquals(List.of(prefix + ".p", prefix + ".p.x", prefix + ".p.x.y"), topics, + "p.** matches p and p., never px"); + } +} diff --git a/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerUnitTest.java b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerUnitTest.java new file mode 100644 index 0000000..606bd85 --- /dev/null +++ b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerUnitTest.java @@ -0,0 +1,225 @@ +package io.amcp.broker.kafka; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import io.amcp.core.Event; +import org.apache.kafka.common.TopicPartition; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.BooleanSupplier; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** Deterministic unit tests for the Kafka broker's worker logic. No Kafka needed: consumers are in-memory fakes. */ +class KafkaEventBrokerUnitTest { + + private KafkaEventBroker broker; + private ListAppender logs; + private Logger brokerLogger; + + @BeforeEach + void attachLog() { + brokerLogger = (Logger) LoggerFactory.getLogger(KafkaEventBroker.class); + logs = new ListAppender<>(); + logs.start(); + brokerLogger.addAppender(logs); + } + + @AfterEach + void cleanup() throws Exception { + brokerLogger.detachAppender(logs); + if (broker != null) { + broker.stop().get(15, TimeUnit.SECONDS); + } + } + + /** Broker whose consumers are fakes: fan-out workers get {@code fanout}, queue workers get {@code queue}. */ + private KafkaEventBroker brokerWith(FakeConsumer fanout, FakeConsumer queue) { + KafkaEventBroker b = new KafkaEventBroker("127.0.0.1:1", "g", "i1"); + b.consumerFactory = props -> String.valueOf(props.get("group.id")).contains(".queue.") ? queue : fanout; + b.baseRetryBackoffMs = 20; + b.batchTimeout = Duration.ofSeconds(5); + broker = b; + return b; + } + + private static void awaitTrue(BooleanSupplier c, long seconds, String what) throws InterruptedException { + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(seconds); + while (!c.getAsBoolean() && System.nanoTime() < end) { + Thread.sleep(10); + } + assertTrue(c.getAsBoolean(), "timed out waiting for: " + what); + } + + private static String json(String topic) { + return KafkaEventBroker.encode(Event.create(topic, Map.of("n", 1))); + } + + private long count(Level level, String fragment) { + synchronized (logs.list) { + return logs.list.stream() + .filter(e -> e.getLevel() == level && e.getFormattedMessage().contains(fragment)).count(); + } + } + + private static String chain(Throwable t) { + StringBuilder sb = new StringBuilder(); + for (; t != null; t = t.getCause()) { + sb.append(t.getMessage()).append(" | "); + } + return sb.toString(); + } + + // ---- item 1 ------------------------------------------------------------------------------------------- + + @Test + void consumerConstructionFailureSurfacesInStart() { + Properties bad = new Properties(); + bad.put("max.poll.records", "abc"); // consumer-only property, invalid value + broker = new KafkaEventBroker("127.0.0.1:1", "g", "i1", bad); + ExecutionException e = assertThrows(ExecutionException.class, + () -> broker.start().get(30, TimeUnit.SECONDS), "start() must fail, not report success"); + assertFalse(broker.isRunning()); + assertEquals(0, broker.workerCount(), "no dead worker may stay in the map"); + assertTrue(chain(e).contains("max.poll.records"), "the cause must name the bad property: " + chain(e)); + } + + @Test + void queueConsumerFailureSurfacesInSubscribeAndLeavesNoDeadWorker() throws Exception { + FakeConsumer fanout = new FakeConsumer(); + broker = new KafkaEventBroker("127.0.0.1:1", "g", "i1"); + broker.consumerFactory = props -> { + if (String.valueOf(props.get("group.id")).contains(".queue.")) { + throw new IllegalArgumentException("queue consumer cannot be built"); + } + return fanout; + }; + broker.start().get(30, TimeUnit.SECONDS); + int before = broker.workerCount(); + IllegalStateException ise = assertThrows(IllegalStateException.class, + () -> broker.subscribeQueue("t.a", "q", e -> { })); + assertInstanceOf(IllegalArgumentException.class, ise.getCause()); + assertEquals(before, broker.workerCount(), "the failed queue worker must not stay in the map"); + } + + // ---- item 2 ------------------------------------------------------------------------------------------- + + @Test + void failedSubscriptionUpdateIsRetriedAndLoggedOnce() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.failSubscribeTimes = 2; + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + List got = Collections.synchronizedList(new ArrayList<>()); + b.subscribeQueue("t.a", "q", got::add); + awaitTrue(() -> got.size() == 1, 8, "event delivered after the subscribe failures are retried"); + assertTrue(queue.subscribeAttempts.get() >= 3, "attempts: " + queue.subscribeAttempts.get()); + assertEquals(1, count(Level.ERROR, "could not update its subscription"), + "one failure episode is logged once, with the cause"); + } + + // ---- item 3 ------------------------------------------------------------------------------------------- + + @Test + void handlerFailureRedeliversAndCommitsOnlyAfterSuccess() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + AtomicInteger calls = new AtomicInteger(); + List ids = Collections.synchronizedList(new ArrayList<>()); + b.subscribeQueue("t.a", "q", e -> { + ids.add(e.getId()); + if (calls.incrementAndGet() == 1) { + throw new IllegalStateException("handler killed mid-batch"); + } + }); + awaitTrue(() -> calls.get() >= 2, 8, "redelivery after the handler threw"); + awaitTrue(() -> queue.commits.size() >= 1, 8, "commit after the successful redelivery"); + assertEquals(2, calls.get()); + assertEquals(ids.get(0), ids.get(1), "the same event is redelivered"); + assertEquals(1, queue.commits.size(), "nothing was committed for the failed attempt"); + assertEquals(1L, queue.commits.get(0).get(new TopicPartition("t.a", 0)).offset()); + } + + @Test + void offsetsAreNotCommittedWhileAHandlerIsStillRunning() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + CountDownLatch entered = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + b.subscribeQueue("t.a", "q", e -> { + entered.countDown(); + try { + release.await(20, TimeUnit.SECONDS); + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + } + }); + assertTrue(entered.await(8, TimeUnit.SECONDS)); + Thread.sleep(500); + assertTrue(queue.commits.isEmpty(), "must not commit before the handler finished (at-most-once bug)"); + release.countDown(); + awaitTrue(() -> queue.commits.size() == 1, 8, "commit after the handler completed"); + } + + @Test + void poisonRecordIsGivenUpAfterBoundedAttemptsAndCommitted() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + AtomicInteger calls = new AtomicInteger(); + b.subscribeQueue("t.a", "q", e -> { + calls.incrementAndGet(); + throw new IllegalStateException("always fails"); + }); + awaitTrue(() -> queue.commits.size() == 1, 10, "give-up commit"); + Thread.sleep(300); + assertEquals(KafkaEventBroker.MAX_DELIVERY_ATTEMPTS, calls.get(), "bounded redelivery, no endless loop"); + assertEquals(1, count(Level.ERROR, "Giving up")); + } + + // ---- item 4 ------------------------------------------------------------------------------------------- + + @Test + void recordWithoutLocalMatchWarnsOncePerTopicAndQueue() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.append("t.other", json("t.other")); + queue.append("t.other", json("t.other")); + queue.append("t.third", json("t.third")); + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + List got = Collections.synchronizedList(new ArrayList<>()); + b.subscribeQueue("t.a", "workers", got::add); + awaitTrue(() -> got.size() == 1, 8, "the matching event is delivered"); + awaitTrue(() -> queue.commits.size() >= 1, 8, "batch committed"); + assertEquals(1, count(Level.WARN, "t.other"), "one WARN for (t.other, workers), not one per record"); + assertEquals(1, count(Level.WARN, "t.third")); + assertTrue(logs.list.stream().anyMatch(e -> e.getLevel() == Level.WARN + && e.getFormattedMessage().contains("workers") && e.getFormattedMessage().contains("t.other"))); + } +} diff --git a/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/TopicPatternsTest.java b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/TopicPatternsTest.java new file mode 100644 index 0000000..d34656b --- /dev/null +++ b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/TopicPatternsTest.java @@ -0,0 +1,111 @@ +package io.amcp.broker.kafka; + +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.regex.Pattern; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** Unit tests for the AMCP-pattern to Kafka-regex translation. No Kafka needed. */ +class TopicPatternsTest { + + private static boolean matches(String amcpPattern, String topic) { + return Pattern.compile(TopicPatterns.toRegex(amcpPattern)).matcher(topic).matches(); + } + + @Test + void exactPatternMatchesOnlyThatTopicAndTreatsDotsLiterally() { + assertTrue(matches("a.b.c", "a.b.c")); + assertFalse(matches("a.b.c", "a.b.cd")); + assertFalse(matches("a.b.c", "aXb.c"), "dots are literal, not regex wildcards"); + assertFalse(matches("a.b.c", "a.b.c.d")); + } + + @Test + void singleStarMatchesExactlyOneSegment() { + assertTrue(matches("p.*", "p.x")); + assertFalse(matches("p.*", "p")); + assertFalse(matches("p.*", "p.x.y")); + assertFalse(matches("p.*", "px.x")); + assertFalse(matches("p.*", "p."), "an empty segment is not a match"); + } + + @Test + void doubleStarMatchesPrefixAndAnyDepthButRespectsSegmentBoundary() { + assertTrue(matches("p.**", "p")); + assertTrue(matches("p.**", "p.x")); + assertTrue(matches("p.**", "p.x.y.z")); + assertFalse(matches("p.**", "px"), "p.** must not match a longer first segment"); + assertFalse(matches("p.**", "phive.x")); + assertFalse(matches("p.**", "x.p")); + } + + @Test + void unionMatchesAnyMemberAndNothingElse() { + Pattern union = TopicPatterns.union(List.of("a.b", "c.*", "d.**")); + assertTrue(union.matcher("a.b").matches()); + assertTrue(union.matcher("c.x").matches()); + assertTrue(union.matcher("d").matches()); + assertTrue(union.matcher("d.e.f").matches()); + assertFalse(union.matcher("c").matches()); + assertFalse(union.matcher("a.c").matches()); + assertFalse(union.matcher("dx").matches()); + } + + @Test + void unionOfNothingIsNull() { + assertNull(TopicPatterns.union(Set.of())); + } + + @Test + void unionAcceptsSortedSetOfPatterns() { + Set wanted = new TreeSet<>(List.of("z.*", "a.**")); + Pattern union = TopicPatterns.union(wanted); + assertTrue(union.matcher("a.q").matches()); + assertTrue(union.matcher("z.q").matches()); + } + + /** Pattern, topic, expected. The SAME table runs through Event.matchesTopic and TopicPatterns.toRegex. */ + private static final String[][] AGREEMENT_TABLE = { + {"a.b.c", "a.b.c", "true"}, + {"a.b.c", "a.b.cd", "false"}, + {"p.*", "p.x", "true"}, + {"p.*", "p", "false"}, + {"p.*", "p.x.y", "false"}, + {"p.*", "px.x", "false"}, + {"p.*", "p.", "false"}, + {"p.*", "p.x.", "false"}, + {"p.*", "p..", "false"}, + {"a.b.*", "a.b.x", "true"}, + {"a.b.*", "a.b.x.", "false"}, + {"p.**", "p", "true"}, + {"p.**", "p.x", "true"}, + {"p.**", "p.x.y", "true"}, + {"p.**", "px", "false"}, + {"p.**", "p.", "false"}, + {"p.**", "p.x.", "false"}, + {"p.**", "p..x", "false"}, + {"p.**", "p.x..y", "false"}, + {"p.", "p.", "true"}, + {"p.", "p", "false"}, + {"p.x.", "p.x.", "true"}, + }; + + @Test + void eventMatchesTopicAndTopicPatternsAgreeOnEveryCase() { + for (String[] row : AGREEMENT_TABLE) { + boolean expected = Boolean.parseBoolean(row[2]); + boolean viaEvent = io.amcp.core.Event.create(row[1], java.util.Map.of("n", 1)).matchesTopic(row[0]); + boolean viaRegex = matches(row[0], row[1]); + org.junit.jupiter.api.Assertions.assertEquals(expected, viaEvent, + "Event.matchesTopic(" + row[0] + ") on topic '" + row[1] + "'"); + org.junit.jupiter.api.Assertions.assertEquals(expected, viaRegex, + "TopicPatterns.toRegex(" + row[0] + ") on topic '" + row[1] + "'"); + } + } +} diff --git a/amcp-core/pom.xml b/amcp-core/pom.xml index 9b06407..bd654ab 100644 --- a/amcp-core/pom.xml +++ b/amcp-core/pom.xml @@ -70,6 +70,9 @@ org.apache.maven.plugins maven-surefire-plugin + + -Xmx512m + diff --git a/amcp-core/src/main/java/io/amcp/core/Event.java b/amcp-core/src/main/java/io/amcp/core/Event.java index 340a44a..df18ef1 100644 --- a/amcp-core/src/main/java/io/amcp/core/Event.java +++ b/amcp-core/src/main/java/io/amcp/core/Event.java @@ -162,30 +162,36 @@ public boolean matchesTopic(String pattern) { return true; } - // Handle wildcards + // Handle wildcards. A wildcard never matches an empty segment, so topics with a trailing dot + // ("p.x.") or a doubled dot ("p..x") match only by exact equality. This is the same rule as + // io.amcp.broker.kafka.TopicPatterns.toRegex. if (pattern.endsWith(".**")) { + // "p.**" matches "p" itself and "p.", never "px". String prefix = pattern.substring(0, pattern.length() - 3); - return topic.startsWith(prefix); + if (topic.equals(prefix)) { + return true; + } + return topic.startsWith(prefix + ".") && nonEmptySegments(topic.substring(prefix.length() + 1)); } - + if (pattern.endsWith(".*")) { + // "p.*" matches "p.". String prefix = pattern.substring(0, pattern.length() - 2); - String[] topicParts = topic.split("\\."); - String[] patternParts = prefix.split("\\."); - - if (topicParts.length == patternParts.length + 1) { - for (int i = 0; i < patternParts.length; i++) { - if (!topicParts[i].equals(patternParts[i])) { - return false; - } - } - return true; + if (!topic.startsWith(prefix + ".")) { + return false; } + String rest = topic.substring(prefix.length() + 1); + return !rest.isEmpty() && rest.indexOf('.') < 0; } - + return false; } + /** True when {@code rest} is one or more dot-separated segments and none of them is empty. */ + private static boolean nonEmptySegments(String rest) { + return !rest.isEmpty() && !rest.startsWith(".") && !rest.endsWith(".") && !rest.contains(".."); + } + @Override public String toString() { return String.format("Event{id=%s, topic=%s, source=%s, time=%s}", diff --git a/amcp-core/src/main/java/io/amcp/core/EventBroker.java b/amcp-core/src/main/java/io/amcp/core/EventBroker.java index 2b0437d..80d9568 100644 --- a/amcp-core/src/main/java/io/amcp/core/EventBroker.java +++ b/amcp-core/src/main/java/io/amcp/core/EventBroker.java @@ -26,6 +26,24 @@ public interface EventBroker extends AutoCloseable { */ String subscribe(String topicPattern, Consumer handler); + /** + * Subscribes to events matching the topic pattern as one member of a named queue + * (competing consumers). Each event is delivered to exactly one subscriber that shares + * the same queue name, instead of to every subscriber as {@link #subscribe} does. + * + *

The default implementation ignores {@code queueName} and falls back to + * {@link #subscribe(String, Consumer)}, which means fan-out delivery. Brokers that can + * distribute work across subscribers should override this method. + * + * @param topicPattern the topic pattern (supports wildcards like "hello.*") + * @param queueName the queue name shared by the competing subscribers + * @param handler the event handler + * @return a subscription ID that can be used to unsubscribe + */ + default String subscribeQueue(String topicPattern, String queueName, Consumer handler) { + return subscribe(topicPattern, handler); + } + /** * Unsubscribes from a previously created subscription. * diff --git a/amcp-core/src/main/java/io/amcp/core/InMemoryEventBroker.java b/amcp-core/src/main/java/io/amcp/core/InMemoryEventBroker.java index b71c781..d35180e 100644 --- a/amcp-core/src/main/java/io/amcp/core/InMemoryEventBroker.java +++ b/amcp-core/src/main/java/io/amcp/core/InMemoryEventBroker.java @@ -3,32 +3,43 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; import java.util.Map; import java.util.UUID; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Consumer; /** * In-memory implementation of EventBroker for development and testing. * Events are routed directly within the JVM without external message broker. + * + *

{@link #subscribe} delivers every event to every matching subscriber (fan-out). + * {@link #subscribeQueue} delivers each event to one subscriber per queue name, + * chosen round-robin among the subscribers that share that queue name. */ public class InMemoryEventBroker implements EventBroker { private static final Logger logger = LoggerFactory.getLogger(InMemoryEventBroker.class); - + private final Map subscriptions = new ConcurrentHashMap<>(); - private final ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor(); + private final Map queueCursors = new ConcurrentHashMap<>(); + private volatile ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor(); private volatile boolean running = false; private static class Subscription { final String topicPattern; final Consumer handler; + final String queueName; // null for fan-out subscriptions - Subscription(String topicPattern, Consumer handler) { + Subscription(String topicPattern, Consumer handler, String queueName) { this.topicPattern = topicPattern; this.handler = handler; + this.queueName = queueName; } } @@ -41,29 +52,58 @@ public CompletableFuture publish(Event event) { return CompletableFuture.runAsync(() -> { logger.debug("Publishing event: {}", event.getTopic()); - - // Find matching subscriptions + + // Fan-out subscriptions all receive the event; queue subscriptions are grouped by + // queue name and each group receives it once, round-robin. + Map> queues = new LinkedHashMap<>(); for (Subscription sub : subscriptions.values()) { - if (event.matchesTopic(sub.topicPattern)) { - try { - executor.submit(() -> sub.handler.accept(event)); - } catch (Exception e) { - logger.error("Error delivering event to subscriber: {}", - e.getMessage(), e); - } + if (!event.matchesTopic(sub.topicPattern)) { + continue; } + if (sub.queueName == null) { + deliver(sub, event); + } else { + queues.computeIfAbsent(sub.queueName, k -> new ArrayList<>()).add(sub); + } + } + for (Map.Entry> q : queues.entrySet()) { + List members = q.getValue(); + int index = Math.floorMod( + queueCursors.computeIfAbsent(q.getKey(), k -> new AtomicInteger()).getAndIncrement(), + members.size()); + deliver(members.get(index), event); } }, executor); } + private void deliver(Subscription sub, Event event) { + try { + executor.submit(() -> sub.handler.accept(event)); + } catch (Exception e) { + logger.error("Error delivering event to subscriber: {}", e.getMessage(), e); + } + } + @Override public String subscribe(String topicPattern, Consumer handler) { String subscriptionId = UUID.randomUUID().toString(); - subscriptions.put(subscriptionId, new Subscription(topicPattern, handler)); + subscriptions.put(subscriptionId, new Subscription(topicPattern, handler, null)); logger.debug("Created subscription {} for pattern {}", subscriptionId, topicPattern); return subscriptionId; } + @Override + public String subscribeQueue(String topicPattern, String queueName, Consumer handler) { + if (queueName == null || queueName.isBlank()) { + throw new IllegalArgumentException("queueName must not be blank"); + } + String subscriptionId = UUID.randomUUID().toString(); + subscriptions.put(subscriptionId, new Subscription(topicPattern, handler, queueName)); + logger.debug("Created queue subscription {} for pattern {} in queue {}", + subscriptionId, topicPattern, queueName); + return subscriptionId; + } + @Override public void unsubscribe(String subscriptionId) { subscriptions.remove(subscriptionId); @@ -71,22 +111,26 @@ public void unsubscribe(String subscriptionId) { } @Override - public CompletableFuture start() { + public synchronized CompletableFuture start() { if (running) { return CompletableFuture.completedFuture(null); } - + + // stop() shuts the executor down for good, so a restart needs a fresh one. + if (executor.isShutdown()) { + executor = Executors.newVirtualThreadPerTaskExecutor(); + } running = true; logger.info("InMemoryEventBroker started"); return CompletableFuture.completedFuture(null); } @Override - public CompletableFuture stop() { + public synchronized CompletableFuture stop() { if (!running) { return CompletableFuture.completedFuture(null); } - + running = false; subscriptions.clear(); executor.shutdown(); diff --git a/amcp-core/src/test/java/io/amcp/core/EventTest.java b/amcp-core/src/test/java/io/amcp/core/EventTest.java index 1479c2a..b900691 100644 --- a/amcp-core/src/test/java/io/amcp/core/EventTest.java +++ b/amcp-core/src/test/java/io/amcp/core/EventTest.java @@ -47,6 +47,16 @@ void testWildcardMatching() { assertTrue(event2.matchesTopic("hello.**")); } + @Test + void testDoubleStarRespectsSegmentBoundary() { + assertTrue(Event.create("p", "test").matchesTopic("p.**"), "p.** matches the bare prefix p"); + assertTrue(Event.create("p.x", "test").matchesTopic("p.**")); + assertTrue(Event.create("p.x.y", "test").matchesTopic("p.**")); + assertFalse(Event.create("px", "test").matchesTopic("p.**"), "p.** must not match phive-style prefixes"); + assertFalse(Event.create("phive.x", "test").matchesTopic("p.**")); + assertFalse(Event.create("other.p", "test").matchesTopic("p.**")); + } + @Test void testComplexPayload() { record TestData(String name, int value) {} diff --git a/amcp-core/src/test/java/io/amcp/core/InMemoryEventBrokerTest.java b/amcp-core/src/test/java/io/amcp/core/InMemoryEventBrokerTest.java new file mode 100644 index 0000000..ab4b436 --- /dev/null +++ b/amcp-core/src/test/java/io/amcp/core/InMemoryEventBrokerTest.java @@ -0,0 +1,76 @@ +package io.amcp.core; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class InMemoryEventBrokerTest { + + private InMemoryEventBroker broker; + + @BeforeEach + void start() throws Exception { + broker = new InMemoryEventBroker(); + broker.start().get(5, TimeUnit.SECONDS); + } + + @AfterEach + void stop() throws Exception { + broker.stop().get(5, TimeUnit.SECONDS); + } + + @Test + void queueSubscribersShareEventsRoundRobin() throws Exception { + List a = Collections.synchronizedList(new ArrayList<>()); + List b = Collections.synchronizedList(new ArrayList<>()); + broker.subscribeQueue("jobs.run", "workers", a::add); + broker.subscribeQueue("jobs.run", "workers", b::add); + + for (int i = 0; i < 4; i++) { + broker.publish(Event.create("jobs.run", Integer.valueOf(i))).get(5, TimeUnit.SECONDS); + } + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (a.size() + b.size() < 4 && System.nanoTime() < end) { + Thread.sleep(20); + } + assertEquals(2, a.size(), "round-robin gives the first queue member half the events"); + assertEquals(2, b.size(), "round-robin gives the second queue member half the events"); + } + + @Test + void fanOutSubscribersAllReceive() throws Exception { + List a = Collections.synchronizedList(new ArrayList<>()); + List b = Collections.synchronizedList(new ArrayList<>()); + broker.subscribe("news.item", a::add); + broker.subscribe("news.item", b::add); + + broker.publish(Event.create("news.item", "x")).get(5, TimeUnit.SECONDS); + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while ((a.isEmpty() || b.isEmpty()) && System.nanoTime() < end) { + Thread.sleep(20); + } + assertEquals(1, a.size()); + assertEquals(1, b.size()); + } + + @Test + void brokerCanBeRestartedAfterStop() throws Exception { + broker.stop().get(5, TimeUnit.SECONDS); + broker.start().get(5, TimeUnit.SECONDS); + List got = Collections.synchronizedList(new ArrayList<>()); + broker.subscribe("again.*", got::add); + broker.publish(Event.create("again.x", Integer.valueOf(1))).get(5, TimeUnit.SECONDS); + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (got.isEmpty() && System.nanoTime() < end) { + Thread.sleep(20); + } + assertEquals(1, got.size(), "publish after stop() then start() must deliver, not throw RejectedExecutionException"); + } +}