diff --git a/CHANGELOG.md b/CHANGELOG.md index 6396a9c..7c464f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,58 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- `EventBroker.subscribeQueue(topicPattern, queueName, handler)`: competing-consumer subscriptions. Each event is + delivered to one subscriber per queue name. The default implementation falls back to `subscribe`, so existing + brokers keep working. `InMemoryEventBroker` implements it round-robin. +- `KafkaEventBroker` constructor `(bootstrap, groupId, instanceId, java.util.Properties extra)`. The extra properties + apply to the producer and to every consumer. The 3-argument constructor is unchanged. +- `KafkaEventBroker` now uses the structured-mode CloudEvents JSON wire format, so topic, id, time, source and + payload survive the round trip. A malformed record is logged and skipped. + +### Fixed + +- Kafka topic names are now the AMCP topic strings verbatim, for both publish and subscribe. Previously publish used + `a.b.c` and subscribe used `a-b-c`, so the two never met. +- A `KafkaEventBroker` consumer now keeps all its subscriptions. A new subscribe no longer replaces earlier ones. +- Wildcard subscriptions use a Kafka `Pattern` (the union of all active patterns) instead of literal topic names. +- Each `KafkaConsumer` is created, polled and closed by one dedicated thread. Subscribe and unsubscribe enqueue + changes and wake that thread. Handlers run on a separate virtual-thread executor. With no subscriptions the poll + loop idles quietly, with no "Consumer is not subscribed" error spam. +- Received events keep their topic. Previously every record was rebuilt as `amcp.event`, so no subscription matched. +- `Event.matchesTopic("p.**")` matches `p` and `p.` only. It no longer matches `phive...`. +- `KafkaEventBroker.publish` completes only after Kafka acknowledges the record, and fails when the broker is not + running. + +- Review follow-ups on the Kafka transport (seven fixes): + 1. A `KafkaConsumer` that cannot be built (for example `max.poll.records=abc` through the extra properties) now makes + `start()` return a failed future, or `subscribeQueue` throw `IllegalStateException` with the cause. Before, the + poll thread died silently and `start()` reported success. A dead worker is never left in the worker map. + 2. A failed `consumer.subscribe/unsubscribe` no longer marks the pattern set as applied: the cause is logged once and + the update is retried with bounded backoff. + 3. Delivery is now at-least-once: manual commit after the handlers of the polled batch finished (bounded wait of + 30 s), rewind and redelivery of failed records, at most 3 attempts per record. Auto-commit used to commit on poll, + before the asynchronous handler ran, which was at-most-once. Handlers must be idempotent. + 4. Queue mode: a record on an owned partition with no matching local subscription is still committed for the group + (a Kafka limitation), but a WARN is logged once per (topic, queue) pair, and the README requires every instance + sharing a queue name to subscribe the same patterns. + 5. `Event.matchesTopic` and `TopicPatterns.toRegex` now agree on topics with empty segments: a wildcard never + matches a trailing dot or doubled dot. A table-driven test runs the same cases through both. + 6. README: the topic-pickup time is stated in terms of `metadata.max.age.ms` (1 s here, plus a rebalance), and a + subscription takes effect shortly after `subscribe()` returns. + 7. `InMemoryEventBroker.stop()` followed by `start()` no longer makes `publish` throw `RejectedExecutionException`. + Not done: the queue-mode drop in item 4 is documented and logged, not prevented. + +### Changed + +- `KafkaEventBroker.subscribe` (fan-out) skips events older than the subscription's creation time minus 5 seconds, + so a new broker instance does not replay history. +- `KafkaEventBroker.stop` wakes and joins every poll thread, closes consumers and producer, and shuts the executor + down. Calling `start` or `stop` more than once is safe. + ## [1.6.0] - 2024-11-10 ### Added diff --git a/amcp-broker-kafka/README.md b/amcp-broker-kafka/README.md new file mode 100644 index 0000000..b2d78e1 --- /dev/null +++ b/amcp-broker-kafka/README.md @@ -0,0 +1,126 @@ +# AMCP Kafka Broker + +`KafkaEventBroker` is an `io.amcp.core.EventBroker` backed by Apache Kafka. Agents on different JVMs publish and +subscribe through the same topics. + +```java +KafkaEventBroker broker = new KafkaEventBroker("127.0.0.1:9092", "mesh", "instance-1"); +broker.start().get(); + +broker.subscribe("bee.*", event -> System.out.println(event.getTopic())); // fan-out +broker.subscribeQueue("jobs.run", "workers", event -> runJob(event)); // competing consumers +broker.publish(Event.create("bee.heartbeat", Map.of("bee", "b1"))).get(); + +broker.stop().get(); +``` + +The 4-argument constructor `(bootstrap, groupId, instanceId, java.util.Properties extra)` passes extra Kafka client +properties to the producer and to every consumer. The bootstrap servers, group ids and serializers are always set by +the broker and override any conflicting extra property. + +## Delivery semantics + +**Correction:** an earlier revision of this broker used Kafka auto-commit while running handlers asynchronously. +Offsets were then committed on poll, before the handler ran, which is at-most-once: a crash or a failing handler +lost the event. That was wrong and is fixed. The broker now commits manually. + +- **At-least-once.** The poll thread hands a polled batch to the handlers, waits for all of them (at most 30 seconds), + and only then commits the offsets of the records whose handlers returned normally. If a handler throws, or does not + finish in time, that record is not committed and the partition is rewound to it, so the record (and the records after + it in the same partition and batch) is delivered again. After 3 failed deliveries of the same record it is logged at + ERROR and skipped, so one poison record cannot block a partition forever. +- **Handlers must be idempotent.** A record can be delivered more than once (a retry after a failure, a rebalance, a + crash between handler completion and commit, a handler that finished after the 30-second wait). In fan-out mode a + retry also repeats the handlers that already succeeded for that record. +- `enable.auto.commit` is always `false`; an extra property cannot turn it back on. +- Because the poll thread waits for the batch, a handler that blocks for a long time delays all later records of that + consumer (the wait is bounded, and Kafka's `max.poll.interval.ms` default of 5 minutes is not at risk). + +The broker offers two delivery modes. Choose the one that matches what the subscriber should see. + +### Fan-out: `subscribe(topicPattern, handler)` + +Every broker instance receives every matching event, which is the same meaning as `InMemoryEventBroker`. + +- Each instance runs one consumer, in consumer group `groupId + ".fanout." + instanceId`. +- The consumer starts at the earliest offset, but it skips any event whose CloudEvent `time` is older than the + subscription's creation time minus 5 seconds. A newly started instance therefore does not replay history. + Events without a `time` are not filtered. +- The 5-second margin is a semantic rule about clocks, not a performance claim. If a producer's clock is more than + about 5 seconds behind the subscriber's clock, its fresh events are skipped by this filter. + +### Competing consumers: `subscribeQueue(topicPattern, queueName, handler)` + +Each event is delivered to exactly one subscriber across all broker instances that share `groupId` and `queueName`. + +- There is one consumer per queue name, in consumer group `groupId + ".queue." + queueName`. Different queue names + each receive a full copy of the stream. +- The consumer starts at the earliest offset and has no history filter. A queue that is new therefore processes + events still retained in the topic. +- Within one instance, if several subscriptions share the queue name, each event goes to one of them, round-robin. +- **Every instance that shares a `queueName` MUST subscribe the same topic patterns.** Kafka assigns partitions of + the union of the group's topics to its members. If a record lands on an instance that has no local subscription + matching its topic, that instance cannot handle it, and its offset is committed for the whole group, so no other + instance sees it: the record is lost for that queue. The broker cannot repair this (Kafka groups do not allow + handing a record back to a chosen member). It logs one WARN per (topic, queueName) pair on the first such record so + the misconfiguration is visible. Use the same patterns everywhere, or use different queue names. +- Calling `subscribeQueue` on an `EventBroker` that does not override it (for example, a custom broker) falls back + to fan-out `subscribe`. + +### Other rules + +- Handlers run on a virtual-thread executor. A slow handler does not block polling. Handler order across events is + not guaranteed. +- `publish(event)` returns a future that completes when Kafka acknowledges the record (`acks=all`). It completes + exceptionally on a send error, and it fails at once with `IllegalStateException` when the broker is not running. +- A subscription takes effect shortly after `subscribe` or `subscribeQueue` returns, not before: the poll thread + applies it, and a queue subscription also waits for the group rebalance. Fan-out consumers start at the earliest + offset and filter by event time, so recent events are still seen. Do not rely on delivery of an event that was + published immediately before or after the call; wait for the assignment first. +- If the Kafka consumer of a worker cannot be created (for example an invalid consumer property in the 4-argument + constructor), `start()` returns a failed future, or `subscribeQueue` throws `IllegalStateException` with the cause. + If a subscription update fails (`subscribe` on the Kafka consumer throws), the broker logs the cause once and retries + with bounded backoff (1 s doubling to 30 s) until it succeeds. + +## Topic naming and patterns + +- The Kafka topic name is the AMCP topic string, verbatim. `publish` sends `bee.heartbeat` to the topic + `bee.heartbeat`. `subscribe("bee.heartbeat", ...)` subscribes to that same topic. +- Kafka topic names may contain only `a-z`, `A-Z`, `0-9`, `.`, `_` and `-`, up to 249 characters. Names that differ + only by `.` versus `_` collide, and Kafka rejects the second topic created. +- Patterns are translated to regular expressions, and the broker subscribes with one `Pattern` that is the union of + all active patterns of the consumer: + +| AMCP pattern | Matches | Does not match | +|---|---|---| +| `a.b.c` | `a.b.c` | `a.b.cd`, `aXb.c` | +| `p.*` | `p.x` (exactly one segment) | `p`, `p.x.y`, `px.x` | +| `p.**` | `p`, `p.x`, `p.x.y` | `px`, `phive.x` | + +- A wildcard never matches an empty segment. A topic with a trailing dot (`p.x.`) or a doubled dot (`p..x`) is matched + only by a pattern equal to the topic, never by `p.*` or `p.**`. `Event.matchesTopic` and the Kafka regex follow the + same rule (a table-driven test runs the same cases through both). +- Kafka subscriptions find topics by metadata refresh. The broker sets `metadata.max.age.ms=1000` (override it through + the extra properties), so a topic created after a pattern subscription is discovered at most about 1 second later; + the group rebalance that follows can add a second or more. Allow 2 to 5 seconds before expecting events from a + brand-new topic. Publishing to a topic that does not exist creates it when the cluster has + `auto.create.topics.enable=true`. + +## Wire format + +Records use the CloudEvents structured JSON format (`application/cloudevents+json`), through +`io.cloudevents.jackson.JsonFormat`. The record key is the CloudEvent extension `partitionkey` when present, and +otherwise the event id. A record that cannot be parsed is logged once by the consumer and skipped. + +## Lifecycle + +- `start()` and `stop()` are idempotent. +- `stop()` wakes and joins each poll thread, closes the consumers and the producer, and shuts the executor down. Each + consumer is created, polled and closed by one dedicated platform thread. + +## Tests + +`KafkaEventBrokerIT` runs against a real broker when `AMCP_KAFKA_BOOTSTRAP` is set, for example +`AMCP_KAFKA_BOOTSTRAP=127.0.0.1:9092 mvn -pl amcp-broker-kafka verify`. Without it, the integration tests are skipped +and the build stays green. `TopicPatternsTest` and `KafkaEventBrokerUnitTest` are plain unit tests that need no broker (the latter drives the +poll logic with an in-memory consumer: redelivery, commit timing, retry, queue WARN, start failure). diff --git a/amcp-broker-kafka/pom.xml b/amcp-broker-kafka/pom.xml index 9e2f78c..758ae55 100644 --- a/amcp-broker-kafka/pom.xml +++ b/amcp-broker-kafka/pom.xml @@ -33,6 +33,11 @@ io.cloudevents cloudevents-kafka + + + io.cloudevents + cloudevents-json-jackson + @@ -57,4 +62,21 @@ test + + + + + org.apache.maven.plugins + maven-surefire-plugin + + + + **/*Test.java + **/*IT.java + + -Xmx512m + + + + diff --git a/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/KafkaEventBroker.java b/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/KafkaEventBroker.java index bfb3e22..9cfa1dd 100644 --- a/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/KafkaEventBroker.java +++ b/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/KafkaEventBroker.java @@ -3,52 +3,535 @@ import io.amcp.core.Event; import io.amcp.core.EventBroker; import io.cloudevents.CloudEvent; +import io.cloudevents.core.format.EventFormat; +import io.cloudevents.core.provider.EventFormatProvider; import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.consumer.OffsetAndMetadata; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.common.errors.WakeupException; import org.apache.kafka.common.serialization.StringDeserializer; import org.apache.kafka.common.serialization.StringSerializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.Instant; import java.time.OffsetDateTime; -import java.util.*; -import java.util.concurrent.*; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Properties; +import java.util.Set; +import java.util.TreeSet; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Consumer; /** - * Kafka-based EventBroker implementation for distributed AMCP mesh. - * - * Features: - * - Distributed message routing via Kafka - * - Topic-based pub/sub - * - Consumer groups for scalability - * - CloudEvents serialization - * - Multi-instance support + * Kafka-based EventBroker for a distributed AMCP mesh. + * + *

Topic names are the AMCP topic strings verbatim. Two delivery semantics are offered: + *

+ * + *

Delivery is at-least-once: offsets are committed manually, after the handlers of a polled batch have + * completed (bounded wait). A failed or unfinished record is rewound and redelivered, at most + * {@link #MAX_DELIVERY_ATTEMPTS} times. Handlers must be idempotent. In queue mode every instance sharing a + * {@code queueName} must subscribe the same patterns; otherwise a record can be committed without any handler. + * + *

Threading: every {@link KafkaConsumer} is created, polled and closed by one dedicated platform + * thread. Subscribe and unsubscribe calls enqueue changes and wake that thread. Handlers run on a + * virtual-thread executor, so a slow handler never blocks polling. */ public class KafkaEventBroker implements EventBroker { private static final Logger logger = LoggerFactory.getLogger(KafkaEventBroker.class); - + + /** Lookback applied to fan-out subscriptions: events older than (creation - 5 s) are skipped. */ + static final Duration FANOUT_HISTORY_GRACE = Duration.ofSeconds(5); + + /** A record whose handler keeps failing is delivered at most this many times, then skipped with an ERROR log. */ + static final int MAX_DELIVERY_ATTEMPTS = 3; + + private static final String FANOUT_KEY = "fanout"; + private static final String QUEUE_KEY_PREFIX = "queue:"; + private static final Duration POLL_TIMEOUT = Duration.ofMillis(500); + private static final long IDLE_WAIT_MS = 500; + private static final long STOP_JOIN_MS = 10_000; + private static final long ERROR_BACKOFF_MS = 1_000; + private static final long MAX_RETRY_BACKOFF_MS = 30_000; + private static final long CONSUMER_CREATE_TIMEOUT_S = 30; + /** metadata.max.age.ms set on every consumer (extra properties may override it). */ + static final int METADATA_MAX_AGE_MS = 1000; + private static final String CLOUDEVENTS_JSON = "application/cloudevents+json"; + + private static final EventFormat JSON_FORMAT = Objects.requireNonNull( + EventFormatProvider.getInstance().resolveFormat(CLOUDEVENTS_JSON), + "CloudEvents JSON format not found on the classpath (cloudevents-json-jackson)"); + private final String bootstrapServers; private final String groupId; private final String instanceId; - - private KafkaProducer producer; - private KafkaConsumer consumer; - private ExecutorService executor; - private volatile boolean running = false; + private final Properties extraProperties; + + /** Guards start, stop, subscribe, unsubscribe and worker creation. Never taken by publish(). */ + private final Object lifecycle = new Object(); private final Map subscriptions = new ConcurrentHashMap<>(); + private final Map workers = new ConcurrentHashMap<>(); + private volatile KafkaProducer producer; + private volatile ExecutorService executor; + private volatile boolean running = false; + + // Test seams (package-private): replaced in unit tests, never by production code. + volatile java.util.function.Function> consumerFactory = + props -> new KafkaConsumer<>(props); + volatile Duration batchTimeout = Duration.ofSeconds(30); + volatile long baseRetryBackoffMs = 1_000; - private static class Subscription { + /** One subscription as registered by the caller. Immutable apart from the poll thread's own copy. */ + private static final class Subscription { + final String id; final String topicPattern; final Consumer handler; + final String queueName; // null for fan-out + final Instant notBefore; // null for queue subscriptions (no history filter) - Subscription(String topicPattern, Consumer handler) { + Subscription(String id, String topicPattern, Consumer handler, String queueName, Instant notBefore) { + this.id = id; this.topicPattern = topicPattern; this.handler = handler; + this.queueName = queueName; + this.notBefore = notBefore; + } + + String workerKey() { + return queueName == null ? FANOUT_KEY : QUEUE_KEY_PREFIX + queueName; + } + + boolean accepts(Event event) { + if (!event.matchesTopic(topicPattern)) { + return false; + } + if (notBefore == null) { + return true; + } + OffsetDateTime time = event.getTime(); + return time == null || !time.toInstant().isBefore(notBefore); + } + } + + private record Change(Subscription subscription, boolean add) { + } + + /** + * Owns one KafkaConsumer and the thread that drives it. Fields marked "poll thread only" are + * touched only by that thread. + */ + private final class PollWorker implements Runnable { + private final String name; + private final String consumerGroup; + private final boolean queue; + private final Thread thread; + /** Completed once the consumer exists, or completed exceptionally when it could not be created. */ + private final CompletableFuture ready = new CompletableFuture<>(); + private final ConcurrentLinkedQueue changes = new ConcurrentLinkedQueue<>(); + private final Object monitor = new Object(); + private final Map active = new LinkedHashMap<>(); // poll thread only + private final AtomicInteger roundRobin = new AtomicInteger(); + private final Set warnedDrops = new HashSet<>(); // poll thread only + private final Map failedAttempts = new HashMap<>(); // poll thread only + private Set appliedPatterns = new TreeSet<>(); // poll thread only + private int applyFailures = 0; // poll thread only + private long applyRetryAtNanos = 0; // poll thread only + private boolean applyFailureLogged = false; // poll thread only + private volatile org.apache.kafka.clients.consumer.Consumer consumer; + private volatile boolean stopping = false; + private volatile int assignedPartitions = 0; + + PollWorker(String name, String consumerGroup, boolean queue) { + this.name = name; + this.consumerGroup = consumerGroup; + this.queue = queue; + this.thread = new Thread(this, "amcp-kafka-" + name); + this.thread.setDaemon(true); + } + + void enqueue(Change change) { + changes.add(change); + synchronized (monitor) { + monitor.notifyAll(); + } + wakeConsumer(); + } + + void requestStop() { + stopping = true; + synchronized (monitor) { + monitor.notifyAll(); + } + wakeConsumer(); + } + + private void wakeConsumer() { + org.apache.kafka.clients.consumer.Consumer c = consumer; + if (c != null) { + try { + c.wakeup(); + } catch (RuntimeException e) { + logger.debug("wakeup on consumer {} ignored: {}", name, e.getMessage()); + } + } + } + + void awaitTermination(long timeoutMs) { + try { + thread.join(timeoutMs); + if (thread.isAlive()) { + logger.warn("Kafka poll thread {} did not stop within {} ms", name, timeoutMs); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + + @Override + public void run() { + org.apache.kafka.clients.consumer.Consumer c; + try { + c = consumerFactory.apply(consumerProperties(consumerGroup)); + } catch (RuntimeException | Error e) { + logger.error("Could not create Kafka consumer {}: {}", name, e.getMessage(), e); + workers.remove(name, this); + ready.completeExceptionally(e); + return; + } + consumer = c; + logger.info("Kafka consumer {} created in group {}", name, consumerGroup); + ready.complete(null); + try { + while (!stopping) { + try { + applyChanges(c); + if (active.isEmpty() || appliedPatterns.isEmpty()) { + // Nothing is subscribed (or the last update failed and is waiting for its retry): + // never call poll() on a consumer without a subscription. + idle(); + continue; + } + ConsumerRecords records = c.poll(POLL_TIMEOUT); + assignedPartitions = c.assignment().size(); + if (!records.isEmpty()) { + processBatch(c, records); + } + } catch (WakeupException expected) { + // Raised on purpose by wakeup(): a change is queued or stop was requested. + } catch (RuntimeException e) { + if (stopping) { + break; + } + logger.error("Kafka poll loop {} error: {}", name, e.getMessage()); + sleepMs(ERROR_BACKOFF_MS); + } + } + } finally { + // Never leave a dead worker in the map: a later subscribe must create a fresh one. + workers.remove(name, this); + consumer = null; + try { + c.close(Duration.ofSeconds(5)); + } catch (RuntimeException e) { + logger.warn("Error closing Kafka consumer {}: {}", name, e.getMessage()); + } + logger.info("Kafka consumer {} closed", name); + } + } + + /** + * Drains queued changes and resubscribes only when the set of patterns actually changed. The applied set + * is updated only after the Kafka call succeeded; on failure the update is retried with bounded backoff. + */ + private void applyChanges(org.apache.kafka.clients.consumer.Consumer c) { + Change change; + while ((change = changes.poll()) != null) { + if (change.add()) { + active.put(change.subscription().id, change.subscription()); + } else { + active.remove(change.subscription().id); + } + } + Set wanted = new TreeSet<>(); + for (Subscription s : active.values()) { + wanted.add(s.topicPattern); + } + if (wanted.equals(appliedPatterns)) { + return; + } + if (applyFailures > 0 && System.nanoTime() - applyRetryAtNanos < 0) { + return; // still backing off after a failed update + } + try { + if (wanted.isEmpty()) { + c.unsubscribe(); + logger.debug("Consumer {} unsubscribed (no active subscriptions)", name); + } else { + c.subscribe(TopicPatterns.union(wanted)); + logger.debug("Consumer {} subscribed to {}", name, wanted); + } + appliedPatterns = wanted; + applyFailures = 0; + applyFailureLogged = false; + } catch (RuntimeException e) { + applyFailures++; + long backoffMs = Math.min(MAX_RETRY_BACKOFF_MS, + baseRetryBackoffMs << Math.min(applyFailures - 1, 10)); + applyRetryAtNanos = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(backoffMs); + if (!applyFailureLogged) { + applyFailureLogged = true; + logger.error("Consumer {} could not update its subscription to {}; the subscription stays " + + "inactive and the update is retried with backoff: {}", name, wanted, e.toString(), e); + } + } + } + + /** Waits without touching the consumer; used when nothing is applied, so poll() is never called. */ + private void idle() { + long waitMs = IDLE_WAIT_MS; + if (applyFailures > 0) { + long untilRetry = TimeUnit.NANOSECONDS.toMillis(applyRetryAtNanos - System.nanoTime()); + waitMs = Math.max(1, Math.min(waitMs, untilRetry)); + } + synchronized (monitor) { + if (changes.isEmpty() && !stopping) { + try { + monitor.wait(waitMs); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + stopping = true; + } + } + } + } + + private void sleepMs(long ms) { + try { + Thread.sleep(ms); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + stopping = true; + } + } + + private record Dispatched(ConsumerRecord record, CompletableFuture done) { + } + + /** + * Hands every record of the polled batch to the handlers, waits (bounded) for all of them, and commits + * only the offsets whose handlers completed. A failed or unfinished record is not committed and the + * partition is rewound to it, so it is redelivered: at-least-once. After {@link #MAX_DELIVERY_ATTEMPTS} + * failed deliveries a record is logged and skipped so one poison record cannot block a partition. + */ + private void processBatch(org.apache.kafka.clients.consumer.Consumer c, + ConsumerRecords records) { + Map> byPartition = new LinkedHashMap<>(); + List> all = new ArrayList<>(); + for (TopicPartition tp : records.partitions()) { + List list = new ArrayList<>(); + for (ConsumerRecord record : records.records(tp)) { + CompletableFuture done = dispatch(record); + list.add(new Dispatched(record, done)); + all.add(done); + } + byPartition.put(tp, list); + } + if (!awaitAll(all)) { + return; // stopping: commit nothing, the batch is redelivered on the next start + } + + Map commit = new LinkedHashMap<>(); + Map rewind = new LinkedHashMap<>(); + for (Map.Entry> entry : byPartition.entrySet()) { + TopicPartition tp = entry.getKey(); + boolean blocked = false; + for (Dispatched d : entry.getValue()) { + ConsumerRecord r = d.record(); + String key = r.topic() + "/" + r.partition() + "/" + r.offset(); + boolean ok = d.done().isDone() && !d.done().isCompletedExceptionally(); + if (ok) { + failedAttempts.remove(key); + } else { + int attempts = failedAttempts.merge(key, 1, Integer::sum); + if (attempts >= MAX_DELIVERY_ATTEMPTS) { + logger.error("Giving up on record {} after {} failed delivery attempts; it is skipped " + + "and its offset is committed", key, attempts); + failedAttempts.remove(key); + ok = true; + } + } + if (blocked) { + continue; + } + if (ok) { + commit.put(tp, new OffsetAndMetadata(r.offset() + 1)); + } else { + blocked = true; + rewind.put(tp, r.offset()); + } + } + } + commitOffsets(c, commit); + if (!rewind.isEmpty()) { + for (Map.Entry e : rewind.entrySet()) { + c.seek(e.getKey(), e.getValue()); + } + logger.warn("Consumer {} will redeliver unfinished or failed records from {}", name, rewind); + sleepMs(baseRetryBackoffMs); + } + } + + private void commitOffsets(org.apache.kafka.clients.consumer.Consumer c, + Map commit) { + if (commit.isEmpty()) { + return; + } + for (int attempt = 0; attempt < 3; attempt++) { + try { + c.commitSync(commit); + return; + } catch (WakeupException wakeup) { + // A subscribe/unsubscribe woke the consumer while we were committing: try again. + } catch (RuntimeException e) { + logger.warn("Consumer {} could not commit offsets {} (the records may be redelivered): {}", + name, commit, e.toString()); + return; + } + } + logger.warn("Consumer {} could not commit offsets {} after repeated wakeups", name, commit); + } + + /** Waits for all handler futures, at most {@code batchTimeout}. Returns false when the worker is stopping. */ + private boolean awaitAll(List> futures) { + CompletableFuture all = CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); + long deadline = System.nanoTime() + batchTimeout.toNanos(); + while (!all.isDone()) { + if (stopping) { + return false; + } + long remaining = deadline - System.nanoTime(); + if (remaining <= 0) { + logger.warn("Consumer {}: handlers did not finish within {} ms; unfinished records are not " + + "committed and will be redelivered", name, batchTimeout.toMillis()); + return true; + } + try { + all.get(Math.min(remaining, TimeUnit.MILLISECONDS.toNanos(100)), TimeUnit.NANOSECONDS); + } catch (TimeoutException | ExecutionException ignored) { + // re-check state; failures are read per record afterwards + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + stopping = true; + return false; + } + } + return true; + } + + /** + * Routes one record. Runs on the poll thread and only hands work to the executor. The returned future + * completes normally when every handler it was given returned normally (or nobody had to handle it). + */ + private CompletableFuture dispatch(ConsumerRecord record) { + Event event; + try { + event = decode(record.value()); + } catch (RuntimeException e) { + logger.warn("Skipping malformed record on topic {} at offset {}: {}", + record.topic(), record.offset(), e.getMessage()); + return CompletableFuture.completedFuture(null); + } + List matches = new ArrayList<>(); + for (Subscription s : active.values()) { + if (s.accepts(event)) { + matches.add(s); + } + } + if (!queue) { + List> futures = new ArrayList<>(); + for (Subscription s : matches) { + futures.add(deliver(s, event)); + } + return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); + } + if (matches.isEmpty()) { + // The group assigned us a partition, but no local subscription matches. The offset is committed + // with the batch, so no other instance will see this record: all instances sharing a queue name + // must subscribe the same patterns. + if (warnedDrops.add(record.topic())) { + logger.warn("Queue '{}' dropped a record on topic '{}': this instance has no local subscription " + + "that matches it, and the offset is committed for the whole queue group. Every " + + "instance sharing a queue name must subscribe the same topic patterns.", + queueNameOf(), record.topic()); + } + return CompletableFuture.completedFuture(null); + } + return deliver(matches.get(Math.floorMod(roundRobin.getAndIncrement(), matches.size())), event); + } + + private String queueNameOf() { + return name.startsWith(QUEUE_KEY_PREFIX) ? name.substring(QUEUE_KEY_PREFIX.length()) : name; + } + + private CompletableFuture deliver(Subscription s, Event event) { + CompletableFuture done = new CompletableFuture<>(); + ExecutorService ex = executor; + if (ex == null) { + done.completeExceptionally(new IllegalStateException("broker is stopping")); + return done; + } + try { + ex.execute(() -> { + try { + s.handler.accept(event); + done.complete(null); + } catch (Throwable t) { + logger.error("Handler for {} failed on event {}: {}", + event.getTopic(), event.getId(), t.getMessage(), t); + done.completeExceptionally(t); + if (t instanceof Error err) { + throw err; + } + } + }); + } catch (RejectedExecutionException e) { + logger.warn("Not delivering event {} because the broker is stopping", event.getId()); + done.completeExceptionally(e); + } + return done; } } @@ -56,143 +539,234 @@ private static class Subscription { * Creates a new KafkaEventBroker. * * @param bootstrapServers Kafka bootstrap servers (e.g., "localhost:9092") - * @param groupId Consumer group ID for this instance + * @param groupId Consumer group base ID shared by the mesh * @param instanceId Unique instance identifier */ public KafkaEventBroker(String bootstrapServers, String groupId, String instanceId) { - this.bootstrapServers = bootstrapServers; - this.groupId = groupId; - this.instanceId = instanceId; + this(bootstrapServers, groupId, instanceId, new Properties()); + } + + /** + * Creates a new KafkaEventBroker with extra Kafka client properties. + * + *

The extra properties apply to the producer and to every consumer. Connection and identity + * settings (bootstrap servers, group id, serializers) are always set by the broker and win over + * the extra properties. + * + * @param bootstrapServers Kafka bootstrap servers + * @param groupId Consumer group base ID shared by the mesh + * @param instanceId Unique instance identifier + * @param extraProperties additional Kafka client properties (may be null) + */ + public KafkaEventBroker(String bootstrapServers, String groupId, String instanceId, Properties extraProperties) { + this.bootstrapServers = Objects.requireNonNull(bootstrapServers, "bootstrapServers"); + this.groupId = Objects.requireNonNull(groupId, "groupId"); + this.instanceId = Objects.requireNonNull(instanceId, "instanceId"); + this.extraProperties = new Properties(); + if (extraProperties != null) { + this.extraProperties.putAll(extraProperties); + } } @Override public CompletableFuture publish(Event event) { - if (!running) { - return CompletableFuture.failedFuture( - new IllegalStateException("Broker not running")); + KafkaProducer p = producer; + ExecutorService ex = executor; + if (!running || p == null || ex == null) { + return CompletableFuture.failedFuture(new IllegalStateException("Broker not running")); } + CompletableFuture acknowledged = new CompletableFuture<>(); + try { + ex.execute(() -> send(p, event, acknowledged)); + } catch (RejectedExecutionException e) { + acknowledged.completeExceptionally(new IllegalStateException("Broker not running", e)); + } + return acknowledged; + } - return CompletableFuture.runAsync(() -> { - try { - String topic = event.getTopic(); - String key = instanceId + "-" + UUID.randomUUID(); - String value = serializeEvent(event); - - ProducerRecord record = new ProducerRecord<>(topic, key, value); - producer.send(record, (metadata, exception) -> { - if (exception != null) { - logger.error("Error publishing to topic {}: {}", topic, exception.getMessage()); - } else { - logger.debug("Published to topic {} partition {}", topic, metadata.partition()); - } - }); - - } catch (Exception e) { - logger.error("Error serializing event: {}", e.getMessage(), e); - } - }, executor); + private void send(KafkaProducer p, Event event, CompletableFuture acknowledged) { + try { + String topic = event.getTopic(); + ProducerRecord record = new ProducerRecord<>(topic, recordKey(event), encode(event)); + p.send(record, (metadata, error) -> { + if (error != null) { + logger.error("Error publishing to topic {}: {}", topic, error.getMessage()); + acknowledged.completeExceptionally(error); + } else { + logger.debug("Published to topic {} partition {} offset {}", + topic, metadata.partition(), metadata.offset()); + acknowledged.complete(null); + } + }); + } catch (RuntimeException e) { + logger.error("Error publishing event {}: {}", event.getId(), e.getMessage(), e); + acknowledged.completeExceptionally(e); + } + } + + /** Record key: the CloudEvent "partitionkey" extension when present, otherwise the event id. */ + private static String recordKey(Event event) { + Object partitionKey = event.getCloudEvent().getExtension("partitionkey"); + return partitionKey != null ? partitionKey.toString() : event.getId(); } @Override public String subscribe(String topicPattern, Consumer handler) { - String subscriptionId = UUID.randomUUID().toString(); - subscriptions.put(subscriptionId, new Subscription(topicPattern, handler)); - - // Convert pattern to Kafka topic - String kafkaTopic = topicPattern.replace("**", "*").replace(".", "-"); - - try { - consumer.subscribe(Collections.singletonList(kafkaTopic)); - logger.debug("Subscribed to topic pattern: {}", topicPattern); - } catch (Exception e) { - logger.error("Error subscribing to topic {}: {}", topicPattern, e.getMessage()); + Instant notBefore = Instant.now().minus(FANOUT_HISTORY_GRACE); + return register(new Subscription(UUID.randomUUID().toString(), topicPattern, handler, null, notBefore)); + } + + @Override + public String subscribeQueue(String topicPattern, String queueName, Consumer handler) { + if (queueName == null || queueName.isBlank()) { + throw new IllegalArgumentException("queueName must not be blank"); } - - return subscriptionId; + return register(new Subscription(UUID.randomUUID().toString(), topicPattern, handler, queueName, null)); + } + + private String register(Subscription subscription) { + synchronized (lifecycle) { + // Create the worker first: if its consumer cannot be built this throws and nothing is registered. + PollWorker worker = running ? workerFor(subscription) : null; + subscriptions.put(subscription.id, subscription); + if (worker != null) { + worker.enqueue(new Change(subscription, true)); + } + } + logger.debug("Registered subscription {} for pattern {} (queue {})", + subscription.id, subscription.topicPattern, subscription.queueName); + return subscription.id; } @Override public void unsubscribe(String subscriptionId) { - subscriptions.remove(subscriptionId); + synchronized (lifecycle) { + Subscription subscription = subscriptions.remove(subscriptionId); + if (subscription != null && running) { + PollWorker worker = workers.get(subscription.workerKey()); + if (worker != null) { + worker.enqueue(new Change(subscription, false)); + } + } + } logger.debug("Removed subscription: {}", subscriptionId); } - @Override - public CompletableFuture start() { - if (running) { - return CompletableFuture.completedFuture(null); + /** Must be called while holding {@code lifecycle} and while the broker is running. */ + private PollWorker workerFor(Subscription subscription) { + return worker(subscription.workerKey(), + subscription.queueName == null + ? groupId + ".fanout." + instanceId + : groupId + ".queue." + subscription.queueName, + subscription.queueName != null); + } + + /** Returns the live worker for {@code key}, creating it and waiting for its consumer. Holds {@code lifecycle}. */ + private PollWorker worker(String key, String consumerGroup, boolean queue) { + PollWorker existing = workers.get(key); + if (existing != null) { + return existing; } + PollWorker w = new PollWorker(key, consumerGroup, queue); + workers.put(key, w); + w.thread.start(); + try { + w.ready.get(CONSUMER_CREATE_TIMEOUT_S, TimeUnit.SECONDS); + } catch (ExecutionException e) { + workers.remove(key, w); + throw new IllegalStateException("Could not create the Kafka consumer for " + key + ": " + + e.getCause().getMessage(), e.getCause()); + } catch (TimeoutException e) { + w.requestStop(); + workers.remove(key, w); + throw new IllegalStateException("Timed out creating the Kafka consumer for " + key, e); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + w.requestStop(); + workers.remove(key, w); + throw new IllegalStateException("Interrupted while creating the Kafka consumer for " + key, e); + } + return w; + } - return CompletableFuture.runAsync(() -> { + @Override + public CompletableFuture start() { + synchronized (lifecycle) { + if (running) { + return CompletableFuture.completedFuture(null); + } try { - // Create producer - Properties producerProps = new Properties(); - producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); - producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); - producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); - producerProps.put(ProducerConfig.ACKS_CONFIG, "all"); - producerProps.put(ProducerConfig.RETRIES_CONFIG, 3); - - producer = new KafkaProducer<>(producerProps); - logger.info("Kafka producer created"); - - // Create consumer - Properties consumerProps = new Properties(); - consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); - consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, groupId); - consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); - consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); - consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); - consumerProps.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true); - - consumer = new KafkaConsumer<>(consumerProps); - logger.info("Kafka consumer created with group: {}", groupId); - - // Start consumer thread + producer = new KafkaProducer<>(producerProperties()); executor = Executors.newVirtualThreadPerTaskExecutor(); + worker(FANOUT_KEY, groupId + ".fanout." + instanceId, false); + for (Subscription s : subscriptions.values()) { + workerFor(s).enqueue(new Change(s, true)); + } running = true; - - executor.submit(this::consumerLoop); - logger.info("KafkaEventBroker started for instance: {}", instanceId); - - } catch (Exception e) { + logger.info("KafkaEventBroker started for instance {} ({} subscriptions)", + instanceId, subscriptions.size()); + return CompletableFuture.completedFuture(null); + } catch (RuntimeException e) { + running = false; + shutdownLocked(); logger.error("Failed to start KafkaEventBroker: {}", e.getMessage(), e); - throw new RuntimeException("Failed to start Kafka broker", e); + return CompletableFuture.failedFuture(new RuntimeException("Failed to start Kafka broker", e)); } - }); + } } @Override public CompletableFuture stop() { - if (!running) { - return CompletableFuture.completedFuture(null); + synchronized (lifecycle) { + if (!running) { + return CompletableFuture.completedFuture(null); + } + running = false; + shutdownLocked(); + logger.info("KafkaEventBroker stopped for instance {}", instanceId); + } + return CompletableFuture.completedFuture(null); + } + + /** Stops every worker, the executor and the producer. Must hold {@code lifecycle}. Safe on a half-started broker. */ + private void shutdownLocked() { + List stopped = new ArrayList<>(workers.values()); + workers.clear(); + for (PollWorker w : stopped) { + w.requestStop(); + } + for (PollWorker w : stopped) { + w.awaitTermination(STOP_JOIN_MS); } - return CompletableFuture.runAsync(() -> { + ExecutorService ex = executor; + executor = null; + if (ex != null) { + ex.shutdown(); try { - running = false; - - if (consumer != null) { - consumer.close(); + if (!ex.awaitTermination(5, TimeUnit.SECONDS)) { + ex.shutdownNow(); } - - if (producer != null) { - producer.close(); - } - - if (executor != null) { - executor.shutdown(); - if (!executor.awaitTermination(5, TimeUnit.SECONDS)) { - executor.shutdownNow(); - } - } - - logger.info("KafkaEventBroker stopped"); - - } catch (Exception e) { - logger.error("Error stopping KafkaEventBroker: {}", e.getMessage(), e); + } catch (InterruptedException e) { + ex.shutdownNow(); + Thread.currentThread().interrupt(); + } + } + + KafkaProducer p = producer; + producer = null; + if (p != null) { + try { + p.close(Duration.ofSeconds(5)); + } catch (RuntimeException e) { + logger.warn("Error closing Kafka producer: {}", e.getMessage()); } - }); + } + } + + /** Test hook: number of live poll workers. */ + int workerCount() { + return workers.size(); } @Override @@ -206,53 +780,61 @@ public String getBrokerType() { } /** - * Consumer loop for processing messages. + * Test hook: waits until the queue consumer for {@code queueName} on this instance has at least one + * partition assigned. Returns false if that does not happen within the timeout. */ - private void consumerLoop() { - while (running) { - try { - var records = consumer.poll(java.time.Duration.ofSeconds(1)); - - records.forEach(record -> { - try { - Event event = deserializeEvent(record.value()); - - // Route to matching subscribers - for (Subscription sub : subscriptions.values()) { - if (event.matchesTopic(sub.topicPattern)) { - try { - executor.submit(() -> sub.handler.accept(event)); - } catch (Exception e) { - logger.error("Error delivering event: {}", e.getMessage()); - } - } - } - } catch (Exception e) { - logger.error("Error processing message: {}", e.getMessage(), e); - } - }); - - } catch (Exception e) { - logger.error("Error in consumer loop: {}", e.getMessage()); + boolean awaitQueueAssignment(String queueName, Duration timeout) throws InterruptedException { + long deadline = System.nanoTime() + timeout.toNanos(); + while (System.nanoTime() < deadline) { + PollWorker w = workers.get(QUEUE_KEY_PREFIX + queueName); + if (w != null && w.assignedPartitions > 0) { + return true; } + Thread.sleep(50); } + return false; } - /** - * Serializes an Event to JSON string. - */ - private String serializeEvent(Event event) { - // Use CloudEvents JSON format - CloudEvent ce = event.getCloudEvent(); - return ce.toString(); + private Properties producerProperties() { + Properties props = new Properties(); + props.put(ProducerConfig.ACKS_CONFIG, "all"); + props.put(ProducerConfig.RETRIES_CONFIG, 3); + props.putAll(extraProperties); + props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); + props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + return props; } - /** - * Deserializes a JSON string to an Event. - */ - private Event deserializeEvent(String json) { - // Parse CloudEvents JSON and reconstruct Event - // This is a simplified version - full implementation would use proper JSON parsing - return Event.create("amcp.event", json); + private Properties consumerProperties(String consumerGroup) { + Properties props = new Properties(); + props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); + // Topics created after a pattern subscription are discovered on the next metadata refresh, which happens + // at most every metadata.max.age.ms; the group rebalance that follows adds a little more. + props.put(ConsumerConfig.METADATA_MAX_AGE_CONFIG, METADATA_MAX_AGE_MS); + props.putAll(extraProperties); + // Offsets are committed by the poll thread after the handlers of a batch finished (at-least-once). + // Auto-commit would commit on poll, before the asynchronous handlers run, so it is never allowed. + props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); + props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); + props.put(ConsumerConfig.GROUP_ID_CONFIG, consumerGroup); + props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); + props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); + return props; + } + + /** Serializes an Event as a CloudEvents structured JSON document. */ + static String encode(Event event) { + byte[] bytes = JSON_FORMAT.serialize(event.getCloudEvent()); + return new String(bytes, StandardCharsets.UTF_8); + } + + /** Parses a CloudEvents structured JSON document into an Event. Throws on malformed input. */ + static Event decode(String json) { + if (json == null) { + throw new IllegalArgumentException("null record value"); + } + CloudEvent ce = JSON_FORMAT.deserialize(json.getBytes(StandardCharsets.UTF_8)); + return new Event(ce); } } diff --git a/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/TopicPatterns.java b/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/TopicPatterns.java new file mode 100644 index 0000000..7afc22c --- /dev/null +++ b/amcp-broker-kafka/src/main/java/io/amcp/broker/kafka/TopicPatterns.java @@ -0,0 +1,56 @@ +package io.amcp.broker.kafka; + +import java.util.Collection; +import java.util.StringJoiner; +import java.util.regex.Pattern; + +/** + * Translates AMCP topic patterns into Kafka topic regexes. + * + *

+ * + *

A wildcard never matches an empty segment: {@code p.x.} and {@code p..x} match {@code p.*} / {@code p.**} not at + * all (they match only a pattern equal to the topic). {@code io.amcp.core.Event#matchesTopic} follows the same rule. + * + *

Kafka matches a subscription pattern against the whole topic name, so the regexes produced + * here are anchored by the caller's {@code matches()} call. + */ +final class TopicPatterns { + + private TopicPatterns() { + } + + /** + * Returns the regex (not compiled) for one AMCP topic pattern. + */ + static String toRegex(String amcpPattern) { + if (amcpPattern.endsWith(".**")) { + String prefix = amcpPattern.substring(0, amcpPattern.length() - 3); + return Pattern.quote(prefix) + "(\\.[^.]+(\\.[^.]+)*)?"; + } + if (amcpPattern.endsWith(".*")) { + String prefix = amcpPattern.substring(0, amcpPattern.length() - 2); + return Pattern.quote(prefix) + "\\.[^.]+"; + } + return Pattern.quote(amcpPattern); + } + + /** + * Returns one compiled pattern that matches any topic matched by any of the given AMCP patterns, + * or {@code null} when the collection is empty. + */ + static Pattern union(Collection amcpPatterns) { + if (amcpPatterns.isEmpty()) { + return null; + } + StringJoiner joiner = new StringJoiner("|", "(?:", ")"); + for (String p : amcpPatterns) { + joiner.add(toRegex(p)); + } + return Pattern.compile(joiner.toString()); + } +} diff --git a/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/FakeConsumer.java b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/FakeConsumer.java new file mode 100644 index 0000000..d0acd76 --- /dev/null +++ b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/FakeConsumer.java @@ -0,0 +1,114 @@ +package io.amcp.broker.kafka; + +import org.apache.kafka.clients.consumer.ConsumerRebalanceListener; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.clients.consumer.MockConsumer; +import org.apache.kafka.clients.consumer.OffsetAndMetadata; +import org.apache.kafka.clients.consumer.OffsetResetStrategy; +import org.apache.kafka.common.TopicPartition; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.regex.Pattern; + +/** + * A deterministic in-memory consumer for unit tests. It behaves like a partitioned log: records stay in the log, a + * seek replays them, and every commit is recorded. subscribe(Pattern) assigns every known partition without checking + * the pattern, which is how a group member can own a partition for a topic it never subscribed to. + */ +class FakeConsumer extends MockConsumer { + final List> log = Collections.synchronizedList(new ArrayList<>()); + final List> commits = Collections.synchronizedList(new ArrayList<>()); + final AtomicInteger subscribeAttempts = new AtomicInteger(); + volatile int failSubscribeTimes = 0; + private final Map nextOffset = new HashMap<>(); + private final Set known = new HashSet<>(); + private boolean subscribed = false; + + FakeConsumer() { + super(OffsetResetStrategy.EARLIEST); + } + + synchronized void append(String topic, String value) { + long offset = nextOffset.merge(topic, 1L, Long::sum) - 1; + ConsumerRecord r = new ConsumerRecord<>(topic, 0, offset, "k", value); + log.add(r); + known.add(new TopicPartition(topic, 0)); + if (subscribed) { + assignKnown(); + addRecord(r); + } + } + + private void assignKnown() { + assign(new HashSet<>(known)); + Map begin = new HashMap<>(); + for (TopicPartition tp : known) { + begin.put(tp, 0L); + } + updateBeginningOffsets(begin); + } + + @Override + public synchronized void subscribe(Pattern pattern, ConsumerRebalanceListener listener) { + subscribeAttempts.incrementAndGet(); + if (failSubscribeTimes > 0) { + failSubscribeTimes--; + throw new IllegalStateException("simulated subscribe failure"); + } + if (!subscribed) { + subscribed = true; + assignKnown(); + for (ConsumerRecord r : new ArrayList<>(log)) { + addRecord(r); + } + } + } + + @Override + public synchronized void subscribe(Pattern pattern) { + subscribe(pattern, new ConsumerRebalanceListener() { + @Override + public void onPartitionsRevoked(java.util.Collection partitions) { + } + + @Override + public void onPartitionsAssigned(java.util.Collection partitions) { + } + }); + } + + @Override + public synchronized ConsumerRecords poll(Duration timeout) { + try { + Thread.sleep(10); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return super.poll(timeout); + } + + @Override + public synchronized void seek(TopicPartition partition, long offset) { + super.seek(partition, offset); + for (ConsumerRecord r : new ArrayList<>(log)) { + if (r.topic().equals(partition.topic()) && r.offset() >= offset) { + addRecord(r); + } + } + } + + @Override + public synchronized void commitSync(Map offsets) { + commits.add(new HashMap<>(offsets)); + super.commitSync(offsets); + } +} diff --git a/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerIT.java b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerIT.java new file mode 100644 index 0000000..147bd2b --- /dev/null +++ b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerIT.java @@ -0,0 +1,240 @@ +package io.amcp.broker.kafka; + +import io.amcp.core.Event; +import org.apache.kafka.clients.admin.Admin; +import org.apache.kafka.clients.admin.NewTopic; +import org.apache.kafka.common.errors.TopicExistsException; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; + +import java.net.InetSocketAddress; +import java.net.Socket; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assumptions.assumeTrue; + +/** + * Integration tests against a real Kafka broker. + * + *

Set {@code AMCP_KAFKA_BOOTSTRAP} (for example {@code 127.0.0.1:9092}) to run them; without it they are skipped, + * so a plain {@code mvn test} stays green on machines without Kafka. Every test uses its own random topic prefix + * and consumer groups, so tests do not see each other's events. + */ +class KafkaEventBrokerIT { + + private static String bootstrap; + private final List brokers = new ArrayList<>(); + private final String prefix = "amcpit" + UUID.randomUUID().toString().replace("-", "").substring(0, 10); + + @BeforeAll + static void requireKafka() { + bootstrap = System.getenv("AMCP_KAFKA_BOOTSTRAP"); + assumeTrue(bootstrap != null && !bootstrap.isBlank(), "AMCP_KAFKA_BOOTSTRAP not set: Kafka integration tests skipped"); + String[] hp = bootstrap.split(",")[0].split(":"); + try (Socket s = new Socket()) { + s.connect(new InetSocketAddress(hp[0], Integer.parseInt(hp[1])), 2000); + } catch (Exception e) { + assumeTrue(false, "Kafka not reachable at " + bootstrap); + } + } + + @AfterEach + void stopBrokers() throws Exception { + for (KafkaEventBroker b : brokers) { + b.stop().get(10, TimeUnit.SECONDS); + } + } + + private KafkaEventBroker broker(String group, String instance) throws Exception { + KafkaEventBroker b = new KafkaEventBroker(bootstrap, prefix + "-" + group, instance); + b.start().get(20, TimeUnit.SECONDS); + brokers.add(b); + return b; + } + + private static List sink() { + return Collections.synchronizedList(new ArrayList<>()); + } + + private static void await(List got, int n, long seconds) throws InterruptedException { + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(seconds); + while (got.size() < n && System.nanoTime() < end) { + Thread.sleep(50); + } + } + + private static void createTopic(String bootstrapServers, String topic, int partitions) throws Exception { + Map config = Map.of("bootstrap.servers", bootstrapServers); + try (Admin admin = Admin.create(config)) { + admin.createTopics(List.of(new NewTopic(topic, partitions, (short) 1))).all().get(30, TimeUnit.SECONDS); + } catch (ExecutionException e) { + if (!(e.getCause() instanceof TopicExistsException)) { + throw e; + } + } + } + + @Test + void roundTripKeepsTopicAndPayload() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List got = sink(); + String topic = prefix + ".bee.heartbeat"; + b.subscribe(topic, got::add); + b.publish(Event.create(topic, Map.of("bee", "b1", "load", 2))).get(10, TimeUnit.SECONDS); + await(got, 1, 30); + assertEquals(1, got.size(), "event delivered once"); + assertEquals(topic, got.get(0).getTopic(), "topic survives the round trip"); + @SuppressWarnings("unchecked") + Map payload = got.get(0).getPayload(Map.class); + assertEquals("b1", payload.get("bee")); + assertEquals(2, ((Number) payload.get("load")).intValue()); + } + + @Test + void severalSubscriptionsOnOneBrokerAllReceive() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List a = sink(); + List c = sink(); + b.subscribe(prefix + ".task.request", a::add); + b.subscribe(prefix + ".task.result", c::add); + b.publish(Event.create(prefix + ".task.request", Map.of("id", 1))).get(10, TimeUnit.SECONDS); + b.publish(Event.create(prefix + ".task.result", Map.of("id", 1))).get(10, TimeUnit.SECONDS); + await(a, 1, 30); + await(c, 1, 30); + assertEquals(1, a.size(), "first subscription still receives after a second subscribe"); + assertEquals(1, c.size(), "second subscription receives"); + } + + @Test + void singleLevelWildcardMatchesOneLevelOnly() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List got = sink(); + b.subscribe(prefix + ".bee.*", got::add); + b.publish(Event.create(prefix + ".bee.alpha", Map.of("n", 1))).get(10, TimeUnit.SECONDS); + b.publish(Event.create(prefix + ".bee.beta", Map.of("n", 2))).get(10, TimeUnit.SECONDS); + b.publish(Event.create(prefix + ".bee.alpha.deep", Map.of("n", 3))).get(10, TimeUnit.SECONDS); + await(got, 2, 30); + Thread.sleep(2000); // give a wrongly matched third event time to arrive + List topics = got.stream().map(Event::getTopic).sorted().toList(); + assertEquals(List.of(prefix + ".bee.alpha", prefix + ".bee.beta"), topics); + } + + @Test + void fanOutReachesEveryBrokerInstance() throws Exception { + KafkaEventBroker b1 = broker("shared", "i1"); + KafkaEventBroker b2 = broker("shared", "i2"); + List g1 = sink(); + List g2 = sink(); + String topic = prefix + ".host.announce"; + b1.subscribe(topic, g1::add); + b2.subscribe(topic, g2::add); + b1.publish(Event.create(topic, Map.of("hello", "mesh"))).get(10, TimeUnit.SECONDS); + await(g1, 1, 30); + await(g2, 1, 30); + assertEquals(1, g1.size(), "instance 1 receives a broadcast"); + assertEquals(1, g2.size(), "instance 2 receives the same broadcast"); + } + + @Test + void publishCompletesOnlyAfterTheBrokerAcknowledges() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + var f = b.publish(Event.create(prefix + ".ack", Map.of("x", 1))); + f.get(10, TimeUnit.SECONDS); + assertTrue(f.isDone() && !f.isCompletedExceptionally()); + } + + @Test + void queueDeliversEachEventOnceAcrossInstances() throws Exception { + String topic = prefix + ".jobs.run"; + createTopic(bootstrap, topic, 2); + KafkaEventBroker b1 = broker("qg", "i1"); + KafkaEventBroker b2 = broker("qg", "i2"); + List ids1 = Collections.synchronizedList(new ArrayList<>()); + List ids2 = Collections.synchronizedList(new ArrayList<>()); + b1.subscribeQueue(topic, "workers", e -> ids1.add(e.getId())); + b2.subscribeQueue(topic, "workers", e -> ids2.add(e.getId())); + assertTrue(b1.awaitQueueAssignment("workers", Duration.ofSeconds(60)), "instance 1 gets a partition"); + assertTrue(b2.awaitQueueAssignment("workers", Duration.ofSeconds(60)), "instance 2 gets a partition"); + + List> sent = new ArrayList<>(); + for (int i = 0; i < 50; i++) { + sent.add(b1.publish(Event.create(topic, Map.of("n", i)))); + } + CompletableFuture.allOf(sent.toArray(new CompletableFuture[0])).get(60, TimeUnit.SECONDS); + + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(60); + while (ids1.size() + ids2.size() < 50 && System.nanoTime() < end) { + Thread.sleep(50); + } + Thread.sleep(1000); // allow a duplicate delivery to show up before counting + Set union = new HashSet<>(ids1); + union.addAll(ids2); + assertEquals(50, ids1.size() + ids2.size(), "every event delivered exactly once in total"); + assertEquals(50, union.size(), "no event id delivered twice"); + assertTrue(!ids1.isEmpty(), "instance 1 received at least one event"); + assertTrue(!ids2.isEmpty(), "instance 2 received at least one event"); + } + + @Test + void topicCreatedAfterSubscribeIsDelivered() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List got = sink(); + String topic = prefix + ".late.topic"; + b.subscribe(prefix + ".late.*", got::add); + Thread.sleep(3000); // the pattern subscription is active before the topic exists + createTopic(bootstrap, topic, 1); + Thread.sleep(3000); // metadata refresh runs every 1 s + b.publish(Event.create(topic, Map.of("late", true))).get(20, TimeUnit.SECONDS); + await(got, 1, 60); + assertEquals(1, got.size(), "event on a topic created after subscribe is delivered"); + assertEquals(topic, got.get(0).getTopic()); + } + + @Test + void newInstanceDoesNotReplayOldHistory() throws Exception { + String topic = prefix + ".hist.event"; + KafkaEventBroker publisher = broker("hg", "old"); + Event old = Event.create(topic, Map.of("when", "old")); + publisher.publish(old).get(20, TimeUnit.SECONDS); + Thread.sleep(7000); + + KafkaEventBroker fresh = broker("hg", "new"); + List got = sink(); + fresh.subscribe(topic, got::add); + Event recent = Event.create(topic, Map.of("when", "new")); + fresh.publish(recent).get(20, TimeUnit.SECONDS); + await(got, 1, 60); + Thread.sleep(2000); // the old event would have arrived before the new one if it were replayed + + List ids = got.stream().map(Event::getId).toList(); + assertEquals(List.of(recent.getId()), ids, "history older than 5 s before subscribe is not replayed"); + } + + @Test + void doubleStarRespectsSegmentBoundary() throws Exception { + KafkaEventBroker b = broker("g", "i1"); + List got = sink(); + b.subscribe(prefix + ".p.**", got::add); + for (String t : List.of(prefix + ".p", prefix + ".p.x", prefix + ".px", prefix + ".p.x.y")) { + b.publish(Event.create(t, Map.of("t", t))).get(10, TimeUnit.SECONDS); + } + await(got, 3, 60); + Thread.sleep(2000); // give a wrongly matched "px" event time to arrive + List topics = got.stream().map(Event::getTopic).sorted().toList(); + assertEquals(List.of(prefix + ".p", prefix + ".p.x", prefix + ".p.x.y"), topics, + "p.** matches p and p., never px"); + } +} diff --git a/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerUnitTest.java b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerUnitTest.java new file mode 100644 index 0000000..606bd85 --- /dev/null +++ b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/KafkaEventBrokerUnitTest.java @@ -0,0 +1,225 @@ +package io.amcp.broker.kafka; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import io.amcp.core.Event; +import org.apache.kafka.common.TopicPartition; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.BooleanSupplier; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** Deterministic unit tests for the Kafka broker's worker logic. No Kafka needed: consumers are in-memory fakes. */ +class KafkaEventBrokerUnitTest { + + private KafkaEventBroker broker; + private ListAppender logs; + private Logger brokerLogger; + + @BeforeEach + void attachLog() { + brokerLogger = (Logger) LoggerFactory.getLogger(KafkaEventBroker.class); + logs = new ListAppender<>(); + logs.start(); + brokerLogger.addAppender(logs); + } + + @AfterEach + void cleanup() throws Exception { + brokerLogger.detachAppender(logs); + if (broker != null) { + broker.stop().get(15, TimeUnit.SECONDS); + } + } + + /** Broker whose consumers are fakes: fan-out workers get {@code fanout}, queue workers get {@code queue}. */ + private KafkaEventBroker brokerWith(FakeConsumer fanout, FakeConsumer queue) { + KafkaEventBroker b = new KafkaEventBroker("127.0.0.1:1", "g", "i1"); + b.consumerFactory = props -> String.valueOf(props.get("group.id")).contains(".queue.") ? queue : fanout; + b.baseRetryBackoffMs = 20; + b.batchTimeout = Duration.ofSeconds(5); + broker = b; + return b; + } + + private static void awaitTrue(BooleanSupplier c, long seconds, String what) throws InterruptedException { + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(seconds); + while (!c.getAsBoolean() && System.nanoTime() < end) { + Thread.sleep(10); + } + assertTrue(c.getAsBoolean(), "timed out waiting for: " + what); + } + + private static String json(String topic) { + return KafkaEventBroker.encode(Event.create(topic, Map.of("n", 1))); + } + + private long count(Level level, String fragment) { + synchronized (logs.list) { + return logs.list.stream() + .filter(e -> e.getLevel() == level && e.getFormattedMessage().contains(fragment)).count(); + } + } + + private static String chain(Throwable t) { + StringBuilder sb = new StringBuilder(); + for (; t != null; t = t.getCause()) { + sb.append(t.getMessage()).append(" | "); + } + return sb.toString(); + } + + // ---- item 1 ------------------------------------------------------------------------------------------- + + @Test + void consumerConstructionFailureSurfacesInStart() { + Properties bad = new Properties(); + bad.put("max.poll.records", "abc"); // consumer-only property, invalid value + broker = new KafkaEventBroker("127.0.0.1:1", "g", "i1", bad); + ExecutionException e = assertThrows(ExecutionException.class, + () -> broker.start().get(30, TimeUnit.SECONDS), "start() must fail, not report success"); + assertFalse(broker.isRunning()); + assertEquals(0, broker.workerCount(), "no dead worker may stay in the map"); + assertTrue(chain(e).contains("max.poll.records"), "the cause must name the bad property: " + chain(e)); + } + + @Test + void queueConsumerFailureSurfacesInSubscribeAndLeavesNoDeadWorker() throws Exception { + FakeConsumer fanout = new FakeConsumer(); + broker = new KafkaEventBroker("127.0.0.1:1", "g", "i1"); + broker.consumerFactory = props -> { + if (String.valueOf(props.get("group.id")).contains(".queue.")) { + throw new IllegalArgumentException("queue consumer cannot be built"); + } + return fanout; + }; + broker.start().get(30, TimeUnit.SECONDS); + int before = broker.workerCount(); + IllegalStateException ise = assertThrows(IllegalStateException.class, + () -> broker.subscribeQueue("t.a", "q", e -> { })); + assertInstanceOf(IllegalArgumentException.class, ise.getCause()); + assertEquals(before, broker.workerCount(), "the failed queue worker must not stay in the map"); + } + + // ---- item 2 ------------------------------------------------------------------------------------------- + + @Test + void failedSubscriptionUpdateIsRetriedAndLoggedOnce() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.failSubscribeTimes = 2; + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + List got = Collections.synchronizedList(new ArrayList<>()); + b.subscribeQueue("t.a", "q", got::add); + awaitTrue(() -> got.size() == 1, 8, "event delivered after the subscribe failures are retried"); + assertTrue(queue.subscribeAttempts.get() >= 3, "attempts: " + queue.subscribeAttempts.get()); + assertEquals(1, count(Level.ERROR, "could not update its subscription"), + "one failure episode is logged once, with the cause"); + } + + // ---- item 3 ------------------------------------------------------------------------------------------- + + @Test + void handlerFailureRedeliversAndCommitsOnlyAfterSuccess() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + AtomicInteger calls = new AtomicInteger(); + List ids = Collections.synchronizedList(new ArrayList<>()); + b.subscribeQueue("t.a", "q", e -> { + ids.add(e.getId()); + if (calls.incrementAndGet() == 1) { + throw new IllegalStateException("handler killed mid-batch"); + } + }); + awaitTrue(() -> calls.get() >= 2, 8, "redelivery after the handler threw"); + awaitTrue(() -> queue.commits.size() >= 1, 8, "commit after the successful redelivery"); + assertEquals(2, calls.get()); + assertEquals(ids.get(0), ids.get(1), "the same event is redelivered"); + assertEquals(1, queue.commits.size(), "nothing was committed for the failed attempt"); + assertEquals(1L, queue.commits.get(0).get(new TopicPartition("t.a", 0)).offset()); + } + + @Test + void offsetsAreNotCommittedWhileAHandlerIsStillRunning() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + CountDownLatch entered = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + b.subscribeQueue("t.a", "q", e -> { + entered.countDown(); + try { + release.await(20, TimeUnit.SECONDS); + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + } + }); + assertTrue(entered.await(8, TimeUnit.SECONDS)); + Thread.sleep(500); + assertTrue(queue.commits.isEmpty(), "must not commit before the handler finished (at-most-once bug)"); + release.countDown(); + awaitTrue(() -> queue.commits.size() == 1, 8, "commit after the handler completed"); + } + + @Test + void poisonRecordIsGivenUpAfterBoundedAttemptsAndCommitted() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + AtomicInteger calls = new AtomicInteger(); + b.subscribeQueue("t.a", "q", e -> { + calls.incrementAndGet(); + throw new IllegalStateException("always fails"); + }); + awaitTrue(() -> queue.commits.size() == 1, 10, "give-up commit"); + Thread.sleep(300); + assertEquals(KafkaEventBroker.MAX_DELIVERY_ATTEMPTS, calls.get(), "bounded redelivery, no endless loop"); + assertEquals(1, count(Level.ERROR, "Giving up")); + } + + // ---- item 4 ------------------------------------------------------------------------------------------- + + @Test + void recordWithoutLocalMatchWarnsOncePerTopicAndQueue() throws Exception { + FakeConsumer queue = new FakeConsumer(); + queue.append("t.other", json("t.other")); + queue.append("t.other", json("t.other")); + queue.append("t.third", json("t.third")); + queue.append("t.a", json("t.a")); + KafkaEventBroker b = brokerWith(new FakeConsumer(), queue); + b.start().get(30, TimeUnit.SECONDS); + List got = Collections.synchronizedList(new ArrayList<>()); + b.subscribeQueue("t.a", "workers", got::add); + awaitTrue(() -> got.size() == 1, 8, "the matching event is delivered"); + awaitTrue(() -> queue.commits.size() >= 1, 8, "batch committed"); + assertEquals(1, count(Level.WARN, "t.other"), "one WARN for (t.other, workers), not one per record"); + assertEquals(1, count(Level.WARN, "t.third")); + assertTrue(logs.list.stream().anyMatch(e -> e.getLevel() == Level.WARN + && e.getFormattedMessage().contains("workers") && e.getFormattedMessage().contains("t.other"))); + } +} diff --git a/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/TopicPatternsTest.java b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/TopicPatternsTest.java new file mode 100644 index 0000000..d34656b --- /dev/null +++ b/amcp-broker-kafka/src/test/java/io/amcp/broker/kafka/TopicPatternsTest.java @@ -0,0 +1,111 @@ +package io.amcp.broker.kafka; + +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.regex.Pattern; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** Unit tests for the AMCP-pattern to Kafka-regex translation. No Kafka needed. */ +class TopicPatternsTest { + + private static boolean matches(String amcpPattern, String topic) { + return Pattern.compile(TopicPatterns.toRegex(amcpPattern)).matcher(topic).matches(); + } + + @Test + void exactPatternMatchesOnlyThatTopicAndTreatsDotsLiterally() { + assertTrue(matches("a.b.c", "a.b.c")); + assertFalse(matches("a.b.c", "a.b.cd")); + assertFalse(matches("a.b.c", "aXb.c"), "dots are literal, not regex wildcards"); + assertFalse(matches("a.b.c", "a.b.c.d")); + } + + @Test + void singleStarMatchesExactlyOneSegment() { + assertTrue(matches("p.*", "p.x")); + assertFalse(matches("p.*", "p")); + assertFalse(matches("p.*", "p.x.y")); + assertFalse(matches("p.*", "px.x")); + assertFalse(matches("p.*", "p."), "an empty segment is not a match"); + } + + @Test + void doubleStarMatchesPrefixAndAnyDepthButRespectsSegmentBoundary() { + assertTrue(matches("p.**", "p")); + assertTrue(matches("p.**", "p.x")); + assertTrue(matches("p.**", "p.x.y.z")); + assertFalse(matches("p.**", "px"), "p.** must not match a longer first segment"); + assertFalse(matches("p.**", "phive.x")); + assertFalse(matches("p.**", "x.p")); + } + + @Test + void unionMatchesAnyMemberAndNothingElse() { + Pattern union = TopicPatterns.union(List.of("a.b", "c.*", "d.**")); + assertTrue(union.matcher("a.b").matches()); + assertTrue(union.matcher("c.x").matches()); + assertTrue(union.matcher("d").matches()); + assertTrue(union.matcher("d.e.f").matches()); + assertFalse(union.matcher("c").matches()); + assertFalse(union.matcher("a.c").matches()); + assertFalse(union.matcher("dx").matches()); + } + + @Test + void unionOfNothingIsNull() { + assertNull(TopicPatterns.union(Set.of())); + } + + @Test + void unionAcceptsSortedSetOfPatterns() { + Set wanted = new TreeSet<>(List.of("z.*", "a.**")); + Pattern union = TopicPatterns.union(wanted); + assertTrue(union.matcher("a.q").matches()); + assertTrue(union.matcher("z.q").matches()); + } + + /** Pattern, topic, expected. The SAME table runs through Event.matchesTopic and TopicPatterns.toRegex. */ + private static final String[][] AGREEMENT_TABLE = { + {"a.b.c", "a.b.c", "true"}, + {"a.b.c", "a.b.cd", "false"}, + {"p.*", "p.x", "true"}, + {"p.*", "p", "false"}, + {"p.*", "p.x.y", "false"}, + {"p.*", "px.x", "false"}, + {"p.*", "p.", "false"}, + {"p.*", "p.x.", "false"}, + {"p.*", "p..", "false"}, + {"a.b.*", "a.b.x", "true"}, + {"a.b.*", "a.b.x.", "false"}, + {"p.**", "p", "true"}, + {"p.**", "p.x", "true"}, + {"p.**", "p.x.y", "true"}, + {"p.**", "px", "false"}, + {"p.**", "p.", "false"}, + {"p.**", "p.x.", "false"}, + {"p.**", "p..x", "false"}, + {"p.**", "p.x..y", "false"}, + {"p.", "p.", "true"}, + {"p.", "p", "false"}, + {"p.x.", "p.x.", "true"}, + }; + + @Test + void eventMatchesTopicAndTopicPatternsAgreeOnEveryCase() { + for (String[] row : AGREEMENT_TABLE) { + boolean expected = Boolean.parseBoolean(row[2]); + boolean viaEvent = io.amcp.core.Event.create(row[1], java.util.Map.of("n", 1)).matchesTopic(row[0]); + boolean viaRegex = matches(row[0], row[1]); + org.junit.jupiter.api.Assertions.assertEquals(expected, viaEvent, + "Event.matchesTopic(" + row[0] + ") on topic '" + row[1] + "'"); + org.junit.jupiter.api.Assertions.assertEquals(expected, viaRegex, + "TopicPatterns.toRegex(" + row[0] + ") on topic '" + row[1] + "'"); + } + } +} diff --git a/amcp-core/pom.xml b/amcp-core/pom.xml index 9b06407..bd654ab 100644 --- a/amcp-core/pom.xml +++ b/amcp-core/pom.xml @@ -70,6 +70,9 @@ org.apache.maven.plugins maven-surefire-plugin + + -Xmx512m + diff --git a/amcp-core/src/main/java/io/amcp/core/Event.java b/amcp-core/src/main/java/io/amcp/core/Event.java index 340a44a..df18ef1 100644 --- a/amcp-core/src/main/java/io/amcp/core/Event.java +++ b/amcp-core/src/main/java/io/amcp/core/Event.java @@ -162,30 +162,36 @@ public boolean matchesTopic(String pattern) { return true; } - // Handle wildcards + // Handle wildcards. A wildcard never matches an empty segment, so topics with a trailing dot + // ("p.x.") or a doubled dot ("p..x") match only by exact equality. This is the same rule as + // io.amcp.broker.kafka.TopicPatterns.toRegex. if (pattern.endsWith(".**")) { + // "p.**" matches "p" itself and "p.", never "px". String prefix = pattern.substring(0, pattern.length() - 3); - return topic.startsWith(prefix); + if (topic.equals(prefix)) { + return true; + } + return topic.startsWith(prefix + ".") && nonEmptySegments(topic.substring(prefix.length() + 1)); } - + if (pattern.endsWith(".*")) { + // "p.*" matches "p.". String prefix = pattern.substring(0, pattern.length() - 2); - String[] topicParts = topic.split("\\."); - String[] patternParts = prefix.split("\\."); - - if (topicParts.length == patternParts.length + 1) { - for (int i = 0; i < patternParts.length; i++) { - if (!topicParts[i].equals(patternParts[i])) { - return false; - } - } - return true; + if (!topic.startsWith(prefix + ".")) { + return false; } + String rest = topic.substring(prefix.length() + 1); + return !rest.isEmpty() && rest.indexOf('.') < 0; } - + return false; } + /** True when {@code rest} is one or more dot-separated segments and none of them is empty. */ + private static boolean nonEmptySegments(String rest) { + return !rest.isEmpty() && !rest.startsWith(".") && !rest.endsWith(".") && !rest.contains(".."); + } + @Override public String toString() { return String.format("Event{id=%s, topic=%s, source=%s, time=%s}", diff --git a/amcp-core/src/main/java/io/amcp/core/EventBroker.java b/amcp-core/src/main/java/io/amcp/core/EventBroker.java index 2b0437d..80d9568 100644 --- a/amcp-core/src/main/java/io/amcp/core/EventBroker.java +++ b/amcp-core/src/main/java/io/amcp/core/EventBroker.java @@ -26,6 +26,24 @@ public interface EventBroker extends AutoCloseable { */ String subscribe(String topicPattern, Consumer handler); + /** + * Subscribes to events matching the topic pattern as one member of a named queue + * (competing consumers). Each event is delivered to exactly one subscriber that shares + * the same queue name, instead of to every subscriber as {@link #subscribe} does. + * + *

The default implementation ignores {@code queueName} and falls back to + * {@link #subscribe(String, Consumer)}, which means fan-out delivery. Brokers that can + * distribute work across subscribers should override this method. + * + * @param topicPattern the topic pattern (supports wildcards like "hello.*") + * @param queueName the queue name shared by the competing subscribers + * @param handler the event handler + * @return a subscription ID that can be used to unsubscribe + */ + default String subscribeQueue(String topicPattern, String queueName, Consumer handler) { + return subscribe(topicPattern, handler); + } + /** * Unsubscribes from a previously created subscription. * diff --git a/amcp-core/src/main/java/io/amcp/core/InMemoryEventBroker.java b/amcp-core/src/main/java/io/amcp/core/InMemoryEventBroker.java index b71c781..d35180e 100644 --- a/amcp-core/src/main/java/io/amcp/core/InMemoryEventBroker.java +++ b/amcp-core/src/main/java/io/amcp/core/InMemoryEventBroker.java @@ -3,32 +3,43 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; import java.util.Map; import java.util.UUID; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Consumer; /** * In-memory implementation of EventBroker for development and testing. * Events are routed directly within the JVM without external message broker. + * + *

{@link #subscribe} delivers every event to every matching subscriber (fan-out). + * {@link #subscribeQueue} delivers each event to one subscriber per queue name, + * chosen round-robin among the subscribers that share that queue name. */ public class InMemoryEventBroker implements EventBroker { private static final Logger logger = LoggerFactory.getLogger(InMemoryEventBroker.class); - + private final Map subscriptions = new ConcurrentHashMap<>(); - private final ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor(); + private final Map queueCursors = new ConcurrentHashMap<>(); + private volatile ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor(); private volatile boolean running = false; private static class Subscription { final String topicPattern; final Consumer handler; + final String queueName; // null for fan-out subscriptions - Subscription(String topicPattern, Consumer handler) { + Subscription(String topicPattern, Consumer handler, String queueName) { this.topicPattern = topicPattern; this.handler = handler; + this.queueName = queueName; } } @@ -41,29 +52,58 @@ public CompletableFuture publish(Event event) { return CompletableFuture.runAsync(() -> { logger.debug("Publishing event: {}", event.getTopic()); - - // Find matching subscriptions + + // Fan-out subscriptions all receive the event; queue subscriptions are grouped by + // queue name and each group receives it once, round-robin. + Map> queues = new LinkedHashMap<>(); for (Subscription sub : subscriptions.values()) { - if (event.matchesTopic(sub.topicPattern)) { - try { - executor.submit(() -> sub.handler.accept(event)); - } catch (Exception e) { - logger.error("Error delivering event to subscriber: {}", - e.getMessage(), e); - } + if (!event.matchesTopic(sub.topicPattern)) { + continue; } + if (sub.queueName == null) { + deliver(sub, event); + } else { + queues.computeIfAbsent(sub.queueName, k -> new ArrayList<>()).add(sub); + } + } + for (Map.Entry> q : queues.entrySet()) { + List members = q.getValue(); + int index = Math.floorMod( + queueCursors.computeIfAbsent(q.getKey(), k -> new AtomicInteger()).getAndIncrement(), + members.size()); + deliver(members.get(index), event); } }, executor); } + private void deliver(Subscription sub, Event event) { + try { + executor.submit(() -> sub.handler.accept(event)); + } catch (Exception e) { + logger.error("Error delivering event to subscriber: {}", e.getMessage(), e); + } + } + @Override public String subscribe(String topicPattern, Consumer handler) { String subscriptionId = UUID.randomUUID().toString(); - subscriptions.put(subscriptionId, new Subscription(topicPattern, handler)); + subscriptions.put(subscriptionId, new Subscription(topicPattern, handler, null)); logger.debug("Created subscription {} for pattern {}", subscriptionId, topicPattern); return subscriptionId; } + @Override + public String subscribeQueue(String topicPattern, String queueName, Consumer handler) { + if (queueName == null || queueName.isBlank()) { + throw new IllegalArgumentException("queueName must not be blank"); + } + String subscriptionId = UUID.randomUUID().toString(); + subscriptions.put(subscriptionId, new Subscription(topicPattern, handler, queueName)); + logger.debug("Created queue subscription {} for pattern {} in queue {}", + subscriptionId, topicPattern, queueName); + return subscriptionId; + } + @Override public void unsubscribe(String subscriptionId) { subscriptions.remove(subscriptionId); @@ -71,22 +111,26 @@ public void unsubscribe(String subscriptionId) { } @Override - public CompletableFuture start() { + public synchronized CompletableFuture start() { if (running) { return CompletableFuture.completedFuture(null); } - + + // stop() shuts the executor down for good, so a restart needs a fresh one. + if (executor.isShutdown()) { + executor = Executors.newVirtualThreadPerTaskExecutor(); + } running = true; logger.info("InMemoryEventBroker started"); return CompletableFuture.completedFuture(null); } @Override - public CompletableFuture stop() { + public synchronized CompletableFuture stop() { if (!running) { return CompletableFuture.completedFuture(null); } - + running = false; subscriptions.clear(); executor.shutdown(); diff --git a/amcp-core/src/test/java/io/amcp/core/EventTest.java b/amcp-core/src/test/java/io/amcp/core/EventTest.java index 1479c2a..b900691 100644 --- a/amcp-core/src/test/java/io/amcp/core/EventTest.java +++ b/amcp-core/src/test/java/io/amcp/core/EventTest.java @@ -47,6 +47,16 @@ void testWildcardMatching() { assertTrue(event2.matchesTopic("hello.**")); } + @Test + void testDoubleStarRespectsSegmentBoundary() { + assertTrue(Event.create("p", "test").matchesTopic("p.**"), "p.** matches the bare prefix p"); + assertTrue(Event.create("p.x", "test").matchesTopic("p.**")); + assertTrue(Event.create("p.x.y", "test").matchesTopic("p.**")); + assertFalse(Event.create("px", "test").matchesTopic("p.**"), "p.** must not match phive-style prefixes"); + assertFalse(Event.create("phive.x", "test").matchesTopic("p.**")); + assertFalse(Event.create("other.p", "test").matchesTopic("p.**")); + } + @Test void testComplexPayload() { record TestData(String name, int value) {} diff --git a/amcp-core/src/test/java/io/amcp/core/InMemoryEventBrokerTest.java b/amcp-core/src/test/java/io/amcp/core/InMemoryEventBrokerTest.java new file mode 100644 index 0000000..ab4b436 --- /dev/null +++ b/amcp-core/src/test/java/io/amcp/core/InMemoryEventBrokerTest.java @@ -0,0 +1,76 @@ +package io.amcp.core; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class InMemoryEventBrokerTest { + + private InMemoryEventBroker broker; + + @BeforeEach + void start() throws Exception { + broker = new InMemoryEventBroker(); + broker.start().get(5, TimeUnit.SECONDS); + } + + @AfterEach + void stop() throws Exception { + broker.stop().get(5, TimeUnit.SECONDS); + } + + @Test + void queueSubscribersShareEventsRoundRobin() throws Exception { + List a = Collections.synchronizedList(new ArrayList<>()); + List b = Collections.synchronizedList(new ArrayList<>()); + broker.subscribeQueue("jobs.run", "workers", a::add); + broker.subscribeQueue("jobs.run", "workers", b::add); + + for (int i = 0; i < 4; i++) { + broker.publish(Event.create("jobs.run", Integer.valueOf(i))).get(5, TimeUnit.SECONDS); + } + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (a.size() + b.size() < 4 && System.nanoTime() < end) { + Thread.sleep(20); + } + assertEquals(2, a.size(), "round-robin gives the first queue member half the events"); + assertEquals(2, b.size(), "round-robin gives the second queue member half the events"); + } + + @Test + void fanOutSubscribersAllReceive() throws Exception { + List a = Collections.synchronizedList(new ArrayList<>()); + List b = Collections.synchronizedList(new ArrayList<>()); + broker.subscribe("news.item", a::add); + broker.subscribe("news.item", b::add); + + broker.publish(Event.create("news.item", "x")).get(5, TimeUnit.SECONDS); + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while ((a.isEmpty() || b.isEmpty()) && System.nanoTime() < end) { + Thread.sleep(20); + } + assertEquals(1, a.size()); + assertEquals(1, b.size()); + } + + @Test + void brokerCanBeRestartedAfterStop() throws Exception { + broker.stop().get(5, TimeUnit.SECONDS); + broker.start().get(5, TimeUnit.SECONDS); + List got = Collections.synchronizedList(new ArrayList<>()); + broker.subscribe("again.*", got::add); + broker.publish(Event.create("again.x", Integer.valueOf(1))).get(5, TimeUnit.SECONDS); + long end = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (got.isEmpty() && System.nanoTime() < end) { + Thread.sleep(20); + } + assertEquals(1, got.size(), "publish after stop() then start() must deliver, not throw RejectedExecutionException"); + } +}