From 124f085509240ffce5b5c72e64b8fa6104d0c97d Mon Sep 17 00:00:00 2001 From: Aayush Kataria Date: Wed, 18 Nov 2026 08:35:52 -0800 Subject: [PATCH 1/3] Retry failed change-feed batches and skip orchestrations that already exist The change-feed trigger dropped a batch that failed, so the turns in it were never counted. The host now retries it up to five times with exponential backoff, then drops it as before. A retry repeats only what failed. Counters already skip turns they have counted. _safe_start now reads an instance status first and leaves an instance that exists alone, because Durable Functions starts a finished instance again when its id is reused. The highest change-feed LSN of the batch joins the instance id hash, so a count reached again after a counter expired is not mistaken for the earlier run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 3 + Docs/architecture/orchestrators.md | 2 + Docs/concepts.md | 3 +- Docs/troubleshooting.md | 5 +- function_app/shared/instance_ids.py | 14 +- function_app/triggers/change_feed.py | 73 +++++-- tests/unit/conftest.py | 8 + tests/unit/function_app/test_change_feed.py | 213 +++++++++++++++++-- tests/unit/function_app/test_instance_ids.py | 21 +- 9 files changed, 300 insertions(+), 42 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f0b951c..4501cae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,9 @@ #### Breaking Changes * **The containers use a new partition key.** `memories`, `memories_turns`, `memories_summaries` and `counter` are keyed `[/tenant_id, /user_id]` instead of `[/user_id, /thread_id]` (the Bicep template and `create_memory_store()`). A partition key can't change in place, so recreate those containers and `leases`, in the order given in the [migration notes](README.md#migration-notes) (every writer stopped, in-flight orchestrations purged, the Function App deployed before the upgraded writers); existing data is not migrated. Record ids must now be unique per owner across threads, not per thread. See [PR:#53](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/53) +#### Bugs Fixed +* **A failed change-feed batch is retried instead of dropped.** The Function App's trigger now retries a batch that fails up to five times, with exponential backoff starting at five seconds, where it used to drop the batch and never count its turns. A retry repeats only what failed: a counter skips the turns it has already counted, and an orchestration whose instance already exists is not started again. + #### Other Changes * Orchestration instance ids in the Function App are now `:[:]`, where the digest is a fixed-length hash of the tenant, owner and thread. They stay within Durable Functions' 100-character limit and no longer contain user ids. See [PR:#53](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/53) diff --git a/Docs/architecture/orchestrators.md b/Docs/architecture/orchestrators.md index a5ef580..f4e18e7 100644 --- a/Docs/architecture/orchestrators.md +++ b/Docs/architecture/orchestrators.md @@ -29,6 +29,8 @@ A batch whose model call fails with a retryable error yields no facts and is cou Every orchestration input and activity payload carries `tenant_id`, which the change feed trigger reads from the changed turn, and each activity runs the pipeline of that tenant (one per tenant, sharing the Cosmos DB and model clients), so one Function App serves every tenant. An orchestration instance ID is `{kind}:{digest}:{count}`, where `kind` is `extract`, `episode`, `thread_summary`, `user_summary` or `procedural` and `digest` is a fixed-length digest of the tenant, owner and (where there is one) thread, so IDs stay within the 100-character limit and hold no user IDs. +A batch the host retries derives the same IDs, and the trigger reads the status of an ID before it starts it: an instance that exists, in any status, is left alone, because Durable Functions alone refuses an ID only while its instance is running and would start a finished one again. A `409` for an instance that has not started running yet, which its status does not show, is skipped the same way. A failed status read fails the batch, so the host retries it. For the orchestrations the change feed starts, the digest also covers the highest `_lsn` of the batch that crossed the threshold, so a count that is reached again after its counter expired (a counter is deleted 90 days after its last update) gets an ID of its own and is not mistaken for the instance of the first time. + Fact IDs are deterministic from user, thread, and normalized content, and episode IDs from user, thread, the segment's first and last turn and the episode's position in it. Thread and user summaries keep their deterministic summary IDs. Procedure IDs derive from the user, kind and name, and a combined fact's ID from the user and its source IDs, so a retried activity finds the record it already wrote (409) instead of writing a second one. `UserSummaryOrchestrator` can start before extraction has persisted anything for a new user. `us_Extract` then returns `no_memories_yet` instead of failing, and the orchestrator polls on a durable timer (App Settings `USER_SUMMARY_WAIT_SECONDS`, default `120`, every `USER_SUMMARY_WAIT_INTERVAL_SECONDS`, default `10`) before it skips that cadence; the next user-summary threshold tries again. `USER_SUMMARY_WAIT_SECONDS=0` makes it a single attempt. diff --git a/Docs/concepts.md b/Docs/concepts.md index b92afef..35ccc45 100644 --- a/Docs/concepts.md +++ b/Docs/concepts.md @@ -370,7 +370,8 @@ on_memory_change trigger 1. The change feed trigger watches the `memories_turns` container for new documents. 2. Only documents with `type == "turn"` are counted. The trigger reads `tenant_id` from each changed turn, next to `user_id` and `thread_id`, and skips a turn whose tenant, owner or thread id is missing or not valid (the rules are in [Tenants and Owners](#tenants-and-owners)) with a warning. 3. Documents in the dedicated `counter` container track how many turns have been seen per scope using ETag-based optimistic concurrency. -4. When a counter crosses a configured threshold, the corresponding Durable Functions orchestration is started automatically, in the tenant of the turns it counted. The Function App needs no tenant setting. A failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch, so every other tenant's counters are still updated; the batch still fails at the end so the host records it. +4. When a counter crosses a configured threshold, the corresponding Durable Functions orchestration is started automatically, in the tenant of the turns it counted. The Function App needs no tenant setting. A failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch, so every other tenant's counters are still updated; the batch still fails at the end, so the host records it and retries it. +5. A batch that fails is retried by the host, up to five times with exponential backoff starting at five seconds (about a minute and a half in all when every retry fails). The batch keeps its lease while it is retried, so the turns written after it wait. If it still fails it is dropped and the turns in it are never counted. A retry repeats only what failed: a counter skips the turns it has already counted, and an orchestration whose instance already exists, in any status, is not started again, so a retry never runs a finished or failed orchestration a second time. ### Threshold settings diff --git a/Docs/troubleshooting.md b/Docs/troubleshooting.md index a545a68..f2b095e 100644 --- a/Docs/troubleshooting.md +++ b/Docs/troubleshooting.md @@ -208,11 +208,12 @@ If nothing fires: - verify the Functions host shows the Cosmos DB trigger - confirm the `leases` container exists -- confirm the `memories`, `memories_turns`, `memories_summaries` and `counter` containers exist and are keyed `[/tenant_id, /user_id]`: the Function App checks them on its first change-feed batch, and a missing or wrongly keyed one fails the batch with a `Topology error` that names the container (for example `Topology error: container '' does not exist`) +- confirm the `memories`, `memories_turns`, `memories_summaries` and `counter` containers exist and are keyed `[/tenant_id, /user_id]`: the Function App checks them on its first change-feed batch, and a missing or wrongly keyed one fails the batch with a `Topology error` that names the container (for example `Topology error: container '' does not exist`). The host retries that batch like any other (see below), and it fails every time until the container is fixed - confirm the `counter` container is writable - confirm `MEMORY_PROCESSOR_OWNER=durable` is set on the Function App - check the Function App log for warnings about skipped turns: a turn needs a valid `tenant_id`, `user_id` and `thread_id` (the SDK stamps the tenant on every record it writes and refuses an invalid owner or thread id), and the trigger skips one that lacks them -- if only some threads or owners fire: a failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch (every other tenant's counters are still updated), but the batch still fails at the end so the host records it +- if only some threads or owners fire: a failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch (every other tenant's counters are still updated), but the batch still fails at the end, so the host records it and retries it +- if a batch fails: the host retries it up to five times, with exponential backoff starting at five seconds, so a transient error (a throttled Cosmos DB request, an unreachable Durable Functions endpoint) shows in the log for each attempt and the batch then succeeds. The batch keeps its lease while it is retried, so turns written after it are not processed until it succeeds or the retries are used up (about a minute and a half when every retry fails). A retry repeats only what failed, so `change-feed: instance of already exists, skipping` for an orchestration the failed attempt had started is expected and is not a duplicate run. If every retry fails, the batch is dropped and the turns in it are never counted: fix the error the log names - insert enough new turn documents to cross the configured threshold - check for generated documents where the Cosmos JSON field is `type="thread_summary"`, `type="fact"`, or `type="user_summary"` diff --git a/function_app/shared/instance_ids.py b/function_app/shared/instance_ids.py index 2c4ef90..71d05ac 100644 --- a/function_app/shared/instance_ids.py +++ b/function_app/shared/instance_ids.py @@ -2,8 +2,9 @@ Durable Functions takes an instance id of 1-100 characters without ``/``, ``\\``, ``#``, ``?`` or control characters, and tenant, user and thread ids are free text, so an instance id never carries them: it is a readable ``kind``, a -fixed-length hash of the owner (and thread) and, for a counter threshold, the count. The same owner, thread and count -always give the same id, which is what makes a replayed change-feed batch start each orchestration once. +fixed-length hash of the owner (and thread) and, for a counter threshold, the count. The same owner, thread, count and +batch always give the same id, which is what lets a replayed change-feed batch find the orchestrations it already +started and leave them alone. """ from __future__ import annotations @@ -11,11 +12,16 @@ from hashlib import sha256 -def orchestration_instance_id(kind: str, *parts: str, count: int | None = None) -> str: +def orchestration_instance_id(kind: str, *parts: str, count: int | None = None, lsn: int | None = None) -> str: """``{kind}:{digest}`` plus ``:{count}`` when a count is given. ``parts`` is ``(tenant_id, user_id)`` or ``(tenant_id, user_id, thread_id)``. They are joined with the unit separator before hashing, so a ``:`` inside an id cannot make two different splits give the same digest. + + ``lsn`` is the highest change-feed sequence number of the batch that reached the count, and joins the hash. A + replay of that batch has the same ``lsn`` and so the same id, while a count reached again after the counter + expired (it lives 90 days) comes from a later batch and gets an id of its own, not that of a finished instance. """ - digest = sha256("\x1f".join(parts).encode("utf-8")).hexdigest()[:32] + hashed = parts if lsn is None else (*parts, str(lsn)) + digest = sha256("\x1f".join(hashed).encode("utf-8")).hexdigest()[:32] return f"{kind}:{digest}" + (f":{count}" if count is not None else "") diff --git a/function_app/triggers/change_feed.py b/function_app/triggers/change_feed.py index 5f7c661..828f4db 100644 --- a/function_app/triggers/change_feed.py +++ b/function_app/triggers/change_feed.py @@ -9,6 +9,9 @@ inputs are per ``(tenant_id, user_id[, thread_id])``, so one Function App serves every tenant that writes to the containers. A failure for one thread or owner does not stop the others in the batch; the batch fails at the end. +A failed batch is retried by the host (see :data:`_RETRY_POLICY`). A retry only repeats what failed: a counter skips +the turns it has already counted, and an orchestration whose instance exists is not started again. + The trigger is registered as a :class:`df.Blueprint` and wired into the top-level :class:`df.DFApp` from ``function_app.py``. The pure-logic entry point :func:`process_changefeed_batch` accepts plain ``dict`` documents and a @@ -269,7 +272,7 @@ async def process_changefeed_batch( if n_thread > 0 and crosses_threshold(old_count, new_count, n_thread): instance_id = orchestration_instance_id( - "thread_summary", tenant_id, user_id, thread_id, count=new_count + "thread_summary", tenant_id, user_id, thread_id, count=new_count, lsn=lsn ) await _safe_start( starter, @@ -280,7 +283,9 @@ async def process_changefeed_batch( ) if n_facts > 0 and crosses_threshold(old_count, new_count, n_facts): - instance_id = orchestration_instance_id("extract", tenant_id, user_id, thread_id, count=new_count) + instance_id = orchestration_instance_id( + "extract", tenant_id, user_id, thread_id, count=new_count, lsn=lsn + ) should_reconcile = bool( n_dedup_turns > 0 and crosses_threshold(old_count, new_count, n_dedup_turns) ) @@ -310,7 +315,9 @@ async def process_changefeed_batch( ) if n_episode > 0 and crosses_threshold(old_count, new_count, n_episode): - instance_id = orchestration_instance_id("episode", tenant_id, user_id, thread_id, count=new_count) + instance_id = orchestration_instance_id( + "episode", tenant_id, user_id, thread_id, count=new_count, lsn=lsn + ) await _safe_start( starter, "ExtractEpisodesOrchestrator", @@ -338,7 +345,9 @@ async def process_changefeed_batch( batch_max_lsn=lsn, ) if crosses_threshold(old_count, new_count, n_user): - instance_id = orchestration_instance_id("user_summary", tenant_id, user_id, count=new_count) + instance_id = orchestration_instance_id( + "user_summary", tenant_id, user_id, count=new_count, lsn=lsn + ) await _safe_start( starter, "UserSummaryOrchestrator", @@ -354,9 +363,10 @@ async def process_changefeed_batch( except Exception as exc: _record_key_failure(exc, key_errors) - # Re-raise so the host records the batch as failed, and retries it when a retry policy is set: a counter skips - # turns it has already counted, so a retry is safe. The host logs this error and any it was raised from, so it - # names the first failure safely instead of chaining the SDK error and its text. + # Re-raise so the host records the batch as failed and, under the retry policy, runs it again. A retry is safe: a + # counter skips turns it has already counted, and ``_safe_start`` leaves an instance that exists alone. The host + # logs this error and any it was raised from, so it names the first failure safely instead of chaining the SDK + # error and its text. if orchestration_errors or key_errors: problems: list[str] = [] if orchestration_errors: @@ -376,23 +386,38 @@ async def _safe_start( client_input: dict, errors: list[Exception], ) -> None: - logger.info( - "change-feed: starting %s instance=%s tenant=%s user=%s thread=%s", - orchestration_name, - instance_id, - client_input["tenant_id"], - client_input["user_id"], - client_input.get("thread_id"), - ) + """Start ``orchestration_name`` as ``instance_id``, unless an instance with that id exists. + + A retried or redelivered batch derives the ids of the orchestrations it already started, so an instance that + exists, in any state, is the run this threshold crossing started and is left as it is. Durable Functions refuses + an id only while its instance is still running and starts a finished one again, which is why the status is read + first. The 409 it still answers for an instance that is starting or running covers two deliveries racing. A + failure of either call is recorded in ``errors``, which fails the batch so the host retries it. + """ try: + if await starter.get_status(instance_id): + logger.info( + "change-feed: instance %s of %s already exists, skipping", + instance_id, + orchestration_name, + ) + return + logger.info( + "change-feed: starting %s instance=%s tenant=%s user=%s thread=%s", + orchestration_name, + instance_id, + client_input["tenant_id"], + client_input["user_id"], + client_input.get("thread_id"), + ) await starter.start_new( orchestration_name, instance_id=instance_id, client_input=client_input, ) except Exception as exc: - # Duplicate-instance-id is the idempotency mechanism for change-feed - # retries: it shows up as a 409 Conflict from the Durable client. + # An instance that is starting or running is refused with a 409 Conflict by the Durable client, the second + # guard after the status read. # The Durable Python SDK currently raises bare ``Exception`` for # everything (no typed class for OrchestrationAlreadyExists), so # we keep two signals - status_code first, then the canonical @@ -429,6 +454,19 @@ async def _safe_start( # Trigger registration # --------------------------------------------------------------------------- +# What the host does with a batch that fails: it runs the batch again, with the same documents, after an exponential +# backoff delay of at least the minimum and at most the maximum. The count is bounded because a failure that does not +# go away, such as a container with the wrong partition key, must be given up on and not retried forever; once the +# retries are used up the batch is dropped, as it was before there was a policy. The batch keeps its lease while it is +# retried, so the changes behind it wait: about a minute and a half when every retry fails. The host takes every value +# as a string. +_RETRY_POLICY = { + "strategy": "exponential_backoff", + "max_retry_count": "5", + "minimum_interval": "00:00:05", + "maximum_interval": "00:05:00", +} + # Bind directly to the turns container so only turn writes can fire this trigger. @bp.cosmos_db_trigger( @@ -439,6 +477,7 @@ async def _safe_start( lease_container_name=config.LEASE_CONTAINER, create_lease_container_if_not_exists=True, ) +@bp.retry(**_RETRY_POLICY) @bp.durable_client_input(client_name="starter") async def on_memory_change(documents: func.DocumentList, starter) -> None: """Change-feed trigger entry point - delegates to :func:`process_changefeed_batch`.""" diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py index 6123d73..2977ffe 100644 --- a/tests/unit/conftest.py +++ b/tests/unit/conftest.py @@ -48,6 +48,14 @@ def register_functions(self, *args, **kwargs): def route(self, *args, **kwargs): return lambda fn: fn + def retry(self, **kwargs): + # Keep what a function registered, so a test can see its retry policy. + def decorator(fn): + fn.retry_policy = kwargs + return fn + + return decorator + def durable_client_input(self, *args, **kwargs): return lambda fn: fn diff --git a/tests/unit/function_app/test_change_feed.py b/tests/unit/function_app/test_change_feed.py index 52add63..5e9e9f1 100644 --- a/tests/unit/function_app/test_change_feed.py +++ b/tests/unit/function_app/test_change_feed.py @@ -50,9 +50,35 @@ def _turn(user_id: str = "u1", thread_id: str = "t1", lsn: int | None = None, te return doc -def _make_starter() -> MagicMock: +class _Status: + """What ``get_status`` returns: falsy for an instance that does not exist, as the Durable client's status is.""" + + def __init__(self, exists: bool) -> None: + self._exists = exists + + def __bool__(self) -> bool: + return self._exists + + +def _make_starter(existing: frozenset[str] | set[str] = frozenset()) -> MagicMock: + """A durable client that knows the ``existing`` instance ids and, like the real one, every id it starts. + + ``get_status`` answers a truthy status for an id it knows and a falsy one for any other, so a test can run a batch + twice and see the second run find the orchestrations of the first. ``instances`` is the set of ids it knows. + """ + instances = set(existing) + + async def start_new(orchestration_name, instance_id=None, client_input=None): + instances.add(instance_id) + return instance_id + + async def get_status(instance_id): + return _Status(instance_id in instances) + starter = MagicMock() - starter.start_new = AsyncMock() + starter.start_new = AsyncMock(side_effect=start_new) + starter.get_status = AsyncMock(side_effect=get_status) + starter.instances = instances return starter @@ -121,15 +147,21 @@ def _counter(container: MagicMock, counter_id: str, *, tenant_id: str = TENANT, def _thread_instance_id( - kind: str, count: int, *, tenant_id: str = TENANT, user_id: str = "u1", thread_id: str = "t1" + kind: str, + count: int, + *, + tenant_id: str = TENANT, + user_id: str = "u1", + thread_id: str = "t1", + lsn: int | None = None, ) -> str: - """The instance id of a thread-scoped orchestration: a hash of the tenant, user and thread, and the count.""" - return orchestration_instance_id(kind, tenant_id, user_id, thread_id, count=count) + """The instance id of a thread-scoped orchestration: a hash of the tenant, user, thread and LSN, and the count.""" + return orchestration_instance_id(kind, tenant_id, user_id, thread_id, count=count, lsn=lsn) -def _user_instance_id(count: int, *, tenant_id: str = TENANT, user_id: str = "u1") -> str: - """The instance id of a user summary: a hash of the tenant and user, and the count.""" - return orchestration_instance_id("user_summary", tenant_id, user_id, count=count) +def _user_instance_id(count: int, *, tenant_id: str = TENANT, user_id: str = "u1", lsn: int | None = None) -> str: + """The instance id of a user summary: a hash of the tenant, user and batch LSN, and the count.""" + return orchestration_instance_id("user_summary", tenant_id, user_id, count=count, lsn=lsn) def _fail_counter_reads(container: MagicMock, error: BaseException, *counters: tuple[str, str, str]) -> None: @@ -1025,14 +1057,12 @@ def test_lsn_replay_does_not_double_increment(): # Layer 1 (LSN replay dedup): the counter must NOT double-increment. assert _counter(container, "thread:u1:t1")["count"] == 4 - # Layer 2 (deterministic instance ID): on replay the counter helper still - # reports the same ``(old, new)`` it returned the first time, so the - # threshold is "crossed" again - but with the IDENTICAL deterministic - # instance id. Azure Durable Functions then dedups the duplicate - # ``start_new`` server-side. We assert the determinism here. + # Layer 2 (existing instance): on replay the counter helper still reports the same ``(old, new)`` it returned the + # first time, so the threshold is "crossed" again - with the IDENTICAL instance id, whose instance the + # first run started. The replay finds it and does not start it again. summary_starts = [c for c in starter.start_new.await_args_list if c.args[0] == "ThreadSummaryOrchestrator"] - assert len(summary_starts) == 2 # same id sent twice - durable dedups - assert all(c.kwargs["instance_id"] == _thread_instance_id("thread_summary", 4) for c in summary_starts) + assert len(summary_starts) == 1 + assert summary_starts[0].kwargs["instance_id"] == _thread_instance_id("thread_summary", 4, lsn=13) @patch.dict( @@ -1218,6 +1248,159 @@ def test_a_cancellation_during_a_counter_update_is_not_swallowed(cancelled): asyncio.run(process_changefeed_batch(_batch_of_three_threads(), starter, counter_container=container)) +# --------------------------------------------------------------------------- +# A retried batch repeats only what failed +# --------------------------------------------------------------------------- + + +def _batch_of_three_threads_with_lsns() -> list[dict]: + """``_batch_of_three_threads`` with the ``_lsn`` the change feed gives each turn: 100 to 105 in listed order.""" + docs = _batch_of_three_threads() + for lsn, doc in enumerate(docs, start=100): + doc["_lsn"] = lsn + return docs + + +# The ids of the five orchestrations that batch starts. Each names the highest LSN of the turns it counted. +_LSN_BATCH_START_IDS = { + _thread_instance_id("thread_summary", 2, tenant_id="acme", thread_id="t1", lsn=101), + _thread_instance_id("thread_summary", 2, tenant_id="acme", thread_id="t2", lsn=103), + _user_instance_id(4, tenant_id="acme", lsn=103), + _thread_instance_id("thread_summary", 2, tenant_id="globex", thread_id="t1", lsn=105), + _user_instance_id(2, tenant_id="globex", lsn=105), +} + + +def _counts(container: MagicMock) -> dict[tuple[str, str, str], int]: + return {key: doc["count"] for key, doc in container._state.items()} + + +@patch.dict(os.environ, _EVERY_TWO_TURNS, clear=False) +def test_a_replayed_batch_starts_no_orchestration_whose_instance_exists(): + """The host runs a failed batch again with the same documents. Every counter reports the threshold crossing it + reported the first time, and each orchestration it names is found and left alone, although Durable Functions + would accept the id of an instance that has finished and run it again.""" + starter = _make_starter() + container = _make_counter_container_starting_at() + docs = _batch_of_three_threads_with_lsns() + + asyncio.run(process_changefeed_batch(docs, starter, counter_container=container)) + asyncio.run(process_changefeed_batch(docs, starter, counter_container=container)) + + assert starter.start_new.await_count == 5 + assert starter.instances == _LSN_BATCH_START_IDS + assert _counts(container) == _BATCH_COUNTS + + +@patch.dict(os.environ, _EVERY_TWO_TURNS, clear=False) +def test_a_retry_starts_only_the_orchestrations_the_failed_attempt_did_not_start(): + starter = _make_starter() + container = _make_counter_container_starting_at() + docs = _batch_of_three_threads_with_lsns() + refused = _thread_instance_id("thread_summary", 2, tenant_id="acme", thread_id="t2", lsn=103) + start = starter.start_new.side_effect + + async def start_but_refuse_one(orchestration_name, instance_id=None, client_input=None): + if instance_id == refused: + raise ConnectionError("starter unavailable") + return await start(orchestration_name, instance_id=instance_id, client_input=client_input) + + starter.start_new.side_effect = start_but_refuse_one + with pytest.raises(RuntimeError, match=r"^Failed to start 1 orchestration\(s\): "): + asyncio.run(process_changefeed_batch(docs, starter, counter_container=container)) + assert starter.instances == _LSN_BATCH_START_IDS - {refused} + + # The retry: the starter works again. + starter.start_new.side_effect = start + starter.start_new.reset_mock() + asyncio.run(process_changefeed_batch(docs, starter, counter_container=container)) + + assert [c.kwargs["instance_id"] for c in starter.start_new.await_args_list] == [refused] + assert starter.instances == _LSN_BATCH_START_IDS + assert _counts(container) == _BATCH_COUNTS + + +@patch.dict( + os.environ, + { + "THREAD_SUMMARY_EVERY_N": "4", + "FACT_EXTRACTION_EVERY_N": "0", + "EPISODE_EVAL_EVERY_N": "0", + "USER_SUMMARY_EVERY_N": "0", + }, + clear=False, +) +def test_a_count_reached_again_after_the_counter_expired_starts_a_new_orchestration(): + """A counter is deleted 90 days after its last turn, so a thread that was idle that long counts from 1 again and + reaches the same count in a later batch. The instance of the first time still exists, and the later batch must + not take it for its own.""" + starter = _make_starter() + container = _make_counter_container_starting_at() + + asyncio.run( + process_changefeed_batch([_turn(lsn=lsn) for lsn in range(10, 14)], starter, counter_container=container) + ) + container._state.clear() + asyncio.run( + process_changefeed_batch([_turn(lsn=lsn) for lsn in range(500, 504)], starter, counter_container=container) + ) + + assert [c.kwargs["instance_id"] for c in starter.start_new.await_args_list] == [ + _thread_instance_id("thread_summary", 4, lsn=13), + _thread_instance_id("thread_summary", 4, lsn=503), + ] + + +@patch.dict(os.environ, _EVERY_TWO_TURNS, clear=False) +def test_a_status_that_cannot_be_read_fails_the_batch_and_starts_nothing(): + """An orchestration whose instance cannot be looked up is not started blind, which could run a finished one again: + the batch fails, and the host retries it.""" + starter = _make_starter() + starter.get_status.side_effect = ConnectionError("status endpoint unavailable") + container = _make_counter_container_starting_at() + + with pytest.raises(RuntimeError, match=r"^Failed to start 5 orchestration\(s\): ConnectionError at "): + asyncio.run(process_changefeed_batch(_batch_of_three_threads_with_lsns(), starter, counter_container=container)) + + starter.start_new.assert_not_awaited() + assert _counts(container) == _BATCH_COUNTS + + +def _conflict_with_a_status_code() -> Exception: + conflict = Exception("conflict") + conflict.status_code = 409 + return conflict + + +@patch.dict(os.environ, _EVERY_TWO_TURNS, clear=False) +@pytest.mark.parametrize( + "conflict", + [ + pytest.param(_conflict_with_a_status_code(), id="status-code"), + pytest.param(Exception("An Orchestration instance with the status Pending already exists."), id="message"), + ], +) +def test_an_instance_that_is_starting_is_skipped_not_an_error(conflict): + """The status of an instance that has not started running yet reads as missing, and Durable Functions then refuses + to start it a second time. That refusal is a skip, not a failed batch.""" + starter = _make_starter() + starter.start_new.side_effect = conflict + container = _make_counter_container_starting_at() + + asyncio.run(process_changefeed_batch(_batch_of_three_threads_with_lsns(), starter, counter_container=container)) + + assert starter.start_new.await_count == 5 + assert _counts(container) == _BATCH_COUNTS + + +def test_the_trigger_retries_a_failed_batch_a_bounded_number_of_times(): + policy = change_feed_module.on_memory_change.retry_policy + + assert policy["strategy"] == "exponential_backoff" + # The host takes the count as a string, and -1 would retry for ever. + assert 1 <= int(policy["max_retry_count"]) <= 10 + + # --------------------------------------------------------------------------- # MEMORY_PROCESSOR_OWNER exclusivity - the change-feed trigger must # respect the owner env var the same way the SDK auto-trigger does, so a diff --git a/tests/unit/function_app/test_instance_ids.py b/tests/unit/function_app/test_instance_ids.py index 96cdb10..5a92103 100644 --- a/tests/unit/function_app/test_instance_ids.py +++ b/tests/unit/function_app/test_instance_ids.py @@ -43,6 +43,20 @@ def test_the_count_is_an_optional_suffix_and_zero_is_a_count(): assert orchestration_instance_id("episode", "acme", "u1", "t1", count=0) == f"{without}:0" +def test_the_batch_lsn_joins_the_hash_and_leaves_the_shape_of_the_id_alone(): + """A replay of a batch has its LSN and so its id; a later batch that reaches the same count gets another id.""" + first = orchestration_instance_id("extract", "acme", "u1", "t1", count=4, lsn=13) + + assert re.fullmatch(r"extract:[0-9a-f]{32}:4", first) + assert first == orchestration_instance_id("extract", "acme", "u1", "t1", count=4, lsn=13) + assert first != orchestration_instance_id("extract", "acme", "u1", "t1", count=4, lsn=503) + assert first != orchestration_instance_id("extract", "acme", "u1", "t1", count=4) + # Without an LSN, which a turn that has none gives, the id is the one it was before the LSN joined the hash. + assert orchestration_instance_id("extract", "acme", "u1", "t1", count=4, lsn=None) == ( + "extract:bdddaa37089972c2babf1b5dda603e63:4" + ) + + def test_the_same_inputs_give_the_same_id(): first = orchestration_instance_id("thread_summary", "acme", "u1", "t1", count=4) second = orchestration_instance_id("thread_summary", "".join(["ac", "me"]), "u1", "t" + "1", count=4) @@ -82,9 +96,10 @@ def test_a_colon_or_a_moved_boundary_cannot_make_two_owners_collide(first, secon @pytest.mark.parametrize("kind", KINDS) def test_the_worst_case_stays_within_the_limit_and_the_allowed_characters(kind): - """Tenant, user and thread of 200 characters and a 10-digit count: the id is as long as for one-character ids.""" - longest = orchestration_instance_id(kind, "t" * 200, "u" * 200, "h" * 200, count=9_999_999_999) - shortest = orchestration_instance_id(kind, "t", "u", "h", count=1_000_000_000) + """Tenant, user and thread of 200 characters, a 10-digit count and the largest LSN: the id is as long as for + one-character ids.""" + longest = orchestration_instance_id(kind, "t" * 200, "u" * 200, "h" * 200, count=9_999_999_999, lsn=2**63 - 1) + shortest = orchestration_instance_id(kind, "t", "u", "h", count=1_000_000_000, lsn=0) assert len(longest) == len(shortest) == len(kind) + 1 + 32 + 1 + 10 assert _is_allowed_instance_id(longest) From 376830137532198f892aabfc1aacbdc70affb72f Mon Sep 17 00:00:00 2001 From: Aayush Kataria Date: Wed, 18 Nov 2026 08:37:18 -0800 Subject: [PATCH 2/3] Link the PR in the CHANGELOG and correct the instance id entry The instance id digest now also covers the change-feed batch, so the entry that describes it says so. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4501cae..c750ee7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,10 +9,10 @@ * **The containers use a new partition key.** `memories`, `memories_turns`, `memories_summaries` and `counter` are keyed `[/tenant_id, /user_id]` instead of `[/user_id, /thread_id]` (the Bicep template and `create_memory_store()`). A partition key can't change in place, so recreate those containers and `leases`, in the order given in the [migration notes](README.md#migration-notes) (every writer stopped, in-flight orchestrations purged, the Function App deployed before the upgraded writers); existing data is not migrated. Record ids must now be unique per owner across threads, not per thread. See [PR:#53](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/53) #### Bugs Fixed -* **A failed change-feed batch is retried instead of dropped.** The Function App's trigger now retries a batch that fails up to five times, with exponential backoff starting at five seconds, where it used to drop the batch and never count its turns. A retry repeats only what failed: a counter skips the turns it has already counted, and an orchestration whose instance already exists is not started again. +* **A failed change-feed batch is retried instead of dropped.** The Function App's trigger retries a batch that fails up to five times, with exponential backoff starting at five seconds, where it used to drop the batch and never count its turns. The turns written after it wait while it is retried. A retry counts no turn twice and starts no orchestration whose instance already exists. See [PR:#55](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/55) #### Other Changes -* Orchestration instance ids in the Function App are now `:[:]`, where the digest is a fixed-length hash of the tenant, owner and thread. They stay within Durable Functions' 100-character limit and no longer contain user ids. See [PR:#53](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/53) +* Orchestration instance ids in the Function App are now `:[:]`, where the digest is a fixed-length hash of the tenant, owner, thread and, for a counter threshold, the change feed batch that reached it. They stay within Durable Functions' 100-character limit and no longer contain user ids. See [PR:#53](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/53), [PR:#55](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/55) ## [0.4.0b1] (2026-11-16) From 2451a53319da529d807bcc55f229065b317eaeeb Mon Sep 17 00:00:00 2001 From: Aayush Kataria Date: Wed, 18 Nov 2026 10:03:51 -0800 Subject: [PATCH 3/3] Resolving comments --- Docs/architecture/orchestrators.md | 4 +- Docs/concepts.md | 4 +- Docs/troubleshooting.md | 4 +- function_app/shared/counters.py | 61 ++++++---- function_app/triggers/change_feed.py | 58 +++++---- tests/unit/function_app/test_change_feed.py | 73 ++++++++++-- tests/unit/function_app/test_counters.py | 123 ++++++++++++++++++-- 7 files changed, 258 insertions(+), 69 deletions(-) diff --git a/Docs/architecture/orchestrators.md b/Docs/architecture/orchestrators.md index f4e18e7..61d5394 100644 --- a/Docs/architecture/orchestrators.md +++ b/Docs/architecture/orchestrators.md @@ -29,7 +29,9 @@ A batch whose model call fails with a retryable error yields no facts and is cou Every orchestration input and activity payload carries `tenant_id`, which the change feed trigger reads from the changed turn, and each activity runs the pipeline of that tenant (one per tenant, sharing the Cosmos DB and model clients), so one Function App serves every tenant. An orchestration instance ID is `{kind}:{digest}:{count}`, where `kind` is `extract`, `episode`, `thread_summary`, `user_summary` or `procedural` and `digest` is a fixed-length digest of the tenant, owner and (where there is one) thread, so IDs stay within the 100-character limit and hold no user IDs. -A batch the host retries derives the same IDs, and the trigger reads the status of an ID before it starts it: an instance that exists, in any status, is left alone, because Durable Functions alone refuses an ID only while its instance is running and would start a finished one again. A `409` for an instance that has not started running yet, which its status does not show, is skipped the same way. A failed status read fails the batch, so the host retries it. For the orchestrations the change feed starts, the digest also covers the highest `_lsn` of the batch that crossed the threshold, so a count that is reached again after its counter expired (a counter is deleted 90 days after its last update) gets an ID of its own and is not mistaken for the instance of the first time. +A batch the host retries derives the same IDs, and the trigger reads the status of an ID before it starts it: an instance that exists, in any status, is left alone, because Durable Functions alone refuses an ID only while its instance is starting or running and would start a finished one again. A `409` for an instance that has not started running yet, which its status does not show, is skipped the same way. A failed status read fails the batch, so the host retries it. For the orchestrations the change feed starts, the digest also covers the highest `_lsn` of the batch that crossed the threshold, so a count that is reached again after its counter expired (a counter is deleted 90 days after its last update) gets an ID of its own and is not mistaken for the instance of the first time. + +The status read and the start are not atomic. Two deliveries of one batch that overlap in time (a lease that moves while the batch runs) can both read a missing instance: the `409` stops the second while the first run is still starting or running, but if the first has already finished when the second's start arrives, Durable Functions accepts the finished ID and runs it again, because its client has no option to refuse one. That takes a stall as long as an orchestration, and the second run writes nothing twice (record IDs are deterministic), so the Function App does not guard against it: closing the gap would take a claim record and crash recovery. Fact IDs are deterministic from user, thread, and normalized content, and episode IDs from user, thread, the segment's first and last turn and the episode's position in it. Thread and user summaries keep their deterministic summary IDs. Procedure IDs derive from the user, kind and name, and a combined fact's ID from the user and its source IDs, so a retried activity finds the record it already wrote (409) instead of writing a second one. diff --git a/Docs/concepts.md b/Docs/concepts.md index 35ccc45..39bf937 100644 --- a/Docs/concepts.md +++ b/Docs/concepts.md @@ -369,9 +369,9 @@ on_memory_change trigger 1. The change feed trigger watches the `memories_turns` container for new documents. 2. Only documents with `type == "turn"` are counted. The trigger reads `tenant_id` from each changed turn, next to `user_id` and `thread_id`, and skips a turn whose tenant, owner or thread id is missing or not valid (the rules are in [Tenants and Owners](#tenants-and-owners)) with a warning. -3. Documents in the dedicated `counter` container track how many turns have been seen per scope using ETag-based optimistic concurrency. +3. Documents in the dedicated `counter` container track how many turns have been seen per scope using ETag-based optimistic concurrency. A counter also keeps the change feed `_lsn` of the last turn it counted and counts only the turns above it, so a batch the change feed delivers again, even one that now holds newer turns as well, adds each turn once. 4. When a counter crosses a configured threshold, the corresponding Durable Functions orchestration is started automatically, in the tenant of the turns it counted. The Function App needs no tenant setting. A failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch, so every other tenant's counters are still updated; the batch still fails at the end, so the host records it and retries it. -5. A batch that fails is retried by the host, up to five times with exponential backoff starting at five seconds (about a minute and a half in all when every retry fails). The batch keeps its lease while it is retried, so the turns written after it wait. If it still fails it is dropped and the turns in it are never counted. A retry repeats only what failed: a counter skips the turns it has already counted, and an orchestration whose instance already exists, in any status, is not started again, so a retry never runs a finished or failed orchestration a second time. +5. A batch that fails is retried by the host, up to five times with exponential backoff starting at five seconds (about a minute and a half in all when every retry fails). The batch keeps its lease while it is retried, so the turns written after it wait. If it still fails it is dropped and the turns in it are never counted. A retry repeats only what failed: a counter skips the turns it has already counted, and an orchestration whose instance already exists, in any status, is not started again, so a retry does not run a finished or failed orchestration a second time. ### Threshold settings diff --git a/Docs/troubleshooting.md b/Docs/troubleshooting.md index f2b095e..644e62f 100644 --- a/Docs/troubleshooting.md +++ b/Docs/troubleshooting.md @@ -212,8 +212,8 @@ If nothing fires: - confirm the `counter` container is writable - confirm `MEMORY_PROCESSOR_OWNER=durable` is set on the Function App - check the Function App log for warnings about skipped turns: a turn needs a valid `tenant_id`, `user_id` and `thread_id` (the SDK stamps the tenant on every record it writes and refuses an invalid owner or thread id), and the trigger skips one that lacks them -- if only some threads or owners fire: a failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch (every other tenant's counters are still updated), but the batch still fails at the end, so the host records it and retries it -- if a batch fails: the host retries it up to five times, with exponential backoff starting at five seconds, so a transient error (a throttled Cosmos DB request, an unreachable Durable Functions endpoint) shows in the log for each attempt and the batch then succeeds. The batch keeps its lease while it is retried, so turns written after it are not processed until it succeeds or the retries are used up (about a minute and a half when every retry fails). A retry repeats only what failed, so `change-feed: instance of already exists, skipping` for an orchestration the failed attempt had started is expected and is not a duplicate run. If every retry fails, the batch is dropped and the turns in it are never counted: fix the error the log names +- if only some threads or owners fire: a failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch (every other tenant's counters are still updated), but the batch still fails at the end, so the host records it and retries it. The log names the tenant, owner and thread of each counter that could not be updated +- if a batch fails: the host retries it up to five times, with exponential backoff starting at five seconds, so a transient error (a throttled Cosmos DB request, an unreachable Durable Functions endpoint) shows in the log for each attempt and the batch then succeeds. The batch keeps its lease while it is retried, so turns written after it are not processed until it succeeds or the retries are used up (about a minute and a half when every retry fails). A retry repeats only what failed, so `change-feed: instance of already exists, skipping` for an orchestration the failed attempt had started is expected and is not a duplicate run. If every retry fails, the batch is dropped and the turns in it are never counted: fix the error the log names. The log of the last attempt names the tenant, owner and thread of each counter it could not update, which tells you whose turns were not counted - insert enough new turn documents to cross the configured threshold - check for generated documents where the Cosmos JSON field is `type="thread_summary"`, `type="fact"`, or `type="user_summary"` diff --git a/function_app/shared/counters.py b/function_app/shared/counters.py index 16dd494..4adf998 100644 --- a/function_app/shared/counters.py +++ b/function_app/shared/counters.py @@ -21,6 +21,7 @@ from __future__ import annotations import logging +from collections.abc import Sequence from datetime import datetime, timezone from typing import Any @@ -88,23 +89,27 @@ async def increment_counter_by( thread_id: str, count: int, *, - batch_max_lsn: int | None = None, + turn_lsns: Sequence[int] = (), owner: str | None = "durable", ) -> tuple[int, int]: - """Atomically increment ``counter_id`` by *count* and return ``(old, new)``. + """Atomically add *count* turns to ``counter_id`` and return ``(old, new)``. * Uses ETag-based optimistic concurrency, retrying up to ``MAX_RETRIES`` times on HTTP 412. * Uses ``create_item`` for the first-write path, retrying on HTTP 409 in case multiple Function workers raced to seed the counter. - * If ``batch_max_lsn`` is *less than or equal to* the LSN persisted on the - existing doc, this is treated as a change-feed replay (immediate or - after a lease re-balance / host crash where checkpoints regressed) and - we return without writing. For the equal case we return the cached - ``(pre_batch_count, current_count)`` so threshold-crossing semantics - hold; for the strict-less case we return ``(current, current)`` (no - crossing) because some other batch already advanced the counter past - this one. + * ``turn_lsns`` holds the change-feed ``_lsn`` of each of the *count* turns. + The counter remembers the highest LSN it has counted (``last_batch_lsn``) + and a turn at or below it was counted already, so it adds nothing. A batch + the change feed delivers again (a host retry, or a lease re-balance or host + crash that left the checkpoint behind) is therefore counted once, even when + newer turns were written since and ride along in the redelivery. A batch + with an LSN for only some of its turns, or for none, is counted whole. + * When every turn is at or below the stored LSN the batch is a replay and + nothing is written. For the equal case we return the cached + ``(pre_batch_count, current_count)`` so threshold-crossing semantics hold; + for the strict-less case we return ``(current, current)`` (no crossing) + because some other batch already advanced the counter past this one. * Preserves SDK-written failure breadcrumbs (``last_failure_at`` / ``last_failure_reason``) so monitors don't flap when the FA writes after an SDK failure stamp. @@ -113,6 +118,9 @@ async def increment_counter_by( double-write configurations across SDK and FA. """ partition_key = partition_key_for(tenant_id, user_id) + # A turn without an LSN cannot be told from a new one, so a turn-by-turn comparison needs one for every turn. + lsns = list(turn_lsns) if count > 0 and len(turn_lsns) == count else [] + batch_max_lsn = max(lsns) if lsns else None for attempt in range(MAX_RETRIES): # ---- Read current counter (or default to 0) ---- @@ -135,20 +143,15 @@ async def increment_counter_by( ) # ---- Replay detection via LSN ---- - # Use ``>=`` not ``==`` so out-of-order redeliveries (lease - # re-balance, host crash → checkpoint regression where another - # batch landed in between) also short-circuit. For the exact - # match we replay the cached result; for the strict-greater case - # we return (current, current) - no threshold crossing - because - # the batch's effect is already absorbed in a later state we have - # no cached pre-batch value for. - if ( - batch_max_lsn is not None - and existing_doc is not None - and existing_doc.get("last_batch_lsn") is not None - and existing_doc["last_batch_lsn"] >= batch_max_lsn - ): - stored_lsn = existing_doc["last_batch_lsn"] + # A turn at or below the stored LSN was counted already. The turns of one delivery can be counted in part: + # the checkpoint stayed behind (lease re-balance, host crash, timeout) while newer turns were written, so the + # redelivery holds both. Each turn is therefore compared with the stored LSN, not only the batch's highest. + # When every turn was counted the batch is a replay: for the exact match we replay the cached result; for the + # strict-less case we return (current, current) - no threshold crossing - because the batch's effect is + # already absorbed in a later state we have no cached pre-batch value for. + stored_lsn = existing_doc.get("last_batch_lsn") if existing_doc is not None else None + counted = sum(1 for lsn in lsns if lsn <= stored_lsn) if stored_lsn is not None else 0 + if lsns and counted == count: if stored_lsn == batch_max_lsn: replay_old = existing_doc.get("last_batch_old_count", old_count) logger.info( @@ -164,8 +167,16 @@ async def increment_counter_by( stored_lsn, ) return (old_count, old_count) + if counted: + logger.info( + "Counter redelivery with newer turns counter_id=%s counted=%d new=%d stored_lsn=%s", + counter_id, + counted, + count - counted, + stored_lsn, + ) - new_count = old_count + count + new_count = old_count + count - counted new_doc = { "id": counter_id, "tenant_id": tenant_id, diff --git a/function_app/triggers/change_feed.py b/function_app/triggers/change_feed.py index 828f4db..b2d1285 100644 --- a/function_app/triggers/change_feed.py +++ b/function_app/triggers/change_feed.py @@ -122,10 +122,16 @@ def _thread_of(doc: dict) -> str | None: return None -def _record_key_failure(exc: Exception, key_errors: list[Exception]) -> None: - """Log one thread's or owner's failed update and remember it, so the rest of the batch still runs.""" +def _record_key_failure(exc: Exception, key: tuple[str, ...], key_errors: list[Exception]) -> None: + """Log one thread's or owner's failed update and remember it, so the rest of the batch still runs. + + The log names the tenant, owner and (for a thread) thread of the counter, which is what an operator needs to know + which turns were not counted once the host has used up its retries and dropped the batch. + """ + scope = " ".join(f"{label}={value}" for label, value in zip(("tenant", "user", "thread"), key)) logger.error( - "change-feed: failed to update a counter (exc_type=%s): %s", + "change-feed: failed to update the counter of %s (exc_type=%s): %s", + scope, type(exc).__name__, describe_exception(exc, stack=True), ) @@ -198,8 +204,9 @@ async def process_changefeed_batch( # ---- Step 1: Filter to turns + group by scope ---- thread_counts: dict[tuple[str, str, str], int] = defaultdict(int) user_counts: dict[tuple[str, str], int] = defaultdict(int) - thread_max_lsn: dict[tuple[str, str, str], int] = {} - user_max_lsn: dict[tuple[str, str], int] = {} + # The change-feed LSN of every counted turn: a counter skips the turns it has counted already, by their LSN. + thread_lsns: dict[tuple[str, str, str], list[int]] = defaultdict(list) + user_lsns: dict[tuple[str, str], list[int]] = defaultdict(list) # Track which threads contributed to each user counter. The user-summary orchestrator # passes them to the pipeline, which only logs how many (``observed_thread_ids``): it @@ -212,8 +219,9 @@ async def process_changefeed_batch( continue # Turns are append-only from this trigger's perspective: the Durable # backend never mutates a turn doc (episodic segmentation advances a - # separate cursor doc; fact extraction is count-based), so each turn is - # delivered - and counted - exactly once, at creation. + # separate cursor doc; fact extraction is count-based), so a turn + # appears in the change feed once, at creation. A redelivered batch + # repeats it, and the counters skip it by its LSN. tenant_id = _tenant_of(doc) user_id = _owner_of(doc) thread_id = _thread_of(doc) @@ -233,8 +241,8 @@ async def process_changefeed_batch( lsn_int = int(lsn) except (TypeError, ValueError): continue - thread_max_lsn[thread_key] = max(thread_max_lsn.get(thread_key, 0), lsn_int) - user_max_lsn[user_key] = max(user_max_lsn.get(user_key, 0), lsn_int) + thread_lsns[thread_key].append(lsn_int) + user_lsns[user_key].append(lsn_int) thread_enabled = n_thread > 0 or n_facts > 0 or n_episode > 0 user_enabled = n_user > 0 @@ -259,7 +267,8 @@ async def process_changefeed_batch( try: tenant_id, user_id, thread_id = thread_key cid = thread_counter_id(user_id, thread_id) - lsn = thread_max_lsn.get(thread_key) + turn_lsns = thread_lsns.get(thread_key, []) + lsn = max(turn_lsns, default=None) old_count, new_count = await increment_counter_by( counter_container, tenant_id, @@ -267,7 +276,7 @@ async def process_changefeed_batch( user_id, thread_id, batch_count, - batch_max_lsn=lsn, + turn_lsns=turn_lsns, ) if n_thread > 0 and crosses_threshold(old_count, new_count, n_thread): @@ -326,7 +335,7 @@ async def process_changefeed_batch( orchestration_errors, ) except Exception as exc: - _record_key_failure(exc, key_errors) + _record_key_failure(exc, thread_key, key_errors) # ---- Step 3: User-scoped counters ---- if user_enabled: @@ -334,7 +343,8 @@ async def process_changefeed_batch( try: tenant_id, user_id = user_key cid = user_counter_id(user_id) - lsn = user_max_lsn.get(user_key) + turn_lsns = user_lsns.get(user_key, []) + lsn = max(turn_lsns, default=None) old_count, new_count = await increment_counter_by( counter_container, tenant_id, @@ -342,7 +352,7 @@ async def process_changefeed_batch( user_id, config.USER_COUNTER_THREAD_ID, batch_count, - batch_max_lsn=lsn, + turn_lsns=turn_lsns, ) if crosses_threshold(old_count, new_count, n_user): instance_id = orchestration_instance_id( @@ -361,7 +371,7 @@ async def process_changefeed_batch( orchestration_errors, ) except Exception as exc: - _record_key_failure(exc, key_errors) + _record_key_failure(exc, user_key, key_errors) # Re-raise so the host records the batch as failed and, under the retry policy, runs it again. A retry is safe: a # counter skips turns it has already counted, and ``_safe_start`` leaves an instance that exists alone. The host @@ -390,9 +400,17 @@ async def _safe_start( A retried or redelivered batch derives the ids of the orchestrations it already started, so an instance that exists, in any state, is the run this threshold crossing started and is left as it is. Durable Functions refuses - an id only while its instance is still running and starts a finished one again, which is why the status is read - first. The 409 it still answers for an instance that is starting or running covers two deliveries racing. A - failure of either call is recorded in ``errors``, which fails the batch so the host retries it. + an id only while its instance is starting or running and starts a finished one again, which is why the status is + read first. + + The status read and the start are not atomic. Of two deliveries of one batch that overlap in time (a lease that + moves while the batch runs), the 409 stops the second only while the first run is still starting or running. If + the first has finished in the gap between the second's status read and its start, the second starts the id again. + That takes a stall as long as an orchestration, and the second run writes nothing twice because record ids are + deterministic. Closing the gap would take a claim record and crash recovery, because the Durable client cannot do + it: ``start_new`` has no option to refuse a finished id. + + A failure of either call is recorded in ``errors``, which fails the batch so the host retries it. """ try: if await starter.get_status(instance_id): @@ -416,8 +434,8 @@ async def _safe_start( client_input=client_input, ) except Exception as exc: - # An instance that is starting or running is refused with a 409 Conflict by the Durable client, the second - # guard after the status read. + # An instance that is starting or running is refused with a 409 Conflict by the Durable client: the second + # guard after the status read, for a delivery that overlaps this one (see the docstring for what it misses). # The Durable Python SDK currently raises bare ``Exception`` for # everything (no typed class for OrchestrationAlreadyExists), so # we keep two signals - status_code first, then the canonical diff --git a/tests/unit/function_app/test_change_feed.py b/tests/unit/function_app/test_change_feed.py index 5e9e9f1..f3367d3 100644 --- a/tests/unit/function_app/test_change_feed.py +++ b/tests/unit/function_app/test_change_feed.py @@ -1095,6 +1095,62 @@ def test_lsns_are_tracked_per_tenant_and_a_replay_counts_nothing_twice(): assert (globex["count"], globex["last_batch_lsn"]) == (1, 50) +@patch.dict( + os.environ, + { + "THREAD_SUMMARY_EVERY_N": "4", + "FACT_EXTRACTION_EVERY_N": "0", + "EPISODE_EVAL_EVERY_N": "0", + "USER_SUMMARY_EVERY_N": "0", + }, + clear=False, +) +def test_a_batch_delivered_again_with_newer_turns_counts_each_turn_once(): + """The host stopped after counting turns 1-3 but before the checkpoint, and turns 4 and 5 were written meanwhile, + so the change feed delivers 1-5. Only 4 and 5 are new: the count is 5 (not 8) and the threshold of 4 is crossed + once, by the summary of count 5. A further delivery of 1-5 changes nothing.""" + starter = _make_starter() + container = _make_counter_container_starting_at() + + asyncio.run(process_changefeed_batch([_turn(lsn=n) for n in (1, 2, 3)], starter, counter_container=container)) + starter.start_new.assert_not_awaited() + + redelivered = [_turn(lsn=n) for n in (1, 2, 3, 4, 5)] + asyncio.run(process_changefeed_batch(redelivered, starter, counter_container=container)) + asyncio.run(process_changefeed_batch(redelivered, starter, counter_container=container)) + + assert _counter(container, "thread:u1:t1")["count"] == 5 + [start] = starter.start_new.await_args_list + assert start.kwargs["instance_id"] == _thread_instance_id("thread_summary", 5, lsn=5) + assert start.kwargs["client_input"]["count"] == 5 + + +@patch.dict( + os.environ, + { + "THREAD_SUMMARY_EVERY_N": "100", + "FACT_EXTRACTION_EVERY_N": "0", + "EPISODE_EVAL_EVERY_N": "0", + "USER_SUMMARY_EVERY_N": "100", + }, + clear=False, +) +def test_every_counter_of_a_redelivered_batch_skips_the_turns_it_counted_itself(): + """Two threads of one owner: each thread's counter and the owner's counter compare the turns of the redelivery + with the LSN it stored, so each counts only the turns after it.""" + starter = _make_starter() + container = _make_counter_container_starting_at() + threads = ("t1", "t2", "t1", "t2", "t1") + turns = [_turn(thread_id=thread, lsn=lsn) for lsn, thread in enumerate(threads, start=1)] + + asyncio.run(process_changefeed_batch(turns[:3], starter, counter_container=container)) + asyncio.run(process_changefeed_batch(turns, starter, counter_container=container)) + + assert _counter(container, "thread:u1:t1")["count"] == 3 + assert _counter(container, "thread:u1:t2")["count"] == 2 + assert _counter(container, "user:u1")["count"] == 5 + + # --------------------------------------------------------------------------- # A failing thread or owner does not stop the others in the batch; the batch fails at the end # --------------------------------------------------------------------------- @@ -1135,15 +1191,18 @@ def _batch_of_three_threads() -> list[dict]: @patch.dict(os.environ, _EVERY_TWO_TURNS, clear=False) @pytest.mark.parametrize( - ("failing", "lost_start"), + ("failing", "lost_start", "named"), [ pytest.param( - (TENANT, "u1", "thread:u1:t1"), _thread_instance_id("thread_summary", 2, thread_id="t1"), id="thread" + (TENANT, "u1", "thread:u1:t1"), + _thread_instance_id("thread_summary", 2, thread_id="t1"), + "tenant=acme user=u1 thread=t1", + id="thread", ), - pytest.param((TENANT, "u1", "user:u1"), _user_instance_id(4), id="owner"), + pytest.param((TENANT, "u1", "user:u1"), _user_instance_id(4), "tenant=acme user=u1", id="owner"), ], ) -def test_a_counter_that_cannot_be_updated_does_not_stop_the_rest_of_the_batch(failing, lost_start, caplog): +def test_a_counter_that_cannot_be_updated_does_not_stop_the_rest_of_the_batch(failing, lost_start, named, caplog): """The first thread, or the first owner, cannot be updated: every other counter is still written and every other threshold crossing still starts its orchestration, and only then does the batch fail.""" caplog.set_level(logging.INFO) @@ -1159,12 +1218,12 @@ def test_a_counter_that_cannot_be_updated_does_not_stop_the_rest_of_the_batch(fa key: count for key, count in _BATCH_COUNTS.items() if key != failing } assert {c.kwargs["instance_id"] for c in starter.start_new.await_args_list} == _BATCH_START_IDS - {lost_start} - # The failure is logged once as its type and code location. Neither the log nor the error the batch raises quotes - # the SDK error's text or chains to it. + # The failure is logged once, naming the tenant, owner and thread of the counter, as its type and code location. + # Neither the log nor the error the batch raises quotes the SDK error's text or chains to it. [record] = [r for r in caplog.records if r.levelno == logging.ERROR] assert record.exc_info is None assert record.getMessage().startswith( - "change-feed: failed to update a counter (exc_type=CosmosHttpResponseError): " + f"change-feed: failed to update the counter of {named} (exc_type=CosmosHttpResponseError): " "azure.cosmos.CosmosHttpResponseError(status=503) at " ) assert (failed.value.__cause__, failed.value.__context__) == (None, None) diff --git a/tests/unit/function_app/test_counters.py b/tests/unit/function_app/test_counters.py index 204e2bf..b21aac2 100644 --- a/tests/unit/function_app/test_counters.py +++ b/tests/unit/function_app/test_counters.py @@ -164,7 +164,7 @@ def test_happy_path_upserts(self): "u1", "t1", 4, - batch_max_lsn=42, + turn_lsns=[39, 40, 41, 42], ) ) @@ -214,9 +214,10 @@ def test_etag_conflict_exhausted_raises(self): assert container.upsert_item.await_count == 3 # MAX_RETRIES def test_lsn_replay_returns_cached_without_writing(self): - """When the doc already records ``last_batch_lsn == batch_max_lsn`` - we treat the change-feed batch as a duplicate delivery: return the - cached ``(pre_batch_count, current_count)`` and DO NOT write.""" + """When every turn is at or below the stored ``last_batch_lsn`` and the + highest is the stored one, we treat the change-feed batch as a duplicate + delivery: return the cached ``(pre_batch_count, current_count)`` and DO + NOT write.""" container = _make_container() container.read_item.return_value = { "id": "thread:u1:t1", @@ -234,7 +235,7 @@ def test_lsn_replay_returns_cached_without_writing(self): "u1", "t1", 4, - batch_max_lsn=100, + turn_lsns=[97, 98, 99, 100], ) ) @@ -242,8 +243,8 @@ def test_lsn_replay_returns_cached_without_writing(self): assert container.upsert_item.await_count == 0 assert container.create_item.await_count == 0 - def test_different_lsn_does_not_trigger_replay(self): - """A different LSN is treated as a fresh batch and increments normally.""" + def test_a_newer_lsn_is_a_fresh_batch(self): + """Turns above the stored LSN are new and increment normally.""" container = _make_container() container.read_item.return_value = { "id": "thread:u1:t1", @@ -261,7 +262,7 @@ def test_different_lsn_does_not_trigger_replay(self): "u1", "t1", 4, - batch_max_lsn=101, + turn_lsns=[101, 102, 103, 104], ) ) @@ -269,12 +270,11 @@ def test_different_lsn_does_not_trigger_replay(self): assert container.upsert_item.await_count == 1 def test_out_of_order_replay_is_noop(self): - """When the redelivered batch's LSN is *less than* the stored LSN + """When every turn of the redelivered batch is *below* the stored LSN (lease re-balance / host crash redelivering an old batch after a newer one already landed), the increment is a no-op - return (current, current) so threshold-crossing logic doesn't fire a - spurious extract/dedup. We use ``>=`` not - ``==`` for replay detection.""" + spurious extract/dedup.""" container = _make_container() container.read_item.return_value = { "id": "thread:u1:t1", @@ -292,7 +292,7 @@ def test_out_of_order_replay_is_noop(self): "u1", "t1", 5, - batch_max_lsn=100, + turn_lsns=[96, 97, 98, 99, 100], ) ) @@ -301,6 +301,105 @@ def test_out_of_order_replay_is_noop(self): assert container.create_item.await_count == 0 +# --------------------------------------------------------------------------- +# A batch delivered again with newer turns counts only the new ones +# --------------------------------------------------------------------------- + + +class TestRedeliveryWithNewerTurns: + @staticmethod + def _counter_at(count: int, last_batch_lsn: int, last_batch_old_count: int = 0) -> dict: + return { + "id": "thread:u1:t1", + "count": count, + "last_batch_lsn": last_batch_lsn, + "last_batch_old_count": last_batch_old_count, + "_etag": "etag-A", + } + + def _increment(self, container, count: int, turn_lsns): + return asyncio.run( + increment_counter_by(container, TENANT, "thread:u1:t1", "u1", "t1", count, turn_lsns=turn_lsns) + ) + + def test_the_turns_already_counted_are_not_counted_again(self): + """Turns 1-3 were counted but the checkpoint stayed behind; turns 4 and 5 were written, so the change feed + now delivers 1-5. Only 4 and 5 are new: the count is 5, not 8.""" + container = _make_container() + container.read_item.return_value = self._counter_at(count=3, last_batch_lsn=3) + + old, new = self._increment(container, 5, [1, 2, 3, 4, 5]) + + assert (old, new) == (3, 5) + body = container.upsert_item.await_args.kwargs["body"] + assert (body["count"], body["last_batch_lsn"], body["last_batch_old_count"]) == (5, 5, 3) + + def test_the_redelivery_that_was_applied_replays_what_the_first_one_returned(self): + """After that, a further delivery of 1-5 is a plain replay: it returns the pair the first one did.""" + container = _make_container() + container.read_item.return_value = self._counter_at(count=5, last_batch_lsn=5, last_batch_old_count=3) + + old, new = self._increment(container, 5, [1, 2, 3, 4, 5]) + + assert (old, new) == (3, 5) + assert container.upsert_item.await_count == 0 + + def test_a_turn_at_the_stored_lsn_was_counted_and_one_above_it_was_not(self): + container = _make_container() + container.read_item.return_value = self._counter_at(count=3, last_batch_lsn=3) + + old, new = self._increment(container, 2, [3, 4]) + + assert (old, new) == (3, 4) + + def test_turns_that_share_an_lsn_are_each_counted(self): + """Documents written in one transactional batch have one LSN: all of them are new, or none is.""" + container = _make_container() + container.read_item.return_value = self._counter_at(count=3, last_batch_lsn=3) + + old, new = self._increment(container, 4, [9, 9, 9, 9]) + + assert (old, new) == (3, 7) + + def test_a_first_write_counts_every_turn(self): + container = _make_container() + container.read_item.side_effect = CosmosResourceNotFoundError(message="404") + + old, new = self._increment(container, 3, [7, 8, 9]) + + assert (old, new) == (0, 3) + assert container.create_item.await_args.kwargs["body"]["last_batch_lsn"] == 9 + + def test_a_counter_without_an_lsn_counts_every_turn(self): + """The SDK's counters carry no ``last_batch_lsn``, so nothing in a batch can be told to be counted.""" + container = _make_container() + container.read_item.return_value = {"id": "thread:u1:t1", "count": 3, "_etag": "etag-A"} + + old, new = self._increment(container, 2, [1, 2]) + + assert (old, new) == (3, 5) + + def test_a_batch_with_an_lsn_for_only_some_turns_is_counted_whole(self): + """A turn without an LSN cannot be told from a new one, so the batch is not compared turn by turn. Like a + batch with no LSNs at all, it leaves none on the counter.""" + container = _make_container() + container.read_item.return_value = self._counter_at(count=3, last_batch_lsn=100) + + old, new = self._increment(container, 3, [90, 91]) + + assert (old, new) == (3, 6) + assert container.upsert_item.await_args.kwargs["body"]["last_batch_lsn"] is None + + def test_a_batch_without_lsns_is_counted_whole(self): + container = _make_container() + container.read_item.return_value = self._counter_at(count=3, last_batch_lsn=100) + + old, new = self._increment(container, 2, ()) + + assert (old, new) == (3, 5) + assert container.upsert_item.await_args.kwargs["body"]["last_batch_lsn"] is None + + # --------------------------------------------------------------------------- # The counter key: the owner's partition [tenant_id, user_id]; thread_id is a plain field # ---------------------------------------------------------------------------