Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,11 @@
#### Breaking Changes
* **The containers use a new partition key.** `memories`, `memories_turns`, `memories_summaries` and `counter` are keyed `[/tenant_id, /user_id]` instead of `[/user_id, /thread_id]` (the Bicep template and `create_memory_store()`). A partition key can't change in place, so recreate those containers and `leases`, in the order given in the [migration notes](README.md#migration-notes) (every writer stopped, in-flight orchestrations purged, the Function App deployed before the upgraded writers); existing data is not migrated. Record ids must now be unique per owner across threads, not per thread. See [PR:#53](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/53)

#### Bugs Fixed
* **A failed change-feed batch is retried instead of dropped.** The Function App's trigger retries a batch that fails up to five times, with exponential backoff starting at five seconds, where it used to drop the batch and never count its turns. The turns written after it wait while it is retried. A retry counts no turn twice and starts no orchestration whose instance already exists. See [PR:#55](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/55)

#### Other Changes
* Orchestration instance ids in the Function App are now `<kind>:<digest>[:<count>]`, where the digest is a fixed-length hash of the tenant, owner and thread. They stay within Durable Functions' 100-character limit and no longer contain user ids. See [PR:#53](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/53)
* Orchestration instance ids in the Function App are now `<kind>:<digest>[:<count>]`, where the digest is a fixed-length hash of the tenant, owner, thread and, for a counter threshold, the change feed batch that reached it. They stay within Durable Functions' 100-character limit and no longer contain user ids. See [PR:#53](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/53), [PR:#55](https://github.com/AzureCosmosDB/AgentMemoryToolkit/pull/55)

## [0.4.0b1] (2026-11-16)

Expand Down
4 changes: 4 additions & 0 deletions Docs/architecture/orchestrators.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ A batch whose model call fails with a retryable error yields no facts and is cou

Every orchestration input and activity payload carries `tenant_id`, which the change feed trigger reads from the changed turn, and each activity runs the pipeline of that tenant (one per tenant, sharing the Cosmos DB and model clients), so one Function App serves every tenant. An orchestration instance ID is `{kind}:{digest}:{count}`, where `kind` is `extract`, `episode`, `thread_summary`, `user_summary` or `procedural` and `digest` is a fixed-length digest of the tenant, owner and (where there is one) thread, so IDs stay within the 100-character limit and hold no user IDs.

A batch the host retries derives the same IDs, and the trigger reads the status of an ID before it starts it: an instance that exists, in any status, is left alone, because Durable Functions alone refuses an ID only while its instance is starting or running and would start a finished one again. A `409` for an instance that has not started running yet, which its status does not show, is skipped the same way. A failed status read fails the batch, so the host retries it. For the orchestrations the change feed starts, the digest also covers the highest `_lsn` of the batch that crossed the threshold, so a count that is reached again after its counter expired (a counter is deleted 90 days after its last update) gets an ID of its own and is not mistaken for the instance of the first time.

The status read and the start are not atomic. Two deliveries of one batch that overlap in time (a lease that moves while the batch runs) can both read a missing instance: the `409` stops the second while the first run is still starting or running, but if the first has already finished when the second's start arrives, Durable Functions accepts the finished ID and runs it again, because its client has no option to refuse one. That takes a stall as long as an orchestration, and the second run writes nothing twice (record IDs are deterministic), so the Function App does not guard against it: closing the gap would take a claim record and crash recovery.

Fact IDs are deterministic from user, thread, and normalized content, and episode IDs from user, thread, the segment's first and last turn and the episode's position in it. Thread and user summaries keep their deterministic summary IDs. Procedure IDs derive from the user, kind and name, and a combined fact's ID from the user and its source IDs, so a retried activity finds the record it already wrote (409) instead of writing a second one.

`UserSummaryOrchestrator` can start before extraction has persisted anything for a new user. `us_Extract` then returns `no_memories_yet` instead of failing, and the orchestrator polls on a durable timer (App Settings `USER_SUMMARY_WAIT_SECONDS`, default `120`, every `USER_SUMMARY_WAIT_INTERVAL_SECONDS`, default `10`) before it skips that cadence; the next user-summary threshold tries again. `USER_SUMMARY_WAIT_SECONDS=0` makes it a single attempt.
Expand Down
5 changes: 3 additions & 2 deletions Docs/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -369,8 +369,9 @@ on_memory_change trigger

1. The change feed trigger watches the `memories_turns` container for new documents.
2. Only documents with `type == "turn"` are counted. The trigger reads `tenant_id` from each changed turn, next to `user_id` and `thread_id`, and skips a turn whose tenant, owner or thread id is missing or not valid (the rules are in [Tenants and Owners](#tenants-and-owners)) with a warning.
3. Documents in the dedicated `counter` container track how many turns have been seen per scope using ETag-based optimistic concurrency.
4. When a counter crosses a configured threshold, the corresponding Durable Functions orchestration is started automatically, in the tenant of the turns it counted. The Function App needs no tenant setting. A failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch, so every other tenant's counters are still updated; the batch still fails at the end so the host records it.
3. Documents in the dedicated `counter` container track how many turns have been seen per scope using ETag-based optimistic concurrency. A counter also keeps the change feed `_lsn` of the last turn it counted and counts only the turns above it, so a batch the change feed delivers again, even one that now holds newer turns as well, adds each turn once.
4. When a counter crosses a configured threshold, the corresponding Durable Functions orchestration is started automatically, in the tenant of the turns it counted. The Function App needs no tenant setting. A failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch, so every other tenant's counters are still updated; the batch still fails at the end, so the host records it and retries it.
5. A batch that fails is retried by the host, up to five times with exponential backoff starting at five seconds (about a minute and a half in all when every retry fails). The batch keeps its lease while it is retried, so the turns written after it wait. If it still fails it is dropped and the turns in it are never counted. A retry repeats only what failed: a counter skips the turns it has already counted, and an orchestration whose instance already exists, in any status, is not started again, so a retry does not run a finished or failed orchestration a second time.

### Threshold settings

Expand Down
5 changes: 3 additions & 2 deletions Docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,11 +208,12 @@ If nothing fires:

- verify the Functions host shows the Cosmos DB trigger
- confirm the `leases` container exists
- confirm the `memories`, `memories_turns`, `memories_summaries` and `counter` containers exist and are keyed `[/tenant_id, /user_id]`: the Function App checks them on its first change-feed batch, and a missing or wrongly keyed one fails the batch with a `Topology error` that names the container (for example `Topology error: container '<name>' does not exist`)
- confirm the `memories`, `memories_turns`, `memories_summaries` and `counter` containers exist and are keyed `[/tenant_id, /user_id]`: the Function App checks them on its first change-feed batch, and a missing or wrongly keyed one fails the batch with a `Topology error` that names the container (for example `Topology error: container '<name>' does not exist`). The host retries that batch like any other (see below), and it fails every time until the container is fixed
- confirm the `counter` container is writable
- confirm `MEMORY_PROCESSOR_OWNER=durable` is set on the Function App
- check the Function App log for warnings about skipped turns: a turn needs a valid `tenant_id`, `user_id` and `thread_id` (the SDK stamps the tenant on every record it writes and refuses an invalid owner or thread id), and the trigger skips one that lacks them
- if only some threads or owners fire: a failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch (every other tenant's counters are still updated), but the batch still fails at the end so the host records it
- if only some threads or owners fire: a failure while counting or starting orchestrations for one thread or owner does not stop the rest of the batch (every other tenant's counters are still updated), but the batch still fails at the end, so the host records it and retries it. The log names the tenant, owner and thread of each counter that could not be updated
- if a batch fails: the host retries it up to five times, with exponential backoff starting at five seconds, so a transient error (a throttled Cosmos DB request, an unreachable Durable Functions endpoint) shows in the log for each attempt and the batch then succeeds. The batch keeps its lease while it is retried, so turns written after it are not processed until it succeeds or the retries are used up (about a minute and a half when every retry fails). A retry repeats only what failed, so `change-feed: instance <id> of <orchestration> already exists, skipping` for an orchestration the failed attempt had started is expected and is not a duplicate run. If every retry fails, the batch is dropped and the turns in it are never counted: fix the error the log names. The log of the last attempt names the tenant, owner and thread of each counter it could not update, which tells you whose turns were not counted
- insert enough new turn documents to cross the configured threshold
- check for generated documents where the Cosmos JSON field is `type="thread_summary"`, `type="fact"`, or `type="user_summary"`

Expand Down
61 changes: 36 additions & 25 deletions function_app/shared/counters.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
from __future__ import annotations

import logging
from collections.abc import Sequence
from datetime import datetime, timezone
from typing import Any

Expand Down Expand Up @@ -88,23 +89,27 @@ async def increment_counter_by(
thread_id: str,
count: int,
*,
batch_max_lsn: int | None = None,
turn_lsns: Sequence[int] = (),
owner: str | None = "durable",
) -> tuple[int, int]:
"""Atomically increment ``counter_id`` by *count* and return ``(old, new)``.
"""Atomically add *count* turns to ``counter_id`` and return ``(old, new)``.

* Uses ETag-based optimistic concurrency, retrying up to ``MAX_RETRIES``
times on HTTP 412.
* Uses ``create_item`` for the first-write path, retrying on HTTP 409 in
case multiple Function workers raced to seed the counter.
* If ``batch_max_lsn`` is *less than or equal to* the LSN persisted on the
existing doc, this is treated as a change-feed replay (immediate or
after a lease re-balance / host crash where checkpoints regressed) and
we return without writing. For the equal case we return the cached
``(pre_batch_count, current_count)`` so threshold-crossing semantics
hold; for the strict-less case we return ``(current, current)`` (no
crossing) because some other batch already advanced the counter past
this one.
* ``turn_lsns`` holds the change-feed ``_lsn`` of each of the *count* turns.
The counter remembers the highest LSN it has counted (``last_batch_lsn``)
and a turn at or below it was counted already, so it adds nothing. A batch
the change feed delivers again (a host retry, or a lease re-balance or host
crash that left the checkpoint behind) is therefore counted once, even when
newer turns were written since and ride along in the redelivery. A batch
with an LSN for only some of its turns, or for none, is counted whole.
* When every turn is at or below the stored LSN the batch is a replay and
nothing is written. For the equal case we return the cached
``(pre_batch_count, current_count)`` so threshold-crossing semantics hold;
for the strict-less case we return ``(current, current)`` (no crossing)
because some other batch already advanced the counter past this one.
* Preserves SDK-written failure breadcrumbs (``last_failure_at`` /
``last_failure_reason``) so monitors don't flap when the FA writes
after an SDK failure stamp.
Expand All @@ -113,6 +118,9 @@ async def increment_counter_by(
double-write configurations across SDK and FA.
"""
partition_key = partition_key_for(tenant_id, user_id)
# A turn without an LSN cannot be told from a new one, so a turn-by-turn comparison needs one for every turn.
lsns = list(turn_lsns) if count > 0 and len(turn_lsns) == count else []
batch_max_lsn = max(lsns) if lsns else None

for attempt in range(MAX_RETRIES):
# ---- Read current counter (or default to 0) ----
Expand All @@ -135,20 +143,15 @@ async def increment_counter_by(
)

# ---- Replay detection via LSN ----
# Use ``>=`` not ``==`` so out-of-order redeliveries (lease
# re-balance, host crash → checkpoint regression where another
# batch landed in between) also short-circuit. For the exact
# match we replay the cached result; for the strict-greater case
# we return (current, current) - no threshold crossing - because
# the batch's effect is already absorbed in a later state we have
# no cached pre-batch value for.
if (
batch_max_lsn is not None
and existing_doc is not None
and existing_doc.get("last_batch_lsn") is not None
and existing_doc["last_batch_lsn"] >= batch_max_lsn
):
stored_lsn = existing_doc["last_batch_lsn"]
# A turn at or below the stored LSN was counted already. The turns of one delivery can be counted in part:
# the checkpoint stayed behind (lease re-balance, host crash, timeout) while newer turns were written, so the
# redelivery holds both. Each turn is therefore compared with the stored LSN, not only the batch's highest.
# When every turn was counted the batch is a replay: for the exact match we replay the cached result; for the
# strict-less case we return (current, current) - no threshold crossing - because the batch's effect is
# already absorbed in a later state we have no cached pre-batch value for.
stored_lsn = existing_doc.get("last_batch_lsn") if existing_doc is not None else None
counted = sum(1 for lsn in lsns if lsn <= stored_lsn) if stored_lsn is not None else 0
if lsns and counted == count:
if stored_lsn == batch_max_lsn:
replay_old = existing_doc.get("last_batch_old_count", old_count)
logger.info(
Expand All @@ -164,8 +167,16 @@ async def increment_counter_by(
stored_lsn,
)
return (old_count, old_count)
if counted:
logger.info(
"Counter redelivery with newer turns counter_id=%s counted=%d new=%d stored_lsn=%s",
counter_id,
counted,
count - counted,
stored_lsn,
)

new_count = old_count + count
new_count = old_count + count - counted
new_doc = {
"id": counter_id,
"tenant_id": tenant_id,
Expand Down
14 changes: 10 additions & 4 deletions function_app/shared/instance_ids.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,26 @@

Durable Functions takes an instance id of 1-100 characters without ``/``, ``\\``, ``#``, ``?`` or control characters,
and tenant, user and thread ids are free text, so an instance id never carries them: it is a readable ``kind``, a
fixed-length hash of the owner (and thread) and, for a counter threshold, the count. The same owner, thread and count
always give the same id, which is what makes a replayed change-feed batch start each orchestration once.
fixed-length hash of the owner (and thread) and, for a counter threshold, the count. The same owner, thread, count and
batch always give the same id, which is what lets a replayed change-feed batch find the orchestrations it already
started and leave them alone.
"""

from __future__ import annotations

from hashlib import sha256


def orchestration_instance_id(kind: str, *parts: str, count: int | None = None) -> str:
def orchestration_instance_id(kind: str, *parts: str, count: int | None = None, lsn: int | None = None) -> str:
"""``{kind}:{digest}`` plus ``:{count}`` when a count is given.

``parts`` is ``(tenant_id, user_id)`` or ``(tenant_id, user_id, thread_id)``. They are joined with the unit
separator before hashing, so a ``:`` inside an id cannot make two different splits give the same digest.

``lsn`` is the highest change-feed sequence number of the batch that reached the count, and joins the hash. A
replay of that batch has the same ``lsn`` and so the same id, while a count reached again after the counter
expired (it lives 90 days) comes from a later batch and gets an id of its own, not that of a finished instance.
"""
digest = sha256("\x1f".join(parts).encode("utf-8")).hexdigest()[:32]
hashed = parts if lsn is None else (*parts, str(lsn))
digest = sha256("\x1f".join(hashed).encode("utf-8")).hexdigest()[:32]
return f"{kind}:{digest}" + (f":{count}" if count is not None else "")
Loading
Loading