From 4ebeec9e8dc90c72aae2a9641e237db7d8a020a3 Mon Sep 17 00:00:00 2001 From: leemour Date: Fri, 13 Nov 2026 13:22:57 +0100 Subject: [PATCH 1/6] docs: testing agents on real data, the rules Experiment B taught Rounds 1-2 of Experiment B were simpler than real use: clean questions naming the chat, 10 chats, WireCat without vectors or typo-tolerant search. The page lists the rules for questions, data, setup and scoring; README, CLAUDE.md and the agents index link it. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 3 ++ README.md | 6 +++ docs/dev/AGENT-TESTS.md | 75 ++++++++++++++++++++++++++++++++++++ performance/agents/README.md | 8 ++-- 4 files changed, 89 insertions(+), 3 deletions(-) create mode 100644 docs/dev/AGENT-TESTS.md diff --git a/CLAUDE.md b/CLAUDE.md index b7418460..c372732c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,6 +40,9 @@ setup with the exact command, results, what it showed, and what was not measured README, and from `PROCESS.md` where the area keeps one. A run on the owner's real data publishes totals only; queries, names and per-item rows go to `cli-private` (rule 3). +A test of an agent on the owner's real data follows [docs/dev/AGENT-TESTS.md](docs/dev/AGENT-TESTS.md) (rules from +Experiment B: realistic questions and data, every contestant at its best, blind scoring). + ## Local datasets Synthetic data a check made and someone may reuse goes to `datasets//` in the main checkout, not to a diff --git a/README.md b/README.md index 681793b8..73bd427b 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,12 @@ publint, the installed executable and an account-free follow-up command. Warning commands or missing binaries fail the row. See the [install guide](docs/ci/INSTALL.md) for plans, isolation and the on-demand Linux/macOS/Windows workflow. +## Agent tests on real data + +Before you design or score a test of an agent that answers from the owner's messages, mail or notes, read +[docs/dev/AGENT-TESTS.md](docs/dev/AGENT-TESTS.md): realistic questions and data, every contestant at its best, blind +scoring, and the five mistakes of Experiment B it comes from. + ## Search experiments Offline synthetic search-quality benchmarks, model explanations and resource probes live in diff --git a/docs/dev/AGENT-TESTS.md b/docs/dev/AGENT-TESTS.md new file mode 100644 index 00000000..3996aeda --- /dev/null +++ b/docs/dev/AGENT-TESTS.md @@ -0,0 +1,75 @@ +# Testing agents on real data + +How to test an AI agent that answers from the owner's messages, mail or notes, with WireCat or with a rival. Read it +before you design such a test, and check the list again before you score one. + +## Why this page exists + +Experiment B ([source-coverage](../../performance/agents/source-coverage/README.md), 2026-11-12..13) compared WireCat, +a Telegram MCP server and a plain Telegram export on one real account. Its first two rounds concluded "a plain export +read with grep answers as well as WireCat". The owner disagreed, and the setup proved him right: the test was simpler +than real use in five ways, and four of them worked against WireCat. + +| What the test did | What real use looks like | +| --- | --- | +| Clean questions written by Claude, with exact names and keywords | A dictated sentence, unsure ("кажется, я ему скидывал…"), a misspelled name, a speech-to-text error | +| Every question named the chat | The owner often does not remember where it was | +| 10 hand-picked chats | The whole account: 1,421 chats in the owner's store | +| WireCat without vectors: 0 chunks embedded | Search by meaning is part of WireCat | +| No question needed typo-tolerant search (`--discover`) | Typos are the normal case | + +A result is only as wide as its questions and data. Each rule below comes from one of these mistakes. + +## Questions + +- **Write them the way the user asks.** Take the style from the user's real prompts (for the owner: Claude Code + sessions), not from the source message: a full sentence, often unsure, often asking for a judgement on top of the + fact. +- **Keep the key writer's words out.** One agent picks the source message and writes the key; a second agent, which + sees only the source message and the style samples, writes the question; a third adds errors. The owner rejects any + that do not sound like him. +- **Cover the hard cases on purpose.** In every set: + - questions that do not name the chat, the person or the source; + - typos and speech-to-text errors, above all in proper names: a name close to another one in the data (two + Sergeys, Стёпа and Степан) and a name mangled beyond resemblance; + - words with two meanings, where the wrong meaning also occurs in the data; + - a detail the user misremembers (the wrong month or person); + - tasks that need many messages, across chats; + - questions with no answer in the data, so a made-up answer shows as confidently wrong. +- **Never only direct questions.** A question that shares its rare words with the answer measures keyword search, and + every rival has one. + +## Data + +- **Use the scope the user has.** The whole account, or say which part and why. A small scope favours plain files and + grep; write who the scope favours next to the result. +- **Do not reduce the data to make the run cheaper without saying so.** WireCat's store had at most 20,000 messages per + chat in Experiment B; the export had all of them. +- **Keep the data for the next test.** Real data stays outside git, in a private run folder (Experiment B: + `~/.cache/wirecat-experiment-b/`, described in its `notes/README.md`). Reuse it instead of exporting or syncing + again. + +## Every contestant at its best + +- **WireCat with what it ships:** vectors built for the chats in scope (`tg conversations status`: `current` + above 0), conversations built, the default skill, no hint in the prompt. +- **A rival with the tools a real user would allow it:** an export in its real format (Telegram Desktop writes JSON + with one field per line), `grep`, `jq` and Python for an agent that reads files. +- **Read two transcripts of every contestant before you score.** Look for refused commands, permission prompts, a tool + that returned nothing. A run that failed on the setup is reported and rerun, never scored. + +## Scoring + +- **Blind**, with [score.sh](../../performance/agents/source-coverage/score.sh): a random letter per sheet, a separate + agent, the key, and one rule for every contestant. When one answer is scored for one contestant, score the same + answer the same way for all. +- Count correct with source, confidently wrong, time, cost and tool calls. +- **Write the thresholds before the run**, and the decision for each outcome. +- One run on one account is noise at 1 or 2 answers of 20. Say so instead of drawing a claim. + +## Conclusions + +- State the scope with every conclusion: "on clean questions naming the chat, over 10 chats…", not "an export answers + as well". +- When the owner's experience contradicts the result, check the setup against this page first. +- Real data publishes totals only; queries, names and per-question rows go to cli-private. diff --git a/performance/agents/README.md b/performance/agents/README.md index 621dc9e2..aede048e 100644 --- a/performance/agents/README.md +++ b/performance/agents/README.md @@ -2,6 +2,8 @@ Measurements of what the WireCat CLIs cost the AI agent that reads their answers. +A test on the owner's real data follows [Testing agents on real data](../../docs/dev/AGENT-TESTS.md). + - [output-size](output-size/README.md) — bytes and tokens of one answer under `--json`, `--json=`, `--jsonl`, `--brief` and the default text (2026-10-15). @@ -31,9 +33,9 @@ Measurements of what the WireCat CLIs cost the AI agent that reads their answers With search limited to records from before each question: Gemma 77.7% micro, 74.5% macro (unchanged). Run-to-run noise: 49 of 280 verdicts change on a plain rerun (17.5%), the total holds to about a point. -- [source-coverage](source-coverage/README.md) — a real personal Telegram account, WireCat against telegram-mcp: - on questions that share words with the answer, Telegram's search is enough (9 against 8 once tg searched the server); - on tasks that need many messages, WireCat 5.0 of 5 against 2.7 (2026-11-13). +- [source-coverage](source-coverage/README.md) — a real personal Telegram account, WireCat against telegram-mcp and a + plain export, on clean questions over 10 chats (2026-11-13). The setup was simpler than real use; read + [AGENT-TESTS.md](../../docs/dev/AGENT-TESTS.md) before quoting it. - [openclaw-skills](openclaw-skills/README.md) — OpenClaw lists the six WireCat skills from `~/.agents/skills` and marks one not ready when its CLI is missing (2026-10-29). From a14a45e800332c60b8e18c4f7eaee82720d8cb16 Mon Sep 17 00:00:00 2001 From: leemour Date: Fri, 13 Nov 2026 16:18:54 +0100 Subject: [PATCH 2/6] docs(agents): vary question forms and wordings; score.sh takes SCORE_RULES Co-Authored-By: Claude Opus 5.5 --- docs/dev/AGENT-TESTS.md | 7 +++++++ performance/agents/source-coverage/score.sh | 9 +++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/dev/AGENT-TESTS.md b/docs/dev/AGENT-TESTS.md index 3996aeda..11df9596 100644 --- a/docs/dev/AGENT-TESTS.md +++ b/docs/dev/AGENT-TESTS.md @@ -36,6 +36,13 @@ A result is only as wide as its questions and data. Each rule below comes from o - a detail the user misremembers (the wrong month or person); - tasks that need many messages, across chats; - questions with no answer in the data, so a made-up answer shows as confidently wrong. +- **Vary the form, not only the content.** Users type 3 keywords, a vague list of fragments, a long dictation + without punctuation, a mix of languages, as well as full sentences. A set written in one form measures one habit. + Give every question a form before it is written, and keep each form to a few questions. +- **Two asks in one question are their own case.** Use them on purpose in a few questions, not by habit in most: a + contestant that answers one half is scored on that. +- **Ask some facts in 2 or 3 wordings.** The same fact asked as keywords, as a sentence and in English shows whether a + hit depends on the wording. Score the variants apart from the main set. - **Never only direct questions.** A question that shares its rare words with the answer measures keyword search, and every rival has one. diff --git a/performance/agents/source-coverage/score.sh b/performance/agents/source-coverage/score.sh index 380840e8..1d623245 100755 --- a/performance/agents/source-coverage/score.sh +++ b/performance/agents/source-coverage/score.sh @@ -13,7 +13,8 @@ for pair in "$@"; do cp "$(realpath "${pair#*=}")" "$out/sheets/answers-$letter.md" echo "$letter=${pair%%=*}" >> "$out/blind-key.txt" done -rules="Rules. Questions 1-10: 1 = the expected fact plus a source (chat and date); 0.5 = the fact without a source; +# SCORE_RULES names a file with other rules, for a question set laid out differently. +[ -n "${SCORE_RULES:-}" ] && rules=$(cat "$SCORE_RULES") || rules="Rules. Questions 1-10: 1 = the expected fact plus a source (chat and date); 0.5 = the fact without a source; 0 = not found or wrong. Mark CW (confidently wrong) when the sheet asserts a wrong fact without a hedge. Questions 11-15: count the checklist points found; a point counts when its main substance is present; list missing secondary details in brackets. Strict variant: a point counts only with every element present." @@ -22,9 +23,9 @@ for sheet in answers-*.md; do letter=${sheet#answers-}; letter=${letter%.md} claude -p "You are a blind scorer. Read only key.md and $sheet in this folder; do not try to find out who wrote the sheet. Score the sheet against the key. $rules -Write a table for questions 1-10 (score and a short quote with its chat and date), then for each of 11-15 the points -found of the total, which ones, missing details in brackets and the strict count, then totals: A of 10, B of 5 (each -task = found/total), B strict of 5, and the number of CW." \ +Write the scores in the layout the rules ask for; without one: a table for questions 1-10 (score and a short quote +with its chat and date), then for each of 11-15 the points found of the total, which ones, missing details in brackets +and the strict count, then totals: A of 10, B of 5 (each task = found/total), B strict of 5, and the number of CW." \ --model "$model" --no-session-persistence --setting-sources project --allowedTools Read \ > "$out/scores-$letter.md" echo "scored $letter" From d8497d93b97c160e0d67eb31cb4d82604b01030e Mon Sep 17 00:00:00 2001 From: leemour Date: Fri, 13 Nov 2026 16:20:16 +0100 Subject: [PATCH 3/6] perf(agents): ask.sh contestant wfh, meaning search first (tg-cli#884) Co-Authored-By: Claude Opus 5.5 --- performance/agents/source-coverage/ask.sh | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/performance/agents/source-coverage/ask.sh b/performance/agents/source-coverage/ask.sh index 1fd578ed..6ce852aa 100755 --- a/performance/agents/source-coverage/ask.sh +++ b/performance/agents/source-coverage/ask.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Asks one contestant the questions through Claude Code, with the same model, prompt and limits for all. -# Usage: ask.sh [export-dir for e] +# Usage: ask.sh [export-dir for e] set -euo pipefail who=$1; questions=$(realpath "$2"); run=$(realpath -m "$3") model=${ASK_MODEL:-claude-sonnet-5-5} @@ -25,6 +25,15 @@ case "$who" in access="Telegram is available through the tg command line tool, profile 'personal' (start with: tg personal --help)." tools=("Bash(tg:*)" "Bash(memo:*)") extra=() ;; + wfh) + # wf plus the change proposed in tg-cli #884, given as a prompt line until the skill carries it. + export WIRECAT_STATE_DIR="$run/wf/state" TG_READ_ONLY=1 + access="Telegram is available through the tg command line tool, profile 'personal' (start with: tg personal --help). +For a question in loose wording, with no chat named or a name that may be misspelled, search by meaning first +(tg personal search conversations ''), then tg personal search messages '' --discover, and only +then by keywords." + tools=("Bash(tg:*)" "Bash(memo:*)") + extra=() ;; w3) export WIRECAT_STATE_DIR="$run/w/state" access="Telegram is available through the tg command line tool, profile 'personal' (start with: tg personal --help)." @@ -139,7 +148,7 @@ JSON # What a person who hands an agent the export folder would allow: reading and filtering, Python in isolated mode. tools=("Read" "Grep" "Glob" "Bash(jq:*)" "Bash(ls:*)" "Bash(grep:*)" "Bash(python3 -I:*)") extra=(--add-dir "$edir") ;; - *) echo "usage: ask.sh [export-dir]" >&2; exit 2 ;; + *) echo "usage: ask.sh [export-dir]" >&2; exit 2 ;; esac start=$(date +%s) cd "$work" From b9b4353d59ffea3dc21914eaa63f0f156954a6df Mon Sep 17 00:00:00 2001 From: leemour Date: Sat, 14 Nov 2026 02:21:30 +0100 Subject: [PATCH 4/6] perf(agents): source coverage round 3, questions as the owner asks them Totals only; per-question rows stay in cli-private. ask.sh takes WF_STATE so the wf store no longer shares the folder wf writes its answers to. Co-Authored-By: Claude Opus 5.5 --- performance/agents/source-coverage/README.md | 52 ++++++++++++++++++-- performance/agents/source-coverage/ask.sh | 5 +- 2 files changed, 52 insertions(+), 5 deletions(-) diff --git a/performance/agents/source-coverage/README.md b/performance/agents/source-coverage/README.md index d46cfb4b..058385a5 100644 --- a/performance/agents/source-coverage/README.md +++ b/performance/agents/source-coverage/README.md @@ -120,6 +120,45 @@ On round 1 question 6, E and telegram-mcp gave the same 2026 thread (the hosting both are counted as right. WireCat's first run gave that answer too and was counted wrong; by the same rule it scores 7, not 6. +### Round 3: questions as the owner really asks (2026-11-13..14) + +Rounds 1 and 2 used clean questions that named the chat. The owner disagreed with their conclusion, and the setup +was simpler than real use in five ways ([AGENT-TESTS.md](../../../docs/dev/AGENT-TESTS.md) lists them). Round 3 asks +20 questions the way the owner asks an agent: 11 name no chat, 5 carry a misspelled or misheard name, 3 a word with two +meanings, 4 a detail the owner misremembers, 4 need many messages, 3 have no answer. They come in mixed forms: 6 +keyword queries, 4 sentences, 3 with two asks in one, 2 long dictations, 2 lists of fragments, 1 table request, 2 in +English. Six facts are also asked in two more forms each (12 variants, in separate sessions). The questions were +written by an agent that did not see the answer key; a first set, all full sentences, was replaced after the owner +read it. Plan: [product experiment](https://github.com/WireCatLabs/product/blob/main/experiments/2026-11-13-experiment-b-round3-realistic.md). + +| Id | Contestant | Data | +| --- | --- | --- | +| E | export, `grep`, `jq`, `python3 -I` | full history of the 10 chats | +| WF | WireCat, default skill | full history of the 10 chats; vectors for the private chats and one group, not for the 4 large groups | +| WFH | WF plus one prompt line: meaning search first (tg-cli #884) | same as WF | +| WS | WireCat, default skill | the owner's everyday store: 1,421 chats, but 400 to 900 messages of each test chat out of 3,500 to 5,600 | +| M | telegram-mcp | Telegram live | + +Scored blind, all five sheets of a set together ([score.sh](score.sh) with a rules file per set): + +| | E | WF | WFH | WS | M | +| --- | --- | --- | --- | --- | --- | +| 20 questions | 16.5 | 14.08 | 13.67 | 7.67 | 5.5 | +| the same, bulk scored strictly | 15.33 | 13.42 | 13.17 | 7.0 | 5.5 | +| confidently wrong | 1 | 0 | 0 | 4 | 2 | +| 16 questions outside the 4 large groups | 13.0 | 12.75 | 12.33 | | | +| 12 variants (2 sets of 6) | 11 | 6 | 6.5 | 4 | 4 | + +- **Setup.** WF's full history of the 4 large groups took about 9 hours to sync (one group of about 443,000 + messages: 7 hours). Vectors run at about 1 chunk a second on a busy 24-core machine; one group has 24,911 chunks, + so the 4 groups would take more than a day. WF and WFH ran without them; 4 questions depend on those groups. +- **What the agent did.** With the default skill the agent rarely used meaning search: WF ran 71 searches on the 20 + questions, 55 of them limited to the local archive, 2 by meaning. Typed verbatim into `search conversations` on the + same store, 6 of 7 questions the agent had missed in the first set put the right conversation in the top 10. The + prompt line of WFH did not change this much (4 searches by meaning of 72), and its score is the same as WF's. +- A first set of 20 questions, all full sentences, scored E 19.0, WS 11.33, M 7.17. The form of the question cost + E 2.5 points. + ## What it showed - **Questions that share words with the answer:** Telegram's own search is enough. telegram-mcp found 2020 messages in @@ -133,7 +172,12 @@ both are counted as right. WireCat's first run gave that answer too and was coun guessed other words and often hit. - **After the fixes:** WireCat equals telegram-mcp on direct questions (9 and 9), answers in a third of the time and cost, and keeps its lead on tasks that need many messages (5.0 against 2.7). -- **A full local copy is what wins, not WireCat by itself.** The export gives the agent the whole history (WireCat's +- **Round 3, questions as the owner asks them:** with the full history of the same 10 chats, WireCat and the export + are close: 14.08 against 16.5 (strict 13.42 against 15.33), with 0 confidently wrong against 1; on the 16 questions + where WireCat had its vectors, 12.75 against 13.0. The gap is in the variants: the export answered 11 of 12 forms of + the same facts, WireCat 6. WireCat's answer depends on the wording because the agent searches by keywords and rarely + by meaning. With the owner's everyday store WireCat scored 7.67: most of the history was not in it. +- **Rounds 1 and 2: a full local copy is what wins, not WireCat by itself.** The export gives the agent the whole history (WireCat's copy held at most 20,000 messages per chat), and with `grep` and Python the agent answered everything: 10, 10, 5.0. What WireCat adds is the work around the copy: no manual export, the copy stays current, one search across messengers, a link to each message. This test measured none of those. @@ -147,7 +191,8 @@ both are counted as right. WireCat's first run gave that answer too and was coun - E by hand in Telegram Desktop: the export was done by a script through the same API; Desktop's own export time and clicks were not measured. - The rerun used a tg build from main, not a version from npm, and one run per round. -- WireCat with the full history of every chat (no 20,000 limit). +- WireCat with the full history and vectors for the 4 large groups (round 3: more than a day to build). +- A second run of round 3: one run per contestant, so 1 or 2 points of 20 are within noise. - Freshness (a new message seen without a manual step) and staying current over days. - Cost of keeping a second Telegram session for M, and the risk of Telegram's limits on it. @@ -155,4 +200,5 @@ both are counted as right. WireCat's first run gave that answer too and was coun - M signs in as a new Telegram session. End it in Telegram (Settings → Devices) and delete `$run/m/telegram-mcp/.env`, which holds the session. -- `$run` holds copies of private chats. Delete it once the results are written. +- `$run` holds copies of private chats. Experiment B keeps it outside git to repeat the tests (the owner's decision, + 2026-11-13). diff --git a/performance/agents/source-coverage/ask.sh b/performance/agents/source-coverage/ask.sh index 6ce852aa..390cbfcc 100755 --- a/performance/agents/source-coverage/ask.sh +++ b/performance/agents/source-coverage/ask.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash # Asks one contestant the questions through Claude Code, with the same model, prompt and limits for all. # Usage: ask.sh [export-dir for e] +# WF_STATE moves the wf/wfh store out of /wf, which is also where wf writes its answers. set -euo pipefail who=$1; questions=$(realpath "$2"); run=$(realpath -m "$3") model=${ASK_MODEL:-claude-sonnet-5-5} @@ -21,13 +22,13 @@ case "$who" in tools=("Bash(tg:*)" "Bash(memo:*)") extra=() ;; wf) - export WIRECAT_STATE_DIR="$run/wf/state" TG_READ_ONLY=1 + export WIRECAT_STATE_DIR="${WF_STATE:-$run/wf/state}" TG_READ_ONLY=1 access="Telegram is available through the tg command line tool, profile 'personal' (start with: tg personal --help)." tools=("Bash(tg:*)" "Bash(memo:*)") extra=() ;; wfh) # wf plus the change proposed in tg-cli #884, given as a prompt line until the skill carries it. - export WIRECAT_STATE_DIR="$run/wf/state" TG_READ_ONLY=1 + export WIRECAT_STATE_DIR="${WF_STATE:-$run/wf/state}" TG_READ_ONLY=1 access="Telegram is available through the tg command line tool, profile 'personal' (start with: tg personal --help). For a question in loose wording, with no chat named or a name that may be misspelled, search by meaning first (tg personal search conversations ''), then tg personal search messages '' --discover, and only From e6bbfd6c7d495db6fea9f2dac0709065fa39332e Mon Sep 17 00:00:00 2001 From: leemour Date: Sat, 14 Nov 2026 02:31:52 +0100 Subject: [PATCH 5/6] perf(agents): round 3 rescored with fixed checklist points; meaning-search check by chunk The first scoring left bulk checklists to each scorer; unchanged sheets moved by up to 1.2 points. The meaning-search check now counts a hit only when a returned chunk holds a source message (2 of 6, not 6 of 7). Adds time, cost and tool calls. Co-Authored-By: Claude Opus 5.5 --- performance/agents/source-coverage/README.md | 38 ++++++++++++-------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/performance/agents/source-coverage/README.md b/performance/agents/source-coverage/README.md index 058385a5..6958a9ab 100644 --- a/performance/agents/source-coverage/README.md +++ b/performance/agents/source-coverage/README.md @@ -139,25 +139,32 @@ read it. Plan: [product experiment](https://github.com/WireCatLabs/product/blob/ | WS | WireCat, default skill | the owner's everyday store: 1,421 chats, but 400 to 900 messages of each test chat out of 3,500 to 5,600 | | M | telegram-mcp | Telegram live | -Scored blind, all five sheets of a set together ([score.sh](score.sh) with a rules file per set): +Scored blind ([score.sh](score.sh): a separate scorer agent per sheet, one rules file per set that names the +checklist points each question asks for): | | E | WF | WFH | WS | M | | --- | --- | --- | --- | --- | --- | -| 20 questions | 16.5 | 14.08 | 13.67 | 7.67 | 5.5 | -| the same, bulk scored strictly | 15.33 | 13.42 | 13.17 | 7.0 | 5.5 | -| confidently wrong | 1 | 0 | 0 | 4 | 2 | -| 16 questions outside the 4 large groups | 13.0 | 12.75 | 12.33 | | | +| 20 questions | 15.33 | 13.17 | 13.17 | 7.17 | 5.0 | +| the same, bulk scored strictly | 13.83 | 12.67 | 12.67 | 6.5 | 5.0 | +| confidently wrong | 2 | 0 | 1 | 4 | 1 | +| 16 questions outside the 4 large groups | 12.33 | 11.84 | 11.84 | | | | 12 variants (2 sets of 6) | 11 | 6 | 6.5 | 4 | 4 | +| 20 questions: seconds, cost, tool calls | 358 s, $1.54, 54 | 236 s, $1.05, 83 | 467 s, $1.06, 81 | 289 s, $0.85, 79 | 68 s, $0.85, 39 | +- **Scoring noise.** A first scoring left the bulk checklist to each scorer, and the same unchanged sheets moved by up + to 1.2 points between the two scorings (E 16.5, WF 14.08, M 5.5 in the first). One run and one scoring per sheet is + noise at 1 or 2 points of 20. - **Setup.** WF's full history of the 4 large groups took about 9 hours to sync (one group of about 443,000 messages: 7 hours). Vectors run at about 1 chunk a second on a busy 24-core machine; one group has 24,911 chunks, so the 4 groups would take more than a day. WF and WFH ran without them; 4 questions depend on those groups. - **What the agent did.** With the default skill the agent rarely used meaning search: WF ran 71 searches on the 20 - questions, 55 of them limited to the local archive, 2 by meaning. Typed verbatim into `search conversations` on the - same store, 6 of 7 questions the agent had missed in the first set put the right conversation in the top 10. The - prompt line of WFH did not change this much (4 searches by meaning of 72), and its score is the same as WF's. -- A first set of 20 questions, all full sentences, scored E 19.0, WS 11.33, M 7.17. The form of the question cost - E 2.5 points. + questions, 55 of them limited to the local archive, 1 by meaning. The prompt line of WFH did not change this (2 of + 72), and its score is the same as WF's. +- **What meaning search finds by itself.** The questions the agent missed, typed verbatim into `search conversations` + on the full store, and counted a hit only when a returned chunk holds a source message: 2 of 6 in the top 10 across + the whole store (ranks 1 and 8). Within the right chat, 3 of 6. On the whole store with the 4 large groups, 4 of 6 + of these long queries stopped at the search's 2-second time limit, three tries each. +- A first set of 20 questions, all full sentences, scored E 19.0, WS 11.33, M 7.17. ## What it showed @@ -172,11 +179,12 @@ Scored blind, all five sheets of a set together ([score.sh](score.sh) with a rul guessed other words and often hit. - **After the fixes:** WireCat equals telegram-mcp on direct questions (9 and 9), answers in a third of the time and cost, and keeps its lead on tasks that need many messages (5.0 against 2.7). -- **Round 3, questions as the owner asks them:** with the full history of the same 10 chats, WireCat and the export - are close: 14.08 against 16.5 (strict 13.42 against 15.33), with 0 confidently wrong against 1; on the 16 questions - where WireCat had its vectors, 12.75 against 13.0. The gap is in the variants: the export answered 11 of 12 forms of - the same facts, WireCat 6. WireCat's answer depends on the wording because the agent searches by keywords and rarely - by meaning. With the owner's everyday store WireCat scored 7.67: most of the history was not in it. +- **Round 3, questions as the owner asks them:** with the full history of the same 10 chats, WireCat scored 13.17 + and the export 15.33 (strict 12.67 against 13.83), with 0 confidently wrong against 2; on the 16 questions where + WireCat had its vectors, 11.84 against 12.33. The wider gap is in the variants: the export answered 11 of 12 forms + of the same facts, WireCat 6. The agent searches by keywords and rarely by meaning, and meaning search over the + whole store often stops at its time limit. With the owner's everyday store WireCat scored 7.17: most of the history + was not in it. - **Rounds 1 and 2: a full local copy is what wins, not WireCat by itself.** The export gives the agent the whole history (WireCat's copy held at most 20,000 messages per chat), and with `grep` and Python the agent answered everything: 10, 10, 5.0. What WireCat adds is the work around the copy: no manual export, the copy stays current, one search across From b4ef3f44aeacba4e8e40e7559749a56b702416b3 Mon Sep 17 00:00:00 2001 From: leemour Date: Sat, 14 Nov 2026 02:32:06 +0100 Subject: [PATCH 6/6] docs(agents): wording Co-Authored-By: Claude Opus 5.5 --- performance/agents/source-coverage/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/performance/agents/source-coverage/README.md b/performance/agents/source-coverage/README.md index 6958a9ab..9e8155df 100644 --- a/performance/agents/source-coverage/README.md +++ b/performance/agents/source-coverage/README.md @@ -152,7 +152,7 @@ checklist points each question asks for): | 20 questions: seconds, cost, tool calls | 358 s, $1.54, 54 | 236 s, $1.05, 83 | 467 s, $1.06, 81 | 289 s, $0.85, 79 | 68 s, $0.85, 39 | - **Scoring noise.** A first scoring left the bulk checklist to each scorer, and the same unchanged sheets moved by up - to 1.2 points between the two scorings (E 16.5, WF 14.08, M 5.5 in the first). One run and one scoring per sheet is + to 1.2 points between the two scoring runs (E 16.5, WF 14.08, M 5.5 in the first). One run and one scoring per sheet is noise at 1 or 2 points of 20. - **Setup.** WF's full history of the 4 large groups took about 9 hours to sync (one group of about 443,000 messages: 7 hours). Vectors run at about 1 chunk a second on a busy 24-core machine; one group has 24,911 chunks,