docs(agents): rules for agent tests on real data; Experiment B round 3

CLAUDE.md

@@ -40,6 +40,9 @@ setup with the exact command, results, what it showed, and what was not measured
 README, and from `PROCESS.md` where the area keeps one. A run on the owner's real data publishes totals only;
 queries, names and per-item rows go to `cli-private` (rule 3).
 
+A test of an agent on the owner's real data follows [docs/dev/AGENT-TESTS.md](docs/dev/AGENT-TESTS.md) (rules from
+Experiment B: realistic questions and data, every contestant at its best, blind scoring).
+
 ## Local datasets
 
 Synthetic data a check made and someone may reuse goes to `datasets/<name>/` in the main checkout, not to a

README.md

@@ -60,6 +60,12 @@ publint, the installed executable and an account-free follow-up command. Warning
 commands or missing binaries fail the row. See the [install guide](docs/ci/INSTALL.md) for plans, isolation
 and the on-demand Linux/macOS/Windows workflow.
 
+## Agent tests on real data
+
+Before you design or score a test of an agent that answers from the owner's messages, mail or notes, read
+[docs/dev/AGENT-TESTS.md](docs/dev/AGENT-TESTS.md): realistic questions and data, every contestant at its best, blind
+scoring, and the five mistakes of Experiment B it comes from.
+
 ## Search experiments
 
 Offline synthetic search-quality benchmarks, model explanations and resource probes live in

docs/dev/AGENT-TESTS.md

@@ -0,0 +1,82 @@
+# Testing agents on real data
+
+How to test an AI agent that answers from the owner's messages, mail or notes, with WireCat or with a rival. Read it
+before you design such a test, and check the list again before you score one.
+
+## Why this page exists
+
+Experiment B ([source-coverage](../../performance/agents/source-coverage/README.md), 2026-11-12..13) compared WireCat,
+a Telegram MCP server and a plain Telegram export on one real account. Its first two rounds concluded "a plain export
+read with grep answers as well as WireCat". The owner disagreed, and the setup proved him right: the test was simpler
+than real use in five ways, and four of them worked against WireCat.
+
+| What the test did | What real use looks like |
+| --- | --- |
+| Clean questions written by Claude, with exact names and keywords | A dictated sentence, unsure ("кажется, я ему скидывал…"), a misspelled name, a speech-to-text error |
+| Every question named the chat | The owner often does not remember where it was |
+| 10 hand-picked chats | The whole account: 1,421 chats in the owner's store |
+| WireCat without vectors: 0 chunks embedded | Search by meaning is part of WireCat |
+| No question needed typo-tolerant search (`--discover`) | Typos are the normal case |
+
+A result is only as wide as its questions and data. Each rule below comes from one of these mistakes.
+
+## Questions
+
+- **Write them the way the user asks.** Take the style from the user's real prompts (for the owner: Claude Code
+  sessions), not from the source message: a full sentence, often unsure, often asking for a judgement on top of the
+  fact.
+- **Keep the key writer's words out.** One agent picks the source message and writes the key; a second agent, which
+  sees only the source message and the style samples, writes the question; a third adds errors. The owner rejects any
+  that do not sound like him.
+- **Cover the hard cases on purpose.** In every set:
+  - questions that do not name the chat, the person or the source;
+  - typos and speech-to-text errors, above all in proper names: a name close to another one in the data (two
+    Sergeys, Стёпа and Степан) and a name mangled beyond resemblance;
+  - words with two meanings, where the wrong meaning also occurs in the data;
+  - a detail the user misremembers (the wrong month or person);
+  - tasks that need many messages, across chats;
+  - questions with no answer in the data, so a made-up answer shows as confidently wrong.
+- **Vary the form, not only the content.** Users type 3 keywords, a vague list of fragments, a long dictation
+  without punctuation, a mix of languages, as well as full sentences. A set written in one form measures one habit.
+  Give every question a form before it is written, and keep each form to a few questions.
+- **Two asks in one question are their own case.** Use them on purpose in a few questions, not by habit in most: a
+  contestant that answers one half is scored on that.
+- **Ask some facts in 2 or 3 wordings.** The same fact asked as keywords, as a sentence and in English shows whether a
+  hit depends on the wording. Score the variants apart from the main set.
+- **Never only direct questions.** A question that shares its rare words with the answer measures keyword search, and
+  every rival has one.
+
+## Data
+
+- **Use the scope the user has.** The whole account, or say which part and why. A small scope favours plain files and
+  grep; write who the scope favours next to the result.
+- **Do not reduce the data to make the run cheaper without saying so.** WireCat's store had at most 20,000 messages per
+  chat in Experiment B; the export had all of them.
+- **Keep the data for the next test.** Real data stays outside git, in a private run folder (Experiment B:
+  `~/.cache/wirecat-experiment-b/`, described in its `notes/README.md`). Reuse it instead of exporting or syncing
+  again.
+
+## Every contestant at its best
+
+- **WireCat with what it ships:** vectors built for the chats in scope (`tg <profile> conversations status`: `current`
+  above 0), conversations built, the default skill, no hint in the prompt.
+- **A rival with the tools a real user would allow it:** an export in its real format (Telegram Desktop writes JSON
+  with one field per line), `grep`, `jq` and Python for an agent that reads files.
+- **Read two transcripts of every contestant before you score.** Look for refused commands, permission prompts, a tool
+  that returned nothing. A run that failed on the setup is reported and rerun, never scored.
+
+## Scoring
+
+- **Blind**, with [score.sh](../../performance/agents/source-coverage/score.sh): a random letter per sheet, a separate
+  agent, the key, and one rule for every contestant. When one answer is scored for one contestant, score the same
+  answer the same way for all.
+- Count correct with source, confidently wrong, time, cost and tool calls.
+- **Write the thresholds before the run**, and the decision for each outcome.
+- One run on one account is noise at 1 or 2 answers of 20. Say so instead of drawing a claim.
+
+## Conclusions
+
+- State the scope with every conclusion: "on clean questions naming the chat, over 10 chats…", not "an export answers
+  as well".
+- When the owner's experience contradicts the result, check the setup against this page first.
+- Real data publishes totals only; queries, names and per-question rows go to cli-private.

performance/agents/README.md

@@ -2,6 +2,8 @@
 
 Measurements of what the WireCat CLIs cost the AI agent that reads their answers.
 
+A test on the owner's real data follows [Testing agents on real data](../../docs/dev/AGENT-TESTS.md).
+
 - [output-size](output-size/README.md) — bytes and tokens of one answer under `--json`, `--json=<fields>`,
   `--jsonl`, `--brief` and the default text (2026-10-15).
 
@@ -31,9 +33,9 @@ Measurements of what the WireCat CLIs cost the AI agent that reads their answers
   With search limited to records from before each question: Gemma 77.7% micro, 74.5% macro (unchanged).
   Run-to-run noise: 49 of 280 verdicts change on a plain rerun (17.5%), the total holds to about a point.
 
-- [source-coverage](source-coverage/README.md) — a real personal Telegram account, WireCat against telegram-mcp:
-  on questions that share words with the answer, Telegram's search is enough (9 against 8 once tg searched the server);
-  on tasks that need many messages, WireCat 5.0 of 5 against 2.7 (2026-11-13).
+- [source-coverage](source-coverage/README.md) — a real personal Telegram account, WireCat against telegram-mcp and a
+  plain export, on clean questions over 10 chats (2026-11-13). The setup was simpler than real use; read
+  [AGENT-TESTS.md](../../docs/dev/AGENT-TESTS.md) before quoting it.
 
 - [openclaw-skills](openclaw-skills/README.md) — OpenClaw lists the six WireCat skills from `~/.agents/skills`
   and marks one not ready when its CLI is missing (2026-10-29).

performance/agents/source-coverage/README.md

@@ -120,6 +120,52 @@ On round 1 question 6, E and telegram-mcp gave the same 2026 thread (the hosting
 both are counted as right. WireCat's first run gave that answer too and was counted wrong; by the same rule it scores
 7, not 6.
 
+### Round 3: questions as the owner really asks (2026-11-13..14)
+
+Rounds 1 and 2 used clean questions that named the chat. The owner disagreed with their conclusion, and the setup
+was simpler than real use in five ways ([AGENT-TESTS.md](../../../docs/dev/AGENT-TESTS.md) lists them). Round 3 asks
+20 questions the way the owner asks an agent: 11 name no chat, 5 carry a misspelled or misheard name, 3 a word with two
+meanings, 4 a detail the owner misremembers, 4 need many messages, 3 have no answer. They come in mixed forms: 6
+keyword queries, 4 sentences, 3 with two asks in one, 2 long dictations, 2 lists of fragments, 1 table request, 2 in
+English. Six facts are also asked in two more forms each (12 variants, in separate sessions). The questions were
+written by an agent that did not see the answer key; a first set, all full sentences, was replaced after the owner
+read it. Plan: [product experiment](https://github.com/WireCatLabs/product/blob/main/experiments/2026-11-13-experiment-b-round3-realistic.md).
+
+| Id | Contestant | Data |
+| --- | --- | --- |
+| E | export, `grep`, `jq`, `python3 -I` | full history of the 10 chats |
+| WF | WireCat, default skill | full history of the 10 chats; vectors for the private chats and one group, not for the 4 large groups |
+| WFH | WF plus one prompt line: meaning search first (tg-cli #884) | same as WF |
+| WS | WireCat, default skill | the owner's everyday store: 1,421 chats, but 400 to 900 messages of each test chat out of 3,500 to 5,600 |
+| M | telegram-mcp | Telegram live |
+
+Scored blind ([score.sh](score.sh): a separate scorer agent per sheet, one rules file per set that names the
+checklist points each question asks for):
+
+| | E | WF | WFH | WS | M |
+| --- | --- | --- | --- | --- | --- |
+| 20 questions | 15.33 | 13.17 | 13.17 | 7.17 | 5.0 |
+| the same, bulk scored strictly | 13.83 | 12.67 | 12.67 | 6.5 | 5.0 |
+| confidently wrong | 2 | 0 | 1 | 4 | 1 |
+| 16 questions outside the 4 large groups | 12.33 | 11.84 | 11.84 | | |
+| 12 variants (2 sets of 6) | 11 | 6 | 6.5 | 4 | 4 |
+| 20 questions: seconds, cost, tool calls | 358 s, $1.54, 54 | 236 s, $1.05, 83 | 467 s, $1.06, 81 | 289 s, $0.85, 79 | 68 s, $0.85, 39 |
+
+- **Scoring noise.** A first scoring left the bulk checklist to each scorer, and the same unchanged sheets moved by up
+  to 1.2 points between the two scoring runs (E 16.5, WF 14.08, M 5.5 in the first). One run and one scoring per sheet is
+  noise at 1 or 2 points of 20.
+- **Setup.** WF's full history of the 4 large groups took about 9 hours to sync (one group of about 443,000
+  messages: 7 hours). Vectors run at about 1 chunk a second on a busy 24-core machine; one group has 24,911 chunks,
+  so the 4 groups would take more than a day. WF and WFH ran without them; 4 questions depend on those groups.
+- **What the agent did.** With the default skill the agent rarely used meaning search: WF ran 71 searches on the 20
+  questions, 55 of them limited to the local archive, 1 by meaning. The prompt line of WFH did not change this (2 of
+  72), and its score is the same as WF's.
+- **What meaning search finds by itself.** The questions the agent missed, typed verbatim into `search conversations`
+  on the full store, and counted a hit only when a returned chunk holds a source message: 2 of 6 in the top 10 across
+  the whole store (ranks 1 and 8). Within the right chat, 3 of 6. On the whole store with the 4 large groups, 4 of 6
+  of these long queries stopped at the search's 2-second time limit, three tries each.
+- A first set of 20 questions, all full sentences, scored E 19.0, WS 11.33, M 7.17.
+
 ## What it showed
 
 - **Questions that share words with the answer:** Telegram's own search is enough. telegram-mcp found 2020 messages in
@@ -133,7 +179,13 @@ both are counted as right. WireCat's first run gave that answer too and was coun
   guessed other words and often hit.
 - **After the fixes:** WireCat equals telegram-mcp on direct questions (9 and 9), answers in a third of the time and
   cost, and keeps its lead on tasks that need many messages (5.0 against 2.7).
-- **A full local copy is what wins, not WireCat by itself.** The export gives the agent the whole history (WireCat's
+- **Round 3, questions as the owner asks them:** with the full history of the same 10 chats, WireCat scored 13.17
+  and the export 15.33 (strict 12.67 against 13.83), with 0 confidently wrong against 2; on the 16 questions where
+  WireCat had its vectors, 11.84 against 12.33. The wider gap is in the variants: the export answered 11 of 12 forms
+  of the same facts, WireCat 6. The agent searches by keywords and rarely by meaning, and meaning search over the
+  whole store often stops at its time limit. With the owner's everyday store WireCat scored 7.17: most of the history
+  was not in it.
+- **Rounds 1 and 2: a full local copy is what wins, not WireCat by itself.** The export gives the agent the whole history (WireCat's
   copy held at most 20,000 messages per chat), and with `grep` and Python the agent answered everything: 10, 10, 5.0.
   What WireCat adds is the work around the copy: no manual export, the copy stays current, one search across
   messengers, a link to each message. This test measured none of those.
@@ -147,12 +199,14 @@ both are counted as right. WireCat's first run gave that answer too and was coun
 - E by hand in Telegram Desktop: the export was done by a script through the same API; Desktop's own export time and
   clicks were not measured.
 - The rerun used a tg build from main, not a version from npm, and one run per round.
-- WireCat with the full history of every chat (no 20,000 limit).
+- WireCat with the full history and vectors for the 4 large groups (round 3: more than a day to build).
+- A second run of round 3: one run per contestant, so 1 or 2 points of 20 are within noise.
 - Freshness (a new message seen without a manual step) and staying current over days.
 - Cost of keeping a second Telegram session for M, and the risk of Telegram's limits on it.
 
 ## Cleanup
 
 - M signs in as a new Telegram session. End it in Telegram (Settings → Devices) and delete
   `$run/m/telegram-mcp/.env`, which holds the session.
-- `$run` holds copies of private chats. Delete it once the results are written.
+- `$run` holds copies of private chats. Experiment B keeps it outside git to repeat the tests (the owner's decision,
+  2026-11-13).

performance/agents/source-coverage/ask.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
 # Asks one contestant the questions through Claude Code, with the same model, prompt and limits for all.
-# Usage: ask.sh <w|w2|w3|w4|ws|ws2|wf|wf2|m|e> <questions-file> <run-dir> [export-dir for e]
+# Usage: ask.sh <w|w2|w3|w4|ws|ws2|wf|wfh|wf2|m|e> <questions-file> <run-dir> [export-dir for e]
+# WF_STATE moves the wf/wfh store out of <run-dir>/wf, which is also where wf writes its answers.
 set -euo pipefail
 who=$1; questions=$(realpath "$2"); run=$(realpath -m "$3")
 model=${ASK_MODEL:-claude-sonnet-5-5}
@@ -21,10 +22,19 @@ case "$who" in
     tools=("Bash(tg:*)" "Bash(memo:*)")
     extra=() ;;
   wf)
-    export WIRECAT_STATE_DIR="$run/wf/state" TG_READ_ONLY=1
+    export WIRECAT_STATE_DIR="${WF_STATE:-$run/wf/state}" TG_READ_ONLY=1
     access="Telegram is available through the tg command line tool, profile 'personal' (start with: tg personal --help)."
     tools=("Bash(tg:*)" "Bash(memo:*)")
     extra=() ;;
+  wfh)
+    # wf plus the change proposed in tg-cli #884, given as a prompt line until the skill carries it.
+    export WIRECAT_STATE_DIR="${WF_STATE:-$run/wf/state}" TG_READ_ONLY=1
+    access="Telegram is available through the tg command line tool, profile 'personal' (start with: tg personal --help).
+For a question in loose wording, with no chat named or a name that may be misspelled, search by meaning first
+(tg personal search conversations '<question>'), then tg personal search messages '<question>' --discover, and only
+then by keywords."
+    tools=("Bash(tg:*)" "Bash(memo:*)")
+    extra=() ;;
   w3)
     export WIRECAT_STATE_DIR="$run/w/state"
     access="Telegram is available through the tg command line tool, profile 'personal' (start with: tg personal --help)."
@@ -139,7 +149,7 @@ JSON
     # What a person who hands an agent the export folder would allow: reading and filtering, Python in isolated mode.
     tools=("Read" "Grep" "Glob" "Bash(jq:*)" "Bash(ls:*)" "Bash(grep:*)" "Bash(python3 -I:*)")
     extra=(--add-dir "$edir") ;;
-  *) echo "usage: ask.sh <w|w2|w3|w4|ws|ws2|wf|wf2|m|e> <questions-file> <run-dir> [export-dir]" >&2; exit 2 ;;
+  *) echo "usage: ask.sh <w|w2|w3|w4|ws|ws2|wf|wfh|wf2|m|e> <questions-file> <run-dir> [export-dir]" >&2; exit 2 ;;
 esac
 start=$(date +%s)
 cd "$work"

performance/agents/source-coverage/score.sh

@@ -13,7 +13,8 @@ for pair in "$@"; do
   cp "$(realpath "${pair#*=}")" "$out/sheets/answers-$letter.md"
   echo "$letter=${pair%%=*}" >> "$out/blind-key.txt"
 done
-rules="Rules. Questions 1-10: 1 = the expected fact plus a source (chat and date); 0.5 = the fact without a source;
+# SCORE_RULES names a file with other rules, for a question set laid out differently.
+[ -n "${SCORE_RULES:-}" ] && rules=$(cat "$SCORE_RULES") || rules="Rules. Questions 1-10: 1 = the expected fact plus a source (chat and date); 0.5 = the fact without a source;
 0 = not found or wrong. Mark CW (confidently wrong) when the sheet asserts a wrong fact without a hedge. Questions
 11-15: count the checklist points found; a point counts when its main substance is present; list missing secondary
 details in brackets. Strict variant: a point counts only with every element present."
@@ -22,9 +23,9 @@ for sheet in answers-*.md; do
   letter=${sheet#answers-}; letter=${letter%.md}
   claude -p "You are a blind scorer. Read only key.md and $sheet in this folder; do not try to find out who wrote the
 sheet. Score the sheet against the key. $rules
-Write a table for questions 1-10 (score and a short quote with its chat and date), then for each of 11-15 the points
-found of the total, which ones, missing details in brackets and the strict count, then totals: A of 10, B of 5 (each
-task = found/total), B strict of 5, and the number of CW." \
+Write the scores in the layout the rules ask for; without one: a table for questions 1-10 (score and a short quote
+with its chat and date), then for each of 11-15 the points found of the total, which ones, missing details in brackets
+and the strict count, then totals: A of 10, B of 5 (each task = found/total), B strict of 5, and the number of CW." \
     --model "$model" --no-session-persistence --setting-sources project --allowedTools Read \
     > "$out/scores-$letter.md"
   echo "scored $letter"