Skip to content

Preserve retained callable history and measure bounded MCP retention (#169, #131) - #208

Merged
trissim merged 6 commits into
mainfrom
fix/memory-session-20260929
Sep 30, 2026
Merged

trissim merged 6 commits into
mainfrom
fix/memory-session-20260929

Conversation

@trissim

@trissim trissim commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

References #169 and #131. Paired upstream draft:
OpenHCSDev/ArrayBridge#1.

  • Repair the reproduced _thread._local retained-history failure via the recorded
    ArrayBridge gitlink: existing ThreadGPUContext owns its runtime handle, rather
    than serializing it through decorator globals. No permissive pickle, history
    dropping, duplicate registry or saved-file reconstruction.
  • Continuous native document/history restart coverage retains current
    declarations, earlier snapshots, undo/head navigation and the runtime handle.
  • python -m openhcs.mcp.memory_diagnostic measures a bounded real MCP request
    sequence with baseline/before/after/full-GC RSS/PSS/private/swap/thread receipts,
    import/library/framework and catalog/custom/history observations. Warm-up is
    distinct from repeated-request slopes; no leak conclusion without measurement.
  • Typed receipt/health decoding, declaration-owned metric projection/selection,
    and sole McpDevToolResult.has_errors() classification remain in place.

Independent review corrections

Addresses both findings in
#208 (comment).

  • BOUND-2: use existing AgentCapabilitySpec.read_only, including mutation-only
    declarations with empty side effects; current Enforce MCP write authority before artifact-plan initialization (#211) #212 artifact-plan inspection is
    rejected. No reconstructed mutation predicate or new capability registry.
  • IMPL-13 / BOUND-2: DiagnosticServerSpec extends normal McpDevServerSpec
    arguments, retaining OpenHCSRuntimeImportAuthority exactly. Entrypoint is now
    inside openhcs.mcp; old benchmark entrypoint, bare child -m, and ambient
    PYTHONPATH copying are deleted without aliases or duplicate bootstrap code.
  • Every memory receipt carries observed diagnostic/OpenHCS paths and the client
    verifies both against its parent import authority, not just health PID.
  • --source-identity provides bounded source-provenance inspection without MCP
    construction. Launch declarations are separated from the minimal entrypoint
    to avoid importing the application-client/scientific dependency graph there.

Actual evidence

  • Latest openhcsdev/main at 283b21275 fetched and merged normally as 32decce9a;
    recorded submodules retained, existing venv imports verified against this tree.
  • 15 focused tests pass in 4.34s: diagnostic and existing import-authority
    tests; actual child launches from a competing cwd with absent/hostile PYTHONPATH;
    both selected source paths asserted; foreign source receipts rejected;
    mutation-only declaration and integrated artifact-plan admission covered.
  • Actual source CLI with PYTHONPATH absent returns selected diagnostic/OpenHCS
    paths: 0.16s, 53,500 KiB peak RSS, zero swaps, exit 0. No MCP/GUI/JVM started.
  • Focused AST guard finds all three returned bypasses at dd5f177f, passes the
    correction; changed-source Ruff correctness and diff checks pass. Scope is
    focused source/AST/catalog review, not a full NRA proof or global debt claim.
  • Earlier durability evidence retained: original ObjectState/callable failure
    reproduced; repaired minimal callable save/load executes (14,424 bytes);
    ArrayBridge 3 tests, initial OpenHCS 12 tests, typed ownership revision
    6 tests including continuous native history all passed at their checkpoints.

Remaining scope and readiness

No Closes claim for #131 or #169. Real MCP retention/slope measurements and the
installed continuous desktop capture/restore/history journey remain pending.
Default discovery measurements cannot exonerate untested catalog/compile/custom
source, execution, JVM or GPU routes. No leak/no-leak finding is claimed yet.

PR208 and paired ArrayBridge PR1 remain draft. Confucius owns the heavy live slot
and frozen installed baseline; no worker MCP/GUI/JVM/large environment startup,
merge/install or skill change. Resource guard before lightweight tests reported
11.8 GiB available RAM with historical-swap warning; that is not leak evidence.
Unrelated #217/BaSiCPy work remains preserved in its separate assigned worktree.
Old H001 saved exports, blind inputs/results, dirty main and installed sources
remain untouched.

See docs/validation/memory-session-20260929.md for detailed evidence, pattern IDs,
deletion/new-case ownership checks, commands and remaining serialized live gates.

@trissim

trissim commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Catalog review at published PR #208 head 7a76b57d3f1b53ac38a12d67ab54e4072eab95d2 distinguishes the actual ownership repair from diagnostic debt:

  • The paired ArrayBridge fix deletes the old global threading.local store and _get_thread_gpu_context helper. Its existing importable ThreadGPUContext owns the runtime handle. That addresses the durable/runtime boundary without dropping history or adding permissive pickle.
  • BOUND-2 / BOUND-1: ProcessMemoryReceipt already declares the sample shape, but sample returns an untyped dict, later consumes ['rss_kib'], and retained_slope_kib reads samples through a string field. Decode the MCP sample once into the existing nominal receipt and keep its metric projections with their owner. /proc parsing is a real external-format boundary and is not condemned merely for string keys.
  • Check duplicate raw status/error classification against the existing McpDevToolResult owner instead of restating its contract.

Linnaeus is the active revision owner and has received these exact witnesses. This stays draft pending the corrected diagnostic and the continuous source/live history acceptance. No leak/no-leak verdict follows from the code or this review alone.

@trissim

trissim commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

Architectural review: two diagnostic consumers bypass existing authorities

Reviewed dd5f177f0c381ebac1d470f9a05f8423c2c40dfb.

  1. Read-only admission is reimplemented incompletely. read_request_sequence, lines 279–285 rejects capability.side_effects but ignores mutating. The existing AgentCapabilitySpec.read_only, lines 976–978 already owns the complete predicate (not mutating and not side_effects). A valid declaration with mutating=True, side_effects=() is accepted by this diagnostic even though the authority says it is not read-only. Use capability.read_only rather than another interpretation of its fields, and test that additional declaration case. This is BOUND-2: the existing owner is available, but the local consumer reconstructs an incomplete answer from its fields. In combination with Enforce MCP write authority before artifact-plan initialization (#211) #212, recheck artifact-plan admission at the integrated head rather than relying on its older incorrect declaration.

  2. Subprocess source identity becomes an environment assumption. DiagnosticServerSpec, lines 321–327 replaces the normal launch projection with bare -m benchmark.mcp_memory_diagnostic and unconditionally reads os.environ["PYTHONPATH"]. The advertised python -m benchmark.mcp_memory_diagnostic command can therefore fail with KeyError when launched from a checkout/venv without PYTHONPATH. More importantly, setting PYTHONPATH does not prove the child's loaded source is the parent's source. The existing McpDevServerSpec.process_args() uses OpenHCSRuntimeImportAuthority.current().module_process_arguments(...) specifically to avoid ambient working-directory/path selection. Keep that import-root authority in the diagnostic launch, exposing a supported diagnostic/module hook there or placing the instrumented entrypoint under the supported OpenHCS launch boundary. Do not just change the subscript to .get() and leave the identity problem intact.

Add a child-launch regression with PYTHONPATH absent and a different working directory, and verify both diagnostic and OpenHCS source paths against the selected head. Health PID equality alone does not establish source identity. These are small source/launch regressions, not requests to rerun the scientific workloads.

@trissim

trissim commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Both returned authority bypasses are fixed and published in d69d65a, following the normal merge of main283b21275 (including #212).

  • BOUND-2 admission: read_request_sequence now uses only capability.read_only. Regression rejects a valid typed declaration with mutating=True, side_effects=() and the integrated current artifact-plan declaration.
  • IMPL-13 / BOUND-2 launch: DiagnosticServerSpec extends McpDevServerSpec.process_args(), retaining OpenHCSRuntimeImportAuthority; only instrumentation/XDG isolation is added. Entrypoint moved to openhcs.mcp.memory_diagnostic; bare child -m, ambient PYTHONPATH copying and the old benchmark entrypoint are deleted, with no alias/bootstrap copy.
  • DiagnosticSourceIdentity records both observed paths in every typed sample; the MCP client validates them against its parent authority, independently of health PID. --source-identity exposes the same observation without server construction.

Actual evidence: 15 focused diagnostic/import-authority tests pass in 4.34s. Real diagnostic children launched using the server spec from a competing cwd pass with PYTHONPATH absent and hostile; both diagnostic and OpenHCS paths must equal the selected source. Either foreign receipt path is rejected. Source CLI with PYTHONPATH absent passes in 0.16s, 53,500 KiB peak RSS, zero swaps. Focused AST guard reproduces all three old bypasses at dd5f177 and passes both corrected modules; Ruff correctness/diff checks pass.

Pattern/new-case and detailed command/resource evidence: validation receipt. This is focused source/launch evidence, not a full NRA proof. Minimal provenance inspection does not import the application-client/scientific dependency graph.

Still pending: real MCP retention slopes and installed continuous declaration/history restore/undo journey. Confucius retains the heavy runtime lock and frozen installed baseline; no new MCP/GUI/JVM/large environment startup, install or skill change. PR208 and ArrayBridge PR1 stay draft; neither #131 nor #169 gets a Closes claim. Separate #217/BaSiCPy source work is preserved, not replaced.

@trissim

trissim commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Parent integration checkpoint: independently reviewed ArrayBridge 06837ec and ran the three dependency cases plus the real ObjectState/document history journey: 4 pass, 3.030s, zero skips. Original packaged ArrayBridge ratchet passes all 38 metrics with zero deltas. I am shipping ArrayBridge PR1 as a source checkpoint without optional CI wait. Your PR208 tree remains untouched; current-main integration and actual installed desktop capture/restore plus MCP retention measurements remain pending. The history test clears/loads the registry in the SAME process: this is not fresh-process or live GUI restart evidence. Issues169/131 remain open. Swap14.6GiB still gives resource-check exit2, so no heavy/native/newparallel run is started. No history-format workaround, filtering or compatibility reader is accepted.

@trissim

trissim commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Current-main integration is now published, preserving your original d69 work and source tree: a5f639e integrates main4a9 normally and records merged ArrayBridge409 (tree identical to reviewed068). Parent independently ran 16 focused diagnostic/import/history source tests: pass5.34s; actual source-identity CLI exits0 in0.13s/25312KiB. Missing-native collection failure is retained; passing source harness reuses unchanged real ABI3 modules under original qualified names after checking C++ source equality, with no build/copy/install/product shim. Authoritative RST receipt contains exact scope. Focused ownership guard passes. Original packaged whole-context ratchet DOES NOT pass: 5101metrics, +5 external Linux smaps decoding reads, +2 canonical read_only/optional argparse predicates. Every site read; no guard alteration, new dispatch/roster or bypassed schema owner. These are classified screening leads, not a global-NRA or guard-pass claim. Per owner checkpoint/CI-deferral instruction, shipping the reviewed coherent source checkpoint rather than cosmetically rewriting valid owner/boundary queries to reduce syntax counts. No installed/GUI restart/MCP slope/blind acceptance is claimed; issues169/131 and full archive guard scope remain open. Integration/install/live remain parent-owned; do not duplicate product edits from the older source tree.

@trissim
trissim marked this pull request as ready for review September 30, 2026 12:07
@trissim
trissim merged commit 65b2ed1 into main Sep 30, 2026
1 of 37 checks passed
@trissim

trissim commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

MERGED source checkpoint verified: GitHub PR merge and independent git ls-remote agree on OpenHCS main65b2ed1017f36db193b6f48b8f9b46495cc915aa, 2026-09-30T12:07:28Z. Reviewed integration heada5f639e80; paired ArrayBridge main409b1e0 verified earlier. No admin override or hosted-CI wait. Sixteen current-main source checks pass; structural screen findings and source-harness limitations remain explicit in the RST receipt. This supersedes the older draft/current-main notes, not the unperformed acceptance work. NOT installed/live-verified; H002 saved/in-memory state unchanged, issues169/131 remain open, actual MCP slope and desktop restart acceptance pending. Resource assertion still exits2 for swap14.6GiB; no heavy/native/new-parallel run or uncertain input replay.

@trissim

trissim commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

INSTALLED checkpoint now verified (supersedes the previous not-installed status): main65b2ed101/ArrayBridge409 installed offline serially in existing .venv,5.40s/144420KiB peak, no dependency/Fiji/JDK downloads or extra runtime env. Actual installed skill-sync dryrun+publish report unchanged/success. Sixteen installed checks pass5.47s with PYTHONPATH unset and real newly built native modules, no preloader. Real installed MCP memory journey completes51events/16.41s with flat post-warmup discovery RSS/privateDirty/swap (issue131 has exact limits); process exits cleanly. Still NOT complete live desktop capture/restore or broad leak acceptance. Protected old H002 viewer geometry mismatch is also reproduced against a fresh healthy installed receiver; issue280 updated, no history dropped or peer forcibly restarted. Persistent active installed tree is /home/ts/wt/openhcs-memory-integration-20260930; do not retire it as an inactive merged worktree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant