diff --git a/CHANGELOG.md b/CHANGELOG.md index 84e4d94..c0e74e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,11 +5,264 @@ All notable changes to this project are documented here. The format follows ## Unreleased +0.14.0 makes the persona layer a library crate, `probbit-persona` (std, and `no_std` + `alloc` checked on +`xtensa-esp32s3-none-elf`), and fits it to a small host: heap-returning setup, turn and resume (one event's stack +high-water on the host proxy 31,824 → 13,808 bytes), resume from a checkpoint or from a record behind the strand, and sealed +strand segments. The expectation keeps an exact cache and solves the event's own program once. The self layer arrives off by +default (RFCs 0013–0016): a reflection computed from a strand's lines alone, an integrity account of the drift from the declared +self, a rest pass over the expectation's memory and a curiosity drive with an optional stir; a declared, replayable self; it +does not feel. A persona that declares none of its keys gets none of its fields or lines. The monitor gains the Mirror tab, the +integrity dial and life-in-segments serving; every CLI output of the library move is byte for byte as 0.13.0. + +Engine gate: strands with the new line kinds (`reflection`, `consolidation`) or the new keys (`integrity`, `curiosity`, a +checkpoint's `expect`, `expectation.rest`, the `identity` self keys) read as unsupported, exit 5, on 0.13.x. + +### The persona layer as a library crate (std, no_std) + +- **`probbit-persona`, the document layer and the strand writer as a library crate** (docs/embedding.md): `json`, `yaml`, + `persona`, `run`, `router` and `evaluate` (feature `decide`), `sys`, and `strand` (the pure half of `probbit live`: `Live`, the + line writers, `Replay`, `verify*`, with no file I/O) move out of probbit-cli, which uses them unchanged; every CLI output is + byte for byte as 0.13.0. probbit-core, probbit-ir, probbit-decide and probbit-persona gain the features `std` (default) and + `nostd` (`no_std` + `alloc`, through the new `probbit-nostd` facade), checked on `xtensa-esp32s3-none-elf`; probbit-wasm + depends on the crate instead of including the CLI's files. +- Development build: the workspace version is `0.14.0-dev` (`probbit --version`, the `engine` of every strand header this + build writes, `verified_by`), so the strands it writes get the 0.14.0 behaviours below (the rounded deviation, the + canonical fold, `expect` on checkpoints, segments of a persona with an expectation block). Strands of 0.13.0 and earlier + replay as their engine wrote them. +- **Cross-platform parity of the deviation** (docs/persona.md section 2.17, docs/embedding.md): the stance's `deviation` + (`self`, `pred`, `input`) is rounded to 6 decimals like every other number of a stance, so a checkpoint line has the same + bytes on every libm (a `nostd` build's `libm` crate differed from the host's by 2 ULP, which broke the chain at the second + checkpoint). Gated on the strand header's engine: a strand written before 0.14.0 keeps its full-precision deviation, and + `live verify` accepts its checkpoint when it equals the replay's at 6 decimals; strands without an expectation block are + unchanged. +- **`turn_trace`** (feature of probbit-persona and probbit-cli, off by default): `persona::trace` records the last event's + microseconds and a host probe's value per phase (input, expect, prep, compile, solve, decode, document, line, fold), so a + firmware measures the same breakdown on its hardware; `probbit live` in such a build prints it on stderr when + `PROBBIT_TURN_TRACE` is set. + +### Small hosts: boxed setup, checkpoint resume, strand segments + +- **The setup path fits a small stack** (docs/embedding.md "Stack budget"): heap-returning setup calls + `persona::build_boxed` (`Box`), `persona::init_boxed` (`Box`), `Live::start_boxed` (`Box` and the header), + `Live::start_to` (the header handed to the host in fixed-size pieces, so a firmware can write it to flash piece by piece with + yields) and `strand::resume_boxed`; `strand::write_header` writes the header line in pieces and returns its sha256. The persona + reader no longer holds a `Persona` by value at each of its layers, and its readers are out of line. On a host proxy of a firmware's setup (a 12 KB persona, `nostd`, x86_64) the stack high-water of parse + build + init + start fell from 39,056 to 12,112 bytes; on Xtensa the reader's frame went from 3,312 to 400 bytes and `Live::start`'s from 1,888 to 160. The by-value + calls are unchanged wrappers; every header, line and digest is byte for byte as before. `persona::sha256` hashes without + copying its input (`persona::Sha256`, fed in pieces). The YAML reader refuses collections nested deeper than 256 (a `limit` + error, as the JSON reader does) instead of recursing without a bound. +- **The turn and `resume` fit a small stack too** (docs/embedding.md "Stack budget"): the turn's steps run out of line and pass + the state boxed (`persona::turn_boxed`, which `Live::event` runs; `turn` is its wrapper), the expectation's programs compile + the same way, and `resume` opens the strand's header once (the chain check builds the persona on the heap and compares the + header in pieces, without a replay). On the host proxy one event's stack high-water fell from 31,824 to 13,808 bytes and + `resume`'s from 30,560 to 15,792; every stance document, strand line and digest is byte for byte as before. +- **Resume robustness** (docs/embedding.md "Resume on a body", docs/persona.md section 5.7 "Resume from a record"): + `strand::Replay::open_boxed` checks the header as the chain check does (the persona built on the heap, the header written + again in pieces) and keeps its `Live` boxed, so a replay from the header fits a small stack (proxy: 24,752 → 16,192 B); + `Replay::open` is the same call. `strand::resume_from_record(text, &p, state, clock, &engine)` resumes from a state record + that is behind the strand (a reboot between a line and its record): the event lines after the record's line are applied + (their turn on the logged inputs, digests and bytes checked; not the expectation's member solves) and the individual equals + the unbroken one, byte for byte; refused with the line for a record the strand does not log, a line that differs, another + persona. `strand::resume_boxed_from(header, checkpoint, line, trailing, &p, state, clock, &engine)` resumes from the header, + the last checkpoint line and the lines after it read one at a time, never the whole strand: cost O(lines since the + checkpoint). `turn_trace`: `trace::last_resume()` (lines walked, lines applied, their µs). No byte of any strand changes. +- **Resume after a checkpoint with an expectation block**: the chain check that `resume` (and `probbit live ... --strand`) runs + validates each checkpoint line (its state read, its event count, its link to the event before) without opening it as a replay + start, which a persona with an `expectation` block cannot do; such a strand resumes after its first checkpoint (it exited 2 + before). Every line and digest as before. +- **Checkpoints carry the expectation's memory** (docs/persona.md section 5.7, engine 0.14.0 on): the canonical fold (every count, + the mean and the variance at 6 decimals; at most `max_combos - 1` named combos, the one moved out adding its count to `rest`, + the empty member's) and one more checkpoint key, `expect`, so `verify --from-checkpoint`, `resume` and `monitor` start at the + last checkpoint of a strand with the block and fold only the events after it. The line is O(max_combos) (6,369 B for the + test fixture at `max_combos` 16; 14,121 B at 64). Strands whose header names an engine before 0.14.0 keep the four-key + checkpoint and the earlier fold, byte for byte; an earlier engine reads `expect` as unsupported (exit 5). +- **Sealed segments** (docs/persona.md section 5.7, docs/embedding.md "Sizing a strand on flash"): `probbit live ... + --segment-every N` seals the strand with a checkpoint line every N events and goes on in `NAME.0001.strand`, ...: a header with + `segment` {index, previous, sealed} and the sealed line itself, so the chain continues across files. `live verify SEGMENT` + checks one segment on its link (summary `segment`, `trusted_link`); `live verify SEGMENT --chain` walks back to the first file + (exit 1 at a break, naming the file; exit 6 when a previous file is missing). `strand::segment_header`, + `strand::segment_name`, `strand::write_header_seg`, `strand::Replay::open_lines`. +- **`probbit live control STRAND seal`** (docs/persona.md section 5.7): a host seals when it decides. Under the writer lock it + replays the newest segment, refuses unless its last line is an event line (exit 2, nothing written), appends the sealing + checkpoint, writes the next segment file and prints it (`file`). `probbit live ... --strand NAME.strand` now always continues + the newest segment that exists, with or without `--segment-every`, so a sealed file is never appended to. + +### The expectation cache and the members + +- **The expectation's cache** (docs/embedding.md "The expectation's cache"): `Live` keeps the exact marginals of the + programs the deviation solved, keyed by the program without its comment, so an event whose members' programs are the + previous event's solves none of them again (a life of one repeated event: `expect` 52% → 37% of the turn on the proxy, 95% + of its programs from the cache). Every stance, strand line and digest is the same with the cache on or off + (`Live::expect_cache`, `persona::ExpectCache::new(false)`); at most `max_combos + 1` rows; a life whose programs move every + event (most lives: the state moves) backs off and costs about what it cost before. `turn_trace` gains `expect_cached` and + `expect_solved`; `persona::deviation_cached`. +- **The expectation's members as models** (docs/embedding.md "The expectation's members: one program text per event"): with + the cache on, the event's own program is the only text the deviation builds and reads; every other member is compiled + without its text and, when the habits in force are the event's, solved as the event's model with its own `h` + (`run::from_json_parts`, `run::Parts::model_with`, `run::enumerated`: the tier `run` answers with, the same calls); another + tier or other rules: its text, written from its compile meta, run by the engine. Same marginals, so every stance, strand line + and digest is the same as the reference path (`ExpectCache::new(false)`); proxy, the body persona's recorded day: `expect` + 74.6% → 67.7% of the turn. `turn_trace` gains `expect_build_us` and `expect_solve_us` (the members' builds and solves). +- **The expectation's members, trimmed** (docs/embedding.md "The expectation's members: what the deviation reuses"): + `Live::event` runs the turn first and, with the cache on, the turn hands the deviation its first program's meta and exact + marginals (`persona::turn_at`, `persona::deviation_at`), so the event's own program is compiled and solved once (when no rest + is due, `src` is taken out alike and the op is `decide`; else as before). A member's prep copies only what it steps and + reads, its compile writes no source names, and the members share one model (`run::Members`, the compile pass's folds + replayed per member). `ExpectCache::with_reuse` keeps the old order or checks the reuse. Every stance, strand line and digest + is the same; proxy, the body persona's recorded day: `expect` 67.7% → 63.9% of the turn (-14% per event). The deviation + runs after the turn, so the event's heap peak (the body persona, proxy) is 97,583 B (89,990 before); `with_reuse(false, false)` + keeps the old order (the deviation before the turn: 89,990 B). +- **The expectation's members before the turn** (docs/embedding.md "The expectation's members: the order with the reuse, and + what is held"): with the cache reusing the turn's program, every member but A_t is solved before the turn (nothing of the + turn held; `persona::deviation_pre`), its exact marginals held as f64 rows, and A_t's mixture runs after the turn + (`persona::deviation_post`). Every member takes the members' one model, read without any program text (the persona's + couplings, a text without its pairs): a member with other rules in force than E_t's takes it with its own caps + (`run::Members::set_caps`), and the components tier's split is computed once per caps and reused per member + (`probbit_ir::exact_components_split`). Every stance, strand line and digest is the same. Proxy, the body persona: event + heap peak 97,583 → 88,844 B, inside `expect` 97,583 → 55,920 B; recorded day: `expect` 64.1% → 57.5% of the turn, + 0 members on the engine's path (253). + +### The self layer (RFCs 0013–0016, off by default) + +- **Self-report** (RFC 0013, docs/rfcs/0013-self-report.md, docs/persona.md §5.8 "Self-report"): `probbit reflect STRAND + [--last N | --since-checkpoint] [--json | --text]` and `probbit_persona::reflect` (std and `nostd`): a canonical account of a + window of a strand's events from its lines alone, no solve: the inputs set, the expectation's surprises (z recomputed from the + logged receipts, the 5 largest |z| with predicted vs measured), per trait the levels taken and the odds' drift, the moods' + path and turning points, the drives' peaks, the habits bound, the credit settled, the learned weights' net change and the + control lines (the stance sections read from the checkpoint lines in the window; `null` without one); every number at 6 + decimals, a sha256 digest, 6-12 sentences from fixed templates. Off by default: nothing is written unless asked. +- **Reflection lines** (engine 0.14.0+, docs/persona.md §5.7): `probbit live control STRAND reflect [--last N]` appends the + reflection as a line; `live verify` recomputes it from the lines before it and refuses a forged one naming the line; an + engine before 0.14.0 reads it as unsupported (exit 5). Every strand without one, and every corpus, is byte for byte unchanged. +- **Integrity** (RFC 0014, docs/rfcs/0014-integrity.md, docs/persona.md §2.19 and §5.9): four optional self keys in the + `identity` block (`keep`, `warn`, `cue`, `traits`; outside the persona digest) turn on, per live event, an exact account of + the drift from the declared self: per trait the total variation between the rest-probe odds of the individual rested (the + channels of the hour at the fresh values, what persists as it is) and of the seed's fresh individual, `bound_used` against + RFC 0004's epsilon at the seed, the dial's `used`, the `keep` habits held or not fired, and the persona's own `cue` past + `warn`; every number at 6 decimals. Library `probbit_persona::integrity::{compute, compute_cached, Integrity}` (std and + `nostd`); the rest probe is cached by its program without the comment (the members' key; a row only for an exact answer) and, + while no row is held, by the rested state's body without `turn`. +- **Integrity on the event line** (engine 0.14.0+): the stance document and the event line carry `integrity`; `live verify` + recomputes it and refuses an edited value naming the line; an engine before 0.14.0 reads such a persona as unsupported + (exit 5). A persona without the self keys writes every strand, stance and corpus byte for byte as before. +- **Integrity's cost** (RFC 0014-b): on a body's recorded day the rest probe is answered from the cache 1,156 of 1,160 times + (from 0): +16.1 % of a turn on the host (from +40.1 %), 13,656 B on the `nostd` build per miss (from 20,952 B). Every + integrity number is byte for byte as before. Still over the 2 % / 8 KB budget: a hit builds the probe's program, a miss is a + turn (docs/embedding.md "Integrity on a body"). +- **Rest pass** (RFC 0015, docs/rfcs/0015-consolidation.md, docs/persona.md §2.17): an optional `rest` key of the + `expectation` block (`half_life`, `bucket`, `surprise_gain`, `decay`; every default moves nothing) declares a memory + consolidation a host runs at rest with `Live::consolidate(at)` (`consolidate_boxed` on a body) or `probbit live control + STRAND rest [--at TIME]`: the window since the last pass (folded in the canonical memory, carried by checkpoints) re-folded + with the rest half-life, counts weighed by the contexts' mean z+, near-duplicate contexts merged, the learned deltas + decayed; one engine-gated `consolidation` line logs the memory's digests before and after, the contexts merged, the weights + moved and the state after. `live verify`, the chain check and both resumes re-run the pass and name the first field of a + forged line; the reflection document gains `consolidation` and one sentence when its window holds such a line. Off by + default: without the key the pass is refused and every document is unchanged. +- **Curiosity** (RFC 0016, docs/rfcs/0016-curiosity.md, docs/persona.md §2.20): an optional `curiosity` block (needs an + expectation block; in the digest) declares a drive fed by the measured surprise: `surprise_recent` (an average of z+ = max(z, 0) with a + half-life in events) and `failed_contexts` (remembered input contexts whose last K visits all missed). Its level, clipped + to a cap at 6 decimals, moves the stance through the engine-owned number input `cur_level`; the stance carries + `drives.curiosity` and the event line a `curiosity` receipt that `live verify` recomputes; the fold rides in the + expectation's checkpoint memory. The optional `stir` gates a self-started idle event: `Live::stir_due(idle_hours)` and + `probbit live control STRAND stir? [--idle-hours H]` (threshold, a rate cap, a slower quiet class while a declared flag + is on); a stir is an ordinary event with `"stir": ""`. The monitor's drives rows show the drive; with a declared stir + the reflection document gains `stirs` and one sentence. Off by default: every document without the block is unchanged. + +### Monitor + +- **The monitor's Mirror tab**: `/reflection?last=N` and `/s/ID/reflection` serve the reflection computed from the strand file + and the latest logged reflection line with whether it recomputes equal; the page shows the sentences, the top surprises, the + habits bound and the moods at the checkpoint samples. +- **The monitor's integrity dial and the Mirror's drift table**: the instrument panel's "room left in the bound" placeholder + becomes the integrity dial when the stance carries `integrity`; the Mirror tab shows the per-trait drift, allowance and share + of the event on view. +- **`probbit monitor --dir` serves a life in segments as one strand**: `NAME.strand` and `NAME.0001.strand`, ... (without a gap) + are listed once, id `NAME`, with the life's event count, the newest file's `segment` index and the base name's sidecar; the + page follows the newest file and goes on in the next one within the 1 s scan after a seal, its event count carrying on. + +### Fixes + +- The test of the served demo's restart allows 45 s for its paced loop (about 13 s of sleeps): on a virtual machine whose sleeps + overshoot up to 2.5 times it failed at 20 s, on this tree and the one before (the replay itself takes about 30 ms). +- `probbit monitor` opens a strand with checkpoint lines when its persona has an `expectation` block: it replays such a + strand from the header (`--once`, `--follow`, `--serve`, and each strand of `--dir` by its own persona) instead of refusing + it at the last checkpoint, which does not carry the expectation's memory. Other personas still start at the last + checkpoint; `live verify` is unchanged. + +### Documentation + - Documentation, from a dress rehearsal of the published v0.13.0 artifacts (the release page's Mac app zip opened as a download, the README and REPRODUCE.md commands with the released binary on Linux and macOS, `probbit-stance` 0.2.0 from npm): `docs/monitor-app.md` says to remove the quarantine mark before the first open and how to recover after a refused one (move the app, or Open Anyway), and that the bundled probbit serves several strands with a switcher; the plugin README names the second warning an older probbit gives; REPRODUCE.md no longer says only 0.12.0 replays the strand. +### Research + +- **RFC 0011, round 7: one math path for the register** (research, `engine.tier: amplitude` stays opt-in; docs/rfc/0011-amplitude-register.md, + "One math path"). The simulated register evaluates cos, sin, sqrt, asin, acos, atan2 and floor through the `libm` crate on the std and + the no_std build alike (a direct dependency of probbit-decide). 100 fresh lives per schedule recorded on the std build re-verify on + the no_std build 100/100 for `tf-linear-16x4`, `tf-linear-64x16` (round 5: 0/100) and `qaoa-p2` (98/100), x86_64; the tutor's + default-schedule strands are byte-identical to before (100/100) and both demo strands re-record byte for byte (a 52-qubit + recorded day's default strands moved in the last bits). Cost on that day: long sweep 1.50x the default. Exact, on a CPU. +- **RFC 0011, round 7: the amplification lands on the long sweep** (research). `amplify_exact` renormalises the swept state once + before the standard iterations: the diffusion about a state of squared norm 1 + eps is not unitary, so the shortfall grew with the + iterations and 165 corpus programs declined under the long sweep; now 0 decline under any schedule (median gap 0.151709, agreement + 0.981671 on the corpus). The receipt's mass before is unchanged; a mass that is not a number declines instead of panicking. + Simulated, exact, on a CPU. +- **RFC 0011, round 7: docs** (research). docs/persona.md's `engine.schedule` paragraph and the demo README state the measured + replay on both builds; docs/persona.md section 5.8: the stance digest is the sha256 of `/doc/N`'s JSON without the trailing newline. +- **RFC 0011, round 6: the demo** (research; `examples/persona/demo/`, docs/rfc/0011-amplitude-register.md, "The demo"). A + fictional tutor reduced to its humour component, with praise and criticism pulling humour in opposite directions, in two + copies: `register.yaml` (`engine: {tier: amplitude, amplify: true, schedule: tf-linear-64x16}`) and `thermal.yaml` (no engine + block). One 17-event life recorded on each (both verify, twice byte for byte, 0 floats past 6 decimals). At event 9 (line 10 + of both strands, the same program) the thermal odds of the top two humour levels sit 0.007412 apart and the simulated + register separates them by 0.494808; the Mood Monitor's two faces at that event are in the RFC. On the corpus's own program + `a7fae99b114ea2d2`: thermal gap 0.010516, long sweep 0.951467. The long sweep replays on the recording build only; the + default schedule stays for every recorded strand. No engine change. Simulated, exact, on a CPU. +- **RFC 0011, round 5: schedules and the indifferent set** (research, `engine.schedule`, absent = the default sweep; + docs/rfc/0011-amplitude-register.md, "Schedules and the indifferent set"). Three names: `tf-linear-16x4` (default), + `tf-linear-64x16` (four times the layers), `qaoa-p2` (two layers, angles fixed on a held-out third of the corpus). On the 10.7 % + of the recorded corpus's trait decisions where the thermal tier is indifferent, the simulated register's top-two gap has median + 0.049 (default), 0.152 (long sweep), 0.094 (qaoa-p2); agreement elsewhere 97.6 / 98.2 / 93.0 %; noise 0.0007. The long sweep + is the rule's choice (1.5× the solve per event on a recorded day). A missed amplification landing (165 programs under the long sweep) now declines to the + default tiers instead of panicking. Failure, stated: the no_std build re-verifies the long sweep's strands 0/100 (qaoa-p2 98/100; + the default 100/100): last-bit math differences, so non-default schedules replay only on the recording build for now. Without + the key every stance and strand is byte-identical to round 4. Exact, on a CPU. +- **RFC 0011, round 4: exact amplitude amplification in place of the projection** (research, `engine.amplify: true`, off by + default; docs/rfc/0011-amplitude-register.md, "Amplification"). `amp::amplify_exact`: the rules as the oracle, the diffusion + about the swept state, k standard iterations and one phase-matched final one, so the simulated register lands on its + feasible plans exactly (mass ≥ 1 − 1e-12, asserted). The plane argument says the result is the projection's distribution; + measured: marginals equal to 1.1e-12 on 220 programs, the same plan on 220/220 and on all 339,389 solves of the recorded corpus (B4 unchanged, 96.29 %), + 287/287 stance lines on a recorded day; + cost 1.02× the projected solve. The receipt adds `engine.amplify` and `feasible_mass` becomes 1. Without the key every stance + and strand is byte-identical to round 3. Simulated exactly on a CPU; no quantum hardware, no speedup claim. +- **RFC 0011, round 3: the verdict, KEEP as an optional tier** (research, off by default; docs/rfc/0011-amplitude-register.md, + status "accepted as optional tier"). Measured on every program the recorded corpus's 653 outputs solve (338,567 exact solves, + 0 declined, the corpus byte-identical with the comparison beside it): the exact simulated register picks the same level as the + exact default tier in 2,095,627 of 2,176,371 trait decisions (96.29 %, bar 90 %); the differing cases sit where the default + tier's own level holds less than half the mass (91 % of them), not where the rules' projection does the work. Three designed + two-path programs differ from the thermal odds by 0.25-0.27 at the node (engine vs reference 1.4e-14). The bound harness with + the tier: 0 of 4,100 cases above a certificate computed with the tier, 1,598 above one computed without it, so `persona bound` + must run with the tier on (docs/persona.md says so). A rule-penalty term was measured and not adopted (it lifts the Born mass + on rule-abiding plans and drops agreement to 83 %). No code change in this round. Simulated exactly on a CPU; no quantum + hardware, no speedup claim. +- **RFC 0011, round 2: multi-level traits on the amplitude register** (draft, research; docs/rfc/0011-amplitude-register.md): + a variable with d allowed values takes ⌈log2 d⌉ qubits (`amp::Encoding`; a three-level trait two qubits, its fourth code + unused), the start is the uniform superposition over the valid codes, the mixer is the complete-graph mixer per variable + (`amp::MIXER`, the identity on unused codes; exp(i t X) for two values, round 1's mixer exactly), and the cost is one general + diagonal phase layer exp(−iγE), E = −log w per basis state (`Register::phase`), so pair tables between three-level traits + need no decomposition (round 1's Walsh transform and its decline are gone). The unused codes keep amplitude 0, checked after + every layer, never renormalised. The tier runs one register per connected component of the program (exact: the state is the + product of the components' states); `engine.q_max` bounds the largest component's encoded qubits. New receipt fields + `engine.components`, `encoding`, `mixer`; `qubits` counts encoded qubits. Measured against an independent mixed-radix + reference: max |Δ marginal| 2.7e-13 on 220 programs. Simulated exactly on a CPU; no quantum hardware, no speedup claim. +- **RFC 0011, round 1: an amplitude register** (draft; docs/rfc/0011-amplitude-register.md): `probbit-decide::amp`, an exact + simulated qubit register on a CPU (q ≤ 16 on the host; 16 · 2^q bytes of amplitudes on the heap; std and no_std + alloc): + gates H, X, Z, RX, RZ, CNOT, CZ, RZZ, a Trotterised transverse-field sweep, QAOA layers, exact marginals, Born-rule + measurement with the engine's RNG, a sha256 digest of the state at 6 decimals. The persona flag `engine.tier: amplitude` + (off by default, `engine.q_max` default 12) solves a turn's program on it when every variable has at most two values; + otherwise the default tiers answer and the stance document says why. Receipt fields `engine.qubits`, `circuit`, + `schedule`, `schedule_name`, `state`, `feasible_mass`. Without the flag no byte of any stance or strand changes. No quantum + hardware, no speedup claim; the comparison with the default tier is the next round. ## 0.13.0 - 2026-10-26 diff --git a/Cargo.lock b/Cargo.lock index cbb7e8c..7b2e5f6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,39 +2,102 @@ # It is not intended for manual editing. version = 3 +[[package]] +name = "critical-section" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b" + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash", +] + +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + +[[package]] +name = "portable-atomic" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85" +dependencies = [ + "critical-section", +] + [[package]] name = "probbit-cli" -version = "0.13.0" +version = "0.14.0-dev" dependencies = [ "probbit-core", "probbit-decide", "probbit-ir", + "probbit-persona", ] [[package]] name = "probbit-core" -version = "0.13.0" +version = "0.14.0-dev" +dependencies = [ + "probbit-nostd", +] [[package]] name = "probbit-decide" -version = "0.13.0" +version = "0.14.0-dev" dependencies = [ + "libm", "probbit-core", "probbit-ir", + "probbit-nostd", ] [[package]] name = "probbit-ir" -version = "0.13.0" +version = "0.14.0-dev" dependencies = [ "probbit-core", + "probbit-nostd", +] + +[[package]] +name = "probbit-nostd" +version = "0.14.0-dev" +dependencies = [ + "hashbrown", + "libm", + "portable-atomic", +] + +[[package]] +name = "probbit-persona" +version = "0.14.0-dev" +dependencies = [ + "probbit-core", + "probbit-decide", + "probbit-ir", + "probbit-nostd", ] [[package]] name = "probbit-wasm" -version = "0.13.0" +version = "0.14.0-dev" dependencies = [ "probbit-core", "probbit-decide", "probbit-ir", + "probbit-persona", ] diff --git a/Cargo.toml b/Cargo.toml index 43994e6..35934af 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,9 +1,9 @@ [workspace] resolver = "2" -members = ["probbit-core", "probbit-ir", "probbit-decide", "probbit-cli", "probbit-wasm"] +members = ["probbit-nostd", "probbit-core", "probbit-ir", "probbit-decide", "probbit-persona", "probbit-cli", "probbit-wasm"] [workspace.package] -version = "0.13.0" +version = "0.14.0-dev" edition = "2021" license = "Apache-2.0" rust-version = "1.78" diff --git a/RELEASING.md b/RELEASING.md index 05fb12b..6b0641c 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -15,7 +15,7 @@ the RFC). `probbit-cli/tests/readme_release.rs` fails when the README's current 1. **Release PR** (branch `release-X.Y.Z`, squash subject `Release probbit X.Y.Z (#N)`), from a green `main`: - `Cargo.toml`'s `version` and the `version` of the internal path dependencies (`probbit-cli`, `probbit-decide`, - `probbit-ir`); `cargo update --workspace --offline` for `Cargo.lock` (CI builds `--locked`); + `probbit-ir`, `probbit-core`, `probbit-persona`: the `probbit-nostd` entries too); `cargo update --workspace --offline` for `Cargo.lock` (CI builds `--locked`); - `cargo build --release --locked -p probbit-cli`, then `target/release/probbit man > docs/probbit.1` (generated, never hand-edited: `probbit-cli/tests/cli_man.rs` compares them); - `npm/package.json`'s `version`; the example tag in `install.sh`, `install.ps1`, `action.yml`, @@ -49,8 +49,9 @@ the RFC). `probbit-cli/tests/readme_release.rs` fails when the README's current `package.json`; no tests), then `npm publish --access public` with the same userconfig; `npm view probbit-stance version` gives the new version. Its `MIN_PROBBIT` (`src/version.ts`) names the oldest probbit it runs. -6. **crates.io**: `cargo publish --workspace --locked` from the tag (`probbit-core`, `probbit-ir`, `probbit-decide`, - `probbit-cli`; `probbit-wasm` has `publish = false`). +6. **crates.io**: `cargo publish --workspace --locked` from the tag, which publishes in dependency order (`probbit-nostd`, + `probbit-core`, `probbit-ir`, `probbit-decide`, `probbit-persona`, `probbit-cli`; `probbit-wasm` has `publish = false`). + `probbit-nostd` goes first: the engine crates name it as an optional dependency with a version. 7. **Back to development**: `main` moves to the next `-dev` version (`Cargo.toml`, the path dependencies, `Cargo.lock`, `docs/probbit.1`); the README goes on naming the released version until the next release PR. diff --git a/docs/embedding.md b/docs/embedding.md new file mode 100644 index 0000000..65a5fde --- /dev/null +++ b/docs/embedding.md @@ -0,0 +1,515 @@ +# Embedding probbit: the `probbit-persona` library, with std or on `no_std` + +The CLI's document layer is a library crate, `probbit-persona`: the JSON and YAML readers, the persona layer (parse, build, +lint, prove, bound, hostlint, the turn, `display.glyphs`), `run` (a probbit-ir JSON program), `router` and `evaluate` (feature +`decide`), and the strand writer and its checks (`strand`). The CLI, probbit-wasm and a firmware run the same code, so a +document or a strand line written by one is the one the others write. + +| feature | default | what it gives | +|---|---|---| +| `std` | yes | builds on std (the CLI, probbit-wasm, any host program) | +| `nostd` | no | builds `no_std` + `alloc` for a target without an operating system; the std subset comes from `probbit-nostd` | +| `decide` | yes | `router` (`probbit decide`) and `evaluate` (the decision-API adapter), through probbit-decide | + +probbit-core, probbit-ir and probbit-decide have the same `std` (default) and `nostd` features. A host program takes the +defaults: + +```toml +[dependencies] +probbit-persona = "0.14" +``` + +## On `no_std` + +```toml +[dependencies] +probbit-persona = { version = "0.14", default-features = false, features = ["nostd"] } # add "decide" for router/evaluate +probbit-core = { version = "0.14", default-features = false, features = ["nostd"] } +probbit-ir = { version = "0.14", default-features = false, features = ["nostd"] } +``` + +The firmware supplies a global allocator (for example `esp-alloc`) and, because `probbit-nostd` builds 64-bit atomics from +`portable-atomic`'s critical-section fallback, a `critical-section` implementation (the HAL's). Checked on +`xtensa-esp32s3-none-elf` (the esp toolchain, `-Zbuild-std=core,alloc`): + +```sh +cargo +esp check -p probbit-persona --no-default-features --features nostd --target xtensa-esp32s3-none-elf -Zbuild-std=core,alloc +``` + +What changes without an operating system: + +- **The facade.** `probbit-nostd` is the subset of `std` the crates use, built from `core`, `alloc`, `hashbrown`, `libm` and + `portable-atomic`. Each crate imports it as `std` on `nostd`, so the code is the same code. Float functions that live in + `std` (`exp`, `ln`, `powf`, `tanh`, ...) come from `libm` through the `FloatExt` trait; `powi` is computed as compiler-rt's + `__powidf2` computes it on the host. `print!`/`println!` go to a console hook: `probbit_nostd::io::set_print(fn(&str))`. +- **The clock.** `probbit_core::rt::Instant` is the clock you install: `probbit_core::rt::set_clock(fn() -> f64)` returning + milliseconds (a firmware passes its system timer). Before any call that measures time, install it. +- **No threads.** `rt::sequential()` is on by default: every chain, gate pass and polish chain runs in order on the calling + thread, as probbit-wasm runs. At fixed work the documents are the ones the CLI writes at any `--threads` (timings and + `telemetry.threads` aside). Set `probbit_ir::GATE_THREADS` to 1 as probbit-wasm does. One core calls the engine. +- **Documents arrive as strings.** There is no file system: `persona::load(path)` returns an error. Read the document with + `persona::parse_doc(text, is_json)` and build it with `persona::build(&doc)` (or `build_boxed`, below). +- **No usage telemetry.** `sys::usage()` and `sys::nice()` return `None`, so `process_cpu_ms`, `peak_rss_mb` and `nice` are + `null` in telemetry. + +## Stack budget: the boxed setup path + +A small target runs the engine on a thread with a fixed stack (tens of KB). The by-value setup calls move `Persona` (about +1.5 KB), `State` (about 1 KB) and `Live` through their frames, and the persona reader recursed through itself once per layer of +the document (`display`, `bound`, `expectation`, the lowered blocks) with all of its readers inlined into that frame. The +heap-returning calls keep each of them on the heap from the moment it is built: + +```rust +use probbit_persona::{persona, strand::{self, Clock, Live}}; +let doc = persona::parse_doc(TEXT, false)?; // Json: a 32-byte value; the tree is on the heap +let p = persona::build_boxed(&doc)?; // Box +let st = persona::init_boxed(&p, None, true, &eng); // Box +// a fresh strand: the header goes to `out` in pieces of at most 1 KB (write each, yield between them), not as one String +let live = Live::start_to(p, &doc, st, Clock::Real(probbit_core::rt::Instant::now()), &strand::engine(), 1024, &mut out); +// or: let (live, header) = Live::start_boxed(p, &doc, st, clock, &engine); // Box and the header whole +// a stored strand: let live = strand::resume_boxed(&text, &p, st, clock)?; // Box +// or from the last checkpoint without the whole text: strand::resume_boxed_from(&header, Some(&cp), line, trailing, &p, st, clock, &eng)? +// a replay from the header: let r = strand::Replay::open_boxed(&header)?; // Box, its Live boxed in it +// each event: live.event(&ev, &eng)? -> (stance, line); the turn inside passes its state boxed (persona::turn_boxed) +``` + +`build`, `init`, `Live::start` and `resume` stay as they were (thin wrappers over the boxed calls) for hosts with a large stack. +The header the pieces make is `Live::start`'s header byte for byte, and `strand::write_header` returns its sha256 (the first +line's `prev`), so the chain is the same however the host cuts it; neither the line nor a copy of the document is held whole. +The YAML reader is bounded like the JSON one: block and flow collections nested deeper than 256 are a `limit` error (the JSON +reader already refused them); on a stack of tens of KB keep documents a few levels deep, as every example is. + +The stack each call needs, measured by a host proxy: probbit-persona `nostd` on x86_64 (release, `lto = "fat"`), each call +run from one frame on a thread whose free stack is painted with a pattern first and scanned after (the high-water below the +calling frame), and a counting allocator for the heap peak above what was live before the call. The persona is a 12 KB body +persona (traits, moods, senses, drives, needs, expectation, display; its header line is 11,985 bytes), read from YAML. + +| call (before → after) | Xtensa frame, B | stack high-water, B | heap peak, B | +|---|---:|---:|---:| +| `parse_doc` (YAML) | 128 → 128 | 4,760 → 5,048 | 93,411 → 93,411 | +| `build` → `build_boxed` | 3,312 → 400 (its reader `build_base` 1,280) | **39,024 → 12,080** | 267,736 → 269,232 | +| `init` → `init_boxed` | 1,184 → 624 (+ `init` 1,184) | 9,240 → 9,240 | 69,365 → 69,365 | +| `Live::start` → `Live::start_to` / `start_boxed` | 1,888 → 160 / 96 (`write_header` 448) | 9,872 → 4,560 | 87,482 → 19,181 | +| **setup: parse + build + init + start** | | **39,056 → 12,112** | 332,669 → 334,165 | +| `resume` → `resume_boxed` | 4,880 → 960 → 896 (`check_chain` 6,320 → 96 + `check_chain_head` 384) | 55,328 → 29,264 (30,560 measured again with the phase rows added) → **15,792** | 397,428 → 398,924 → 356,129 | +| one event (`Live::event`) | 2,688 → 2,672 → 960 (`turn_src` 6,160 → 528; its steps `turn_prep` 448, `turn_solve` 1,312, `turn_doc` 704) | 31,904 → 31,824 → **13,808** | 112,990 → 112,990 → 88,149 | +| the solve alone (in the event) | | 6,856 | 29,009 | +| a checkpoint line | 160 | 5,408 → **3,264** (the key sort in place) | 42,594 | + +Three columns where a row has them: before the boxed setup, after it, and after the boxed turn and resume. The turn's steps +(`turn_prep`, `turn_solve`, `turn_doc`, the state steps, the decoder) are out of line and pass the state boxed, so the deepest +frame is one step's; the expectation's programs (`program_json`) compile the same way. `resume` opens the header once: the chain +check builds its persona on the heap, writes the header again in pieces against the stored line (no `Live`, no whole line) and +`resume` continues with that persona instead of building it a second time; the expectation's fold reads one line at a time out +of line. The heap peak of setup and of `resume` is the reader's (the document tree and the lowered documents); the event's +(88,149 B) is the live set of a turn: the expectation's exact solves, then the program as a document, the engine's answer and +the stance under construction (per phase, the peak above the event's start: the expectation's solves 66,879, prep 20,221, +compile 73,957, the solve 88,149, decode 76,529, document 60,295; the program tree is held from the compiler to the engine and +the engine's answer to the decoder). + +**Size the engine stack from the turn** (and from `resume` for a host that boots on a stored strand), not from setup: setup, one +event and `resume` now need about the same (12,112, 13,808 and 15,792 B on the proxy), so the largest of the three plus the +host's own frames is the thread's stack. + +### The turn's phases (`turn_trace`) + +With the feature `turn_trace` (off by default; without it the hooks are empty inline functions) `Live::event` records per +phase the microseconds spent and the largest value of a probe the host installs, so a firmware reads the same breakdown on +its own clock and counters: + +```rust +use probbit_persona::trace::{self, Phase}; +fn probe(_: Phase) -> u64 { /* e.g. the allocator's peak since the last call, reset; or the task's stack high-water */ 0 } +trace::set_probe(Some(probe)); // optional: called at the end of each phase's segment +let (stance, line) = live.event(&ev, &eng)?; +let t = trace::last(); // t.us[Phase::Solve as usize], t.bytes[..], t.total_us, t.solves +``` + +The phases: `input` (the event stamped), `expect` (the deviation and the trajectory: exact solves on the state before the +event), `prep` (provenance, payments, rest, senses / needs / drives / pace, credit, the surprise gate), `compile`, `solve` +(the engine's calls), `decode`, `document` (the stance's blocks, credit, glyph, digest), `line` (the strand line and a rest +line due before it) and `fold` (the expectation's memory). The parse of the event line and the store of the strand line are +the host's, before and after the call. `probbit live` built with `--features turn_trace` prints one `trace` line per event +on stderr when `PROBBIT_TURN_TRACE` is set (with the parse and the append it timed); stdout and the strand are unchanged. + +On the proxy (the body persona's last recorded event, x86_64, median of 21 runs) the turn takes 1,249 µs: expect 727, compile +130, solve 117, decode 102, document 97, prep 49, line 13, fold 4 (the host's parse of the event line and its append to a buffer +are about 1 µs each). The solve is 9.4% of it; the expectation's exact solves are 58%. Stack high-water and heap peak per phase +on the proxy: expect 15,520 / 66,879, prep 14,880 / 20,221, decode 12,464 / 76,529, solve 11,544 / 88,149, compile 8,816 / +73,957, document 6,192 / 60,295, fold 4,592 / 35,154, line 4,088 / 35,745, input 3,424 / 1,005 (with the feature compiled in, +which adds about 1.7 KB to the event's stack: 15,520 against 13,808 without it). + +### The expectation's cache + +`expect` (RFC 0009, a persona with an expectation block) solves, every event, the event's own program, the empty program and +one program per remembered combination (exact tier each). `Live` keeps the non-twin exact marginals of each program it solved +(`Live::expect_cache`, a `persona::ExpectCache`, on by default), keyed by the sha256 of the program's structure without its +`comment` (the only line that names the turn and the seed; each number by its bits, -0 as 0, as the program's text writes +it). The exact tier's marginals are a function of the program's model alone, so a row is the value the solve would return: +the stance, its `deviation`, every strand line and digest are the same with the cache on or off (`ExpectCache::new(false)`). +After each event only the rows that event used are kept: at most `max_combos + 1`. Decay never invalidates a row (marginals +do not read counts); a row stops being used when its program changes, which happens whenever the state the program is built +from moves (moods, needs, senses: every event of most lives; the state's digest itself carries the turn, so it is never a +key). The cache skips solves, not builds (the next section removes most of the build). A life whose programs move +every event gets no hit; after an event without one the cache is not consulted for the next 1, 2, 4, ... 32 events (a hit +resets the pause), so such a life keys two events in 34. `turn_trace` counts `expect_cached` and `expect_solved` per event. + +On the proxy (median of 3–7 runs, cache off → on, `expect`'s share of the turn): a life of one event repeated 500 times +(the expectation fixture persona, max_combos 16): 52.2% → 37.4%, 95.2% of programs from the cache; 500 events of many +combos: 80.9% → 81.2% (max_combos 16) and 93.6% → 93.7% (64), no hit; the body persona's recorded day (134 events, max_combos +6): 74.2% → 74.5%, 0.7% of programs from the cache. The rows' heap on the proxy (x86_64): 13,771 B at most for the body +persona (7 rows), 46,471 B (17 rows) and 178,327 B (65 rows) for the fixture persona at max_combos 16 and 64; a 32-bit target +holds about two thirds of that (its `String` and `Vec` headers are 12 bytes, not 24). `ExpectCache::bytes` reports it on the +target. A host short of heap at a large `max_combos` turns the cache off; the bytes do not move. With the cache the body +persona's turn on the proxy takes 13,840 B of stack (13,808 before) and its heap peak is 89,990 B (88,153 before: the rows +the individual holds between events). + +### The expectation's members: one program text per event + +Measured before (proxy, the body persona's recorded day, 134 events, max_combos 6; `turn_trace`'s `expect_build_us` and +`expect_solve_us`): 6.63 programs per event, each about 105 µs to build (prep 40, compile 61) and 110 µs to solve (`from_json` +47, the tier ladder with its answer document 36.5, the document's finite check 10, the cache key's hash over the text). The +exact solve itself is the small part; the program as text and back is about half of every member. + +A member's program is the event's own program A_t with another `h`: the persona's values, variables, pairs and steps are the +same, and so are the rules whenever the habits in force are A_t's. So, with the cache on (`Live`'s default): + +- A_t is built as before (its text and the compiler's meta) and read once into the model's parts before the compile pass + (`run::from_json_parts`), on the first program the cache does not answer. +- Every other member (the empty program, each remembered combination) runs prep, credit and the gate, then the compiler's + arithmetic without writing its text (the field per variable and the rules in force). With A_t's rules, its model is A_t's + parts with its own `h` (each level at the value index its name has in the text, zeros where the text omits a weight, the + twins copied, each weight checked as the text's is), then the compile pass and `Model::new` as `from_json` runs them + (`run::Parts::model_with`): the model the engine would build from its text. A_t's own field mapped this way is checked + against the `h` its text carries before any member uses the mapping. +- The model is solved by the exact tier `run` answers a `decide` with (`run::enumerated`): over `exact_limit` raw plans the + components tier, within it the whole-program enumeration, the same calls with the same arguments, the same verdicts, the + same finite check; the marginals are read as from the answer document. Another tier (one of those declining, or `sweeps: 0` + over the limit, where `run` stops on the wall clock), or other rules in force: the member's text is written from its meta + (no second build) and run by the engine, as before. +- The cache keys these programs by A_t's text without its comment and its vars' `h`, then the program's field: equal keys, + equal texts, as the cache's own key. + +No member's text, digest or marginals is in a stance or a strand line (only the deviation's numbers, from the marginals), and +the marginals are the same numbers: every stance, strand line and digest is the same as with the cache off, which is the +reference path (`ExpectCache::new(false)`, `persona::deviation`: every member's text built and run by the engine). The engine +argument is not called for these members; the deviation already reads members as the engine's exact tier (the cache returns +rows without calling it), and every engine in the repository is `persona::run_program`. + +On the proxy (median of 3–5 runs, before → after, `expect`'s share of the turn, cache on): the body persona's recorded day +74.6% → 67.7% (`expect` 1,467 → 1,047 µs per event; per member build 105 → 84 µs, solve 110 → 65 µs); its two recorded events +50.6% → 41.8%; the fixture persona, one event repeated 500 times, 37.1% → 35.2%; 500 events of many combinations 81.6% → +81.9% (most of its members have other habits in force than A_t: they keep the engine's path, without a second build). What is +left per member on the body is its prep (about 40 µs: the senses, needs and drives step on the member's own readings) and the +compiler's arithmetic, then the components tier; with M members per event this is the floor of any exact path that solves +every member. The body persona's turn on the proxy takes 13,824 B of stack (13,840 before) and its heap peak is unchanged +(89,990 B, at the turn's solve; inside `expect` the peak rises from 68,476 to 87,214 B: A_t's parts and one member's model). + +### The expectation's members: what the deviation reuses + +The event's own program A_t was compiled and solved twice per event: once by the turn, once as the deviation's own member. +Since the turn's first compile reads the same state and inputs as the deviation's A_t, `Live::event` now runs the turn first +and, with the cache on, the turn hands the deviation its first program's compile meta and that answer's exact marginals +(`persona::turn_at`, `persona::deviation_at`). The two programs are the same text when no rest line is due before the event, +`src` is taken out of the inputs alike (a `reward_from` without a senses, credit or needs block keeps it in the turn only) and +the persona's op is `decide`; the answer's marginals are the deviation's when its verdict is exact (an answer without an exact +verdict means no deviation, either way). Any other event: the deviation builds and solves A_t itself, as before. A_t's text is +written again from its meta (no prep, no compile) only to read the members' skeleton and key. `ExpectCache::with_reuse(false, +_)` keeps the old order (A_t built and solved by the deviation); `with_reuse(true, true)` builds and solves A_t a second time +as well and compares the two, text for text and bit for bit (a test's instrument). The phases' order in `turn_trace` changes: +the turn's phases come before `expect`. + +The member path's cost after that: prep copies only what it steps and reads (the senses and drive states it replaces are not +copied first; a member's copy leaves out the persona's name, the agenda, the resting stance, the learned-from counts and the +digest); the compiler's contributions carry no source name for a member (`Opts::meta_only`: the names exist for the turn's +`why`); and the members share one model (`run::Members`: A_t's parts moved into it, the compile pass's folds of separable +tables kept apart and replayed per member in the pass's order, checked bit for bit on A_t's own `h`), so no second copy of the +pairs is built per member; A_t's text is dropped once it is read. The trajectory runs before the turn, the deviation after +it. Every stance, strand line and digest is the same as before (and as the cache-off reference). + +On the proxy (median of 3–5 runs, cache on, `expect`'s share of the turn, before → after): the body persona's recorded day +67.7% → 63.9% (`expect` 1,042 → 894 µs per event, -14%; the rest of the turn about 500 µs; 5.63 member programs per event, +each about 79 µs to build and 69 µs to solve); its two recorded events 43.3% → 26.3%; the fixture persona, one event repeated, +35.8% → 32.9%; 500 events of many combinations 81.9% → 81.7%. Each step: A_t once 65.2%, the prep copy 64.7%, no source names +64.4%, one model 63.8%. Stack: the body persona's turn 13,584 B (13,824 before). Heap (the body persona's two recorded events, the +last one): the deviation now runs while the turn's stance document and next state are held (about 36.7 KB), so the event's +peak is 97,583 B, inside `expect` (89,990 before, at the turn's solve; inside `expect` 87,214); the deviation's own peak above +what it finds held fell from 86,209 to 60,857 B (one model instead of A_t's parts and a member's model; a member that keeps the +engine's path still builds its text and the engine's answer). A host short of heap keeps the old order with +`ExpectCache::with_reuse(false, false)`: the deviation before the turn, A_t built and solved by it, the same bytes; the body +persona's event peak is then 89,990 B, at the turn's solve (inside `expect` 69,332 B), at the cost of A_t's second build and solve. + +### The expectation's members: the order with the reuse, and what is held + +With the cache reusing the turn's program (the default), `Live::event` runs the deviation in two parts around the turn +(`persona::deviation_pre`, `persona::deviation_post`): + +1. **Before the turn** (nothing of the turn held): every member but A_t is solved from the pre-event state, E_t first (when E_t + has no exact answer there is no deviation and no other member is solved), then the remembered combos in the mixture's + order. Each member's exact marginals are kept as a row of f64 in the order of its layout (about 0.4 KB per member on the + body persona; the layout is kept only as its sha256: each variable's id and its levels' names in order). The cache keeps + only this event's rows through the turn. +2. **The turn**, which hands A_t's compile meta and exact marginals (T3-b). +3. **After the turn**: A_t's layout must hash to the held one; the rows are named again from it (the same names, values and + order), A_t's row goes into the cache, and the mixture runs as before. A layout that differs solves the members again; an + event the turn does not hand A_t for (a rest due, another op) builds and solves A_t after the turn. + +The members share one model, read without any program text: the skeleton comes from E_t's text without its pairs (a few KB), +and the pairs are the persona's couplings read as `run::from_json` reads the text's (r6 of each table entry at its values' +indices, each weight checked as the text's). A member whose rules in force differ from E_t's (other habits, rules on inputs) +takes the same model with its own caps: its rules lowered as `from_json` lowers a text's (from its text without the pairs), +the compile pass's never-binding caps dropped, `Model::new` on the same parts (`run::Members::set_caps`, no copy of the pairs); +its cache key is the skeleton's, its rules' and its field's. The components tier's split of that model (the components, their +kinds, the sub-models of the components with caps or cycles) does not read `h`, so it is computed once per caps and reused by +every member (`probbit_ir::exact_components_split`, each sub-model's `h` copied in; the same tiers on the same numbers). + +Every stance, strand line and digest is the same as with the old order and as with the cache off +(`the_members_before_the_turn_move_no_byte`; the IR test `one_split_over_many_h_is_bit_identical` for the split). + +On the proxy (cache on; the body persona: its recorded day for time, its last recorded event for heap; one node session): + +| | T3-b head | members before the turn | every member a model | the split once | +|---|---:|---:|---:|---:| +| `expect` share of the turn, recorded day | 64.1% | 63.2% | 57.9% | 57.5% | +| per member: build / solve (µs) | 80.7 / 71.0 | 71.7 / 65.7 | 66.9 / 41.6 | 67.5 / 40.5 | +| members on the engine's path, recorded day | 253 | 159 | 0 | 0 | +| event heap peak (B) | 97,583 | 88,844 | 88,844 | 88,844 | +| heap inside `expect` (B) | 97,583 | 72,568 | 55,920 | 55,920 | +| turn stack (B) | 13,584 | 13,504 | 13,552 | 13,552 | + +The event's heap peak is now the turn's solve (its own allocations as before, plus about 1.2 KB of held rows); inside `expect` +the peak is the members' (no program text, no answer document). `ExpectCache::with_reuse(false, false)` keeps the old order +(the whole deviation before the turn, A_t built and solved by it): event peak 89,990 B, inside `expect` 72,003 B, same bytes; +the reuse is the better default now on both. On the fixture persona's 500 varied events the members are solved before the +turn knows whether A_t has an exact answer, so events without one solve their members for nothing (about 3.6 programs per +event); the model path still brings that life's share from 81.7% to 74.8% (its steady life 33.2% → 19.4%). + +Silicon numbers (the same calls on the target's engine thread) are measured by the firmware and will follow; the x86_64 proxy +and an Xtensa build differ in frame layout, so read the proxy as the shape and the ratio, not the target's bytes. + +## The strand writer, and how the host checks it + +`probbit_persona::strand` has no file I/O: each call returns the line, and the caller stores it (the CLI appends to a file under +a writer lock, `probbit-cli/src/live.rs`; a firmware writes flash). + +```rust +use probbit_persona::{json, persona, strand::{self, Clock, Live}}; + +// how a turn runs its program: in order on this thread (as the CLI's tests and probbit-wasm run it) +let engine = |prog: &json::Json, f: &persona::Flags| persona::run_program(prog, f, 1, 100, 0); +let doc = persona::parse_doc(PERSONA_YAML, false)?; // Err(String): the reader's message +let p = persona::build(&doc)?; // Err(json::InErr): path, code, message +let st = persona::init(&p, Some(seed), true, &engine); +let (mut live, header) = Live::start(p, &doc, st, Clock::Fixed, &strand::engine()); +store(&header); +let (stance, line) = live.event(&json::parse(r#"{"praise": true, "elapsed_hours": 0.5}"#)?, &engine)?; +store(&line); +if live.n % strand::CHECKPOINT_EVERY == 0 { store(&live.checkpoint(&stance)); } +``` + +- `Clock::Fixed`: each event carries its own `elapsed_hours`, so the strand is a pure function of its events. `Clock::Real` + holds an `rt::Instant` (the installed clock on `nostd`). +- `live.control(what, by, reason, at)` writes pause, resume and retire lines; on `nostd` the caller passes `at` (there is no + `strand::utc_now`). +- Every line carries `prev`, the sha256 of the line before, so the strand is one chain from its header. + +The host reads the strand back (over serial, from flash) and checks it with the same code: `probbit live verify STRAND`, or +`strand::verify(&text, &engine)` in a program. Verify replays every event from the header's persona and initial state and +names the earliest line that differs (a changed, removed or reordered line, or a stance or state digest that the replay does +not reproduce), or the earliest field this engine does not know (`unsupported`: not verified, and not a divergence). +`strand::Replay` steps a strand line by line, and `strand::resume(text, &p, st, clock)` continues one from its last state. + +A `nostd` build takes its transcendentals (`ln`, `exp`) from the `libm` crate, which can differ from the host's libm in the +last bits (2 ULP measured on one deviation). Every number a strand line carries is rounded (6 decimals, or 5 significant +digits for the deviation receipt), so the device's strand and the host's replay have the same bytes; from engine 0.14.0 this +includes the stance's `deviation` on checkpoint lines (`strand::DEV_R6_SINCE`). A strand whose header names an earlier engine +keeps its full-precision deviation, and verify accepts it when it equals the replay at 6 decimals. + +**The engine string.** A strand's header names the engine that wrote it, and the rules a replay applies follow that name, not +the replaying engine: from `probbit 0.14.0` on (a `-dev` or other suffix ignored: `probbit 0.14.0-dev` is 0.14.0) a strand gets +the rounded deviation, the canonical fold of the expectation's memory and `expect` on checkpoints; an earlier name keeps the +earlier rules, byte for byte. A firmware passes its engine string to `Live::start` (the header's `engine`) and should pass the +version of the crate it builds with (`strand::engine()` gives `probbit ` + that version; a development build says `probbit 0.14.0-dev`), +so a strand it writes replays on the host under the same rules. A strand that keeps naming an older engine keeps the older +fold, and its boot folds the expectation over every line (no checkpoint carries the memory). + +An engine older than the strand verifies what it can read: it does not compare engine names. At the first line with a field it +does not know it stops with `unsupported by this engine: FIELD` (exit 5: a newer engine's strand, not verified, and not a +divergence), so 0.13.0 answers exit 5 at the first checkpoint that carries `expect`. A 0.14.0 strand without such a line (no +expectation block, or no checkpoint yet) has nothing 0.13.0 does not know, and 0.13.0 replays it under its own rules (ok while +those give the same bytes). A segment file after the first (`NAME.0001.strand`, ...) is not readable before 0.14.0: its +second line chains into the previous file, and 0.13.0 reports a divergence at line 2 (exit 1) before it reaches the header's +`segment` key. Verify strands of 0.14.0 with 0.14.0 or later. + +### Sizing a strand on flash: checkpoints and segments + +A strand grows by one event line per event and one checkpoint line every K events (`live.checkpoint`), and it never shrinks. +With E bytes per event line, C per checkpoint line and H for the header, n events take about H + n x (E + C / K) bytes. The +expectation fixture persona of the CLI's tests (a 6,312-byte header, a 10-input tutor with an `expectation` block, 500 events +over many input combos) measures E <= 498 B and, from engine 0.14.0 (the checkpoint carries the expectation's memory), +C = 6,369 B with `max_combos` 16 and C = 14,121 B with `max_combos` 64, the memory's largest window (63 named combos); before +0.14.0 the same checkpoint was 3,888 B and the memory it did not carry had grown to 387 combos. At K = 50 and one event a +minute that is about 37 KB an hour: 1 MB lasts about 27 hours. A larger persona scales both: one measured at about 0.85 KB per +event and 8.2 KB per checkpoint fills 1 MB in 3.5 to 4 hours at K = 50 and its event rate. + +Segments bound the file a life is stored in: `--segment-every S` (or a host's own seal: a checkpoint line, then +`strand::segment_header(&live, &doc, engine, k, previous_name, &sha256_of_that_line)` as the first line of a new file and the +checkpoint line as its second) costs one header and one repeated checkpoint per segment, so a segment file is about +H + C + S x (E + C / K) bytes (for the fixture at S = 500, K = 50: about 325 KB). A host that keeps the newest two segments and +deletes the older ones holds a life of any length in about twice that; `verify --chain` then reports the first missing file +(exit 6) instead of a divergence, and each kept segment still verifies on its link. Resume works on the newest segment alone: +`strand::resume` reads its header, its sealed line (the state and, from 0.14.0, the expectation's memory) and the lines after +it: a boot parses and hashes the lines of that one segment, folds the expectation's memory over the lines after its last +checkpoint only, and never solves a stored event again. + +### Resume on a body: the call and the three recoveries + +A body keeps three things between boots: the persona document (it rides in the strand's header line, which also names the +engine and the initial state), **the byte offset of the last checkpoint line** it wrote, and **its last state record** (the +state after the event it last answered, in its own small log). On boot it reads the header line, the checkpoint line at the +stored offset and then the lines after it one at a time, and calls + +```rust +// trailing: an iterator over the lines after the checkpoint (each without its line ending), read from flash as it goes +let live = strand::resume_boxed_from(&header, Some(&checkpoint), checkpoint_line_number, trailing, &p, record, clock, &eng)?; +// before the first checkpoint: resume_boxed_from(&header, None, 1, lines_after_the_header, ...) +``` + +The header is checked as `verify` checks it (the persona built once on the heap, the header written again in pieces against +the stored line); the checkpoint line by its own fields (canonical JSON, its state and the expectation's memory read, an event +count), its link to the line before taken on the host's offset, as a segment's sealed line is taken on its link; every line +after it is chain-checked as `resume` checks it. The whole strand is never loaded, and nothing before the checkpoint is read +again. The three recoveries: + +1. **The record is the strand's last state** (the normal boot): the walk reads the trailing lines, finds the record at the last + event line and returns the individual; no turn runs. +2. **The record is k lines behind** (a reboot between a line and its record): the record is found at the line that logs it and + the k event lines after it are applied (each its turn's main solve, not the expectation's member solves; docs/persona.md + §5.7 "Resume from a record"). The result is the individual the unbroken life has, byte for byte; nothing needs the host to + resync. A record older than the checkpoint is refused (`state`): resume from the whole strand with + `strand::resume_from_record(text, ...)`, which starts at the last checkpoint before the record. +3. **No record** (a first boot on a copied strand, a lost state log): replay from the header with `strand::Replay::open_boxed` + (every event solved again, the cost of `verify`), or start at the checkpoint's state: a checkpoint line carries the whole + state, so a record taken from it (`State::read` of its `state`) resumes at once and applies the lines after it. + +The cost on the host proxy (probbit-persona `nostd` on x86_64, the 12 KB body persona; the strand: 50 events, one checkpoint, +then T event lines; record last; median of 7): `resume_boxed_from` = **1.46 ms fixed + 13.4 µs per trailing line** (T = 0, 58, +200, 449: 1,459, 2,328, 4,284, 7,456 µs), against the whole-text `resume_boxed` at 2.3 ms + about 15 µs for every line of the +strand. The fixed part is the header's check and the persona's build plus the checkpoint's state and memory read. An applied +line (recovery 2) costs about 612 µs where the turn costs 2,580 µs on that life (0.24 of a turn: the expectation's member +solves, most of this persona's turn, are not run). A body measured the whole-text resume at about 488 ms fixed + 8 ms per +line, so its boot cost now follows the lines since its last checkpoint, not the length of its life (silicon numbers for the +new call are the body's to measure, with `turn_trace`: `trace::last_resume()` gives the lines walked, the lines applied and +their µs). + +Stack on the proxy (the same harness as the table above, 20 KB thread run too): `Replay::open_boxed` and its steps over a +500-event strand **16,192 B** (the by-value `Replay::open` was 24,752 B: it built the persona and the `Live` by value and held +the `Live` in the replay; `open` is now `open_boxed` moved out of its box, 16,736 B with the replay in the caller's frame); +`resume_from_record` with a record 1, 5 or 49 lines behind **16,080 B** (13,648 with the record last); `resume_boxed_from` a +checkpoint + 58 lines **12,656 B** (the whole-text `resume_boxed` of the same strand 15,072 B). All three ran on a 20 KB thread. + +### Self-report in standby (RFC 0013) + +A body that idles can give an account of its recent events without running a turn: `reflect::reflect_window(&lines, w)` reads +the strand's lines and returns the canonical `reflection` document (docs/rfcs/0013-self-report.md); `reflect::line(prev, &r)` +is the line to append (`{"prev", "reflection"}`), which `live verify` recomputes. No solve, no randomness, no model: one pass +over the lines from the expectation fold's start (the last checkpoint before the window carrying the expectation's memory), +each event line parsed once and each checkpoint line in the window parsed once, so the cost is O(lines read), dominated by +the checkpoint lines' size. Measured on the x86_64 host (release, LTO; median of 21 calls), with the stack high-water and the +heap peak of one call on the `nostd` build (the stack painted below the caller and scanned after the call, as for the setup +path above): + +| Strand | Window | Lines | Host µs (std) | Host µs (`nostd`) | Stack high-water | Heap peak | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| a recorded day, 1,160 events, 23 checkpoints (0.99 MB) | last 50 | 1,184 | 664 | 601 | 5,552 B | 217,674 B | +| the same | last 200 | 1,184 | 1,299 | 1,248 | 5,552 B | 223,787 B | +| the same | all | 1,184 | 5,303 | 5,283 | 5,552 B | 251,144 B | +| the tutor with an expectation block, 500 events, checkpoint every 50 | last 50 | 511 | 335 | 344 | 6,568 B | 128,292 B | +| the same | all | 511 | 1,221 | 1,320 | 6,568 B | 140,547 B | + +The heap peak is the parsed checkpoint lines (a checkpoint line of the recorded day is about 20 KB of JSON, held parsed while +it is read); a body short of heap reflects a window with fewer checkpoint lines in it. The document is byte for byte the same +from the std and `nostd` builds and the CLI (6/6 windows over the two strands). Not measured on silicon. + +### Integrity on a body (RFC 0014) + +With the identity self keys (docs/persona.md §2.19), `Live::event` computes `integrity` after the turn +(`probbit_persona::integrity::compute_cached`, std and `nostd`). Once per seed it runs the fresh individual's rest probe and +computes the allowances (RFC 0004's vertex search at that seed). Then, per event, it handles the rest probe on the rested +state. The probe's turn steps run, its first program is compiled and keyed by structure without its `comment`, and an equal key +answers from the row the last exact probe left. Otherwise the turn finishes on that program. A row is kept only for an exact +answer with no yield, hold or conflict, so every number is the uncached computation's: with the self keys declared, the strands +of the lives below are byte for byte those of the build before the program key. A hit costs one program build; a miss costs one +turn. The probe's path is boxed (the rested state, the fresh individual, the cache step and the document each in a frame of +their own). Measured on the x86_64 host (release, best of 3 lives), and the stack high-water and heap peak of one call on the +`nostd` build (painted below the caller, as above): + +| Life | Events | Host ms with / without | Share of the turn | Answered from the cache | Stack (miss / hit) | Heap peak (miss / hit) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| a recorded day of a body (moods, senses, drives, needs, a learning block; no feedback that day) | 1,160 | 1,849.9 / 1,593.5 (std), 1,889.5 / 1,627.9 (`nostd`) | +16.1 % / +16.1 % | 1,156 of 1,160 (the resting program moved on 4 events) | 13,656 B / 13,656 B | 93,190 B / 68,748 B | +| the tutor with learning, feedback on most events | 200 | 195.9 / 97.0 (std) | +102.0 % | 88 of 200 (a learning step moves the program) | 12,784 B / 11,008 B | 119,933 B / 75,476 B | + +The bare rest-probe turn (`turn_boxed` on the rested state, no cache) reaches 12,520 B on the same build, so a miss is a turn's +stack plus 1,136 B of the probe's own frames. A hit by program runs the turn's steps before the solve, the deepest part of +a turn, so it reaches the same 13,656 B; a hit by state (a life where nothing that persists moved) 5,104 B. Before the program key the day cost +40.1 % / +36.8 % with 0 hits, +20,952 B per miss. This is still over RFC 0014's budget (2 % of a turn, 8 KB per miss): a hit still builds the probe's +program, and a miss is a turn. Integrity is off by default and costs nothing without the self keys. Not measured on silicon. + +### Curiosity and the stir on a body (RFC 0016) + +With a `curiosity` block (docs/persona.md §2.20), `Live::event` reads the drive's level from the expectation's memory before +the turn and folds the event into it after the turn: an average, one short miss list per remembered context (at most +`max_combos`), the hours since the last stir. A body's idle loop polls `Live::stir_due(idle_hours)` (the hours since its last +event, from its own clock): the call reads that fold and returns `None` or a `Stir` (class, speaks, asleep, level, minutes +since the last stir). It solves nothing and writes nothing (its one allocation is the class name it returns), so a body may poll it every +minute; it logs nothing either. When the body acts on a stir it feeds an ordinary event with `"stir": ""` and its current readings, so +the stir is one strand line like any event, and a habit in force on those readings (a night flag) binds it. +Measured on the x86_64 host and on the `nostd` build (the chip's proxy, painted as above), on a body's recorded day (134 +events) with a curiosity block added, against the same individual with the drive cleared (best of 3): + +| What | std | `nostd` | +| --- | ---: | ---: | +| the day, drive on / off | 174.023 / 174.688 ms | 178.970 / 178.274 ms | +| the drive's share of a turn (fold + level + receipt) | -0.38 % (noise) | +0.39 % | +| `stir_due`, one call | 0.000060 ms | 0.000060 ms | +| `stir_due`'s share, one poll per event | 0.0046 % | 0.0045 % | +| `stir_due` stack high-water / heap peak | | 2,768 B / 8 B | +| `expect_step` with the drive, stack / heap | | 3,040 B / 2,184 B | + +The drive's cost is within the run-to-run noise of a turn; polling once a minute through a night adds about 60 calls an hour. +Not measured on silicon. + +### The rest pass on a body (RFC 0015) + +With an `expectation.rest` key (docs/persona.md §2.17), `Live::event` folds each event into the pass's window (one row of four +numbers per remembered context, at most `max_combos`), and a body in standby calls `Live::consolidate_boxed(&mut live, at)` +with its own time stamp: the pass works on the boxed individual in place (no `State` or `Live` by value on a frame), solves +nothing, and returns the one `consolidation` line the body appends to its strand (the line logs the memory's digests before and +after, the contexts merged, the weights moved, the decay and the state digest after). The expectation's cache is emptied (its +rows are rebuilt by the next events). Measured on the x86_64 host and on the `nostd` build (painted as above), on a body's +recorded day (134 events, one pass at its end over a window of 134 events; `half_life` 64, `bucket` 2, `surprise_gain` 1, +`decay` 0.5), against the same persona without the key (best of 3): + +| What | std | `nostd` | +| --- | ---: | ---: | +| the day, with / without the key | 218.383 / 224.205 ms | 193.365 / 190.726 ms | +| the window fold, per event | within noise (-43.449 µs) | +19.694 µs (+1.4 %) | +| one pass over the day's window | 121.492 µs | 120.649 µs (0.900 µs per event of the window) | +| `consolidate_boxed` stack high-water / heap peak | | 3,760 B / 17,345 B | + +The pass is O(contexts) per event of the window plus O(contexts²) for the merge (at most 64 contexts); its heap peak is the +memory written twice for its digests. Not measured on silicon. + +### Measured on one board (ESP32-S3) + +First numbers from one ESP32-S3 at 240 MHz, the engine on one core shared with a 16 fps renderer, with the boxed setup path +above: the engine task's stack high-water for the setup plus one to three turns was 17,612 to 18,096 bytes (28,304 bytes for +the setup alone before the boxed constructors); the strand header was 11,985 bytes, byte for byte the host's; one stance turn +took 0.60 to 0.93 s with 17 to 23 KB of transient heap per event; the engine's PSRAM use was 50 to 112 KB during setup and run; +the internal heap's floor was 30,216 bytes through the header write. These are one board's measurements, not bounds. + +## The host hooks + +`probbit_persona::hooks` lets a host see the decision pipeline: `set_monitor(tier, gate_seen)` (the CLI's `--top` view) and +`set_config_echo(f)` (the telemetry fields that name the configuration in effect). Unset, the monitor calls do nothing and +the echo is empty. diff --git a/docs/monitor-app.md b/docs/monitor-app.md index 178503d..95362f4 100644 --- a/docs/monitor-app.md +++ b/docs/monitor-app.md @@ -45,6 +45,9 @@ either kind opens like any app. A copy you build yourself (below) has no quarant labels ([persona.md §5.8](persona.md#58-monitor-watch-an-individuals-inner-state)). A probbit that watches one strand at a time (0.12.1 and earlier, if you build the app around one) refuses that; the app then shows the first strand you chose (for a folder, its most recently changed `*.strand` file) and says so under the window title. +- **Integrity** (engine 0.14.0+, a persona with the identity self keys, [persona.md §5.9](persona.md#59-integrity-the-drift-from-the-declared-self-rfc-0014)): + the instrument panel shows the integrity dial (the drift from the declared self as a share of the certificate's allowance) + where "room left in the bound" stood, and the Mirror tab adds the per-trait drift table of the event on view. - The window title is the page's title. When the page's title starts with a glyph, the Dock icon shows that glyph as its badge. - File → Open Recent lists what you opened; the app reopens the last selection when it starts with nothing else to open. - View → Reload (⌘R) reloads the page; **Follow Latest** (⌘L) loads the page's root again, the latest event with no event diff --git a/docs/persona.md b/docs/persona.md index 2d8e2f6..4040554 100644 --- a/docs/persona.md +++ b/docs/persona.md @@ -16,7 +16,7 @@ reads it as its instructions for how to write; an agent can read it to know its none, verify first, ask first). The model still writes every word. This document is normative for the file format, the compilation, the stance document and the state. It is implemented by -`probbit persona` (probbit-cli/src/persona.rs and yaml.rs, no dependency) and served on every surface with the same documents: the +`probbit persona` (probbit-persona/src/persona.rs and yaml.rs, no dependency) and served on every surface with the same documents: the CLI, `probbit mcp` (tools `probbit_persona_init`, `probbit_persona_turn`), the Python wrapper (`probbit.persona_init`, `persona_turn`, `persona_replay`) and the browser module (probbit-wasm ops 4 and 5). `probbit live` (section 5.7, live.rs) keeps an individual running on a clock, with learning (section 2.8) and a replayable log; `probbit monitor` (section 5.8, monitor.rs) @@ -230,12 +230,40 @@ engine: twin: true # also solve the habit-free twin in the same program, to report which habits changed the stance values: positional # positional (level i -> l: small pair tables, fast) | semantic (level names: readable programs) on_conflict: fallback # fallback | yield: when habits contradict, the lower-priority one yields (section 5.5) + # tier: amplitude # research (RFC 0011, draft): an exact simulated qubit register on a CPU; absent = the default tiers + # q_max: 12 # the most encoded qubits (ceil(log2 d) per variable) in one component the amplitude tier takes (1-16) + # amplify: false # research (RFC 0011, round 4): with the tier, exact amplitude amplification in place of the projection + # schedule: tf-linear-16x4 # research (RFC 0011, round 5): the register's evolution: tf-linear-16x4 | tf-linear-64x16 | qaoa-p2 line: max_tokens: 40 # estimated tokens (max of chars / 4 and 4/3 per word) order: [verbosity, caution, directness, warmth, humour, initiative, formality, curiosity] prefix: "Stance: " ``` +**`engine.tier: amplitude` (research, off by default; docs/rfc/0011-amplitude-register.md).** The turn's program is solved by +an exact simulated qubit register on a CPU: a variable with d allowed values takes ⌈log2 d⌉ qubits (a three-level trait two, +its fourth code unused and kept at amplitude 0), one register per connected component of the program (2^q complex +amplitudes each), a fixed transverse-field sweep with a general diagonal phase layer, the habits as a projection onto the +plans that obey them, when the largest component has at most `q_max` encoded qubits; otherwise the default tiers answer and +`engine.amplitude` says why (`"declined: ..."`). With the tier, the stance document's `engine` carries `tier: "amplitude"`, +`qubits`, `components`, `encoding`, `mixer`, `circuit`, `schedule`, `schedule_name`, `state` (sha256 digests of the encoding, +the layer list, the schedule and the final amplitudes at 6 decimals) and +`feasible_mass`; `live verify` recomputes them from the strand. Without the key nothing changes, byte for byte. An engine +without the tier reports a strand that uses it as `unsupported by this engine: engine.tier` (exit 5). `engine.amplify: true` +(research, round 4; needs the tier) replaces the projection by exact amplitude amplification, simulated on a CPU: the +rules are the oracle, the register is rotated onto the plans that obey them, the stance is the projection's (the same plan and odds, measured equal to +1.1e-12) and `engine.amplify` logs the iterations, the final phases and the mass before and after (`feasible_mass` 1). `engine.schedule` +(research, round 5; needs the tier; absent = `tf-linear-16x4`) names the evolution of the register, simulated exactly on a CPU: `tf-linear-64x16` (four times +the layers) separates the top two levels where the thermal tier is a near tie (median gap 0.152 vs 0.049 on the recorded corpus) and +agrees more elsewhere; `qaoa-p2` is two fixed QAOA layers (RFC 0011, "Schedules and the indifferent set"). Since round 7 the +register computes its math functions with one library (`libm`) on the std and the no_std build: 100 fresh lives per schedule +recorded on the std build re-verify on the no_std build 100 of 100 and re-write byte for byte, on each of the three schedules +(x86_64; arm64 not measured). The math change is in the last bits: the tutor's and the demo's default-schedule strands are +byte-identical to before, a larger persona's can move. The register is a +simulation: no quantum hardware, no speedup claim; its stances agree with the default tier in 96 % of trait decisions on the +recorded corpus (RFC 0011, round 3). With the key, `persona bound` probes through the tier: compute the certificate again with +it, since one computed without the key does not cover the tier. + ### 2.8 Learning (optional) ```yaml @@ -951,10 +979,39 @@ half-life; read after the cold start, from the logged value); `registers` = z >= `"deviation": {"input", "pred", "self"}` at 5 significant digits (with `"cold": true` while cold); `live verify` recomputes it ("the deviation differs" on an edit), `resume` folds the same memory from the logged lines, and the stance digest the line logs excludes `deviation`. Bound (by construction): per variable, `pred_i <= input` for a member combo and -`pred_i <= self_i + input` otherwise. `persona turn` and `persona replay` carry no memory and add nothing. A strand with -the block verifies from its header (`--from-checkpoint` refuses: the memory reads every event). The deviation is absent +`pred_i <= self_i + input` otherwise. `persona turn` and `persona replay` carry no memory and add nothing. Before engine 0.14.0 a +strand with the block verifies from its header (`--from-checkpoint` refuses: the memory reads every event); from 0.14.0 the +memory is the canonical fold and every checkpoint line carries it (section 5.7, "Checkpoints"), so a replay and a resume start +at the last checkpoint. The deviation is absent (`null`) when a program is not solved by the exact tier. +Every number of the stance's `deviation` is rounded to 6 decimals (the rule every other number of a stance and a state +follows), so a checkpoint line, which carries the stance, has the same bytes on every platform: `self`, `pred` and `input` come +from `ln`, whose last bits differ between libm implementations (macOS, glibc, the `libm` crate of a `nostd` build differ by a +few ULP). Engines before 0.14.0 wrote `self`, `pred` and `input` at full precision; a strand whose header names such an engine +keeps that formatting when it is replayed or continued, and `live verify` accepts its checkpoint line when it equals the +replay's at 6 decimals (the chain goes on from the logged bytes), so a strand written on one platform verifies on another. +The event line's receipt (5 significant digits) is checked exactly, as before. + +**Rest pass (RFC 0015).** An optional `rest` key of the block declares the rest pass (memory consolidation) a host may run at +rest (standby, night, idle) with `Live::consolidate(at)` or `probbit live control STRAND rest [--at TIME]`: + +```yaml +expectation: + rest: # optional; every key optional; {} = the pass with nothing moving + half_life: 64 # the window since the last pass re-folded with this half-life in events (default half_life_events) + bucket: 2 # bins of a declared number input in the merge key, 1-3 (default 3 = the thirds: nothing merges) + surprise_gain: 1 # weight gain of a context's mean z+ in the window, 0-4 (default 0) + decay: 0.5 # factor on the learned deltas, 0-1 (default 1) +``` + +With the key, the canonical memory (0.14.0 on) carries a `window` (per remembered context [a, b, k, zs] in key order and +`since`; checkpoint lines carry it). The pass re-folds each count as count − a + b, weighs counts by 1 + g · min(zs/k, 1.5)/1.5 +(total kept), merges contexts with the same key under B bins, multiplies the learned deltas by `decay`, and appends one +`consolidation` line: `{at, consolidation {after, before, decay, events, merged {count, keys}, moved}, n, prev, state}`. +`live verify` re-runs the pass and names the first field of a forged line; an engine before 0.14.0 refuses the line. Without +the key the pass is refused and every document is byte for byte as before. Full definition: docs/rfcs/0015-consolidation.md. + **Trajectory.** p_beta, proportional to p^beta, of the event's program at the k betas, each solved by the exact tier on `Model::scaled(beta)`: no sampler, a pure function of the program. It is derived and never logged: `probbit monitor` shows a `TRAJECTORY` row (mean marginal entropy per step) and its JSON frame a `trajectory` key; `probbit persona @@ -1021,6 +1078,63 @@ point (and its selector). What it is: a lookup from the stance's mood levels, active habits and deviation to a symbol, so a person reading the chat or the monitor sees the engine's state at a glance. What it is not: a measurement of anything beyond those fields. +### 2.19 Identity: the self keys (optional; RFC 0014) + +Engine 0.14.0 and later; an engine before it refuses a persona that declares them (`identity.keep: unknown field`; `live +verify` of such a strand exits 5, `unsupported`). Four optional keys in the `identity` block declare which part of the +individual is "the self" and turn on integrity (section 5.9). They move nothing (effects {}) and stay out of the persona +digest, as `bound` does: declaring them on a persona with a recorded life leaves every state as it was. + +```yaml +identity: + name: Pip + version: 1.1.0 + seed: 1 + keep: [no_jokes_on_loss, after_error_check] # identity-bearing habits: reported held or not fired per event + warn: 0.5 # a share of the allowance in (0, 1], default 0.8 + cue: hold the line # the persona's own words, handed to the host past warn (1-120 characters) + traits: [humour, verbosity] # the traits measured; default learning.traits, else every trait +``` + +Refusals: a `keep` entry that is not a habit id of the persona, a `traits` entry that is not a trait (a mood is refused), an +entry listed twice, `warn` outside (0, 1], an empty or multi-line `cue`. The allowance per trait comes from the persona's +`bound` block (section 2.16); without one every allowance is null and `bound_used` reads 0. + +### 2.20 Curiosity and the stir (optional; RFC 0016) + +Engine 0.14.0 and later (an engine before it refuses the block, `curiosity: unknown field`; `live verify` of such a strand +exits 5, `unsupported`). A drive named `curiosity`, fed by the surprise the expectation block (section 2.17) measures, and +an optional self-started idle event, the **stir**, that it gates. It needs an expectation block and is in the persona digest +(its level moves the stance). Full definition, refusals and bars: docs/rfcs/0016-curiosity.md. + +```yaml +curiosity: + half_life_events: 8 # surprise_recent: the average of z+ = max(z, 0), its half-life in events (starts at quiet_z) + quiet_z: 0.5 # below it nothing is new + miss_z: 1.5 # a context's visit misses when z+ > miss_z + last_k: 3 # failed_contexts: the contexts whose last K visits all missed + weights: {quiet: 1.0, failed: 0.5} + cap: 2 + effects: {initiative: 0.6} # per unit of the level, on traits or moods (never pursue) + stir: {threshold: 1.0, every_minutes: 30, class: check, when_asleep: {flag: night, every_minutes: 120, class: glance}} +``` + +**The level** = clip(quiet x max(0, 1 - surprise_recent / quiet_z) + failed x failed_contexts, 0, cap), 6 decimals. Event t's +turn reads the level after event t-1 as the engine-owned number input `cur_level` (box [0, cap], carrying `effects`; a host +that sends it is refused at `inputs.cur_level`; the stance's `inputs` leave it out). The stance gains +`drives.curiosity: {failed_contexts, level, surprise_recent}` and the event line the same object under `curiosity` +(`live verify` recomputes it). The drive's state is folded from the logged lines inside the expectation's memory, so a +checkpoint carries it (`expect.curiosity`) and resume and replay from a checkpoint read it back. `persona turn` and +`persona replay` carry no memory: the level is 0 there. + +**The stir.** `Live::stir_due(idle_hours)` (and `probbit live control STRAND stir? [--idle-hours H]`) answers whether a +self-started event is due: the level at least `threshold`, at most one per `every_minutes` since the last stir; while the +last event had the `when_asleep.flag` on, only with `when_asleep`, at its slower rate and with its class, which never +speaks. A host that acts on it feeds an ordinary event with `"stir": ""` and its own current readings; a habit in +force on those readings binds the stir's stance as any other (a persona that sleeps on a flag names that flag in +`when_asleep`). A stir the host ignores leaves no line. Without the block nothing changes: no `cur_level`, no `drives` +key, no line field, every document as before. + ## 3. Compilation: persona + state + inputs → one probbit IR program For turn t of an individual (seed s), with inputs x_t: @@ -1535,7 +1649,92 @@ event line: `{"checkpoint":n,"prev":"sha256:...","stance":{...},"state":{...}}`, and the whole state after it, chained like the others. A full `verify` replays from the header and checks every checkpoint against the replay (its `"checkpoints": N` is in the summary); `verify --from-checkpoint` reads the header, checks the last checkpoint against the event line before it (prev, the stance digest, the state digest, the state read as any state is) and -replays only what follows (`"from_checkpoint": n`); the lines before it are trusted, so run a full `verify` to check them. +replays only what follows (`"from_checkpoint": n`); the lines before it are trusted, so run a full `verify` to check them. Every +number a checkpoint line carries is rounded (6 decimals; genes 4), so the line is the same on every platform's libm; for the +stance's `deviation` this holds from engine 0.14.0 on (section 2.17). + +**Checkpoints with the expectation's memory (engine 0.14.0 on).** A persona with an `expectation` block (RFC 0009) folds every +event into a memory: per input combo its decayed count and the inputs of its latest event, whether a non-time combo was seen, +and the learned normal of the deviation. From 0.14.0 the fold is canonical: after every event each count, the mean and the +variance are at 6 decimals (a count at 0 is dropped), and at most `max_combos - 1` named combos are kept: when one more enters, +the last in the mixture's order (count descending, then key) moves its count to `rest`, which decays like a count and is added to +the empty member's weight, where the mixture already put every combo past the kept ones (the one change: a combo that comes +back after it was moved out starts at count 1). The checkpoint line then carries the memory as one more key: +`"expect": {"combos": [[count, inputs], ...], "mean", "n", "rest", "var", "warm"}` (combos in key order; the key is computed from +the inputs). A full `verify` checks it byte for byte against the replay's; `verify --from-checkpoint`, `resume` and `monitor` +start at the last checkpoint and fold only the events after it. The line's size is O(max_combos): the `expect` object is at +most about 64 + (max_combos - 1) x (I + 24) bytes for events whose inputs are at most I bytes. A strand whose header names an +engine before 0.14.0 keeps the four-key checkpoint and the fold before it, byte for byte; such a strand with the block still +replays from its header. An engine before 0.14.0 reads `expect` as `unsupported by this engine: expect` (exit 5). + +**Resume from a record.** A host that keeps the individual's state in its own record (a state log beside the strand) can +reboot between a strand line and its record, so the record is one or more lines behind the strand. `strand::resume` refuses a +state that is not the strand's last; `strand::resume_from_record(text, &p, state, clock, &engine)` takes any state the strand +logs (an event line's `state`, a checkpoint's, the header's) and brings it to the strand's last line. What it promises: the +individual it returns is the one a header replay reaches at the last line (state, chain head, the expectation's memory, the +status after control lines), byte for byte and digest for digest, and the next event writes the line the unbroken life writes. +How: the whole strand is chain-checked as by `resume`; the walk starts at the last checkpoint at or before the record's line +(the header without one, and for a persona with an expectation block before 0.14.0, whose checkpoints do not carry its +memory); every event line after the record's line is applied: its turn run on the logged inputs (stamped as the live writer +stamped them), its stance and state digests compared, the line written again from the turn's values and compared byte for +byte. An event line logs its stance's digest, not the stance, and the next state is decoded from the solve (the levels and +odds the moods' inertia and the credit read), so an applied line costs its turn's main solve; it does not run the +expectation's member solves: the line's logged `deviation` (and the `glyph` that reads it) is taken as written and folded into +the memory as `resume` folds it. Rest lines are recomputed against the logged ones, control lines applied as the live writer +applies them (the credit zeroed, `erase` returning the learned weights to zero). Refused, with the line: a record that is no +state the strand logs (`state`), a line whose application differs (the stance, the state, the rest line or the line's bytes: +a forged line, re-chained or not), a chain that does not hold, another persona. What it does not check: the logged deviation +and glyph of the applied lines (only `verify` recomputes the expectation); the lines before the walk's start beyond the chain +check `resume` makes. + +`strand::resume_boxed_from(header, Some(checkpoint), line, trailing, &p, state, clock, &engine)` is the same walk without the +whole text (docs/embedding.md "Resume on a body"): the header line, the last checkpoint line (taken on the link the host kept, +like a segment's sealed line) and the lines after it one at a time. The record may be any state logged from that checkpoint on; +an older one is refused (`state`: resume from the whole strand). + +**Reflection lines (engine 0.14.0 on, RFC 0013).** `probbit live control STRAND reflect [--last N | --since-checkpoint]` +appends the strand's self-report (section 5.8, "Self-report") as one line after its last line, under the writer lock: +`{"prev": ..., "reflection": }`. A host decides when; a body calls `reflect::reflect_window` and `reflect::line` +and appends the line itself. `verify` recomputes the document from the lines before it over the window it names and refuses one +that differs (exit 1, the line named: "the reflection differs from the lines before it"); replay, the chain check and resume +pass over the line with no state change, and `live` continues a strand that ends on one (a checkpoint or a seal still follows +an event line). The writer refuses a strand of an engine before 0.14.0; an engine before 0.14.0 reads the line as +`unsupported by this engine: reflection` (exit 5). No line is written unless asked. + +**Segments.** `probbit live ... --strand NAME.strand --segment-every N` seals the strand after every N-th event and goes on in +a new file: the sealing line is a checkpoint line (the last line of the segment); the next file, `NAME.0001.strand`, then +`NAME.0002.strand` and so on (four digits at least), starts with a header that has one more key at its end, +`"segment": {"index": k, "previous": "", "sealed": "sha256:"}`, its state the +sealed state, and its second line is the sealing line itself, byte for byte. The chain goes on across the files unchanged: the +lines of all segments, each segment's header and repeated sealing line set aside, are the strand the same life writes to one +file with a checkpoint at each seal. The labels sidecar is the first file's (`NAME.labels.json`); a run on `--strand +NAME.strand` continues the newest segment that exists (a sealed file is never appended to). Before 0.14.0 a persona with an +expectation block is not segmented (its checkpoint does not carry the memory). An engine before 0.14.0 cannot read a segment +file after the first: its second line chains into the previous file, so such an engine reports a divergence at line 2 (exit +1) instead of the unknown `segment` key. + +A host that decides when to seal runs `probbit live control NAME.strand seal` (under the writer lock, like any control): it +replays the newest segment (from its last checkpoint), refuses unless that file's last line is an event line (a seal follows +an event: a seal right after a seal or a checkpoint exits 2 and writes nothing), appends the sealing checkpoint, writes the +next segment file (its header, then the sealing line) and prints `{"ok": true, "control": "seal", "segment": k, "file": +"", "line", "head"}`. It writes no control line, so it takes no `--by` or `--reason`. + +`probbit monitor --dir DIR --serve` lists a life's files (`NAME.strand`, then `NAME.0001.strand`, ... without a gap) as one +strand, id `NAME`, with the life's event count, `"segment": k` (the newest file's index) and the base name's sidecar; its page +follows the newest file and goes on in the next one within the 1 s scan after a seal (the new file opens at its sealing line, +the event already shown). A segment file given on its own still opens at its sealing line. + +- `live verify SEGMENT` checks one segment on its link: the header is as written, the second line's sha256 is the header's + `sealed`, that line is a valid checkpoint (its state is the header's, its memory reads), and every line after it replays. + Exit 0 says: this file continues, faithfully, whatever line has that sha256. The summary adds `"segment": k` and + `"trusted_link": "sha256:..."`. +- `live verify SEGMENT --chain` also walks back through `previous` to the first file: each one verifies, its last line's + sha256 is the `sealed` the next one links and its index is one less. Exit 1 at the first break, with `"segment_file"` + naming the file; exit 6 when a previous file does not exist (`{"ok": false, "missing": "", "segment_file": ""}`): a segment rotated away is a known state, not a divergence. The summary adds `"segments": N`. + +Compaction (replacing old segments by one synthetic start) is not part of the format: it would drop the history a verify +checks, and the decision to trust a dropped history belongs to the host. **Senses in live** (section 2.11). An event carries `senses` as `persona turn` does; the strand logs each reading as used, with its source, and `verify` recomputes the normals from the logged readings: its summary adds `"normals": {"": {"mean", "n", @@ -1573,7 +1772,11 @@ check that inference. Hashes alone do not authenticate a writer or detect a whol anchor. Historical measurement before this hardening, on an Apple M4: a 10,000-event strand of a drives persona (4 goals, a floor, learning; 3,126,444 bytes, 10 checkpoint lines) opens in `monitor --once` from its last checkpoint in 0.005 s (load 3.3), where a full `verify` takes 192 s. A replay from a checkpoint costs what the events after it cost: 500 events past the last one took 26 s to draw -(load 6-13, about 50 ms an event for this persona), so K bounds the wait; a smaller K costs a few KB per checkpoint line. A strand without checkpoint or control lines (shorter than K events, or `--checkpoint-every 0`) is the +(load 6-13, about 50 ms an event for this persona), so K bounds the wait; a smaller K costs a few KB per checkpoint line. Before engine 0.14.0 a persona with an `expectation` block (RFC 0009) +is the exception: its expectation memory is a fold of every event line, and no checkpoint line carries it, so `monitor` +(`--once`, `--follow`, `--serve`, each strand of `--dir` by its own persona) replays such a strand from its header, every +line, and `verify --from-checkpoint` still refuses it. The cost is a full replay when the strand is opened: on a 16-vCPU Linux VM, a 5,000-event life of the tutor with the block (10 checkpoint lines) opens in `monitor --once` in 7.4 to 7.8 s, about 650 events a second, where the same life without the block opens from its last checkpoint in under 0.05 s; after +that, appended lines replay one by one from the state in memory, as for any strand. A strand without checkpoint or control lines (shorter than K events, or `--checkpoint-every 0`) is the same format as in 0.8.0. New strand headers record the current engine version, so their chain hashes differ; existing 0.8.0 strands remain replayable. Binaries before 0.8.0 do not read checkpoint or control lines. @@ -1649,8 +1852,8 @@ cannot read the documents: |---|---| | `/` | the page: one file embedded in the binary (under 72 KB, styles and script inline; it fetches no fonts, styles or scripts), with a Content-Security-Policy that lets it connect to its own server and nowhere else | | `/events` | server-sent events: `meta` (the layout: the strand, the engine, the persona, its traits, moods, inputs and history features in the persona's order, and `face`, the two moods the face's meters show), the latest `frame`, then a frame per event: `{n, doc, given, learned, spark, diverges, note}` (with `engine.trajectory` also `trajectory` and `settle`, below), that is the stance document, the inputs as the strand logs them, the learned deltas, the moods' recent positions, the line that differs and a warning; a heartbeat comment after 15 quiet seconds; events that land within one poll (100 ms), or while a page falls behind, come as the latest frame | -| `/doc/N` | event N's stance document, canonical JSON and a newline, as `probbit live` printed it (its sha256 is the `stance` digest of the strand's line N + 1); the latest 10,000 are kept | -| `/strands` | the strands served, canonical JSON: `{demo, dir, strands: [{id, strand, labels, persona, n, glyph, age_s, labels_error?}]}` (the file name, never its directory; `age_s`: seconds since the strand file was last written); the demo lists none | +| `/doc/N` | event N's stance document, canonical JSON and a newline, as `probbit live` printed it (the sha256 of the JSON without that newline is the `stance` digest of the strand's line N + 1); the latest 10,000 are kept | +| `/strands` | the strands served, canonical JSON: `{demo, dir, strands: [{id, strand, labels, persona, n, glyph, age_s, labels_error?, segment?}]}` (the file name, never its directory; `age_s`: seconds since the strand file was last written, a life in segments: its newest file; `segment`: that file's index, when it is not the first); the demo lists none | | `/s/ID/events`, `/s/ID/doc/N` | as `/events` and `/doc/N`, for strand ID; `/events` and `/doc/N` are the first strand's, so a single-strand page is as before | **Who and where: labels, many strands.** A strand says which persona and engine; it does not say which agent, on which @@ -1754,6 +1957,55 @@ canonical replay documents are unchanged. Demo bars are scripted engine state, n Exit codes: 0 when every line replays, 1 at a line that differs (the frame names it and shows the event before it), 2 for a bad flag, a port that cannot be had, or a file that cannot be read or is not a strand. +#### Self-report (RFC 0013) + +`probbit reflect STRAND [--last N | --since-checkpoint] [--json | --text]` prints a canonical account of a window of the +strand's events, computed from its lines alone: no turn is run, nothing is solved, nothing is random. Per event line it reads +what the line logs (the inputs, the expectation's receipt, the glyph): the inputs set and how often, and with an expectation +block each receipt's z, recomputed from the logged receipts (the expectation's mean and variance folded as the engine folds +them), the events that registered and the K = 5 largest |z| with what the expectation predicted (`pred`) and what the event +measured (`self`). Per checkpoint line in the window it reads the stance and state the line carries: per trait the levels taken +and the odds at the first and last sample, the moods' path (each mood's position along its levels, the monitor's sparkline +value) with its turning points, the drives' peaks, the habits that bound, the learned weights' net change. Feedback that settled +the credit, control, rest and reflection lines are counted. A window without a checkpoint line reports no levels, moods, +drives or habits: those sections are `null` (an event line logs digests, not the stance), never invented. Every number is at +6 decimals, `digest` is the sha256 of the canonical text without it, and `summary` holds 6 to 12 sentences (13 at most with a declared stir, RFC 0016) from fixed +templates, part of the document. `--text` prints the sentences only. The library call is `probbit_persona::reflect::{reflect, +reflect_since_checkpoint, reflect_window}` (std and `nostd`). The format is docs/rfcs/0013-self-report.md; section 5.7, +"Reflection lines", appends it to the strand. + +**Mirror.** The served page has a Mirror tab: the reflection of the last 50 events computed by the server from the strand file +at each request (`/reflection?last=N`, `/s/ID/reflection` per strand), and the latest reflection line the strand carries with +whether it recomputes equal; the tab shows the sentences, the top surprises with predicted vs measured, the habits that bound +and the moods at the checkpoint samples, refreshed with the page's scan of the strand list. + +### 5.9 Integrity: the drift from the declared self (RFC 0014) + +With the self keys (section 2.19), every live stance document and every event line carries `integrity`, computed after the +turn from the state it produced (docs/rfcs/0014-integrity.md): + +```json +{"allowance":0.902968,"crossed":[],"cue":null,"drift":0.214,"habits_held":{"held":[],"not_fired":["after_error_check","no_jokes_on_loss"]}, + "traits":{"humour":{"allowance":0.462081,"bound_used":0.21,"drift":0.097},"verbosity":{"allowance":0.440887,"bound_used":0.27,"drift":0.117}}, + "used":0.237,"warn":0.5} +``` + +- `traits..drift`: the total variation between the odds at the rest probe (section 2.16) of the individual rested (the + channels of the hour, such as moods with a half-life, the credit awaiting feedback, the last stance and the agenda's step, at the fresh individual's + values; everything that persists as it is) and of the fresh individual of the same seed. `drift` is the sum over the traits. +- `allowance`: epsilon at the individual's seed (null without a `bound` block or for an unlearned trait); `bound_used` = + drift / allowance; `used` = the summed drift over the traits with an allowance over the summed allowance: the monitor's + integrity dial. By RFC 0004's theorem `bound_used` stays at or below 1 (up to the 1e-6 rounding of amendment A7). +- `crossed`: the traits at or past `warn`; `cue`: `identity.cue` while `crossed` is not empty, else null. The host appends the + cue to the line it hands the model; the engine does not edit `line`, which the state's remembered stance digest covers. +- `habits_held`: of the `keep` habits, those that bound at this event and those that did not fire. + +Every number is at 6 decimals. Both stance digests leave `integrity` out; `live verify` recomputes it on every event line and +refuses an edited value naming the line ("the integrity differs"); `resume` takes it as written. The library call is +`probbit_persona::integrity::{compute, compute_cached, Integrity}` (std and `nostd`). In the monitor the integrity dial takes +the place of "room left in the bound" on the instrument panel, and the Mirror tab adds the per-trait drift table of the event +on view. + ## 6. What a persona can NOT do - It does not write, read or check text. Text-level rules (banned phrases, exact formats, facts) stay in the prompt or in @@ -1816,7 +2068,7 @@ previous event, the restart included). `probbit_live_event` (MCP) and Python's ` | `probbit persona compile PERSONA --state STATE [--inputs ...]` | the turn's probbit-ir program; its sha256 is the stance's `engine.program` | | `probbit live PERSONA [--seed N \| --state FILE] [--strand FILE] [--events FILE [--watch]] [--clock real\|fixed]` | a resident individual: JSONL events in, one stance per event out, `elapsed_hours` stamped by the clock, the life logged to a strand (section 5.7); `--state` is rewritten after every event | | `probbit live PERSONA --demo week [--seed N] [--strand FILE] [--plain]` | one individual's scripted week on the fixed clock: learning to its cap, a quiet night, habits that hold (section 5.7) | -| `probbit live verify STRAND [--from-checkpoint] [--actions FILE]` | replay a strand from its header alone: ok, the earliest line that differs (exit 1), or the earliest line with a field or line kind this engine does not know (`unsupported`, exit 5: a newer engine's strand, not a divergence); with `--actions`, also the earliest turn whose host action differs from the level its stance released (section 5.7) | +| `probbit live verify STRAND [--from-checkpoint] [--chain] [--actions FILE]` | replay a strand from its header alone: ok, the earliest line that differs (exit 1), or the earliest line with a field or line kind this engine does not know (`unsupported`, exit 5: a newer engine's strand, not a divergence); with `--actions`, also the earliest turn whose host action differs from the level its stance released; with `--chain`, a segment and every previous segment file (exit 6: one is missing) (section 5.7) | | `probbit monitor STRAND [--follow] [--once] [--plain] [--fps N] [--serve] [--open] [--port N]`, `probbit monitor --demo [...]` | watch an individual's inner state: the strand replayed as live bars in the terminal, or as a page on 127.0.0.1 (section 5.8) | A script is a JSON list of per-turn input objects (or `{"turns": [...]}`; a turn may be `{"inputs": {...}, ...}`). Exit codes diff --git a/docs/probbit-ir-json.md b/docs/probbit-ir-json.md index 5c7246c..c12a81b 100644 --- a/docs/probbit-ir-json.md +++ b/docs/probbit-ir-json.md @@ -123,7 +123,7 @@ size limits in "Limits in v1", stdin above `--max-input-mb`). Unreadable stdin a the offending field (`""` = the whole document, as for JSON syntax errors and non-finite numbers, whose message gives the byte offset). The object goes to stdout when stdout is not a terminal or with `--json` (0.8.2; at a terminal the stderr line alone). Flag errors keep their convention: exit 2, a message on stderr, nothing on stdout (with `--json`: one object, code `flag`). The same strict reader parses -both front-ends (`probbit-cli/src/json.rs`; CLI test `input_contract_rejects_malformed_documents`, which includes the 2026-09-30 external +both front-ends (`probbit-persona/src/json.rs`; CLI test `input_contract_rejects_malformed_documents`, which includes the 2026-09-30 external review's cases: `"allowed": "A"` used to enable every scored worker and answer `exact`; a cap object where the caps array belongs was dropped; duplicate value names emitted duplicate keys). @@ -134,7 +134,7 @@ returns `Err(String)` for structural errors (lengths, out-of-range indices, `i = ## Numeric contract - **Inputs.** Every score / weight (`h`, `potts`, `table`, router `scores` and `affinity`) is a finite number with - |x| <= 1e9 (`limit` error otherwise; `probbit-cli/src/json.rs` `MAX_WEIGHT`). Why 1e9: weights are natural-log odds, and a + |x| <= 1e9 (`limit` error otherwise; `probbit-persona/src/json.rs` `MAX_WEIGHT`). Why 1e9: weights are natural-log odds, and a weight difference beyond ~745 already decides (e^745 is past the range of double-precision probability ratios), so the limit removes no distribution you can express; it keeps every log-space quantity (log w sums over variables, pairs and same-group pairs; log Z) below ~1e9 x (input size), far inside the double range (1.8e308). Caps and limits are integers diff --git a/docs/probbit.1 b/docs/probbit.1 index 308f4de..35d255c 100644 --- a/docs/probbit.1 +++ b/docs/probbit.1 @@ -1,4 +1,4 @@ -.TH PROBBIT 1 "" "probbit 0.13.0" "probbit manual" +.TH PROBBIT 1 "" "probbit 0.14.0-dev" "probbit manual" .SH NAME probbit \- a virtual probabilistic (p\-bit) processor: joint decisions under hard rules, with gated odds .SH SYNOPSIS @@ -7,13 +7,16 @@ probbit \- a virtual probabilistic (p\-bit) processor: joint decisions under har (SUB: init turn replay explain diff lint fuzz prove check compile describe) \fBprobbit\fR live PERSONA [\-\-seed N | \-\-state FILE] [\-\-strand FILE [\-\-label KEY=VALUE]...] [\-\-events FILE [\-\-watch]] - [\-\-clock real|fixed] [\-\-checkpoint\-every K] + [\-\-clock real|fixed] [\-\-checkpoint\-every K] [\-\-segment\-every N] \fBprobbit\fR live PERSONA \-\-demo week [\-\-seed N] [\-\-strand FILE [\-\-label KEY=VALUE]...] [\-\-plain] -\fBprobbit\fR live verify STRAND [\-\-from\-checkpoint] [\-\-actions FILE] - [\-\-blocks FILE] +\fBprobbit\fR live verify STRAND [\-\-from\-checkpoint] [\-\-chain] + [\-\-actions FILE] [\-\-blocks FILE] \fBprobbit\fR live control STRAND pause|resume|retire|erase \-\-by WHO \-\-reason TEXT [\-\-at TIME] +\fBprobbit\fR live control STRAND reflect [\-\-last N | \-\-since\-checkpoint] +\fBprobbit\fR live control STRAND stir? [\-\-idle\-hours H] +\fBprobbit\fR live control STRAND rest [\-\-at TIME] [\-\-state FILE] \fBprobbit\fR check PERSONA [\-\-props FILE] [\-\-strand FILE]... [\-\-seeds LIST] [\-\-depth N] [\-\-hours LIST] [\-\-json] \fBprobbit\fR monitor STRAND [\-\-follow] [\-\-once] [\-\-plain] [\-\-fps N] [\-\-serve] @@ -23,6 +26,7 @@ probbit \- a virtual probabilistic (p\-bit) processor: joint decisions under har [\-\-open] [\-\-port N] \fBprobbit\fR monitor \-\-demo [week|drives] [\-\-once] [\-\-plain] [\-\-serve] [\-\-open] [\-\-port N] +\fBprobbit\fR reflect STRAND [\-\-last N | \-\-since\-checkpoint] [\-\-json | \-\-text] \fBprobbit\fR decide [flags] < router.json \fBprobbit\fR run [flags] < program.json \fBprobbit\fR evaluate [flags] < request.json @@ -294,13 +298,16 @@ a resident individual: JSONL events in, one stance per event out .nf \fBprobbit\fR live PERSONA [\-\-seed N | \-\-state FILE] [\-\-strand FILE [\-\-label KEY=VALUE]...] [\-\-events FILE [\-\-watch]] - [\-\-clock real|fixed] [\-\-checkpoint\-every K] + [\-\-clock real|fixed] [\-\-checkpoint\-every K] [\-\-segment\-every N] \fBprobbit\fR live PERSONA \-\-demo week [\-\-seed N] [\-\-strand FILE [\-\-label KEY=VALUE]...] [\-\-plain] -\fBprobbit\fR live verify STRAND [\-\-from\-checkpoint] [\-\-actions FILE] - [\-\-blocks FILE] +\fBprobbit\fR live verify STRAND [\-\-from\-checkpoint] [\-\-chain] + [\-\-actions FILE] [\-\-blocks FILE] \fBprobbit\fR live control STRAND pause|resume|retire|erase \-\-by WHO \-\-reason TEXT [\-\-at TIME] +\fBprobbit\fR live control STRAND reflect [\-\-last N | \-\-since\-checkpoint] +\fBprobbit\fR live control STRAND stir? [\-\-idle\-hours H] +\fBprobbit\fR live control STRAND rest [\-\-at TIME] [\-\-state FILE] .fi .PP A resident individual (docs/persona.md \[u00A7]5.7): JSONL events (one object of inputs per line) from \-\-events FILE or stdin, one stance per event on stdout (canonical JSON). The clock stamps each event's elapsed_hours (quantised to 1e\-6 h), so moods decay by their half\-lives between events; feedback moves the learned deltas of a persona with a learning block (\[u00A7]2.8). @@ -337,6 +344,9 @@ no colour, no animation (as NO_COLOR=1 or PROBBIT_THEME=plain); output otherwise \fB\-\-checkpoint\-every\fR \fIK\fR with \-\-strand: a checkpoint line (the whole state) after every K\-th event (default 1000; 0: none); `verify \-\-from\-checkpoint` and `monitor` start at the last one .TP +\fB\-\-segment\-every\fR \fIN\fR +with \-\-strand NAME.strand: seal the strand after every N\-th event (a sealing checkpoint line) and go on in the next segment file NAME.0001.strand, NAME.0002.strand, ...: its header links the previous file (its name, the sealed line's sha256, the index) and its second line is the sealed line, so the chain continues across files; a run continues the newest segment (default 0: one file) +.TP \fB\-\-label\fR \fIKEY=VALUE\fR with \-\-strand, repeatable: a presentation label for the monitor (agent, host, lane, session, or a free key; at most 8 keys, values of 1 to 64 printable characters) merged into the strand's sidecar FILE.labels.json (FILE without its extension); the strand itself, its chain and its digests are not touched .RE @@ -344,13 +354,16 @@ with \-\-strand, repeatable: a presentation label for the monitor (agent, host, Subcommands: .RS 4 .TP -\fBverify\fR STRAND [\-\-from\-checkpoint] [\-\-actions FILE] [\-\-blocks FILE] +\fBverify\fR STRAND [\-\-from\-checkpoint] [\-\-chain] [\-\-actions FILE] [\-\-blocks FILE] replay a strand: the earliest line that differs (exit 1), or the earliest line with a field or line kind this engine does not know (`unsupported by this engine: FIELD`, exit 5: a newer engine's strand, not a divergence); from the header (every checkpoint checked), or from the last checkpoint; with \-\-actions, also the earliest turn whose host action differs from the level its stance released .RS 4 .TP \fB\-\-from\-checkpoint\fR start at the last checkpoint instead of the header .TP +\fB\-\-chain\fR +a segment (section 5.7): verify it and walk back through the previous segment files to the first, each verified and its last line the sealed line the next one links; exit 1 at the first break (the segment file named), 6 when a previous segment file is missing +.TP \fB\-\-actions\fR \fIFILE\fR the host's action log, one {"turn": T, "action": {trait: level}} per line: compared with each turn's released levels (summary `actions`: checked, differs); exit 1 when one differs .TP @@ -358,24 +371,36 @@ the host's action log, one {"turn": T, "action": {trait: level}} per line: compa write each event's block (the text a host injects, RFC 0008 E: every trait and mood as `name level`, then the viability line) to FILE, one {"block", "n"} per line; replays from the header .RE .TP -\fBcontrol\fR STRAND pause|resume|retire|erase \-\-by WHO \-\-reason TEXT [\-\-at TIME] -pause, resume or retire an individual, or erase its learned weights (RFC 0004: every learned weight back to 0, the status unchanged); appends a control line under the lock: while paused or retired every event is refused (exit 4); no credit crosses a control line; retire is final; `verify` replays them and reports the status +\fBcontrol\fR STRAND pause|resume|retire|erase \-\-by WHO \-\-reason TEXT [\-\-at TIME] | STRAND seal | STRAND reflect [\-\-last N | \-\-since\-checkpoint] | STRAND stir? [\-\-idle\-hours H] | STRAND rest [\-\-at TIME] [\-\-state FILE] +pause, resume or retire an individual, or erase its learned weights (RFC 0004: every learned weight back to 0, the status unchanged); appends a control line under the lock: while paused or retired every event is refused (exit 4); no credit crosses a control line; retire is final; `verify` replays them and reports the status. `seal` (section 5.7) seals the newest segment of STRAND: refused unless its last line is an event line, it appends the sealing checkpoint and writes the next segment file (its header links the sealed line, then the sealed line itself), and prints that file in `file` .RS 4 .TP \fB\-\-by\fR \fIWHO\fR -who decided it: human:ID or env:ID (required) +who decided it: human:ID or env:ID (required, except for seal) .TP \fB\-\-reason\fR \fITEXT\fR -why (required) +why (required, except for seal) .TP \fB\-\-at\fR \fITIME\fR when (default: now, UTC) +.TP +\fB\-\-last\fR \fIN\fR +reflect: the last N events (default: every event) +.TP +\fB\-\-since\-checkpoint\fR +reflect: the events after the last checkpoint line +.TP +\fB\-\-idle\-hours\fR \fIH\fR +stir? (RFC 0016): the hours since the strand's last event (default 0); the answer is {due, stir}, nothing is written +.TP +\fB\-\-state\fR \fIFILE\fR +rest (RFC 0015): the stored individual (the state after the strand's last line), rewritten after the pass .RE .RE .PP \fIthe safety kit\fR: (docs/persona.md \[u00A7]5.7) one writer per strand: `\-\-strand` takes STRAND.lock (a second writer exits 4 and changes nothing; a lock whose process is gone is taken over). With `reward_from` in the persona (\[u00A7]2.10), a reward from `src: self`, without a src or from an undeclared one is refused like a bad event. .PP -\fIexit\fR: 0 done (live: every event answered; verify: every line replays; control: the line appended), 1 verify: a line differs (its number and what) or, with \-\-actions, a host action differs from its turn's released level, 2 a bad persona, state, event (one {"error"} object on stdout per bad event when stdout is not a terminal or with \-\-json; the run goes on), control line or flag, 4 refused: the strand's writer lock is held by another writer, or the individual is paused or retired (one {"error"} object per refused event, as for 2; nothing is written), 5 verify: unsupported, a line has a field or line kind this engine does not know (a newer engine's strand: {"ok": false, "line", "unsupported": the field, "engine", "strand_engine"}; not verified, and not a divergence). +\fIexit\fR: 0 done (live: every event answered; verify: every line replays; control: the line appended), 1 verify: a line differs (its number and what) or, with \-\-actions, a host action differs from its turn's released level, 2 a bad persona, state, event (one {"error"} object on stdout per bad event when stdout is not a terminal or with \-\-json; the run goes on), control line or flag, 4 refused: the strand's writer lock is held by another writer, or the individual is paused or retired (one {"error"} object per refused event, as for 2; nothing is written), 5 verify: unsupported, a line has a field or line kind this engine does not know (a newer engine's strand: {"ok": false, "line", "unsupported": the field, "engine", "strand_engine"}; not verified, and not a divergence), 6 verify \-\-chain: a previous segment file is missing ({"ok": false, "missing": its file name, "segment_file": the segment that links it}). .PP Examples: .RS 4 @@ -493,6 +518,40 @@ probbit monitor pip.strand \-\-follow probbit monitor pip.strand \-\-serve \-\-port 8080 .fi .RE +.SS reflect +RFC 0013 self\-report: a canonical account of a strand's last N events, from its lines alone +.PP +.nf +\fBprobbit\fR reflect STRAND [\-\-last N | \-\-since\-checkpoint] [\-\-json | \-\-text] +.fi +.PP +The self\-report (docs/persona.md \[u00A7]5.8): a canonical document computed from the strand's lines alone (no turn is run, nothing is solved, nothing is random): over the window, the inputs set, the expectation's surprises (each receipt's z recomputed from the logged receipts; the K = 5 largest |z| with the predicted and measured values), per trait the levels taken and the odds' drift, the moods' path with its turning points, the drives' peaks and the habits that bound (read from the checkpoint lines in the window: a window without one reports them as null), the credit settled, the learned weights' net movement and the control lines; every number at 6 decimals, `digest` = the sha256 of the canonical text, and `summary`: 6 to 12 sentences from fixed templates. `probbit live control STRAND reflect` appends it as a line (0.14.0 strands), which `live verify` recomputes. +.PP +Flags: +.RS 4 +.TP +\fB\-\-last\fR \fIN\fR +the last N events (default: every event) +.TP +\fB\-\-since\-checkpoint\fR +the events after the last checkpoint line (that checkpoint is the first sample) +.TP +\fB\-\-json\fR +the canonical document (the default) +.TP +\fB\-\-text\fR +the summary's sentences only, one per line +.RE +.PP +\fIexit\fR: 0 the reflection printed, 2 a bad flag, a file that cannot be read, or a strand with no event line in the window (one {"error"} object on stdout). +.PP +Examples: +.RS 4 +.nf +probbit reflect pip.strand \-\-last 50 \-\-text +probbit reflect pip.strand \-\-since\-checkpoint +.fi +.RE .SS decide route tasks to workers: a router document in, a JSON decision out .PP diff --git a/docs/rfc/0011-amplitude-register.md b/docs/rfc/0011-amplitude-register.md new file mode 100644 index 0000000..5d26c24 --- /dev/null +++ b/docs/rfc/0011-amplitude-register.md @@ -0,0 +1,291 @@ +# RFC 0011: an amplitude register, an exact simulated qubit register as an optional solver tier for small stance programs + +Status: **accepted as optional tier (research; off by default)**, round 3. Verdict by the registered rule (B4 ≥ 90 %): 96.29 % +of 2,176,371 trait decisions on the recorded corpus agree with the exact default tier; B5 passes (three designed programs differ +from the thermal odds by 0.25-0.27 at the node; 0 of 4,100 bound cases exceed a certificate computed with the tier). Nothing in +this RFC changes a release or `main`; the tier stays off unless a persona asks for it. Turning it on changes the bound +certificate: compute it again with the tier (section 5). Round 4 adds `engine.amplify` (off by default): exact amplitude +amplification in place of the projection, which lands on the projection's numbers with feasible mass 1 (section 5, +"Amplification"). Round 5 adds `engine.schedule` (absent = the default sweep): a longer sweep separates the top two levels +where the thermal tier is indifferent and agrees more elsewhere (section 5, "Schedules and the indifferent set"). Round 6 shows +one such decision end to end, with a demo persona, its two recorded strands and the monitor's two faces (section 5, "The demo"). + +## 1. What it is, in one paragraph + +A register of q qubits (q ≤ 16 on the host, q ≤ 12 on a small microcontroller) simulated **exactly** on a CPU as a state +vector of 2^q complex amplitudes: unitary gates with phase, interference between basis states, measurement by the Born rule, +every step logged so the strand replays byte for byte. It is a simulation of a tiny quantum system and nothing else: no +quantum hardware, no speedup claim, no scaling past what 2^q amplitudes allow (10 qubits 16 KB, 16 qubits 1 MB, 20 qubits +16 MB; a microcontroller with 8 MB of PSRAM reaches about 16; a laptop about 25-28). Sampling a thermal distribution cannot +produce this behaviour (amplitudes carry signs and phases that probabilities do not); the register gets it by paying 2^q +memory and time, which is fine for the programs probbit actually solves (tens of two-valued variables, most of them not +coupled all at once). + +## 2. Why bother (the question the measurement answers) + +probbit's stance is one exact program over coupled p-bits per turn: biases and couplings → exact marginals → a level per +trait with its odds → a stance line, habits as hard rules on top. The p-bit tier gives a thermal (Boltzmann) distribution. +An amplitude register fed the same program as a Hamiltonian gives a **coherent** distribution: options can cancel (destructive +interference) as well as add, and a short unitary evolution (a transverse-field sweep or a few QAOA-style layers) reaches its +distribution by dynamics, not by equilibrium. Two concrete things that might matter to an agent: + +1. **Conflict handling.** When two drives push a trait opposite ways, the thermal answer is a flattened mixture; a coherent + answer can be a sharpened choice or a cancellation. Whether that is better for a stance is exactly what has to be measured. +2. **Reflex arcs.** A sub-turn surge needs a cheap, deterministic, bounded state shift. A q-qubit register evolved by a fixed + small circuit is O(2^q · gates) with no sampling noise and no convergence question, which is a candidate mechanism for the + arc, to be compared with a rule-based register. + +It might also matter to nobody: the register may reproduce the p-bit marginals within noise, in which case it is a toy and the +RFC is closed with that number. Both outcomes are publishable with receipts. + +## 3. Design (minimal) + +- Module `probbit-decide::amp` (std, and no_std with `alloc`; the state vector on the heap): `Register::new(q)`, gates + `H, X, Z, RX(θ), RZ(θ), CNOT, CZ, RZZ(θ)` (enough for Ising-type programs and QAOA layers), `evolve_ising(program, + schedule)` (a Trotterised transverse-field sweep, fixed step count), `qaoa(program, p, params)`, `marginals()` (exact, no + sampling), `measure(seed)` (the Born rule with the engine's deterministic RNG), `digest()` (sha256 of the canonical state at + 6 decimals, like every other emitted float). Amplitudes are two f64: 16 · 2^q bytes exactly. +- A solver tier `amplitude` beside `components` / `knapsack` / `exact`: selected by the persona flag `engine.tier: amplitude` + (off when absent) and only when every variable has an allowed value and the largest connected component of the program + has at most `engine.q_max` encoded qubits (default 12, 1-16); otherwise the existing tiers run and the stance document says + why (`engine.amplitude: "declined: ..."`). +- How a program becomes registers (round 2). **Encoding:** a variable with d candidate values (allowed, and the clamp) + takes ⌈log2 d⌉ qubits; code c < d is its c-th value, the codes ≥ d are unused (d = 3: codes 00, 01, 10; 11 unused). The start + is the uniform superposition over the valid codes (for two-valued variables, H on every qubit). **Mixer** (`complete-graph`, + fixed): per variable exp(i t A_d) with A_d = J − I, the adjacency of the complete graph on its d codes, and the identity on + the unused codes; closed form e^{−it}(I + (e^{itd} − 1)/d · J), O(d) per block; for d = 2 it is exp(i t X), round 1's mixer + exactly. **Phase layer:** one general diagonal layer, a_x ← e^{−iγE(x)} a_x with E = −log w of the plan x encodes, on every + valid basis state (O(2^q) per layer). Pair tables between multi-level variables give 3- and 4-body terms in any qubit + encoding; the diagonal layer takes them as they are, with no Walsh transform, no residual and no decline. Round 1's RZ/RZZ + decomposition of the cost is gone: on a 16-variable two-valued chain one general phase layer took 333 µs against 1,938 µs + for the RZ/RZZ layer, and the whole sweep 48.4 ms against 51.9 ms (x86_64, median of 20); the gates stay in the module. The unused codes keep amplitude exactly 0: the + engine checks their mass (≤ 1e-15) after every layer and never renormalises. **Components:** the program is split into its + connected components (two variables are linked by a pair or by sharing a capacity rule, as the `components` tier splits + it) and each component runs its own register. This is exact: E is a sum over components and the mixer acts per variable, + so the joint state is the product of the components' states (the repository's example personas need 52-70 encoded qubits + as one register, 10-15 in their largest component). The sweep is the fixed default `tf-linear-16x4` (16 Trotter steps, + total time 4, `H(s) = −(1−s) Σ A_i + s E`). The rules (habits, caps, clamps) are a projection per component: the Born + probabilities of the plans that break one are set to 0 and the rest renormalised, so a stance obeys every habit in force + exactly as on the other tiers. Plan = per component its most probable feasible plan. +- Receipt fields in the stance document: `engine.tier = "amplitude"`, `engine.qubits` (encoded qubits, summed over the + components), `engine.components` (each component's qubits), `engine.encoding` (digest of the variable → qubits map with each + variable's number of valid codes), `engine.mixer` (`complete-graph`), `engine.circuit` (digest of the layer list), + `engine.schedule` (digest) and `engine.schedule_name`, `engine.state` (digest of the final amplitudes at 6 decimals), + `engine.feasible_mass` (the Born mass on plans that obey the rules), and the marginals at 6 decimals (the stance odds). + `live verify` recomputes all of it from the strand alone (the same rule as every other tier). An engine without the tier + reads the persona's `engine.tier` as a field it does not know: `unsupported by this engine` (exit 5), never a divergence. +- No change to habits (hard rules), the strand format for other tiers, or any public wording. The bound certificate is + computed from the persona file, so with `engine.tier: amplitude` in the file `persona bound` probes through the tier and + its epsilon changes; a certificate computed without the flag does not cover the tier (measured in round 3, section 5). Public text, if any: "an exact simulated qubit register as an optional + solver tier for small programs; measured against the default tier; no quantum hardware". +- **Amplification (round 4, `engine.amplify: true`, off by default).** Instead of projecting, each component's swept state + |ψ⟩ is rotated onto its feasible plans: the rules are the oracle (a phase on the plans that obey them), the diffusion is + about the stored |ψ⟩, k = ⌊π/(4θ) − 1/2⌋ standard iterations with θ = asin √a₀ (a₀ the feasible mass) and one final + iteration whose two phases are chosen so that it lands exactly (`amp::amplify_exact`; mass after asserted ≥ 1 − 1e-12). + Plan and marginals are read from the register's own Born probabilities. The receipt adds `engine.amplify` (iterations and + the final phases per component, the mass before and after) and `feasible_mass` is the mass after (1.000000); the layer list + logs `oracle` and `diffuse` lines. The register holds a second copy of |ψ⟩ while it amplifies (16 · 2^q bytes more). +- Scope (rounds 1-2): the tier answers the turn's solves (the stance and its habit checks). The rest position at the start, the + expectation's members and the trajectory stay on the existing tiers. + +## 4. Bars (registered before any run; PASS/FAIL per bar, failures printed as failures) + +- **B1 exactness.** Marginals from the register agree with an independent reference (a NumPy state-vector implementation + outside the repo) within 1e-9 for q ≤ 12 on 200 random programs and 20 designed ones; gates unitary to 1e-12. +- **B2 replay.** 100 strands with the amplitude tier replay byte for byte on x86_64 and on the no_std build; the 6-decimal + rounding rule holds for every emitted float; a strand recorded with the tier verifies on an engine without it with the + documented `unsupported by this engine` exit (5), never a false divergence. +- **B3 cost.** Host: µs per gate and per `evolve_ising` for q = 8, 12, 16; memory = 16 · 2^q bytes exactly; no_std stack + high-water for q = 8 and 12 ≤ 8 KB (the vector on the heap). Silicon numbers come later. +- **B4 agreement with the p-bit tier (the honest comparison).** On the recorded corpus (653 programs): the level chosen per + trait agrees with the exact p-bit tier in ≥ 90 % of trait decisions under the default schedule, and the cases that differ are + listed with both odds. KEEP as an optional tier if ≥ 90 %; DISCARD as a product tier if < 70 % (kept as a lab experiment); + in between, the verdict is "needs a schedule study", stated as such. +- **B5 interference exists and is bounded.** Three designed programs where two paths cancel: the register's odds differ from the + thermal odds by ≥ 0.2 at the designed node, and the persuadability bound still holds on every trait (0 exceedances on the + 247-program certificate set). +- **B6 the language.** Every public sentence uses "simulated", "exact", "on a CPU"; no claim of quantum hardware or of an + advantage over classical computation; no claim of experience. + +## 5. Measurements so far + +Round 1 (two-valued programs only) and round 2 (multi-level) results are recorded in the change log's Research entries and +the pull request; the verdict on B4 and B5 belongs to round 3. + +Round 2 (multi-level programs, simulated exactly on a CPU; x86_64 host, median of 20): + +| bar | measured | +|---|---| +| B1 exactness | 220 programs mixing two- and three-level variables (q ≤ 12 encoded; 20 designed, among them one where the unused code would be favoured if it were allowed) against an independent mixed-radix reference (one tensor axis of size d per variable, mixer from an eigendecomposition): max \|Δ marginal\| per level 2.7e-13 (bar 1e-9); unused-code mass after every layer 0 (bar 1e-15); mixer unitarity 8.9e-16 for d = 1-8 (bar 1e-12); round 1's 220 two-valued programs through the new layers: 5.3e-15 | +| B2 replay | 100 strands of the tutor example with the tier (three-level traits; 60 encoded qubits in 12 components, the largest 10): recorded twice byte-equal 100/100, verify 100/100 on x86_64 and on the no_std build, re-written byte-equal 100/100, 0 floats past 6 decimals; an engine without the tier: exit 5 on 100/100 | +| B3 cost | one sweep of 16 steps with level marginals: 42 µs (q 8, four three-level traits), 0.82 ms (q 12, six), 16.3 ms (q 16, eight); all two-valued: 97 µs, 2.1 ms, 43 ms; memory 16 · 2^q bytes of amplitudes plus 8 · 2^q of energies; no_std stack high-water 2,016 B at q 8 and 12 | + +The Born mass on plans that obey the rules can be small after the default sweep (as low as 7.3e-05 on the tutor example: +the product over its 12 components; its four-step order component alone allows 24 of its 256 codes); the projection renormalises it. B4 (agreement with the default +tiers, its first pass recorded with the pull request) and B5 have their verdict in round 3. + +Round 3 (the verdict round; simulated exactly on a CPU, x86_64 host): + +| bar | measured | result | +|---|---|---| +| B4 agreement | every program the recorded corpus's 653 outputs solve on the default tiers (replay, live, init, lint, prove, fuzz, the continuation; strand verification left out), solved again by the tier at q_max 16: 338,567 exact solves (4,304 not exact on the default tiers, excluded), 0 declined; the level per trait agrees in **2,095,627 of 2,176,371 decisions = 96.29 %** (the stance path alone, replay and live, 96.52 %; distinct programs 95.94 %; worst traits valence 81.8 %, emoji 86.6 %, arousal 89.7 %). The corpus itself stays byte-identical with the comparison running beside it (653 outputs, 0 differ) | **PASS (≥ 90 %)** | +| B4 where they differ | 80,744 differing decisions, each listed with both odds. By the default tier's odds on its own level: ≥ 0.60 → 99.97 % agree, 0.50-0.60 → 98.3 %, 0.40-0.50 → 93.6 %, < 0.40 → 90.9 %; in 91 % of the differing cases the default level holds less than half the mass (median 0.385): the two disagree where the thermal answer is a plurality, not a majority. By the trait's component's feasible mass: 99.1 % of the differing cases sit in components with mass ≥ 0.1, so they are not an artefact of the projection | stated | +| feasible mass | the receipt's product over components is < 1e-2 on 36.7 % of the solves (min 2e-06), but per component on 0.04 % (min 2.6e-04, median 1): the product multiplies up to 12 components. A rule-penalty term (E' = E + 4 · violations, the projection kept) measured in the lab lifts every component above 7e-03 and drops B4 to 83.44 % (the order rules' step variables move most): not adopted; the projection stays | measured, not changed | +| B5 interference | three designed programs where two paths into a node differ in energy by D (the default sweep turns the difference into a relative phase), designed and predicted by the reference before the engine ran: register vs thermal odds at the node 0.748 vs 1.000, 0.744 vs 0.990, 0.697 vs 0.969 (gaps 0.252, 0.246, 0.272; the same programs at D = 0: 0.001, 0.000, 0.037); engine vs reference within 1.4e-14 | **PASS** | +| B5 bound | the bound harness (7 personas, seeds 0-9, greedy, mixed and self-sourced campaigns) with the tier on: certificate computed from the persona file with the flag, **0 of 4,100 (campaign, trait) cases above epsilon + r** (largest ratio 1.000), self-sourced events refused 10/10; with the certificate computed without the flag, 1,598 of 4,100 above (largest ratio 1.46): the tier moves resting odds further than the thermal tier, so a certificate must be computed with it. Tightness (≥ 0.9 of epsilon reached) fails on one fixture with the tier (0.85) | **PASS** (with that caveat) | + +Verdict: **KEEP as an optional tier** (research; off by default). It reproduces the stance semantics in 96 % of decisions and +differs where the thermal answer is nearly indifferent; it adds interference where designed programs have two paths. Not +verified: silicon cost and replay on a chip. + +### Amplification (round 4) + +Why it changes nothing in the stance, and why that is the point. Write the swept state as |ψ⟩ = sin θ |g⟩ + cos θ |b⟩, with +|g⟩ its normalised part on the plans that obey the rules and |b⟩ the rest. A phase on the feasible plans and the diffusion +about |ψ⟩ both map the plane of |g⟩ and |b⟩ into itself, so every state the amplification visits is α|g⟩ + β|b⟩; one final +iteration with matched phases makes β exactly 0, and the result is |g⟩ up to a global phase. Its Born probability on a +feasible plan x is |ψ_x|² / a₀, which is what the projection computes by renormalising. So the plan and the odds are the +projection's, and the feasible mass is 1: the projection was never an approximation layered over the register, it is the +distribution exact amplification produces by a unitary evolution, simulated exactly on a CPU. Nothing in the argument depends +on the program or the rules; it needs only the exact landing. + +Measured (x86_64 host): on 220 programs (200 random mixed two- and three-level programs and 20 designed ones, a rule each: +seeded random plan subsets at densities 0.03-0.8 or "at most one variable at its top level"; a₀ from 1.7e-06 to 1, 0 to 611 +iterations) the marginals with amplification differ from the projection's by at most 1.1e-12 (bar 1e-9), the feasible mass +after is at least 1 − 3.3e-13 (bar 1 − 1e-12; the largest drift, 1.1e-12 above 1, after 611 iterations), and the most +probable plan is the same on 220 of 220. On every program the recorded corpus solves (339,389 exact solves, each run both +ways) the plans are equal on 339,389 of 339,389 and the marginals differ by at most 2.1e-13, so B4 is round 3's to the +decision: 2,095,627 of 2,176,371 = 96.29 %. A 287-event recorded +day of a 52-qubit persona gives the same stance line on 287 of 287 events (per component 1-2 iterations: the per-component +mass is mostly near 1 and the receipt's small product, median 0.058, multiplies 11 components). Cost: 1.02× the projected +solve on that day (1.19 vs 1.17 ms per event) and 1.06× on a 16-qubit component at a₀ = 0.022 (5 iterations); no_std stack +high-water 2,016 B at q 8 and 12. 100 lives with amplification replay byte for byte on x86_64 and on the no_std build, 0 +floats past 6 decimals, and an engine without the tier exits 5. Turning it on therefore buys a receipt whose register holds feasible mass 1 and logs +how it got there; it does not buy a different stance. + +### Schedules and the indifferent set (round 5) + +Round 4 showed amplification computes what the projection computed, so the only place the register can decide differently is +its evolution. `engine.schedule` names it (needs the tier; absent = `tf-linear-16x4`, byte for byte): `tf-linear-16x4` (the +default sweep, 16 steps over time 4), `tf-linear-64x16` (the same transverse-field sweep, 64 steps over time 16: four times the +layers) or `qaoa-p2` (two QAOA layers, γ 0.500000 β 0.500000 then γ 1.500000 β 0.250000, fixed by a grid search on a held-out +third of the corpus's programs and then frozen). The receipt's `schedule` digest and `schedule_name` name it; an unknown name +is refused. + +The study: on the recorded corpus (339,389 exact solves, 130,088 distinct programs, 2,183,612 trait decisions), the thermal +tier is **indifferent** (its top two levels' odds within 0.05; its own gap a median 0.028) on **234,004 decisions = 10.72 %** +(fuzz 9.9 %, replay and live 15.5-15.7 %, lint 17.5 %). On those decisions, measured on the two thirds of programs the grid +did not see, the register's gap between its top two levels: + +| schedule | median gap | gap > 0.10 | gap > 0.25 | agreement where the thermal tier is not indifferent | +|---|---|---|---|---| +| tf-linear-16x4 (default) | 0.048628 | 9.5 % | 0.1 % | 97.58 % | +| tf-linear-64x16 | **0.151709** | **64.6 %** | **16.0 %** | **98.17 %** | +| qaoa-p2 | 0.094292 | 47.4 % | 7.0 % | 92.99 % | + +Two disjoint samples of programs give medians within 0.000697 of each other, so the long sweep's margin (0.103) is about 150 +times the noise. By the registered rule (the largest median gap subject to agreement ≥ 90 % on the decisions that are not +indifferent) the choice is `tf-linear-64x16`; it is not the default, because the default is what every strand recorded so far +replays. What the longer sweep buys a decision, stated plainly: a slower sweep keeps more of the simulated register's amplitude +on the most probable plans, so it commits where the thermal answer is a near tie (it takes the thermal plurality on 78.5 % of +those decisions, the runner-up on 20.1 %, a third level on 1.4 %), and it keeps the thermal choice where there is one. It does +not create a preference the weights do not hold. Its cost is the layers: 1.5× the default solve per event on a recorded day of a +52-qubit persona (3.1-3.5× on a lone 8-12 qubit component), 0.8× for `qaoa-p2`; no_std stack at most 2,960 B on every schedule. With amplification on, 165 of the 130,088 +programs reach a component mass near 5e-5 under the long sweep and 100+ iterations, where rounding leaves the landing about +1e-12 short of its bar; that component now declines to the default tiers (`amplification missed its 1 - 1e-12 landing`) +instead of stopping the turn (0.12 % of decisions; 0 with the other two schedules). Every number here is a register simulated exactly on a CPU; +the gap is a property of the evolution's length and phases, not a claim about any device. +Replay, measured, and a failure printed as one: 100 lives per schedule record twice byte for byte, verify on the same build and +emit 0 floats past 6 decimals; an engine without the tier exits 5. The no_std build (the chip's math library) re-verifies +the default schedule's lives 100 of 100 on x86_64, but **the long sweep's 0 of 100 and qaoa-p2's 98 of 100**: the platform's +math functions and the no_std library differ in the last bits, and over four times the layers a 6-decimal odd flips. On an arm64 +host even the default fails there (0 of 5 on round 4's own path), so that part is not new. Round 7 (below, "One math path") +makes the tier compute with the same math functions on every build: all three schedules now replay 100 of 100 on the no_std +build on x86_64. + +### The demo (round 6) + +One decision where the thermal tier is a near tie, shown end to end on the recording (std) build: the program, the two odds, +the receipts and the monitor's two faces. Every number is a register simulated exactly on a CPU. + +**The program, from the corpus.** The recorded corpus's tutor fixture (seed 1, script 2, the 58th exact solve of its replay) +solves program `a7fae99b114ea2d2` (first 8 bytes of the sha256 of its compact JSON). Humour, regenerated from the corpus: +thermal `playful` 0.452774 vs `light` 0.442258 (gap 0.010516); the register with the default sweep, gap 0.080846; with the +long sweep, `playful` 0.975625 (gap 0.951467). The runner-up, `07d909f824f224e4` (the same fixture, script 3): thermal +`light` 0.444788 vs `playful` 0.425243 (gap 0.019545), long sweep `playful` 0.972868 (gap 0.945965): here the register takes +the thermal runner-up, as it does on 20.1 % of the indifferent decisions. + +**The persona and the life** (`examples/persona/demo/`). A fictional tutor reduced to its humour component (warmth, humour, +formality, emoji, the valence mood, their couplings and the two habits that bind them) and the two drives that pull humour +in opposite directions: praise up, criticism (a correction) down. `register.yaml` carries `engine: {tier: amplitude, amplify: +true, schedule: tf-linear-64x16}`; `thermal.yaml` is the same persona without the engine block (`amplify` and `schedule` need +the tier). The life, 17 events at seed 1 on a fixed clock, opens with a run of corrections; at event 9 praise and a correction +arrive together. Both strands verify, record twice byte for byte, emit 0 floats past 6 decimals; event 9 is line 10 of both, +and its two documents solve the same program. + +| event 9, humour | none | light | playful | top-two gap | +|---|---|---|---|---| +| thermal (default tiers) | 0.059462 | 0.466563 | 0.473975 | **0.007412** | +| register, default sweep (comparison) | 0.013180 | 0.566151 | 0.420670 | 0.145481 | +| register, long sweep (`register.yaml`) | 0.007171 | 0.249010 | 0.743818 | **0.494808** | + +The register document's receipts at event 9: `tier` amplitude, `qubits` 20 (`components` [10, 10]), `mixer` complete-graph, +`schedule_name` tf-linear-64x16, `amplify` iterations [10, 1] with the mass before 0.006196 and after 1, `feasible_mass` 1, the +`state` digest `sha256:6dbfd5a4…1bbe11`, the program digest `sha256:06517240…ab9a66` (the thermal document's too). Events 2 and +11 are near ties of the same kind (thermal gaps 0.033093 and 0.048852, register gaps 0.656343 and 0.725300). The register is not +always the more decisive face: at event 14 the thermal gap is 0.072051 and the register's 0.009221. At event 9 the default sweep +leans to `light` (gap 0.145481); the long sweep commits to `playful`, the thermal plurality. + +**The two faces.** The Mood Monitor (`probbit monitor`, section 5.8 of docs/persona.md) on each strand at event 9, replayed +from the strand alone ("replay verified"; the document it serves for event 9 is the recorded one, byte for byte), humour +selected. Thermal: `playful` 47.4 %, `light` 46.7 %, the panel's words "playful, slightly". Register: `playful` 74.4 %, +`light` 24.9 %, "playful, clearly". + +![The Mood Monitor on the thermal strand at event 9, humour selected](0011-demo-thermal.png) + +![The Mood Monitor on the register strand at event 9, humour selected](0011-demo-register.png) + +**The sentence.** On one recorded decision of a fictional tutor, the default solver's odds for its two leading humour levels +sit 0.007412 apart; the same program, solved by a qubit register simulated exactly on a CPU, separates them by 0.494808. +Since round 7 both strands re-record byte for byte and re-verify on the no_std build as well ("One math path", below). + +**Cost** (x86_64, best of 5, the life repeated 20 times, 340 events): thermal 2,184 µs per event, the register with the long +sweep 1,822 µs (0.83× the thermal copy), the register with the default sweep 913 µs (the long sweep 2.00× the default sweep). +On this five-variable persona the thermal copy's exact enumeration costs more than the register. + +### One math path (round 7) + +What an optional tier needs: the same odds on every build, no declines, the numbers re-measured. The register, simulated +exactly on a CPU, now evaluates every function past + − × ÷ (cos, sin, sqrt, asin, acos, atan2, floor) through the `libm` +crate on the std and the no_std build alike: pure Rust, the same code and evaluation order on every target, no platform math +library, no fused multiply-add. Measured on x86_64 (100 fresh tutor lives per schedule recorded on the std build, each +re-verified and re-written by a no_std build of the persona crate on the host): + +| schedule | recorded twice byte-equal | verify (std) | verify (no_std) | no_std re-write byte-equal | strands equal to the round-6 engine's | +|---|---|---|---|---|---| +| `tf-linear-16x4` (default) | 100/100 | 100/100 | **100/100** (round 5: 100/100) | 100/100 | 100/100 | +| `tf-linear-64x16` | 100/100 | 100/100 | **100/100** (round 5: 0/100) | 100/100 | 0/100 | +| `qaoa-p2` | 100/100 | 100/100 | **100/100** (round 5: 98/100) | 100/100 | 98/100 | + +The tutor's default-schedule strands are byte-identical to the round-6 engine's and both demo strands re-record byte for byte, +but the change is in the last bits and a larger persona's default strands can move: on a recorded day of a 52-qubit persona +they did (qaoa-p2's did not); the long sweep's tutor strands all moved. An unreleased tier allows it; no shipped fixture moved. +The study of round 5 re-run on this tree: 0 programs decline under any schedule (round 5: 165 under the long sweep), the long +sweep's median gap on the indifferent decisions 0.151709 and agreement on the others 0.981671 (round 5: 0.151709, 0.981717, +now over the formerly declined decisions too), the default's and qaoa-p2's rows unchanged to 6 decimals; cost per event on that +day 1.50x the default for the long sweep (branch vs the round-6 engine 0.95-1.00x); no_std stack at most 2,944 B per schedule. Both demo strands re-record byte for byte. The arm64 leg is not measured (no arm64 runner for the replay). + +The missed landing: the diffusion about a swept state of squared norm 1 + ε is not unitary (|Dv|² = |v|² + 4ε|⟨ψ|v⟩|²), so +the error grows with every iteration, linearly in their number; at a tiny feasible mass (hundreds of iterations) the long +sweep's ε left 165 corpus programs about 1e-12 short of the landing, and the same ε of the other sign is round 4's drift above +1. `amplify_exact` now renormalises the swept state once before the standard iterations (squared norm by a compensated sum; +only when there are iterations, so the one matched step of a mass above 1/4 is unchanged), and a mass that is not a number +declines like a miss. A rotation in the plane would have hidden the cause; renormalising keeps every iteration the same +unitary step. The receipt's mass before stays the swept state's; `feasible_mass` stays 1. + +## 6. What this RFC does not claim + +That a CPU can do what quantum hardware does at scale (it cannot: section 1), that the register has experiences (it is +arithmetic), or that it beats the p-bit tier (B4 decides). What it can honestly give: superposition with real phases and +interference inside the agent's decision, exact and replayable, simulated on a CPU at a size a laptop and a small chip can +afford. diff --git a/docs/rfc/0011-demo-register.png b/docs/rfc/0011-demo-register.png new file mode 100644 index 0000000..5356fc2 Binary files /dev/null and b/docs/rfc/0011-demo-register.png differ diff --git a/docs/rfc/0011-demo-thermal.png b/docs/rfc/0011-demo-thermal.png new file mode 100644 index 0000000..4fe8a48 Binary files /dev/null and b/docs/rfc/0011-demo-thermal.png differ diff --git a/docs/rfcs/0013-self-report.md b/docs/rfcs/0013-self-report.md new file mode 100644 index 0000000..9d346f1 --- /dev/null +++ b/docs/rfcs/0013-self-report.md @@ -0,0 +1,83 @@ +# RFC 0013: self-report, a canonical account of the last N events from the strand alone + +Status: **implemented, off by default** (0.14.0-dev: `probbit reflect`, the library's `reflect` module, the engine-gated +`reflection` line, the monitor's Mirror tab) · Changes: docs/persona.md §5.7 (one more line kind, 0.14.0+) and §5.8 +("Self-report", "Mirror"), `probbit live control`, `live verify`, `probbit monitor` · Depends on: RFC 0009 (the expectation +block and its receipts) + +Mechanism and simulation only. The self layer is a declared, exact, replayable account an individual gives of its own recent +events: arithmetic over the lines of its strand, recomputable by anyone who holds the strand. Nothing here is run, solved or +sampled; nothing is written unless a host asks. + +## What it is + +`reflect(lines, window)` reads a strand's lines and returns one canonical document, `reflection`, over a window of its event +lines: the last N (`--last N`), the events after the last checkpoint line (`--since-checkpoint`, that checkpoint the first +sample), or every event. It reads what each line logs and nothing else: + +| Section | Read from | What | +| --- | --- | --- | +| `window` | every line | first and last event n, the events, the lines read, the checkpoint samples, the strand's engine, the window asked for | +| `inputs` | event lines | per input, the events where it was true or a non-zero number (history and sensed readings are not counted) | +| `surprise` | event lines (`deviation` receipts) | per receipt z recomputed from the expectation's memory, a fold of the logged receipts (its mean and variance only, as `persona::expect_step` folds them); the events scored, the ones that registered (z ≥ 1.5), the K = 5 largest \|z\| with `pred` (what the expectation predicted), `self` (what the event measured), `input`, the glyph and the inputs set; `null` without an expectation block | +| `traits` | checkpoint lines in the window | per trait (and `pursue`) the levels taken, the odds at the first and the last sample, the smallest and largest p of the level taken | +| `moods` | checkpoint lines | per mood its position along its levels (0 the lowest, 1 the highest: the monitor's sparkline value) at each sample: start, end, min, max, the turning points (sign changes of the first difference; all counted, the first 8 listed) and the last 50 samples | +| `drives`, `habits` | checkpoint lines | per goal the largest want and its event; per habit the samples where it bound | +| `credit` | event and control lines | feedback that settled the credit (reward, correction, both on: cancelled) and the control lines that zeroed it; `null` without a learning block | +| `learned` | checkpoint lines | the learned weights' net change between the first and the last sample, and the sum of its absolute values | +| `controls` | control, rest, reflection lines, the header | pause, resume, retire, erase; `zero_credit` (every control line zeroes the credit); rest lines; earlier reflection lines; the segment index | +| `summary` | the document's numbers | 6 to 12 sentences from fixed templates (13 at most with a declared stir, RFC 0016), part of the canonical document | + +A section the window has no line for is `null`: an event line logs digests, not the stance, so a window without a checkpoint +line reports no levels, moods, drives or habits (absent, never invented). A strand of an older engine is read the same way; +the fields its lines lack are absent. Every number is at 6 decimals (the P0-b rule), the document is canonical JSON, and +`digest` is the sha256 of the canonical text without `digest`. + +The z of a receipt is the expectation's own: on a 0.14.0 strand the fold starts at the last checkpoint before the window that +carries the expectation's memory (its mean and variance are the fold's, at 6 decimals), else at the header; it equals the z +the checkpoint lines' stances carry (a unit test checks every checkpoint of the fixture). + +## The templates + +The sentences are the document's numbers in fixed wording, so a model can voice them or a person can read them; nothing in +them is generated. They describe mechanism. Over the expectation fixture of the test suite +(`probbit-cli/tests/fixtures/monitor/tutor-expect.strand`, 22 events, a checkpoint every 5): "This account covers events 1 to +22: 22 events and 26 lines read, 4 of them checkpoint samples.", "The expectation scored 22 events and 11 of them registered as +surprising (z at least 1.500000).", "The largest surprise was event 2: z 6.710933, the expectation predicted 2.842300 and the +event measured 2.211600.", "The mood valence moved from 0.705807 to 0.465881 on a scale of 0 to 1 (min 0.255617, max 0.705807, +2 turning points)." A unit test +holds every template's output to a list of words it never contains. + +## The reflection line (0.14.0+) + +`probbit live control STRAND reflect [--last N | --since-checkpoint]` appends the reflection as one line after the strand's +last line, under the writer lock: `{"prev": , "reflection": }`, canonical, chained like +every line. A host decides when (after a checkpoint, at the end of a day); a body in standby calls `reflect::reflect_window` +and `reflect::line` and appends the line itself. + +- `live verify` recomputes the document from the lines before it, over the window it names, and refuses one that differs (a + changed number, a changed sentence, a moved window): exit 1 naming the line, "the reflection differs from the lines before + it". Replay treats the line like a control line: no state changes, the chain goes on from it; `live` continues a strand that + ends on one. +- Engine-gated: the writer refuses a strand whose header names an engine before 0.14.0, and a 0.14.0 replay of such a strand + diverges at a reflection line. An engine before 0.14.0 reads the line as a key it does not know: `unsupported by this engine: + reflection`, exit 5 (not verified, and not a divergence). +- No line is written unless asked: every strand without reflection lines, and every corpus, is byte for byte what it was. + +## The Mirror tab + +`probbit monitor --serve` serves `/reflection?last=N` (and `/s/ID/reflection` per strand): the reflection of the last N events +(default 50) computed from the strand file as it is at the request, and the latest reflection line the strand carries with +whether it recomputes equal. The page's Mirror tab shows the sentences, the top surprises with predicted vs measured, the +habits that bound and the moods at the checkpoint samples, refreshed with the page's scan. + +## Cost + +One pass over the lines from the fold's start: each event line parsed once, each checkpoint line in the window parsed once, +no solve. Host microseconds and the stack high-water on the no_std proxy are in docs/embedding.md ("Self-report in standby"). +Silicon is not measured. + +## Not in this RFC + +Integrity (how far the individual drifted from the persona it declared), consolidation and curiosity are later RFCs of the +self layer. The reflection judges nothing: it counts and folds what the strand logs. diff --git a/docs/rfcs/0014-integrity.md b/docs/rfcs/0014-integrity.md new file mode 100644 index 0000000..889ccb6 --- /dev/null +++ b/docs/rfcs/0014-integrity.md @@ -0,0 +1,93 @@ +# RFC 0014: Integrity, the drift from the declared self + +Status: implemented (library and strand line), off by default; engine 0.14.0-dev on a draft PR · Changes: docs/persona.md +(the `identity` block's self keys, additive and outside the digest), one engine-gated field on event lines · Depends on: +RFC 0004 (the allowance is its epsilon), RFC 0013 (the Mirror tab, where the per-trait table is to be shown) + +## Summary + +Per event, one exact number and its decomposition, from the state and the persona document alone: how far the individual's +resting odds have moved from those of the self its persona declares, and how much of the persuadability certificate's +allowance that movement uses. Labelled **mechanism**: arithmetic over declared numbers, replayable from the strand, +recomputed by `live verify`. It adds nothing to the state and changes no turn. + +## The declaration + +Four optional keys in the existing `identity` block (the "self keys"): + +- `keep: [habit ids]`: the habits that are identity-bearing. +- `warn: `: a share of the allowance in (0, 1], default 0.8. +- `cue: `: a short line (1-120 characters) in the persona's own words, handed to the host when a trait reaches `warn`. + The engine never writes cue text of its own. +- `traits: [trait ids]`: the traits measured; default the learned traits (`learning.traits`), else every trait. + +Declaring any of them turns integrity on. The keys are read against the persona built without them and stay out of the +persona digest (as RFC 0004's `bound`, amendment A1): every state is byte for byte as without them. A persona that declares +none of them emits nothing, so every existing strand, stance and corpus is unchanged. + +## The numbers + +- **The rest probe** (RFC 0004's): one event of `bound.rest_hours` idle hours (default 30 × the longest declared half-life, + 24 without one), no inputs. +- **The rested individual**: the current state with the channels of the hour at the fresh individual's values (moods with + `half_life_hours`, the pace average, the load, the rest pressure, the eligibility trace, the credit awaiting feedback and + its open decisions, the levels and stance taken last, the agenda's step) and everything that persists as it is (learned + deltas, history, moods without a half-life, drives, senses, the surprise gates, genes). The credit is of the hour: it is + what the next feedback would act on, zeroed by every control line, and a rest probe has no feedback. +- **drift**, per trait: the total variation, half the L1 distance over the trait's levels, between the probe's odds for the + rested individual and for the fresh individual of the same seed (the declared persona at birth); `drift` at the top is the + sum over the traits. The fresh individual is its own rested state, so its drift is 0 exactly. Total variation is chosen over the + plain L1 because it is the certificate's own unit: `bound_used` is then a ratio of like quantities, and the theorem of + RFC 0004 reads directly as `bound_used ≤ 1` (up to the residue r of decayed channels and the 1e-6 rounding of amendment A7). +- **allowance**, per trait: epsilon at the individual's own seed, computed once per seed from the persona's `bound` block + (the same vertex search as `persona bound --audit`); null for a trait that is not learned or without a `bound` block. +- **bound_used** = drift / allowance (0 without an allowance). **used** (the dial) = the summed drift over the traits with an + allowance, divided by the summed allowance (0 without one). +- **crossed**: the traits whose `bound_used` reached `warn`; **cue**: `identity.cue` when `crossed` is not empty, else null. +- **habits_held**: of the `keep` habits, `held` (bound this event) and `not_fired`. + +Every number is rounded to 6 decimals. + +## Where it appears + +- The live stance document gets `integrity`; the event line carries the same document as its `integrity` field. Both stance + digests leave it out (it is computed after the turn, from the state the turn produced, so the state's remembered stance + digest and `for` credit binding are unchanged). +- `live verify` recomputes it on every event line and refuses an edited value naming the line ("the integrity differs"). + `resume` takes it as written, as it takes `deviation` and `glyph`. +- Engine-gated: a persona with self keys is refused by an engine before 0.14.0 as an unknown field, which `live verify` + reports as `unsupported` (exit 5), never as a divergence. + +## Cost + +At most one extra solve per event: the rest probe of the rested individual. The fresh individual's probe and the +allowances are computed once per seed. The probe is cached twice (RFC 0014-b): + +- **By program.** The probe's turn steps run as the turn runs them, its first program is compiled and keyed by its structure + without the `comment` (the key the expectation's cache gives its members). An equal key answers from the row. Otherwise the + turn finishes on that same compiled program, so a miss compiles once. A row is kept only when the program's answer was exact + and the stance is that answer's (no yield, no hold's clamped solve, no conflict): exact marginals read the program's model + alone, so a row is the uncached computation's value bit for bit. A body whose senses, drives or needs move every event hits + whenever its resting program did not move. +- **By state**, as before: the rested state's body without `turn` and `digest`, used while no program row is held (after a + sampled answer). While a row is held an equal rested state would compile the same program, so the state key is not built. + +Measured (docs/embedding.md "Integrity on a body"): on a body's recorded day of 1,160 events the resting program moved on 4 +events (the `pursue` goal's drive terms), so the probe was answered from the cache 1,156 times (99.66 %). The cost fell from ++40.1 % to +16.1 % of a turn on the host (std and the `nostd` proxy alike): what remains on a hit is building the probe's program (the turn's steps before the +compiler, the compile, the key). A miss is a whole turn and carries its stack (13,656 B on the `nostd` build, against +12,520 B for the bare rest-probe turn); a hit by program reaches the same depth (13,656 B): the turn's steps before the solve +are the deepest part of a turn. The bar of this RFC (2 % of a turn, 8 KB per miss) is still not +met. The floor of a program key is one program build per event. The remaining lever is to skip the build when nothing the +probe's program reads has moved (a key on the compiler's inputs: the learned deltas, the credit, the history windows, the drives' +terms, the senses' gains). That key is not built here: its exactness needs the slice proved field by field, the way the +expectation's members would need it (T2's NEXT 1). A per-miss stack under 8 KB needs a rest probe lighter than a turn. + +## Not in this build + +- The cost bar (2 % of a turn, 8 KB per miss): the cache now hits on a body's ordinary event; what remains is one program + build per event and a turn's stack per miss (see Cost). + +- Appending the cue to the stance's `line` (the line is inside the stance digest the state remembers); the host appends + `integrity.cue` itself. +- The monitor's integrity dial and the Mirror's per-trait table. diff --git a/docs/rfcs/0015-consolidation.md b/docs/rfcs/0015-consolidation.md new file mode 100644 index 0000000..08392af --- /dev/null +++ b/docs/rfcs/0015-consolidation.md @@ -0,0 +1,103 @@ +# RFC 0015: Memory consolidation, a rest pass that compacts what was lived and logs what changed + +Status: implemented (library `Live::consolidate` / `consolidate_boxed`, the `consolidation` strand line, `live control STRAND +rest`), off by default; engine 0.14.0-dev on a draft PR · Changes: docs/persona.md (§2.17, an optional `rest` key of the +`expectation` block, outside the digest as the whole block is), one engine-gated line kind, one key in the expectation's checkpoint memory · Depends on: +RFC 0009 (the expectation's memory and its z), RFC 0013 (the reflection reports the pass), RFC 0016 (its fold is left as it is) + +## Summary + +At rest, when a host decides (standby, night, idle), the engine runs a **rest pass** over the expectation's memory: it +re-folds what the events since the last pass added with a declared half-life, weighs each remembered context by how much it +kept surprising, merges near-duplicate contexts, and applies a declared decay to the learned deltas. It writes one +`consolidation` line that logs the memory's digest before and after, the contexts merged, the weights moved and the decay. +Labelled **mechanism and simulation**: the pass is arithmetic over numbers the strand already logs; the line is +checkpoint-like and `live verify` re-runs the pass from the lines before it. No new state the strand cannot recompute. + +## The declaration + +```yaml +expectation: + half_life_events: 32 + max_combos: 6 + rest: # optional; without it the pass is refused and nothing is folded for it + half_life: 64 # the window re-folded with this half-life in events (default: half_life_events, a no-op) + bucket: 2 # bins of a declared number input in the merge key, 1-3 (default 3: the waking key, nothing merges) + surprise_gain: 1 # weight gain of a context's mean z+ (0-4; default 0: none) + decay: 0.5 # factor on the learned deltas, 0-1 (default 1: the state is left as it is) +``` + +Every key is optional and every default changes nothing: `rest: {}` declares the pass, and a pass under it logs `moved 0`, +`merged 0` and leaves the state digest as it was. The key is outside the persona digest, as the whole `expectation` block +is: the pass runs only when a host calls it, and the strand's header carries the document every replay and resume reads, +so the line's `state` digest pins what a `decay` did. Two documents that differ only in `rest` share a digest. + +## The window + +With a `rest` block, the canonical memory (engine 0.14.0 on) carries one more key, `window`: `since` (events since the last +pass) and, per remembered context in key order, `[a, b, k, zs]`: `a` the context's visits in the window decayed by the +expectation's half-life (the part of its count the window added), `b` the same visits decayed by `rest.half_life`, `k` the +visits, `zs` the sum of the visits' z+ = r6(max(z, 0)) (z as the deviation computed it: the logged prediction against the +learned normal before the event; a cold event adds 0). Each row is at 6 decimals after every event; a context the memory +evicts or prunes leaves the window with it. A checkpoint carries the window, so a resume and `verify --from-checkpoint` start +there. + +## The pass + +In this order, on the memory as the last event left it: + +1. **Re-fold**: each count becomes r6(max(0, count − a + b)): the window's part re-weighted with the rest half-life (a longer + half-life keeps more of the window's early visits; the expectation's own half-life leaves the count as it is). +2. **Surprise**: with gain g > 0, each context gets the weight f = 1 + g · min(zs/k, 1.5)/1.5 (1.5 = the deviation's + registration threshold; a context without visits in the window has f = 1), and the counts are scaled back to their + total before (r6 each): contexts that kept surprising keep more weight, the total is unchanged. +3. **Merge**: with `bucket` B < 3, contexts whose key is the same with each declared number input in B equal bins (instead of + the waking key's thirds) become one: the inputs of the one with the largest count (ties: key order), the counts summed. +4. **Decay**: every learned delta becomes r6(decay · Δ) (the state digest moves; 1 leaves the state as it is). + +The window is then empty. The expectation's learned normal (mean, variance), its `rest` count and the curiosity fold (RFC +0016) are left as they are; a context merged away leaves the curiosity fold's failed set at the next event by the fold's own +prune. The memory holds at most `max_combos − 1` named contexts after every pass (the merge only removes, the other steps +only reweigh). The expectation's cache is emptied (its rows are keyed by a program's text, so no receipt changes). + +## The line + +```json +{"at":"2026-10-28T04:00:00Z","consolidation":{"after":"sha256:...","before":"sha256:...","decay":0.5,"events":50, + "merged":{"count":1,"keys":["{...}"]},"moved":2.186238},"n":50,"prev":"sha256:...","state":"sha256:..."} +``` + +`at` the host's time stamp (1-64 characters, taken as written); `n` the events before the pass; `before`/`after` the sha256 of +the memory's checkpoint form; `events` the window's length; `merged` the keys merged away; `moved` the sum over every key of +|count after − count before| (6 decimals); `state` the digest after the decay. Engine-gated: a strand of an engine before +0.14.0 refuses the line (`CONSOLIDATE_OLD`), and an engine before 0.14.0 reads a persona with `expectation.rest` as unsupported +(`verify` exit 5). `live verify`, the chain check, `resume` and `resume_from_record` re-run the pass; a forged line is refused +at its line with the first field that differs (`the consolidation line differs at consolidation.moved`). + +## Interactions + +- RFC 0009 expect cache: invalidated by the pass; every receipt is the same with the cache on or off. +- RFC 0013 reflection: a window holding a consolidation line gets `consolidation {lines, merged, moved}` and one sentence + ("The rest pass ran N times in this window: M contexts merged, the remembered weights moved by X in total."); every other + reflection document keeps its bytes. +- RFC 0016 curiosity: its fold is not touched by the pass. +- RFC 0014 integrity: reads the state after the pass as any state (a decay moves the learned deltas and so the drift). + +## Surfaces + +- Library: `Live::consolidate(at)`; `Live::consolidate_boxed(&mut Box, at)` on a body (the same call, in place). +- CLI: `probbit live control STRAND rest [--at TIME] [--state FILE]`: the strand replayed from its header, the line appended + under the writer lock; `--state` names the host's stored individual (`live --state FILE`), checked against the strand's + last state and rewritten after the pass, so the next `live --state FILE` continues the strand. +- Monitor: a `REST` row with the last pass (time stamp, event, line, contexts merged, weights moved, decay, window length), + and `last_rest` in the page's frame, once the strand holds a consolidation line. + +## Cost + +O(contexts) per event for the window fold (at most `max_combos` rows) and O(contexts²) per pass for the merge (at most 64 +contexts); no solve. Numbers on the host and the `no_std` proxy are in docs/embedding.md; chip numbers are the body lane's. + +## Not built + +`rest.decay` naming the curiosity fold (a decay of failed contexts at rest) is not built: the pass leaves the drive's fold as +it is. The monitor shows the last rest (a `REST` row and the frame's `last_rest`); its history view does not mark rest lines. diff --git a/docs/rfcs/0016-curiosity.md b/docs/rfcs/0016-curiosity.md new file mode 100644 index 0000000..6e1472c --- /dev/null +++ b/docs/rfcs/0016-curiosity.md @@ -0,0 +1,118 @@ +# RFC 0016: Curiosity, a drive fed by the measured surprise, and the self-start it gates + +Status: implemented (library, strand line, `live control STRAND stir?`), off by default; engine 0.14.0-dev on a draft PR · +Changes: docs/persona.md (§2.20, a new optional top-level block, in the digest), one engine-gated field on event lines, one +key in the expectation's checkpoint memory · Depends on: RFC 0009 (the deviation's z and the memory fold), RFC 0004 (the +bound), RFC 0013 (the Mirror's account) + +## Summary + +A persona may declare a drive named `curiosity`. Its level is arithmetic over two numbers the expectation block already +measures: `surprise_recent`, an exponentially weighted average of z+ = max(z, 0), and `failed_contexts`, the remembered input contexts +whose last K visits all missed. The level rises while nothing new arrives and while contexts keep failing, and it falls when +a surprising event arrives. It moves the stance through one engine-owned number input, as `pace` does. It can also gate a +self-started idle event (a **stir**): `Live::stir_due(idle_hours)` answers, exactly from the drive's state and the idle +hours, whether one is due, at a declared rate, with a quieter class and a lower rate while the individual is asleep. +Labelled **mechanism and simulation**: the name is the drive's declared name; the drive is a function of logged numbers, +replayable from the strand and recomputed by `live verify`. + +## The declaration + +```yaml +curiosity: # needs an expectation block (the surprise it reads) + half_life_events: 8 # the average of z+ = max(z, 0): its half-life in events (>= 1) + quiet_z: 0.5 # surprise_recent below this is "nothing new" (> 0); the average starts here + miss_z: 1.5 # a visit misses when z+ > miss_z (> 0) + last_k: 3 # a context fails when its last K visits all missed (1-16) + weights: {quiet: 1.0, failed: 0.5} # 0-50 each + cap: 2 # the level stays in [0, cap] (> 0) + effects: {initiative: 0.6} # required: ordinal shift per unit of the level, on traits or moods (never pursue) + stir: # optional: the self-start + threshold: 1.0 # due only at a level >= threshold, in (0, cap] + every_minutes: 30 # at most one stir per 30 minutes (>= 1) + class: check # the stir's class (an identifier; default check) + speaks: true # whether that class may speak (default true) + when_asleep: # optional; without it no stir while asleep + flag: night # a declared flag input: the last event had it on = asleep + every_minutes: 120 # >= every_minutes + class: glance # the asleep class (default glance); it never speaks +``` + +The block is in the persona digest (it moves the stance). Refused when the persona is read, at the field's path: no +expectation block (C-R1); a bound outside its range, `effects` empty or on `pursue` or on an unknown id, a `when_asleep.flag` +that is not a declared flag input, an asleep rate faster than the awake one, an asleep class equal to a speaking awake class +(C-R2); a declared input named `cur_level` or `stir`, a habit condition on either (C-R3). + +## The drive + +Per event, from the logged line alone (so `verify`, `resume` and a restart from a checkpoint fold the same numbers): + + z = (ln(1 + pred) - mean) / sqrt(var + 1e-4) the deviation's z: the logged pred against the normal before the event + z+ = r6(max(z, 0)) one-sided, as `registers`: an error below the normal is no surprise + s <- r6(lam x s + (1 - lam) x z+), lam = 0.5^(1 / half_life_events) (cold events: unchanged) + miss = z+ > miss_z, appended to the event's context (its expectation combo), the last K kept + F = the contexts whose last K visits all missed (contexts the expectation drops are dropped) + level = r6(clip(quiet x max(0, 1 - s / quiet_z) + failed x F, 0, cap)) + +Event t's turn reads the level after event t-1 as the number input `cur_level` (box [0, cap], carrying `effects`), and so +does every member of the expectation's mixture (the drive is part of what the engine expects of itself). A host never sends +`cur_level` (an event that does is refused at `inputs.cur_level`); the stance's `inputs` leave it out. `persona turn` and +`persona replay` carry no memory: the level is 0 there. + +The fold lives in the expectation's canonical memory (engine 0.14.0 on): a checkpoint line's `expect` gains +`curiosity: {asleep, ctx: [[key, [miss, ...]], ...], s, since}`, every number at 6 decimals, so a replay and a resume start at +the last checkpoint. A strand of an engine before 0.14.0 has no drive (its level is 0). + +## Where it appears + +- The stance: `drives.curiosity = {failed_contexts, level, surprise_recent}` (the values the turn read; on a stir also + `stir: `). Outside the stance digest, as `deviation`. +- The event line: the same object under `curiosity`; `live verify` recomputes it ("the line differs" on an edit). An engine + before this one reports the line as unsupported (exit 5), never as a divergence. +- `probbit monitor`: the drives rows show `curiosity` with its level, its two inputs, and the stir's class on a stir. +- The Mirror account (RFC 0013): with a declared stir, the reflection document gains `stirs: {class: count}` over the window + and one sentence (count, classes); every other reflection document is byte for byte as before. + +**Why one-sided (measured).** The first build averaged |z|, as the draft named it. On a steady quiet stream the learned +normal's floor (var + 1e-4) turns a slowly falling prediction error into z of about -1 on every event, so |z| read a dark +night as surprising and the quiet term never fired (unit test: the level stayed at or below 0.064833 over 48 identical +night events). A prediction error below the learned normal is the opposite of surprise; the drive reads z+. + +## At rest: integrity and the bound + +The level moves per event, not with the hours, so it is a channel of the moment. RFC 0014's rested state takes the fresh +individual's level (0): integrity's drift measures what persists, not the drive. RFC 0004's certificate cannot assume the +level decays over the rest hours (it does not), so with the block its scope adds the drive's whole shift to the residual: +`d_r += 2 x sum(|effects| on the trait's component) x cap`, as `pace` adds its decayed shift. A persona without the block +gets the certificate of before. + +## The stir + +`Live::stir_due(idle_hours) -> Option` (and `probbit live control STRAND stir? [--idle-hours H]`, which replays the strand +and prints `{due, events, stir}`; nothing is written): + +1. None without a stir block, while paused or retired, or when the level is below `threshold`. +2. Asleep (the last event had `when_asleep.flag` on): None without `when_asleep`; else the asleep class (never speaks) at the + asleep rate. Awake: the declared class at the declared rate. +3. The rate cap: due when no stir was fed yet, or when the hours since the last stir (folded from the stamped hours) plus + `idle_hours` reach the class's minutes. + +A host that acts on a stir feeds an ordinary event carrying `"stir": ""` together with its own current readings (a +body's sensed inputs, the night flag); the turn runs as usual, the inputs on the line carry the marker (the self-started +event, source self), and the fold restarts the rate cap there. A stir the host ignores leaves no line. **Habit precedence:** +the stir is an event like any other, so a habit in force on its inputs binds its stance; a persona that sleeps on a flag +declares the same flag in `when_asleep`, so the only class offered while that habit binds is the quiet one. + +## Bars (measured on a draft PR; numbers in the PR section) + +- B1 exact and replayable: personas without the block unchanged byte for byte (corpora); with it, lives replay byte for byte + on x86_64 and the no_std proxy, checkpoint resume equal, the previous engine reports the strand unsupported (exit 5). +- B2 a dark night: the sensed fixture with a quiet-at-night habit and the drive, over a synthetic night of identical sensed + events: stirs rise from 0 to the declared asleep rate, and the habit binds on every stir. +- B3 the bound: the RFC 0004 certificate recomputed with the drive declared holds (0 exceedances). +- B4 cost: the fold and `stir_due` against a turn. + +## Not in this build + +A drive toward a chosen context (the level is global, not per context); a stir class that names what to look at; a body's +silicon measurement (the stack and cost numbers are from the proxy build). diff --git a/examples/persona/demo/README.md b/examples/persona/demo/README.md new file mode 100644 index 0000000..8539d19 --- /dev/null +++ b/examples/persona/demo/README.md @@ -0,0 +1,51 @@ +# The amplitude register demo (research, RFC 0011) + +One decision, shown twice. A fictional tutor reduced to its humour component lives the same 17 events twice: once on the +default tiers (`thermal.yaml`, the thermal face) and once on the amplitude register (`register.yaml`), an exact simulated +qubit register on a CPU, run with the long sweep (`tf-linear-64x16`) and exact amplification onto the plans its habits allow. +No quantum hardware is involved and no speedup is claimed: every number below is arithmetic, exact and replayable, simulated +on a CPU. + +Two opposed drives pull humour: praise raises it (with valence and emoji), criticism, a correction, lowers it. The life +(`life.jsonl`, seed 1, fixed clock) opens with a run of corrections; at **event 9** praise and a correction arrive together. + +| event 9 (strand line 10), humour | none | light | playful | top two gap | +|---|---|---|---|---| +| thermal (default tiers) | 0.059462 | 0.466563 | 0.473975 | **0.007412** | +| register, default sweep (`tf-linear-16x4`, for comparison) | 0.013180 | 0.566151 | 0.420670 | 0.145481 | +| register, long sweep (`tf-linear-64x16`, this copy) | 0.007171 | 0.249010 | 0.743818 | **0.494808** | + +On the same decision the thermal odds of the top two humour levels sit 0.007412 apart (a near tie: the drives cancel), and +the simulated register separates them by 0.494808: it commits to `playful`, the thermal plurality (the default sweep, four +times shorter, leans the other way, to `light`, by 0.145481). It does not invent a preference the weights do not hold; a +slower sweep keeps more of the register's amplitude on the most probable plans (docs/rfc/0011-amplitude-register.md, +"Schedules and the indifferent set" and "The demo"). Events 2 and 11 are near ties of the same kind (praise and a correction +together: thermal gaps 0.033093 and 0.048852, register gaps 0.656343 and 0.725300); the register is not always the more +decisive face (event 14: thermal gap 0.072051, register gap 0.009221). + +## Files + +| file | what | +|---|---| +| `register.yaml` | the persona with `engine: {tier: amplitude, amplify: true, schedule: tf-linear-64x16}` | +| `thermal.yaml` | the same persona without the engine block (`amplify` and `schedule` need the tier, so all three go) | +| `life.jsonl` | the 17 events | +| `register.strand`, `thermal.strand` | the two recorded lives (`probbit live verify` checks each); event 9 is line 10 of both | + +Record and check them again: + +```sh +probbit live examples/persona/demo/register.yaml --seed 1 --clock fixed --events examples/persona/demo/life.jsonl --strand register.strand +probbit live examples/persona/demo/thermal.yaml --seed 1 --clock fixed --events examples/persona/demo/life.jsonl --strand thermal.strand +probbit live verify register.strand +probbit monitor register.strand --open # the Mood Monitor (docs/persona.md, section 5.8); thermal.strand beside it +``` + +The register strand's documents carry the receipts: `engine.tier` `amplitude`, `engine.schedule_name` `tf-linear-64x16`, +`engine.amplify` (iterations, phases, the mass before and after), `feasible_mass` 1.000000 and the `state` digest of the final +amplitudes at 6 decimals. + +**Replay on every build.** Since round 7 the register computes its math functions with one library on the std and the no_std +build, so the long sweep replays on both: 100 fresh lives per schedule recorded on the std build re-verify on the no_std build +100 of 100 (x86_64; arm64 not measured), and both strands here re-record byte for byte and verify on the no_std build. An +engine without the tier (0.13.0) refuses `register.strand` (exit 5) and verifies `thermal.strand`. diff --git a/examples/persona/demo/life.jsonl b/examples/persona/demo/life.jsonl new file mode 100644 index 0000000..6906ae8 --- /dev/null +++ b/examples/persona/demo/life.jsonl @@ -0,0 +1,17 @@ +{"elapsed_hours":2} +{"criticism":true,"elapsed_hours":4,"praise":true} +{"criticism":true,"elapsed_hours":1} +{"criticism":true,"elapsed_hours":1} +{"criticism":true,"elapsed_hours":4} +{"criticism":true,"elapsed_hours":1} +{"criticism":true,"elapsed_hours":1} +{"criticism":true,"elapsed_hours":1} +{"criticism":true,"elapsed_hours":0,"praise":true} +{"elapsed_hours":1,"praise":true} +{"criticism":true,"elapsed_hours":2,"praise":true} +{"elapsed_hours":0,"praise":true} +{"elapsed_hours":0} +{"criticism":true,"elapsed_hours":1} +{"elapsed_hours":1} +{"criticism":true,"elapsed_hours":4} +{"criticism":true,"elapsed_hours":4} diff --git a/examples/persona/demo/register.strand b/examples/persona/demo/register.strand new file mode 100644 index 0000000..0e7df81 --- /dev/null +++ b/examples/persona/demo/register.strand @@ -0,0 +1,18 @@ +{"probbit_strand":1,"engine":"probbit 0.14.0-dev","persona":{"name":"demo-tutor","version":"0.1.0","digest":"sha256:0f90e48d4690395fb90aa16ca0b71d1cbb34c9aa411d880fe996ca487762d6d3"},"seed":1,"state":{"digest":"sha256:c84048d4325525a85d790f4a919c3226cf5a2c2ff97c274656445bb4adfa8ea7","genes":{"react":{"praise":0.7987},"shift":{"emoji":0.4947,"formality":0.3187,"humour":0.2479,"valence":0.0793,"warmth":0.6261}},"history":{},"mood":{"valence":[0,0,0]},"persona":{"digest":"sha256:0f90e48d4690395fb90aa16ca0b71d1cbb34c9aa411d880fe996ca487762d6d3","name":"demo-tutor","version":"0.1.0"},"prev":{"emoji":"sparse","formality":"casual","humour":"light","valence":"even","warmth":"warm"},"probbit_persona_state":1,"rest":{"emoji":"sparse","formality":"casual","humour":"playful","valence":"up","warmth":"warm"},"seed":1,"turn":0},"document":{"probbit_persona":1,"identity":{"name":"demo-tutor","version":"0.1.0","seed":1,"summary":"a tutor reduced to its humour component, for the amplitude register demo"},"traits":[{"id":"warmth","levels":["cool","neutral","warm"],"prior":[0.05,0.3,0.65],"spread":0.5,"say":["plain","","warm and encouraging"]},{"id":"humour","levels":["none","light","playful"],"prior":[0.15,0.45,0.4],"spread":0.6,"say":["no jokes","a light joke is welcome","playful"]},{"id":"formality","levels":["casual","neutral","formal"],"prior":[0.6,0.35,0.05],"spread":0.3,"say":["casual","","formal"]},{"id":"emoji","levels":["none","sparse","rich"],"prior":[0.2,0.5,0.3],"spread":0.5,"say":["no emoji","one emoji at most","emoji welcome"]}],"moods":[{"id":"valence","levels":["down","even","up"],"prior":[0.1,0.5,0.4],"spread":0.4,"inertia":0.5,"half_life_hours":12}],"couplings":[{"vars":["valence","humour"],"align":1},{"vars":["valence","emoji"],"align":0.6},{"vars":["warmth","formality"],"align":-0.6},{"vars":["humour","formality"],"align":-0.6},{"vars":["humour","emoji"],"align":0.4}],"inputs":[{"id":"praise","kind":"flag","say":"praise","reactivity_spread":0.4,"effects":{"valence":1.2,"humour":0.5,"emoji":0.5}},{"id":"criticism","kind":"flag","say":"criticism","effects":{"valence":-0.6,"humour":-0.4}}],"habits":[{"id":"flourish_budget","rules":{"linear":[{"terms":[["humour","playful",2],["humour","light",1],["emoji","rich",2],["emoji","sparse",1]],"limit":3}]},"say":"one flourish at a time"},{"id":"no_formal_emoji","rules":{"tables":[{"vars":["formality","emoji"],"forbid":[["formal","rich"],["formal","sparse"]]}]},"say":"no emoji when formal"}],"engine":{"tier":"amplitude","amplify":true,"schedule":"tf-linear-64x16"},"line":{"max_tokens":40,"order":["humour","warmth","emoji","formality"]}}} +{"inputs":{"elapsed_hours":2},"n":1,"prev":"sha256:f7f918deac79a089d818d078c2107af9a8551cf0c1e13f501880b1f0310f3055","stance":"sha256:0ea75eaaaad2925f1d840f8f244f15184767e5abb0893693e1c421b9843b293f","state":"sha256:96bd28501be3af6b52e55cb44312577d77bd48e3ff1580fec4dec25e77bcde91"} +{"inputs":{"criticism":true,"elapsed_hours":4,"praise":true},"n":2,"prev":"sha256:0d3c0f851600e4502ce2cd66f5bc3912cb2828a268c09eba4c8cf433e2a23b64","stance":"sha256:15d64c824bef92c0baea4d6217217b6b5ad66cdc52627ec094dcb2543668cd73","state":"sha256:9bae644f633fecbfb9860f19c648ccf991646b2024d60a4faee8a7c0909e9a0f"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":3,"prev":"sha256:69f5fe05b842eb256f07ef09992e352afd0d88722714dfeaafa3b793fb20d26d","stance":"sha256:558208d4b53faa28280b7c8e2a1ad5180d67caeb4dcebf9f9f223fa71a25995d","state":"sha256:5aebd4d81ef411cea573cf951c23455fa1308423461d10e45b9f3501d783eb06"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":4,"prev":"sha256:04621bc267dde076be8068938e9575c66a17ec6536f037f8dad3199a3e034492","stance":"sha256:2ad864a848ec499e8657c289c67b1d8e8afee3e5271eb2d24aeaecbf325f0987","state":"sha256:558b0ccd0cbf5a67d31ad9a991dfae9165536b689c4facba7210edb0ed7fc71d"} +{"inputs":{"criticism":true,"elapsed_hours":4},"n":5,"prev":"sha256:c4c9250a865415d3eb56da2fdb8762dd3b4fc05da37f0b70fd77151fa3352ee1","stance":"sha256:6fc813956b1e08708ee757b12409fbca5eed52fc21a3aa01c005cf5ba6ebba68","state":"sha256:5abdbb412a343850083273df229de2dfb644ee8ddf349e78d4f0b18148652403"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":6,"prev":"sha256:8481f42add2e7de64387488ad014b129612b554f7fdddaf67bb04adeb3c59bcf","stance":"sha256:6f0cc3ce009e3109f4b20452957596790000b4d44650b49d610eab7bc5dd3350","state":"sha256:eb93d29cfb413b01a34dccfa2a4a61f29b6be597de7da6a04ed2913a0c8ed5c8"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":7,"prev":"sha256:81394f399b0e08559d9c0714de9c14094631cf238c0c35faee58365de220a647","stance":"sha256:476759a6ca2e3202f9d25b85fa4da5e5c187f5dc8c6eed104180c4555906f267","state":"sha256:ca12842fe89b3d9ee78cb3244c8540dd600f9d3878e5fa948014e8b1d2502c02"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":8,"prev":"sha256:15d826747f1967a0bd7f7c81a51163b99d31601d87c0d7b3befe6ac356368915","stance":"sha256:2073a8a79dc30b3d012216e987d165b978cbfa35120c3ad537a8bd54490b95a9","state":"sha256:ef66448369e444c52d61d46f091397b0e5cb38bdd964aeda5431dac6742e6da2"} +{"inputs":{"criticism":true,"elapsed_hours":0,"praise":true},"n":9,"prev":"sha256:a9b010dc46e7b37094605c515f7942e38d6e12367ec44bc5e2118da5ed974da7","stance":"sha256:390469686dc6805079079259e69c8d6363cbc6f6a07790687bbc741ce631d641","state":"sha256:852c7969fde1a0ce103be8a354e6e69a2c399c3711be1e5c1e125fac22ab0b87"} +{"inputs":{"elapsed_hours":1,"praise":true},"n":10,"prev":"sha256:75e5cd3fe5f0e41c40adca3a193c766c214d752fd37e7f4119a4d1bc47c33127","stance":"sha256:7061e8197c0bb2cdeae248bafd3479ce2af83b370550f4a6aab2e5dfd6ff2a61","state":"sha256:e57e940272213655514b18d06079e24ab3793cc3a39a8b152c0a8b5756bbeda9"} +{"inputs":{"criticism":true,"elapsed_hours":2,"praise":true},"n":11,"prev":"sha256:3de8cb027dd83fe5b20af21e1b81f7ccfdd57a076528a60bc7b29b055ea95e93","stance":"sha256:9f2eb1cd70dacd6bc0fc1703823ed8705812ca8436806cf5e3163b7023b3f711","state":"sha256:9e3b37108bbe2459b25ca8b57bdbe224370d39780d271209f9be93819dab0fbd"} +{"inputs":{"elapsed_hours":0,"praise":true},"n":12,"prev":"sha256:c6abea7b6d9889993c1fb7d5ed0f08ee8d5c0dafb0dccebcafc52b0fcfda6b65","stance":"sha256:14a8c85a0c15bac274c20df42eea43c1624f617effcee960e4b0406cfb14959a","state":"sha256:5fc5e75832bf7d4f6cc41fd8e17b309c944ae5bf6b83adc1da3b405fb81c7a5e"} +{"inputs":{"elapsed_hours":0},"n":13,"prev":"sha256:81a906e06d49b7e2c4eda69967beeb6d5cd72ba05f65aa6f63f748ec1f76a0a8","stance":"sha256:f8435ba6ab5dc84b145acdafbb4991bec830f457acb113ab486198608112144f","state":"sha256:862d484cbab209ddc35e92f663d4d4df4e64c4fdbf46ede4f446197c5ece2466"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":14,"prev":"sha256:35edb356815aa9c4c0d93ff43fa18e110b8010acdcadae5cccb552d30d7eaa0b","stance":"sha256:27a6f948c0adb0921ed8889cb4462f7d496cb4718951776c4f0170354f9768dc","state":"sha256:d77a30a5c2a0a72569e41cd19fd9f9f9bb58f88c93a90937c397d496ddbe54de"} +{"inputs":{"elapsed_hours":1},"n":15,"prev":"sha256:abb69d92e54ee697203e354c08f423d8b2287d7e1e45d88f0a34173f82ea0b21","stance":"sha256:ab8a1ae0a39f34c8adc6fc22d8dc518d4d703fbea724d88c9447a00ede3fb829","state":"sha256:1cb4c8f07d6084365a1d5014605a004123d40afb218ecef33be5a0f4d696490f"} +{"inputs":{"criticism":true,"elapsed_hours":4},"n":16,"prev":"sha256:b817992a9ee62b66b8fe03c7e17ad829a19ae84483a807ff7411ceb7e4a065e2","stance":"sha256:bddc0d05e986a851acd237f66b6b05c7ab0faf1d5a116b76367d901837607e7a","state":"sha256:00948463ada05fad5d4874e98383cb471a1698fc08e526af423bf75703601925"} +{"inputs":{"criticism":true,"elapsed_hours":4},"n":17,"prev":"sha256:d85613c8c1c24c6b4d6670fff73593305807d9d6e6020a44dd98abe388d6ba15","stance":"sha256:4be10566f4e0e16ca456d04c82e4706005f88525959c58cd0af5aeae98360e51","state":"sha256:e63a0e17d618492147c7a573b28db4f70f893b1350f9e403b7dffe037cbf2201"} diff --git a/examples/persona/demo/register.yaml b/examples/persona/demo/register.yaml new file mode 100644 index 0000000..bc4c3da --- /dev/null +++ b/examples/persona/demo/register.yaml @@ -0,0 +1,78 @@ +# A fictional persona for the RFC 0011 demo (research, off by default elsewhere): a tutor reduced to its humour component. +# probbit persona format 1 (docs/persona.md). Two opposed drives pull humour: praise raises it, criticism (a correction) +# lowers it. This copy solves each turn on the amplitude register: an exact simulated qubit register on a CPU, run with the +# long sweep (tf-linear-64x16) and exact amplification onto the plans the habits allow. thermal.yaml is the same persona on +# the default tiers. No quantum hardware, no speedup claim. See README.md and docs/rfc/0011-amplitude-register.md, "The demo". +probbit_persona: 1 +identity: + name: demo-tutor + version: 0.1.0 + seed: 1 + summary: a tutor reduced to its humour component, for the amplitude register demo + +traits: + - id: warmth + levels: [cool, neutral, warm] + prior: [0.05, 0.3, 0.65] + spread: 0.5 + say: [plain, "", warm and encouraging] + - id: humour + levels: [none, light, playful] + prior: [0.15, 0.45, 0.4] + spread: 0.6 + say: [no jokes, a light joke is welcome, playful] + - id: formality + levels: [casual, neutral, formal] + prior: [0.6, 0.35, 0.05] + spread: 0.3 + say: [casual, "", formal] + - id: emoji + levels: [none, sparse, rich] + prior: [0.2, 0.5, 0.3] + spread: 0.5 + say: [no emoji, one emoji at most, emoji welcome] + +moods: + - id: valence + levels: [down, even, up] + prior: [0.1, 0.5, 0.4] + spread: 0.4 + inertia: 0.5 + half_life_hours: 12 + +couplings: + - {vars: [valence, humour], align: 1.0} + - {vars: [valence, emoji], align: 0.6} + - {vars: [warmth, formality], align: -0.6} + - {vars: [humour, formality], align: -0.6} + - {vars: [humour, emoji], align: 0.4} + +inputs: + - id: praise + kind: flag + say: praise + reactivity_spread: 0.4 + effects: {valence: 1.2, humour: 0.5, emoji: 0.5} + - id: criticism + kind: flag + say: criticism + effects: {valence: -0.6, humour: -0.4} + +habits: + - id: flourish_budget + rules: + linear: [{terms: [[humour, playful, 2], [humour, light, 1], [emoji, rich, 2], [emoji, sparse, 1]], limit: 3}] + say: one flourish at a time + - id: no_formal_emoji + rules: + tables: [{vars: [formality, emoji], forbid: [[formal, rich], [formal, sparse]]}] + say: no emoji when formal + +engine: + tier: amplitude # research (RFC 0011): an exact simulated qubit register on a CPU + amplify: true # research (round 4): exact amplification onto the plans the habits allow (a receipt change) + schedule: tf-linear-64x16 # research (round 5): the long sweep; replays on the recording (std) build only for now + +line: + max_tokens: 40 + order: [humour, warmth, emoji, formality] diff --git a/examples/persona/demo/thermal.strand b/examples/persona/demo/thermal.strand new file mode 100644 index 0000000..13d64e9 --- /dev/null +++ b/examples/persona/demo/thermal.strand @@ -0,0 +1,18 @@ +{"probbit_strand":1,"engine":"probbit 0.14.0-dev","persona":{"name":"demo-tutor","version":"0.1.0","digest":"sha256:f63a2a990dce5c0290a3fb9d3148f945b9ba985b9a50897b33ac7e9589f44f47"},"seed":1,"state":{"digest":"sha256:8442eba81d5ab11311bd7367c272ef54da3963c18e0b6f7a044e788588202f2c","genes":{"react":{"praise":0.7987},"shift":{"emoji":0.4947,"formality":0.3187,"humour":0.2479,"valence":0.0793,"warmth":0.6261}},"history":{},"mood":{"valence":[0,0,0]},"persona":{"digest":"sha256:f63a2a990dce5c0290a3fb9d3148f945b9ba985b9a50897b33ac7e9589f44f47","name":"demo-tutor","version":"0.1.0"},"prev":{"emoji":"sparse","formality":"casual","humour":"light","valence":"even","warmth":"warm"},"probbit_persona_state":1,"rest":{"emoji":"sparse","formality":"casual","humour":"playful","valence":"up","warmth":"warm"},"seed":1,"turn":0},"document":{"probbit_persona":1,"identity":{"name":"demo-tutor","version":"0.1.0","seed":1,"summary":"a tutor reduced to its humour component, for the amplitude register demo"},"traits":[{"id":"warmth","levels":["cool","neutral","warm"],"prior":[0.05,0.3,0.65],"spread":0.5,"say":["plain","","warm and encouraging"]},{"id":"humour","levels":["none","light","playful"],"prior":[0.15,0.45,0.4],"spread":0.6,"say":["no jokes","a light joke is welcome","playful"]},{"id":"formality","levels":["casual","neutral","formal"],"prior":[0.6,0.35,0.05],"spread":0.3,"say":["casual","","formal"]},{"id":"emoji","levels":["none","sparse","rich"],"prior":[0.2,0.5,0.3],"spread":0.5,"say":["no emoji","one emoji at most","emoji welcome"]}],"moods":[{"id":"valence","levels":["down","even","up"],"prior":[0.1,0.5,0.4],"spread":0.4,"inertia":0.5,"half_life_hours":12}],"couplings":[{"vars":["valence","humour"],"align":1},{"vars":["valence","emoji"],"align":0.6},{"vars":["warmth","formality"],"align":-0.6},{"vars":["humour","formality"],"align":-0.6},{"vars":["humour","emoji"],"align":0.4}],"inputs":[{"id":"praise","kind":"flag","say":"praise","reactivity_spread":0.4,"effects":{"valence":1.2,"humour":0.5,"emoji":0.5}},{"id":"criticism","kind":"flag","say":"criticism","effects":{"valence":-0.6,"humour":-0.4}}],"habits":[{"id":"flourish_budget","rules":{"linear":[{"terms":[["humour","playful",2],["humour","light",1],["emoji","rich",2],["emoji","sparse",1]],"limit":3}]},"say":"one flourish at a time"},{"id":"no_formal_emoji","rules":{"tables":[{"vars":["formality","emoji"],"forbid":[["formal","rich"],["formal","sparse"]]}]},"say":"no emoji when formal"}],"line":{"max_tokens":40,"order":["humour","warmth","emoji","formality"]}}} +{"inputs":{"elapsed_hours":2},"n":1,"prev":"sha256:67aef832803e96fc96d37f73fe25d8c26effc0a8c43ce9d4d2a613607017a2bc","stance":"sha256:838aefc4a6cae9d87efd37366f83697fb07c0f09d8b319cb279df92b48af7699","state":"sha256:961f1e5a1e85c4f904a997dbddc194406c40153d6bfc35c72fc2a47eec4e2e28"} +{"inputs":{"criticism":true,"elapsed_hours":4,"praise":true},"n":2,"prev":"sha256:684c4505b81eaa3d095c0356fdcf2032a848889d3b956ca3bfdaef584b654d84","stance":"sha256:a11cf3d94c1397aa720ec97b9ec0061f5c1ae2a4abd3b9eb25f415e8e1e3c3ab","state":"sha256:2a54fbe4288035defa4bf85e00017631062f1813d047bb5626051a144921ee05"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":3,"prev":"sha256:a5e8b2ccf957d240cd780eda5de1b3fd3ea2555fa0b4e3d9645aa42647e373ca","stance":"sha256:3df181cb189f1e8d55f8c68b95db7ec1dd29084e9ac7d545100d527dad863462","state":"sha256:4ccb76820044ec00bbde3741e4a840639369a14e06fc4b848cd81f2ab377e331"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":4,"prev":"sha256:90297f13f0eaeb0e349aabf0cf9ce67cd427928f0b08bf5a35c627af1d42903d","stance":"sha256:06141011e3c0aaba94fb6b8d7810a14a87b139d4a00486a2696742802993c943","state":"sha256:52510a0e4fea2f04196846b2d02953473ffd0948b0e237936ba2b33430b86605"} +{"inputs":{"criticism":true,"elapsed_hours":4},"n":5,"prev":"sha256:89268a6942a74765a9272176f884894490e9b7f3198c93ba154900a3da840c79","stance":"sha256:e966521e32cadcbf9b6602499935f5bae877eb9904b1a4b37931e99d3234c471","state":"sha256:947106cafbed21b56cea4b100e2d28910063801192112a325b11d9c76e38fe6a"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":6,"prev":"sha256:62fefaeeb8143826f699f4e30d6013963273ee6eee71cf618120fe0726beb2b0","stance":"sha256:e4b1dbe7aa9da19d39b93c66bce25b951fba68f74d017971dbc1a1af35e7cbfe","state":"sha256:c86b66ef6d7b2ad86c471b729c4cc8ef20b236d131eabf12125c861ec67f6a40"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":7,"prev":"sha256:5507d0ecd8ae5252fde735d41766b5210158cb768c810f740b611aa69eadb908","stance":"sha256:39ed1c3015adf01cc7acd31447d0e68dfaadcd921f5c7f4c300f8dac309b50e0","state":"sha256:15a93dec6649e8761ff6988bdf375eaddc3ef4ece242265d408144039d77c8f6"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":8,"prev":"sha256:676480fca7616f81bb0cfbbc626c0ff5f5e22afe4eb0e0080effbbcf430a0d9f","stance":"sha256:27e0f52ebe689dc3c99fc394b1cb95dda66326412c1f434878ed0ca30d311e40","state":"sha256:70d7f743627196ec4045afe54bb3b0f0f1de2e75697a543075e4ff48517490c3"} +{"inputs":{"criticism":true,"elapsed_hours":0,"praise":true},"n":9,"prev":"sha256:bc04904c165cfe76b8c59f226c01e8357b32bbd34783d4d450b19f0640b828fc","stance":"sha256:944c0a26385c27a5ccf35b0e21459b1cc5f5f60f3320a5fbaad22aa50552930a","state":"sha256:7e35e79810e249b30af48a8c6457028a0455a8ec25d3fc84a45e3261f7d08d43"} +{"inputs":{"elapsed_hours":1,"praise":true},"n":10,"prev":"sha256:1f6de80d7bdb4fdd26baec686f120d9916c346951e8c796eb99440466e603768","stance":"sha256:00e1ac558fc6fbf6c04fab4bf406a4dd4da3f1ca9070471a8044a23d4a6dfc1b","state":"sha256:df0cf433b0c57044e3d9edfa1e9f521d1f7d6fa49dd71e7d3df37899420bbb76"} +{"inputs":{"criticism":true,"elapsed_hours":2,"praise":true},"n":11,"prev":"sha256:6335744d8462b3905ced3cc0981affc24e56ce1998764b018aa35893067b6516","stance":"sha256:ef62f91b701fc242b26b9992780d97b4c28084d01754b90eef3bef7ae0acef39","state":"sha256:3579d66c48d8cf50df61c9ab1bd4fffc0002e7225e9db3e3ae08f35ce73c91cd"} +{"inputs":{"elapsed_hours":0,"praise":true},"n":12,"prev":"sha256:25aa44fb4fe28fe86b62641afeb4157882a5554a3db23e7e1fe231481d06f79e","stance":"sha256:01cb4eca2af8d84540275cbea13364903342fe663755a950d4f341a186243786","state":"sha256:ed8b0c8dd993f2541cdfe986ee16b7f304af5dd28f818a808f0ebf54937a9023"} +{"inputs":{"elapsed_hours":0},"n":13,"prev":"sha256:cb5a0222b37e05c37bfe2b1414d4916dc0d28b2c6bbcc9c0a634a83e7995260d","stance":"sha256:106891ba89bdb843de3ac3990f3fe2d9cb4fdbf430270a13b6b44589aeb89abe","state":"sha256:e304fc76b0218fb96bfdd691e34a224acf47ee100527b325d160c266b8c651af"} +{"inputs":{"criticism":true,"elapsed_hours":1},"n":14,"prev":"sha256:4fa4c61ee945583a8f7e469ada36992374a36c665e35ce4db5ca0c6abd08ecf5","stance":"sha256:738deedd442f853a783f9e9d8622ef230c38e665f878114cbdaf1c0c47d3fc4a","state":"sha256:c21e698198fa9034915279a26974ab54faabfa11c6b5dde1c8e94a1a8e81e699"} +{"inputs":{"elapsed_hours":1},"n":15,"prev":"sha256:1689bbc90fd8f3bc326167de45536deb549351c7e6a22e8152b7036103c72db7","stance":"sha256:74ba46799b86f3174f9aa957fbb3f9016bb31175805c4ad13429f7122b28e5c8","state":"sha256:dba20f77fc79cc516040614baf77d9be139d316e0a71c22590963385a39d5fbe"} +{"inputs":{"criticism":true,"elapsed_hours":4},"n":16,"prev":"sha256:9238433fe1e0a0eff7592a1e5d15b45c6ab0e7176abdfc3c2462991c5ba2ffd4","stance":"sha256:b961d8cc9db2e80c4f264be1789c5a5fc39035af6bf68e6ab57bea2bb84506d5","state":"sha256:be90cd8467f85d215724231e1df674f74c12a57c47995fcb66c18d900311cdef"} +{"inputs":{"criticism":true,"elapsed_hours":4},"n":17,"prev":"sha256:6b75ed2e5ea4ac0c2c6d22eba956c12aa3c6d1d1536296676c33ba1d6235669b","stance":"sha256:89fe74fb1b8a54c0bdd346313fd0600f43036611a23fa815b8da96afae00a8a9","state":"sha256:fabd63a75c5564c24e68b015c5ffb8428de219888226d72f5c5f6534eb28e87f"} diff --git a/examples/persona/demo/thermal.yaml b/examples/persona/demo/thermal.yaml new file mode 100644 index 0000000..503f652 --- /dev/null +++ b/examples/persona/demo/thermal.yaml @@ -0,0 +1,71 @@ +# A fictional persona for the RFC 0011 demo: register.yaml on the default tiers (the thermal face). probbit persona format 1. +# Identical to register.yaml except that the engine block is absent: engine.amplify and engine.schedule need engine.tier, so +# removing the tier removes all three. Every engine since 0.13.0 runs this copy. See README.md. +probbit_persona: 1 +identity: + name: demo-tutor + version: 0.1.0 + seed: 1 + summary: a tutor reduced to its humour component, for the amplitude register demo + +traits: + - id: warmth + levels: [cool, neutral, warm] + prior: [0.05, 0.3, 0.65] + spread: 0.5 + say: [plain, "", warm and encouraging] + - id: humour + levels: [none, light, playful] + prior: [0.15, 0.45, 0.4] + spread: 0.6 + say: [no jokes, a light joke is welcome, playful] + - id: formality + levels: [casual, neutral, formal] + prior: [0.6, 0.35, 0.05] + spread: 0.3 + say: [casual, "", formal] + - id: emoji + levels: [none, sparse, rich] + prior: [0.2, 0.5, 0.3] + spread: 0.5 + say: [no emoji, one emoji at most, emoji welcome] + +moods: + - id: valence + levels: [down, even, up] + prior: [0.1, 0.5, 0.4] + spread: 0.4 + inertia: 0.5 + half_life_hours: 12 + +couplings: + - {vars: [valence, humour], align: 1.0} + - {vars: [valence, emoji], align: 0.6} + - {vars: [warmth, formality], align: -0.6} + - {vars: [humour, formality], align: -0.6} + - {vars: [humour, emoji], align: 0.4} + +inputs: + - id: praise + kind: flag + say: praise + reactivity_spread: 0.4 + effects: {valence: 1.2, humour: 0.5, emoji: 0.5} + - id: criticism + kind: flag + say: criticism + effects: {valence: -0.6, humour: -0.4} + +habits: + - id: flourish_budget + rules: + linear: [{terms: [[humour, playful, 2], [humour, light, 1], [emoji, rich, 2], [emoji, sparse, 1]], limit: 3}] + say: one flourish at a time + - id: no_formal_emoji + rules: + tables: [{vars: [formality, emoji], forbid: [[formal, rich], [formal, sparse]]}] + say: no emoji when formal + +line: + max_tokens: 40 + order: [humour, warmth, emoji, formality] diff --git a/probbit-cli/Cargo.toml b/probbit-cli/Cargo.toml index 134e69a..c8c342e 100644 --- a/probbit-cli/Cargo.toml +++ b/probbit-cli/Cargo.toml @@ -14,7 +14,12 @@ categories = ["command-line-utilities", "algorithms", "science"] name = "probbit" path = "src/main.rs" +[features] +# `probbit live` with PROBBIT_TURN_TRACE set prints each event's phases to stderr (probbit-persona `turn_trace`) +turn_trace = ["probbit-persona/turn_trace"] + [dependencies] -probbit-core = { path = "../probbit-core", version = "0.13.0" } -probbit-decide = { path = "../probbit-decide", version = "0.13.0" } -probbit-ir = { path = "../probbit-ir", version = "0.13.0" } +probbit-core = { path = "../probbit-core", version = "0.14.0-dev" } +probbit-decide = { path = "../probbit-decide", version = "0.14.0-dev" } +probbit-ir = { path = "../probbit-ir", version = "0.14.0-dev" } +probbit-persona = { path = "../probbit-persona", version = "0.14.0-dev" } diff --git a/probbit-cli/src/live.rs b/probbit-cli/src/live.rs index c84badd..339bb88 100644 --- a/probbit-cli/src/live.rs +++ b/probbit-cli/src/live.rs @@ -9,557 +9,14 @@ //! The safety kit (§5.7): one writer per strand (`STRAND.lock`), control lines (pause, resume, retire: appended by `probbit live //! control`, never by an event; no credit crosses them) and checkpoint lines (the whole state every K events, so a replay can //! start there). `verify` replays all three. +//! +//! The pure half (the line writers, `Live`, `Replay`, `verify`, the chain checks) is `probbit_persona::strand`, so a firmware or +//! another runtime writes and checks the same strand; this file is the CLI's half: strand files, the writer lock, the tools of +//! `probbit mcp` and the demo week. +pub use probbit_persona::strand::*; use crate::json::{self, InErr, Json}; use crate::persona::{self, perr, Engine, Persona, State}; -/// The strand format -const FORMAT: f64 = 1.0; -/// The clock. `Real`: a monotonic clock started with the run; each event's elapsed hours are the time since the previous event -/// (the opening event: since the start). `Fixed`: time comes from the events, each carrying its own `elapsed_hours` (default 0), so a -/// run is a pure function of its events. -pub enum Clock { Real(std::time::Instant), Fixed } -/// Hours quantised to 1e-6 h (3.6 ms): the value the turn uses is the value the strand logs -fn q6(h: f64) -> f64 { persona::r6(h) } - -/// A strand's life as its control lines leave it: events are answered only while `Active` -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub enum Status { Active, Paused, Retired } -impl Status { - pub fn name(self) -> &'static str { match self { Status::Active => "active", Status::Paused => "paused", Status::Retired => "retired" } } - /// The status a control line leads to: pause an active individual, resume a paused one, retire either; nothing after retire - pub fn after(self, what: &str) -> Result { - match (what, self) { - (_, Status::Retired) => Err(refusal(Status::Retired)), - ("pause", Status::Active) => Ok(Status::Paused), ("resume", Status::Paused) => Ok(Status::Active), ("retire", _) => Ok(Status::Retired), - // RFC 0004 amendment A2: `erase` returns the learned weights to zero and leaves the status as it was - ("erase", s) => Ok(s), - ("pause", _) => Err(perr("control", "already paused")), ("resume", _) => Err(perr("control", "not paused (resume follows a pause)")), - _ => Err(perr("control", "pause | resume | retire")) } - } -} -/// The refusal of an event (or a control line) by a paused or retired individual: code `paused` / `retired` (`live` exits 4) -fn refusal(s: Status) -> InErr { - let msg = if s == Status::Retired { "the individual is retired: every event is refused, for good (probbit live control ... retire)" } - else { "the individual is paused: every event is refused until `probbit live control STRAND resume`" }; - InErr { code: s.name(), path: "strand".into(), msg: msg.into() } -} -/// The default number of events between two checkpoint lines -pub const CHECKPOINT_EVERY: u64 = 1000; - -/// A live individual: the persona, the current state, the clock, the strand's chain and its status (control lines) -pub struct Live { pub p: Persona, pub st: State, clock: Clock, last: f64, prev: String, pub n: u64, pub status: Status, pub controls: u64, pub checkpoints: u64, pub rests: u64, - /// RFC 0009: the expectation's memory (a fold of the logged events) and the last event's exact trajectory (derived, never logged) - pub expect: Option, pub traj: Option } -/// A document with its keys sorted at every level (`persona::text` of it is its canonical JSON) -fn sorted(j: &Json) -> Json { - match j { Json::Obj(v) => { let mut o: Vec<(String, Json)> = v.iter().map(|(k, x)| (k.clone(), sorted(x))).collect(); o.sort_by(|a, b| a.0.cmp(&b.0)); Json::Obj(o) } - Json::Arr(v) => Json::Arr(v.iter().map(sorted).collect()), x => x.clone() } -} -/// One line of a strand without its line ending: strands are written with `\n`; a `\r\n` copy (Windows text mode, an editor) -/// reads the same, since a canonical JSON line never ends in `\r` itself -fn bare(l: &str) -> &str { l.strip_suffix('\r').unwrap_or(l) } -impl Live { - /// Start a life from `st` (an `init` state or a stored one) -> (the individual, the strand's header line). The header keeps - /// the persona document in its own key order (the compiled program follows it, and `engine.program` with it) and the - /// state as canonical JSON (keys sorted), so a state read from a file and the same state from `init` give one header; the - /// header is compact JSON, not key-sorted. - pub fn start(p: Persona, doc: &Json, st: State, clock: Clock, engine: &str) -> (Live, String) { - let s = |x: &str| Json::Str(x.to_string()); - let header = persona::text(&Json::Obj(vec![("probbit_strand".into(), Json::Num(FORMAT)), ("engine".into(), s(engine)), - ("persona".into(), Json::Obj(vec![("name".into(), s(&p.name)), ("version".into(), s(&p.version)), ("digest".into(), s(&p.digest))])), - ("seed".into(), Json::Num(st.seed as f64)), ("state".into(), sorted(&st.to_json(&p))), ("document".into(), doc.clone())])); - let prev = persona::digest_of(&header); - let expect = p.expect.as_ref().map(|_| persona::ExpectMem::new()); - (Live { p, st, clock, last: 0.0, prev, n: 0, status: Status::Active, controls: 0, checkpoints: 0, rests: 0, expect, traj: None }, header) - } - /// One event: stamp its elapsed hours, run the turn, chain the strand line -> (the stance document, the strand line). A bad - /// event is an error and changes nothing (no turn, no line); so is every event while the individual is paused or retired. - pub fn event(&mut self, ev: &Json, eng: Engine) -> Result<(Json, String), InErr> { - if self.status != Status::Active { return Err(refusal(self.status)); } - let Json::Obj(kv) = ev else { return Err(perr("event", "must be a JSON object of inputs")) }; - let mut kv: Vec<(String, Json)> = kv.iter().filter(|(k, _)| k != "elapsed_hours").cloned().collect(); - let given = ev.get("elapsed_hours"); - let (elapsed, now) = match &self.clock { - Clock::Real(t0) => { if given.is_some() { return Err(perr("event.elapsed_hours", "the clock stamps it (--clock fixed reads it from the events)")); } - let now = t0.elapsed().as_secs_f64() / 3600.0; (q6((now - self.last).max(0.0)), now) } - Clock::Fixed => { let e = match given { None => 0.0, Some(Json::Num(x)) if x.is_finite() && *x >= 0.0 => *x, _ => return Err(perr("event.elapsed_hours", "a number >= 0")) }; (q6(e), self.last) } }; - kv.push(("elapsed_hours".into(), Json::Num(elapsed))); - let inputs = Json::Obj(kv); - // RFC 0009: the deviation and the trajectory are read from the state before the event (effects {}: the turn is as without them) - let dev = self.expect.as_ref().and_then(|m| persona::deviation(&self.p, &self.st, &inputs, m, eng)); - let traj = persona::trajectory(&self.p, &self.st, &inputs); - let (mut stance, ns) = persona::turn(&self.p, &self.st, &inputs, false, eng, false)?; - let sd = persona::stance_sha(&stance); - if self.expect.is_some() { if let Json::Obj(v) = &mut stance { v.push(("deviation".into(), dev.as_ref().map_or(Json::Null, persona::Deviation::to_json))); } } - // §2.18: the glyph reads the finished document, `deviation` included (the surprise modifier) - persona::set_glyph(&self.p, &mut stance); - // RFC 0002 part 2: a rest line due before this event is its own line, chained before the event line (the turn applied it) - let (rest, prev) = match persona::rest_line(&self.p, &self.st, &inputs) { - Some((r, rs)) => { let l = rest_line(&self.prev, r, &rs.digest); let d = persona::digest_of(&l); (Some(l), d) } - None => (None, self.prev.clone()) }; - let mut lj = vec![("n".into(), Json::Num((self.n + 1) as f64)), ("inputs".into(), inputs.clone()), ("stance".into(), Json::Str(sd)), - ("state".into(), Json::Str(ns.digest.clone())), ("prev".into(), Json::Str(prev))]; - if let Some(d) = &dev { lj.push(("deviation".into(), d.receipt())); } - if let Some(g) = stance.get("glyph") { lj.push(("glyph".into(), g.clone())); } - let line = persona::canon(&Json::Obj(lj)); - if let Some(m) = self.expect.as_mut() { persona::expect_step(&self.p, m, &inputs, dev.as_ref().and_then(persona::Deviation::logged_pred)); } - self.traj = traj; - self.n += 1; self.last = now; self.prev = persona::digest_of(&line); self.st = ns; - if let Some(r) = &rest { self.rests += 1; return Ok((stance, format!("{r}\n{line}"))); } - Ok((stance, line)) - } - /// A control line (pause, resume or retire) `by` a person or a sensor (`human[:id]`, `env[:id]`) for `reason`, stamped `at`: - /// the status moves, the credit is zeroed (no feedback after the line credits a stance before it) and the line is chained -> - /// the line. Nothing else changes: moods, drives, learned weights and the clock are the individual's as before. - pub fn control(&mut self, what: &str, by: &str, reason: &str, at: &str) -> Result { - let to = self.status.after(what)?; - let line = control_line(&self.prev, what, by, reason, at)?; - self.st = self.st.zero_credit(&self.p); if what == "erase" { self.st = self.st.erased(&self.p); } self.status = to; self.controls += 1; self.prev = persona::digest_of(&line); - Ok(line) - } - /// A checkpoint line after event n: the event count, the whole state after it and that event's stance document, chained -> - /// the line. A replay can start there (the state) and a board can be drawn there (the stance) without the events before. - pub fn checkpoint(&mut self, stance: &Json) -> String { - let line = persona::canon(&Json::Obj(vec![("checkpoint".into(), Json::Num(self.n as f64)), ("prev".into(), Json::Str(self.prev.clone())), ("stance".into(), stance.clone()), - ("state".into(), self.st.to_json(&self.p))])); - self.checkpoints += 1; self.prev = persona::digest_of(&line); line - } - /// The sha256 of the strand's last line (the header's before any event) - pub fn head(&self) -> &str { &self.prev } -} -/// A control line after the line whose sha256 is `prev` (canonical JSON: at, by, control, prev, reason). `by` names who: a -/// person or a sensor, never the individual or the clock; the reason is 1-500 characters. -pub fn control_line(prev: &str, what: &str, by: &str, reason: &str, at: &str) -> Result { - if !["pause", "resume", "retire", "erase"].contains(&what) { return Err(perr("control", "pause | resume | retire")); } - if !matches!(persona::src_kind(by), Some("human" | "env")) { return Err(perr("control.by", "who: human[:id] or env[:id] (id: 1-64 of A-Z a-z 0-9 _ . - @ / :); the individual cannot control itself")); } - if reason.trim().is_empty() || reason.chars().count() > 500 { return Err(perr("control.reason", "1-500 characters")); } - if at.is_empty() || at.len() > 64 { return Err(perr("control.at", "a time stamp, 1-64 characters (RFC 3339 by default)")); } - let s = |x: &str| Json::Str(x.to_string()); - Ok(persona::canon(&Json::Obj(vec![("at".into(), s(at)), ("by".into(), s(by)), ("control".into(), s(what)), ("prev".into(), s(prev)), ("reason".into(), s(reason))]))) -} -/// A rest line (RFC 0002 part 2) after the line whose sha256 is `prev` (canonical JSON: prev, rest, state): what the line did -/// (downscale, prune, the pressure, the learned weights after it, each gate as kept) and the digest of the state after it. It is -/// written only together with the event it precedes, and that event's line chains to it. -fn rest_line(prev: &str, rest: Json, state: &str) -> String { - persona::canon(&Json::Obj(vec![("prev".into(), Json::Str(prev.into())), ("rest".into(), rest), ("state".into(), Json::Str(state.into()))])) -} -/// What kind of strand line `j` is -enum Kind { Event, Control, Checkpoint, Rest } -fn kind(j: &Json) -> Kind { if j.get("control").is_some() { Kind::Control } else if j.get("checkpoint").is_some() { Kind::Checkpoint } else if j.get("rest").is_some() { Kind::Rest } else { Kind::Event } } -/// The header's keys, as this engine writes them -const HEADER: [&str; 6] = ["probbit_strand", "engine", "persona", "seed", "state", "document"]; -/// The earliest key of a line after the header that its kind does not have (the line's keys in canonical order): a line kind -/// this engine does not know (a line without `control` or `checkpoint` is read as an event line, so `{"prev": ..., "nap": ...}` -/// gives `nap`) or a field it does not know on a known one -fn unknown_key(j: &Json) -> Option<&str> { - let known: &[&str] = match kind(j) { Kind::Event => &["deviation", "glyph", "inputs", "n", "prev", "stance", "state"], Kind::Control => &["at", "by", "control", "prev", "reason"], - Kind::Checkpoint => &["checkpoint", "prev", "stance", "state"], Kind::Rest => &["prev", "rest", "state"] }; - j.as_obj()?.iter().map(|(k, _)| k.as_str()).find(|k| !known.contains(k)) -} -/// The start of a replay's `why` that is not a divergence: the line has a field or a line kind this engine does not know (a -/// strand from a newer engine). Nothing from that line on was checked: the strand is not verified, and not shown to differ. -/// `verify` reports it as `unsupported` (exit 5), never as `diverges`; a changed value of a field this engine knows still -/// diverges. -pub const UNSUPPORTED: &str = "unsupported by this engine: "; -/// The `why` of an unsupported field: UNSUPPORTED, then its path (a persona field as the persona validator names it, a state -/// field under `state.`, otherwise its path in the strand line) -fn unsupported(path: &str) -> String { format!("{UNSUPPORTED}{path}") } -/// The field path of an unsupported `why`; None for a divergence -pub fn unsupported_field(why: &str) -> Option<&str> { why.strip_prefix(UNSUPPORTED) } -/// A persona, state or event error for a field this engine does not know (the validators' "unknown field" at its path) -> the -/// unsupported `why`; any other error -> `what` + the error, a divergence -fn read_err(e: InErr, what: &str) -> String { if e.msg.starts_with("unknown field") { unsupported(&e.path) } else { format!("{what}{}: {}", e.path, e.msg) } } -/// Now as RFC 3339 UTC to the second (the control lines' default `at`) -pub fn utc_now() -> String { - let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_secs()); - let (z, t) = ((secs / 86_400) as i64 + 719_468, secs % 86_400); // days since 0000-03-01 (civil-from-days) - let (era, doe) = (z.div_euclid(146_097), z.rem_euclid(146_097)); let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365; - let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); let mp = (5 * doy + 2) / 153; let d = doy - (153 * mp + 2) / 5 + 1; - let m = if mp < 10 { mp + 3 } else { mp - 9 }; let y = yoe + era * 400 + i64::from(m <= 2); - format!("{y:04}-{m:02}-{d:02}T{:02}:{:02}:{:02}Z", t / 3600, t % 3600 / 60, t % 60) -} - -/// A strand replayed line by line: `verify` and `probbit monitor` (§5.8) share it, so both read a strand with one set of -/// rules. `open` checks the header and rebuilds the individual; each `step` replays one event line on the fixed clock and -/// checks it (`prev`, `n`, the stance and state digests, the bytes). After an error the replay is spent: the chain past the -/// line that differs cannot be checked. -pub struct Replay { pub live: Live, pub header: Json, pub doc: Json, pub line: usize, pub from: Option, last: Option, checkpoint_ready: bool, norms: Normals, at_cap: Vec, seen: (f64, Vec), viab: Option<(f64, Option)>, rest: Option } -/// A senses block's normals over the replay (§2.11): per sense the means after the last 50 events, the largest move of a mean -/// within 50 events so far, and the latest event's move (the monitor's alert: over drift_cap / 2) -#[derive(Default)] -pub struct Normals { window: Vec>, pub max_move: Vec, pub moved: Vec } -impl Normals { - fn start(p: &Persona, st: &State) -> Normals { - let n = persona::normals(p, st); - Normals { window: n.iter().map(|x| std::collections::VecDeque::from(vec![x.1])).collect(), max_move: vec![0.0; n.len()], moved: vec![0.0; n.len()] } - } - fn track(&mut self, p: &Persona, st: &State) { - for (k, (_, m, _, _)) in persona::normals(p, st).iter().enumerate() { let q = &mut self.window[k]; - let mv = persona::r6(q.iter().fold(0.0f64, |a, x| a.max((m - x).abs()))); - self.moved[k] = mv; self.max_move[k] = self.max_move[k].max(mv); q.push_back(*m); if q.len() > 50 { q.pop_front(); } } - } -} -impl Replay { - /// The header line (without its line ending) -> the replay, ready for event lines; Err((1, what differs)), or Err((1, the - /// unsupported `why`)) for a header key, persona field or state field this engine does not know - pub fn open(head: &str) -> Result { - let h = json::parse(head).map_err(|e| (1, format!("the header is not JSON: {}", e.msg)))?; - if h.get("probbit_strand").and_then(Json::as_f64) != Some(FORMAT) { return Err((1, "not a probbit strand (format 1)".into())); } - if let Some((k, _)) = h.as_obj().and_then(|kv| kv.iter().find(|(k, _)| !HEADER.contains(&k.as_str()))) { return Err((1, unsupported(k))); } - let doc = h.get("document").cloned().ok_or((1, "the header has no persona document".to_string()))?; - let p = persona::build(&doc).map_err(|e| (1, read_err(e, "the persona document: ")))?; - if h.get("persona").and_then(|x| x.get("digest")).and_then(Json::as_str) != Some(p.digest.as_str()) { return Err((1, "the persona digest does not match the document".into())); } - let st0 = State::read(&p, h.get("state").unwrap_or(&Json::Null)).map_err(|e| (1, read_err(e, "the initial state: ")))?; - let (live, header) = Live::start(p, &doc, st0, Clock::Fixed, h.get("engine").and_then(Json::as_str).unwrap_or("")); - if header != head { return Err((1, "the header is not as written".into())); } - let norms = Normals::start(&live.p, &live.st); let at_cap = vec![0.0; persona::deficits(&live.p, &live.st).len()]; let seen = (0.0, vec![0.0; persona::gate_gains(&live.p, &live.st).len()]); - let viab = viab_start(&live); - Ok(Replay { live, header: h, doc, line: 1, from: None, last: None, checkpoint_ready: false, norms, at_cap, seen, viab, rest: None }) - } - /// The replay from a checkpoint line (`cp`, without its line ending, at 1-based line `at`; `before` = the event line before it) - /// instead of the header: the header is read as by `open`; the checkpoint must follow `before` (prev), at its event count, with - /// its stance (the digest `before` logs) and its state (read as any state is, with the digest `before` logs); the chain goes on - /// from the checkpoint line. The lines before it are taken as written: a full `verify` replays them and checks the checkpoint. - pub fn open_at(head: &str, before: &str, cp: &str, at: usize) -> Result { - let mut r = Replay::open(head)?; - // RFC 0009: the expectation's memory is a fold of every event line, so a strand with the block replays from its header - if r.live.p.expect.is_some() { return Err((at, "the persona's expectation block reads every event before this checkpoint: verify this strand from its header".into())); } - let j = json::parse(cp).map_err(|e| (at, format!("not JSON: {}", e.msg)))?; - let b = json::parse(before).map_err(|e| (at - 1, format!("not JSON: {}", e.msg)))?; - if persona::canon(&j) != cp { return Err((at, "not canonical JSON".into())); } - if let Some(k) = unknown_key(&j) { return Err((at, unsupported(k))); } - if !j.as_obj().is_some_and(|kv| kv.len() == 4 && kv.iter().all(|(k, _)| ["checkpoint", "prev", "stance", "state"].contains(&k.as_str()))) { - return Err((at, "checkpoint fields must be checkpoint, prev, stance and state".into())); - } - // §2.18: an event line of a persona with glyphs also carries `glyph` - if !b.as_obj().is_some_and(|kv| kv.len() == 5 + usize::from(kv.iter().any(|(k, _)| k == "glyph")) && kv.iter().all(|(k, _)| ["n", "inputs", "prev", "stance", "state", "glyph"].contains(&k.as_str()))) { - return Err((at - 1, "a checkpoint must immediately follow an event line".into())); - } - if j.get("prev").and_then(Json::as_str) != Some(persona::digest_of(before).as_str()) { return Err((at, "prev is not the sha256 of the line before".into())); } - let n = j.get("checkpoint").and_then(Json::as_f64).filter(|x| *x >= 1.0 && x.fract() == 0.0).ok_or((at, "not a checkpoint line".to_string()))? as u64; - if b.get("n").and_then(Json::as_f64) != Some(n as f64) { return Err((at, format!("the line before is not event {n}"))); } - let stance = j.get("stance").cloned().unwrap_or(Json::Null); - if b.get("stance").and_then(Json::as_str) != Some(persona::stance_sha(&stance).as_str()) { return Err((at, "the checkpoint's stance is not the one event {n} logs".replace("{n}", &n.to_string()))); } - let st = State::read(&r.live.p, j.get("state").unwrap_or(&Json::Null)).map_err(|e| (at, read_err(e, "the checkpoint's state: ")))?; - if b.get("state").and_then(Json::as_str) != Some(st.digest.as_str()) { return Err((at, format!("the checkpoint's state is not the one event {n} logs"))); } - r.live.st = st; r.live.n = n; r.live.prev = persona::digest_of(cp); r.live.checkpoints = 1; r.line = at; r.from = Some(n); r.last = Some(stance); r.checkpoint_ready = false; - r.norms = Normals::start(&r.live.p, &r.live.st); r.at_cap = vec![0.0; r.at_cap.len()]; r.seen = (0.0, vec![0.0; r.seen.1.len()]); r.viab = viab_start(&r.live); - Ok(r) - } - /// The stance document of the last event replayed (or of the checkpoint the replay started from) - pub fn last_stance(&self) -> Option<&Json> { self.last.as_ref() } - /// The senses' normals over the replay so far (§2.11) - pub fn normals(&self) -> &Normals { &self.norms } - /// One line (without its line ending) -> the stance document an event line replays to (None for a control or checkpoint - /// line); Err((its 1-based line number, what differs)). A control line must be a valid move of the status and the line its - /// fields give; a checkpoint line must immediately follow an event while active and carry - /// the event count and the state the replay reached. The last stance remains available to - /// the monitor after controls/checkpoints, but that does not authorize another checkpoint. A chained line with a key its - /// kind does not have (a newer line kind, a newer field), or an event whose inputs have a field this engine does not know, - /// is Err((its line number, the unsupported `why`)). - pub fn step(&mut self, l: &str, eng: Engine) -> Result, (usize, String)> { - let (i, live) = (self.line + 1, &mut self.live); - let j = json::parse(l).map_err(|e| (i, format!("not JSON: {}", e.msg)))?; - if persona::canon(&j) != l { return Err((i, "not canonical JSON".into())); } - let head = self.rest.as_deref().map_or(live.head().to_string(), persona::digest_of); - if j.get("prev").and_then(Json::as_str) != Some(head.as_str()) { return Err((i, PREV.into())); } - if let Some(k) = unknown_key(&j) { return Err((i, unsupported(k))); } - if self.rest.is_some() && !matches!(kind(&j), Kind::Event) { return Err((i, "a rest line must be followed by its event line".into())); } - match kind(&j) { - // RFC 0002 part 2: a rest line is checked with the event after it (the replay recomputes it there) - Kind::Rest => { if live.status != Status::Active { return Err((i, format!("rest line while {}", live.status.name()))); } - self.rest = Some(l.to_string()); self.line = i; self.checkpoint_ready = false; return Ok(None) } - Kind::Control => { let f = |k: &str| j.get(k).and_then(Json::as_str).unwrap_or(""); - let line = live.control(f("control"), f("by"), f("reason"), f("at")).map_err(|e| (i, format!("control line: {}: {}", e.path, e.msg)))?; - if line != l { return Err((i, "the control line differs".into())); } - self.line = i; self.checkpoint_ready = false; return Ok(None) } - Kind::Checkpoint => { - if live.status != Status::Active { return Err((i, format!("checkpoint while {}", live.status.name()))); } - if !self.checkpoint_ready { return Err((i, "a checkpoint must immediately follow an event line".into())); } - if j.get("checkpoint").and_then(Json::as_f64) != Some(live.n as f64) { return Err((i, format!("the checkpoint is not at event {}", live.n))); } - let Some(stance) = self.last.as_ref() else { return Err((i, "a checkpoint follows an event line".into())) }; - if live.checkpoint(stance) != l { return Err((i, "the checkpoint's state or stance differs from the replay".into())); } - self.line = i; self.checkpoint_ready = false; return Ok(None) } - Kind::Event => {} } - if j.get("n").and_then(Json::as_f64) != Some((live.n + 1) as f64) { return Err((i, format!("n is not {}", live.n + 1))); } - let ev = j.get("inputs").ok_or((i, "no inputs".to_string()))?; - // a needs block (RFC 0002): the stamped hours of events that found a deficit at its cap (it sat there since the event before) - let before = persona::deficits(&live.p, &live.st); let h = ev.get("elapsed_hours").and_then(Json::as_f64).unwrap_or(0.0); - let logged = self.rest.take(); - let (stance, both) = live.event(ev, eng).map_err(|e| (i, read_err(e, "")))?; - let line = match (both.split_once('\n'), logged) { - (Some((r, e)), Some(lr)) => { if r != lr { return Err((i - 1, "the rest line differs".into())); } e.to_string() } - (Some(_), None) => return Err((i, "a rest line is due before this event and missing".into())), - (None, Some(_)) => return Err((i - 1, "a rest line where none is due".into())), - (None, None) => both }; - for (k, (_, x, cap)) in before.iter().enumerate() { if *x >= cap - 5e-7 { self.at_cap[k] = persona::r6(self.at_cap[k] + h); } } - if j.get("stance").and_then(Json::as_str) != Some(persona::stance_sha(&stance).as_str()) { return Err((i, "the stance differs".into())); } - // RFC 0009: the logged deviation is recomputed (an edited receipt diverges here) - if json::parse(&line).ok().and_then(|x| x.get("deviation").cloned()) != j.get("deviation").cloned() { return Err((i, "the deviation differs".into())); } - // §2.18: the logged glyph is recomputed from the replayed stance (an edited glyph diverges here) - if stance.get("glyph") != j.get("glyph") { return Err((i, "the glyph differs".into())); } - if j.get("state").and_then(Json::as_str) != Some(live.st.digest.as_str()) { return Err((i, "the state differs".into())); } - if line != l { return Err((i, "the line differs".into())); } - self.line = i; self.last = Some(stance.clone()); self.checkpoint_ready = true; self.norms.track(&self.live.p, &self.live.st); - // RFC 0002: the largest R̄ (Design C) and per learned trait the largest gain (Design B) after any event replayed - if let Some((r, _, _)) = persona::pace_rbar(&self.live.p, &self.live.st) { self.seen.0 = self.seen.0.max(r); } - for (k, (_, g, _)) in persona::gate_gains(&self.live.p, &self.live.st).iter().enumerate() { if let Some(m) = self.seen.1.get_mut(k) { *m = m.max(*g); } } - // RFC 0008 E: the smallest margin_min after any event replayed, and the event at which viability was lost - if let (Some((seen, lost_at)), Some((_, m, lost))) = (self.viab.as_mut(), persona::viability(&self.live.p, &self.live.st)) { - *seen = seen.min(m); if lost && lost_at.is_none() { *lost_at = Some(self.live.n); } } - Ok(Some(stance)) - } - /// The end of the strand: a rest line (RFC 0002 part 2) is only ever written with the event after it, so a strand that ends - /// on one diverges at it - pub fn end(&self) -> Result<(), (usize, String)> { - if self.rest.is_some() { return Err((self.line, "a rest line must be followed by its event line".into())); } Ok(()) - } - /// `verify`'s summary of the lines replayed so far: `engine` is the strand header's (the engine that wrote it), `verified_by` - /// the engine that replayed it - pub fn summary(&self) -> Json { - let (s, live) = (|x: &str| Json::Str(x.to_string()), &self.live); - let mut v = vec![("ok".into(), Json::Bool(true)), ("events".into(), Json::Num(live.n as f64)), ("persona".into(), s(&live.p.name)), ("seed".into(), Json::Num(live.st.seed as f64)), - ("engine".into(), self.header.get("engine").cloned().unwrap_or(Json::Null)), ("verified_by".into(), Json::Str(engine())), ("final_state".into(), s(&live.st.digest)), - ("last_line".into(), s(live.head()))]; - // a strand with control or checkpoint lines says so (a strand without them gives the summary of before) - if live.controls > 0 { v.push(("controls".into(), Json::Num(live.controls as f64))); v.push(("status".into(), s(live.status.name()))); } - if live.checkpoints > 0 { v.push(("checkpoints".into(), Json::Num(live.checkpoints as f64))); } - // RFC 0002 part 2: rest lines replayed, the pressure now; the load now - if live.rests > 0 { v.push(("rest_lines".into(), Json::Num(live.rests as f64))); } - if let Some((sp, pend)) = persona::rest_pressure(&live.st) { v.push(("pressure".into(), Json::Obj(vec![("s".into(), Json::Num(sp)), ("pending".into(), Json::Bool(pend))]))); } - if let Some((l, _)) = persona::load_level(&live.p, &live.st) { v.push(("load".into(), Json::Obj(vec![("l".into(), Json::Num(l))]))); } - if let Some(n) = self.from { v.push(("from_checkpoint".into(), Json::Num(n as f64))); } - // a senses block (§2.11): per sense the normal now and the largest move of its mean within 50 events - let nm = persona::normals(&live.p, &live.st); - if !nm.is_empty() { v.push(("normals".into(), Json::Obj(nm.iter().enumerate().map(|(k, (id, m, n, _))| (id.clone(), Json::Obj(vec![("mean".into(), Json::Num(*m)), - ("n".into(), Json::Num(*n as f64)), ("max_move_50".into(), Json::Num(self.norms.max_move.get(k).copied().unwrap_or(0.0)))]))).collect()))); } - // a needs block (RFC 0002): per need its deficit now and the stamped hours it sat at its cap - let dx = persona::deficits(&live.p, &live.st); - if !dx.is_empty() { v.push(("needs".into(), Json::Obj(dx.iter().enumerate().map(|(k, (id, x, _))| (id.clone(), Json::Obj(vec![("deficit".into(), Json::Num(*x)), - ("at_cap_hours".into(), Json::Num(self.at_cap.get(k).copied().unwrap_or(0.0)))]))).collect()))); } - // RFC 0002 Design C: R̄ now and the largest after any event replayed; Design B: per learned trait the largest gain seen - if let Some((r, _, _)) = persona::pace_rbar(&live.p, &live.st) { v.push(("pace".into(), Json::Obj(vec![("rbar".into(), Json::Num(r)), ("max".into(), Json::Num(self.seen.0))]))); } - let gg = persona::gate_gains(&live.p, &live.st); - if !gg.is_empty() { v.push(("surprise".into(), Json::Obj(gg.iter().enumerate().map(|(k, (t, _, _))| (t.clone(), - Json::Obj(vec![("g_max_seen".into(), Json::Num(self.seen.1.get(k).copied().unwrap_or(0.0)))]))).collect()))); } - // RFC 0008 E: with a bound, recomputed from the logged readings and amounts - if let Some((seen, lost_at)) = self.viab { v.push(("viability".into(), Json::Obj(vec![("margin_min_seen".into(), Json::Num(seen)), ("lost_at".into(), lost_at.map_or(Json::Null, |n| Json::Num(n as f64)))]))); } - // a credit block (§2.8): the decisions still awaiting an outcome - if let Some(o) = persona::open_decisions(&live.p, &live.st) { v.push(("open".into(), Json::Arr(o.iter().map(|(t, _)| Json::Num(*t as f64)).collect()))); } - Json::Obj(v) - } -} -/// RFC 0008 E: the viability record a replay starts from: the margin_min of its first state, and lost_at (the event count when the -/// state is already lost, as after a checkpoint); none without a bound -fn viab_start(live: &Live) -> Option<(f64, Option)> { persona::viability(&live.p, &live.st).map(|(_, m, lost)| (m, lost.then_some(live.n))) } -/// Per event line its n and its block -pub type Blocks = Vec<(u64, String)>; -/// `live verify --blocks`: replay from the header -> (the summary, per event line its n and the block its stance renders, RFC 0008 E) -pub fn verify_blocks(text: &str, eng: Engine) -> Result<(Json, Blocks), (usize, String)> { - let lines = lines(text); - let replay = || -> Result<(Json, Blocks), (usize, String)> { - let mut r = Replay::open(lines.first().copied().unwrap_or(""))?; let mut out = vec![]; - for l in lines.iter().skip(1) { if let Some(doc) = r.step(l, eng)? { out.push((r.live.n, persona::block(&r.live.p, &doc))); } } - Ok((r.summary(), out)) }; - replay().map_err(|e| chain_first(&lines, e)) -} -/// A strand's text -> its lines without their line endings (a final newline ends the last line; it does not start another). -/// One leading UTF-8 byte-order mark is skipped, as `\r` before `\n` is: a Windows editor adds both on save, and neither is -/// part of the line text the chain hashes. -pub fn lines(text: &str) -> Vec<&str> { - let mut lines: Vec<&str> = text.strip_prefix('\u{feff}').unwrap_or(text).split('\n').map(bare).collect(); - if lines.last() == Some(&"") { lines.pop(); } - lines -} -/// `probbit live verify`: replay a strand from its header -> Ok(summary) when every line is as recorded; Err((1-based line -/// number, what differs)) at the earliest line that is not. Lines end in `\n` or `\r\n`; the chain hashes the line text alone. -/// A line with a field or line kind this engine does not know stops the replay as unsupported, unless a later line breaks the -/// chain (`chain_first`). -pub fn verify(text: &str, eng: Engine) -> Result { - let lines = lines(text); - let replay = || -> Result { - let mut r = Replay::open(lines.first().copied().unwrap_or(""))?; - for l in lines.iter().skip(1) { r.step(l, eng)?; } - r.end()?; Ok(r.summary()) }; - replay().map_err(|e| chain_first(&lines, e)) -} -/// The divergence the hash chain shows after an unsupported line: a strand this engine cannot replay to the end is still checked -/// for what the format alone says, so an edited, removed or reordered line after it (or the unsupported line itself, edited: -/// the next line's `prev`) diverges at the earliest line that breaks the chain (not JSON, not canonical JSON, or a `prev` that is -/// not the sha256 of the line before), as in a replay. `unsupported` is left only for a strand whose chain is whole; a -/// divergence passes unchanged. -fn chain_first(lines: &[&str], (line, why): (usize, String)) -> (usize, String) { - if unsupported_field(&why).is_none() { return (line, why); } - for i in line.max(1)..lines.len() { - let j = match json::parse(lines[i]) { Ok(j) => j, Err(e) => return (i + 1, format!("not JSON: {}", e.msg)) }; - if persona::canon(&j) != lines[i] { return (i + 1, "not canonical JSON".into()); } - if j.get("prev").and_then(Json::as_str) != Some(persona::digest_of(lines[i - 1]).as_str()) { return (i + 1, PREV.into()); } - } - (line, why) -} -/// The divergence of a line whose `prev` is not the sha256 of the line before -const PREV: &str = "prev is not the sha256 of the line before (a line before it was changed, removed or reordered)"; -/// The 0-based index of the strand's last checkpoint line followed by at least `after` lines, if any (a cheap scan: the key) -pub fn last_checkpoint(lines: &[&str], after: usize) -> Option { - (1..lines.len().saturating_sub(after)).rev().find(|&i| lines[i].starts_with(r#"{"checkpoint":"#)) -} -/// `verify --from-checkpoint`: replay from the last checkpoint line (the header read as always), not from the header -pub fn verify_from_checkpoint(text: &str, eng: Engine) -> Result { - let lines = lines(text); - let Some(k) = last_checkpoint(&lines, 0) else { return verify(text, eng) }; - let replay = || -> Result { - let mut r = Replay::open_at(lines[0], lines[k - 1], lines[k], k + 1)?; - for l in lines.iter().skip(k + 1) { r.step(l, eng)?; } - r.end()?; Ok(r.summary()) }; - replay().map_err(|e| chain_first(&lines, e)) -} -/// `verify` from the header, keeping the stance documents of the given turns (what `--actions` compares with) -pub fn verify_keep(text: &str, eng: Engine, keep: &[u64]) -> Result<(Json, Vec<(u64, Json)>, Persona), (usize, String)> { - let lines = lines(text); - let replay = || -> Result<(Json, Vec<(u64, Json)>, Persona), (usize, String)> { - let mut r = Replay::open(lines.first().copied().unwrap_or(""))?; let mut kept = vec![]; - for l in lines.iter().skip(1) { if let Some(doc) = r.step(l, eng)? { - if let Some(t) = doc.get("turn").and_then(Json::as_f64).map(|x| x as u64).filter(|t| keep.contains(t)) { kept.push((t, doc)); } } } - r.end()?; Ok((r.summary(), kept, r.live.p.clone())) }; - replay().map_err(|e| chain_first(&lines, e)) -} -/// One line of a host's action log (`live verify --actions FILE`, §5.7): the decision's turn and, per trait the host acted on, -/// the level it acted on -pub struct Action { pub line: usize, pub turn: u64, pub levels: Vec<(String, String)> } -/// A host's action log: one JSON object per line, {"turn": T, "action": {trait: level, ...}} (blank lines skipped) -pub fn read_actions(text: &str) -> Result, InErr> { - let mut out = vec![]; - for (i, l) in text.split('\n').map(bare).enumerate() { if l.trim().is_empty() { continue; } let path = format!("actions.line{}", i + 1); - let bad = |m: &str| perr(&path, m.to_string()); - let j = json::parse(l).map_err(|e| perr(&path, format!("not JSON: {}", e.msg)))?; - let kv = j.as_obj().filter(|kv| kv.len() == 2 && kv.iter().all(|(k, _)| k == "turn" || k == "action")).ok_or_else(|| bad("an object {\"turn\": T, \"action\": {trait: level}}"))?; - let turn = kv.iter().find(|(k, _)| k == "turn").and_then(|(_, x)| x.as_f64()).filter(|x| *x >= 0.0 && x.fract() == 0.0 && *x <= 9_007_199_254_740_992.0) - .ok_or_else(|| bad("turn: a whole number >= 0 (the stance's turn)"))? as u64; - let act = kv.iter().find(|(k, _)| k == "action").and_then(|(_, x)| x.as_obj()).filter(|a| !a.is_empty()).ok_or_else(|| bad("action: an object trait -> level"))?; - let levels = act.iter().map(|(t, x)| x.as_str().map(|l| (t.clone(), l.to_string())).ok_or_else(|| bad("action: each level a string"))).collect::, InErr>>()?; - out.push(Action { line: i + 1, turn, levels }); } - Ok(out) -} -/// The earliest turn (then the earliest line) whose host action differs from the level its stance released -> (actions -/// checked, the difference: {turn, line, trait, action, released (null: not released or no such turn), why}); an action on a -/// trait the persona does not have is an input error -pub fn first_difference(p: &Persona, actions: &[Action], stances: &[(u64, Json)]) -> Result<(usize, Option), InErr> { - let mut order: Vec<&Action> = actions.iter().collect(); order.sort_by_key(|a| (a.turn, a.line)); - let s = |x: &str| Json::Str(x.to_string()); let mut checked = 0; - for a in order { checked += 1; let doc = stances.iter().find(|(t, _)| *t == a.turn).map(|x| &x.1); - for (t, level) in &a.levels { - let e = match doc { Some(d) => persona::doc_entry(p, d, t), None => None }; - if doc.is_some() && e.is_none() { return Err(perr(&format!("actions.line{}.action.{t}", a.line), "not a trait of the persona")); } - let released = e.as_ref().filter(|e| e.get("released") == Some(&Json::Bool(true))).and_then(|e| e.get("level")).and_then(Json::as_str).map(str::to_string); - let why = match (doc, &released) { (None, _) => "the strand has no stance with this turn", (_, None) => "the stance did not release this trait", - (_, Some(r)) if r != level => "the host acted on another level", _ => continue }; - return Ok((checked, Some(Json::Obj(vec![("turn".into(), Json::Num(a.turn as f64)), ("line".into(), Json::Num(a.line as f64)), ("trait".into(), s(t)), ("action".into(), s(level)), - ("released".into(), released.map_or(Json::Null, |r| s(&r))), ("why".into(), s(why))])))); } } - Ok((checked, None)) -} -/// `verify` as one document: the summary, or `failure`'s document -pub fn verify_doc(text: &str, eng: Engine) -> Json { verify(text, eng).unwrap_or_else(|(line, why)| failure(text, line, &why)) } -/// The document of a strand that does not verify at `line`: {ok: false, line, diverges: what differs}, or, for a field or line -/// kind this engine does not know, {ok: false, line, unsupported: the field's path, engine: this engine, strand_engine: the -/// header's `engine` (null if it names none)} -pub fn failure(text: &str, line: usize, why: &str) -> Json { - let mut v = vec![("ok".into(), Json::Bool(false)), ("line".into(), Json::Num(line as f64))]; - match unsupported_field(why) { - Some(f) => { v.push(("unsupported".into(), Json::Str(f.to_string()))); v.push(("engine".into(), Json::Str(engine()))); v.push(("strand_engine".into(), strand_engine(text))); } - None => v.push(("diverges".into(), Json::Str(why.to_string()))) } - Json::Obj(v) -} -/// The engine a strand's header names: its `engine` string, or null when the header is not JSON or has none -pub fn strand_engine(text: &str) -> Json { - json::parse(lines(text).into_iter().next().unwrap_or("")).ok().and_then(|h| h.get("engine").cloned()).filter(|e| matches!(e, Json::Str(_))).unwrap_or(Json::Null) -} -/// This binary's engine (the strand header's `engine`) -pub fn engine() -> String { format!("probbit {}", env!("CARGO_PKG_VERSION")) } - -/// Validate the stored chain and lifecycle before extending it. This is deliberately not -/// inference replay or authentication: verify still recomputes every event. In particular, a -/// checkpoint cannot reactivate a retired individual or hide an invalid earlier control. -fn check_chain(text: &str) -> Result { - let ls = lines(text); - let err = |line: usize, why: String| perr("strand", format!("line {line}: {why}")); - let r = Replay::open(ls.first().copied().unwrap_or("")).map_err(|(i, why)| err(i, why))?; - let mut status = Status::Active; - let mut n = 0u64; - for i in 1..ls.len() { - let j = json::parse(ls[i]).map_err(|e| err(i + 1, e.msg))?; - if persona::canon(&j) != ls[i] { return Err(err(i + 1, "not canonical JSON".into())); } - if j.get("prev").and_then(Json::as_str) != Some(persona::digest_of(ls[i - 1]).as_str()) { - return Err(err(i + 1, "prev is not the sha256 of the line before".into())); - } - if let Some(k) = unknown_key(&j) { return Err(err(i + 1, unsupported(k))); } - match kind(&j) { - Kind::Control => { - let f = |k: &str| j.get(k).and_then(Json::as_str).unwrap_or(""); - status = status.after(f("control")).map_err(|e| err(i + 1, e.msg))?; - let line = control_line(f("prev"), f("control"), f("by"), f("reason"), f("at")).map_err(|e| err(i + 1, e.msg))?; - if line != ls[i] { return Err(err(i + 1, "the control line differs".into())); } - } - Kind::Event => { - if status != Status::Active { return Err(err(i + 1, format!("event while {}", status.name()))); } - n += 1; - if j.get("n").and_then(Json::as_f64) != Some(n as f64) { return Err(err(i + 1, format!("n is not {n}"))); } - let Some(Json::Obj(_)) = j.get("inputs") else { return Err(err(i + 1, "no input object".into())) }; - for k in ["state", "stance"] { - if !j.get(k).and_then(Json::as_str).is_some_and(|s| s.len() == 71 && s.starts_with("sha256:") && s[7..].bytes().all(|c| c.is_ascii_hexdigit())) { - return Err(err(i + 1, format!("invalid {k} digest"))); - } - } - } - Kind::Rest => { - if status != Status::Active { return Err(err(i + 1, format!("rest line while {}", status.name()))); } - if !ls.get(i + 1).and_then(|x| json::parse(x).ok()).is_some_and(|x| matches!(kind(&x), Kind::Event)) { return Err(err(i + 1, "a rest line must be followed by its event line".into())); } - } - Kind::Checkpoint => { - if status != Status::Active { return Err(err(i + 1, format!("checkpoint while {}", status.name()))); } - // Full state validation plus links to the immediately preceding event; no inference. - let cp = Replay::open_at(ls[0], ls[i - 1], ls[i], i + 1).map_err(|(at, why)| err(at, why))?; - if cp.live.n != n { return Err(err(i + 1, "checkpoint event count differs".into())); } - // Keep the header persona/state check independent of the caller's state. - debug_assert_eq!(cp.live.p.digest, r.live.p.digest); - } - } - } - Ok(status) -} - -/// Continue a strand: `text` is the strand so far and `st` the individual after its last line -> the individual, ready for the -/// next event (no header to write). The header's persona must be `p` (the same digest); the run uses the header's document, the -/// key order the strand replays with. The lines before are not replayed here (`verify` does that). -pub fn resume(text: &str, p: &Persona, st: State, clock: Clock) -> Result { - let bad = |m: String| perr("strand", m); - if !text.ends_with('\n') { return Err(bad("its last line is incomplete (no newline at the end)".into())); } - check_chain(text)?; - let lines = lines(text); - let h = json::parse(lines.first().copied().unwrap_or("")).map_err(|e| bad(format!("the header is not JSON: {}", e.msg)))?; - if h.get("probbit_strand").and_then(Json::as_f64) != Some(FORMAT) { return Err(bad("not a probbit strand (format 1)".into())); } - let q = persona::build(h.get("document").unwrap_or(&Json::Null)).map_err(|e| bad(format!("the header's persona: {}: {}", e.path, e.msg)))?; - if q.digest != p.digest { return Err(bad(format!("it is another persona's ({} != {})", &q.digest[..19], &p.digest[..19]))); } - let last = lines[lines.len() - 1]; - // the control lines after the last event or checkpoint line (the anchor; the header without either) set the status: an - // event or a checkpoint is only ever written while active - let (mut a, mut tail) = (lines.len() - 1, vec![]); - while a > 0 { let j = json::parse(lines[a]).map_err(|e| bad(format!("line {} is not JSON: {}", a + 1, e.msg)))?; - if !matches!(kind(&j), Kind::Control) { break; } tail.push(j); a -= 1; } - let (n, want) = if a == 0 { (0, h.get("state").and_then(|s| s.get("digest")).and_then(Json::as_str).unwrap_or("").to_string()) } else { - let j = json::parse(lines[a]).map_err(|e| bad(format!("line {} is not JSON: {}", a + 1, e.msg)))?; - match kind(&j) { Kind::Checkpoint => (j.get("checkpoint").and_then(Json::as_f64).unwrap_or(0.0) as u64, j.get("state").and_then(|s| s.get("digest")).and_then(Json::as_str).unwrap_or("").to_string()), - _ => (j.get("n").and_then(Json::as_f64).unwrap_or(0.0) as u64, j.get("state").and_then(Json::as_str).unwrap_or("").to_string()) } }; - if st.digest != want { return Err(perr("state", format!("not the strand's last state ({} != {})", &st.digest[..st.digest.len().min(19)], &want[..want.len().min(19)]))); } - // RFC 0009: the expectation's memory, folded from every event line's inputs and logged deviation - let expect = q.expect.as_ref().map(|_| { let mut m = persona::ExpectMem::new(); - for l in lines.iter().skip(1) { if let Ok(j) = json::parse(l) { if matches!(kind(&j), Kind::Event) { let d = j.get("deviation"); - let lp = d.filter(|d| d.get("cold").is_none()).and_then(|d| d.get("pred")).and_then(Json::as_f64); - persona::expect_step(&q, &mut m, j.get("inputs").unwrap_or(&Json::Null), lp); } } } m }); - let mut lv = Live { p: q, st, clock, last: 0.0, prev: persona::digest_of(last), n, status: Status::Active, controls: 0, checkpoints: 0, rests: 0, expect, traj: None }; - for c in tail.iter().rev() { let what = c.get("control").and_then(Json::as_str).unwrap_or(""); - lv.status = lv.status.after(what).unwrap_or(Status::Retired); lv.st = lv.st.zero_credit(&lv.p); lv.controls += 1; } - Ok(lv) -} /// A strand file to log to: a new file gets the header (returned, to write before the events); an existing strand is continued from `st`, /// which must be its last state pub fn open(path: &str, p: Persona, doc: &Json, st: State, clock: Clock) -> Result<(Live, Option), InErr> { @@ -670,6 +127,99 @@ pub fn control_cmd(path: &str, what: &str, by: &str, reason: &str, at: &str) -> ("head".into(), s(&persona::digest_of(&line)))])) } +/// `probbit live control STRAND reflect [--last N | --since-checkpoint]` (RFC 0013): a host decides when. Under the writer lock, +/// the reflection of the strand's lines (no solve) is appended as one line after its last line, chained like every line -> +/// {ok, control: reflect, digest, events, line, head}. Engine-gated: refused on a strand of an engine before 0.14.0. +pub fn reflect_cmd(path: &str, w: probbit_persona::reflect::Window) -> Result { + let lock = Lock::take(path)?; + let text = std::fs::read_to_string(path).map_err(|e| perr("strand", format!("cannot read {path}: {e}")))?; + if !text.ends_with('\n') { return Err(perr("strand", "its last line is incomplete (no newline at the end)")); } + let ls = lines(&text); + let h = json::parse(ls.first().copied().unwrap_or("")).ok(); + if h.as_ref().and_then(|h| h.get("probbit_strand").and_then(Json::as_f64)) != Some(FORMAT) { return Err(perr("strand", "not a probbit strand (format 1)")); } + if engine_version(h.as_ref().and_then(|h| h.get("engine")).and_then(Json::as_str).unwrap_or("")).map_or(true, |v| v < probbit_persona::reflect::REFLECT_SINCE) { + return Err(perr("strand", "a reflection line needs a strand of engine 0.14.0 or later (`probbit reflect STRAND` reads an older one)")); } + check_chain(&text)?; + let r = probbit_persona::reflect::reflect_window(&ls, w)?; + let line = probbit_persona::reflect::line(&persona::digest_of(ls[ls.len() - 1]), &r); + if !lock.held() { return Err(locked(None, &format!("{path}.lock"))); } + append(path, &[&line])?; + let s = |x: &str| Json::Str(x.to_string()); + Ok(Json::Obj(vec![("ok".into(), Json::Bool(true)), ("control".into(), s("reflect")), ("digest".into(), s(&r.digest)), + ("events".into(), r.doc.get("window").and_then(|x| x.get("events")).cloned().unwrap_or(Json::Null)), ("line".into(), Json::Num((ls.len() + 1) as f64)), + ("head".into(), s(&persona::digest_of(&line)))])) +} + +/// `probbit live control STRAND stir? [--idle-hours H]` (RFC 0016): the strand replayed from its header (every line checked), then +/// the individual's `stir_due(H)` -> {due, stir (its class, speaks, asleep, level, since_minutes; null when none is due), events}. +/// Writes nothing: a host that acts on a due stir feeds `{"stir": class}` with its own readings as an event. +pub fn stir_cmd(path: &str, idle_hours: f64, eng: Engine) -> Result { + let text = std::fs::read_to_string(path).map_err(|e| perr("strand", format!("cannot read {path}: {e}")))?; + let ls = lines(&text); + let (mut r, start) = probbit_persona::strand::Replay::open_lines(&ls).map_err(|(i, why)| perr("strand", format!("line {i}: {why}")))?; + for i in start..ls.len() { r.step_at(&ls, i, eng).map_err(|(i, why)| perr("strand", format!("line {i}: {why}")))?; } + let st = r.live.stir_due(idle_hours); + Ok(Json::Obj(vec![("due".into(), Json::Bool(st.is_some())), ("events".into(), Json::Num(r.live.n as f64)), ("stir".into(), st.map_or(Json::Null, |s| s.to_json()))])) +} +/// `probbit live control STRAND rest [--at TIME]` (RFC 0015): a host decides when (standby, night, idle). Under the writer lock, +/// the strand replayed from its header (every line checked), then the rest pass on the replayed individual and its one +/// `consolidation` line appended -> {ok, control: rest, consolidation, line, head}. Refused without an `expectation.rest` block, +/// before engine 0.14.0, and once retired. `state`: the host's stored individual (`live --state FILE`), which must be the state +/// after the strand's last line; it is rewritten with the state after the pass (a `decay` moves it), so `live --state FILE` +/// continues the strand. +pub fn rest_cmd(path: &str, at: &str, state: Option<&str>, eng: Engine) -> Result { + let lock = Lock::take(path)?; + let text = std::fs::read_to_string(path).map_err(|e| perr("strand", format!("cannot read {path}: {e}")))?; + if !text.ends_with('\n') { return Err(perr("strand", "its last line is incomplete (no newline at the end)")); } + let ls = lines(&text); + let (mut r, start) = probbit_persona::strand::Replay::open_lines(&ls).map_err(|(i, why)| perr("strand", format!("line {i}: {why}")))?; + for i in start..ls.len() { r.step_at(&ls, i, eng).map_err(|(i, why)| perr("strand", format!("line {i}: {why}")))?; } + if r.live.dev_full { return Err(perr("strand", "a consolidation line needs a strand of engine 0.14.0 or later")); } + if let Some(f) = state { let t = std::fs::read_to_string(f).map_err(|e| perr("state", format!("cannot read {f}: {e}")))?; + let j = json::parse(t.strip_prefix('\u{feff}').unwrap_or(&t)).map_err(|e| perr("state", format!("not JSON: {}", e.msg)))?; + let st = State::read(&r.live.p, &j)?; + if st.digest != r.live.st.digest { return Err(perr("state", format!("not the state after the strand's last line ({} != {})", &st.digest[..19], &r.live.st.digest[..19]))); } } + let line = r.live.consolidate(at)?; + if !lock.held() { return Err(locked(None, &format!("{path}.lock"))); } + append(path, &[&line])?; + if let Some(f) = state { let tmp = format!("{f}.{}.tmp", std::process::id()); + std::fs::write(&tmp, persona::canon(&r.live.st.to_json(&r.live.p))).and_then(|_| std::fs::rename(&tmp, f)).map_err(|e| perr("state", format!("cannot write {f}: {e}")))?; } + let s = |x: &str| Json::Str(x.to_string()); + Ok(Json::Obj(vec![("ok".into(), Json::Bool(true)), ("control".into(), s("rest")), + ("consolidation".into(), json::parse(&line).ok().and_then(|j| j.get("consolidation").cloned()).unwrap_or(Json::Null)), + ("line".into(), Json::Num((ls.len() + 1) as f64)), ("head".into(), s(&persona::digest_of(&line)))])) +} +/// `probbit live control STRAND seal` (§5.7, "Segments"): a host decides when to seal. Under the writer lock, the newest segment +/// of STRAND (`NAME.strand`, `NAME.0001.strand`, ...) is replayed (from its last checkpoint, else from its start); refused unless +/// its last line is an event line (a seal follows an event); then the sealing checkpoint is appended to it and the next segment +/// written: its header (`segment` {index, previous, sealed}) and the sealing line -> {ok, control: seal, segment, file, line, head} +pub fn seal_cmd(path: &str, eng: Engine) -> Result { + let lock = Lock::take(path)?; + let mut k = 0u64; while std::path::Path::new(&segment_name(path, k + 1)).exists() { k += 1; } + let cur = segment_name(path, k); + let text = std::fs::read_to_string(&cur).map_err(|e| perr("strand", format!("cannot read {cur}: {e}")))?; + if !text.ends_with('\n') { return Err(perr("strand", format!("{cur}: its last line is incomplete (no newline at the end)"))); } + let ls = lines(&text); + let last = ls.last().and_then(|l| json::parse(l).ok()).filter(|_| ls.len() > 1); + if !last.as_ref().is_some_and(|j| matches!(kind(j), Kind::Event)) { return Err(perr("strand", format!("{cur}: its last line is not an event line (a seal follows an event)"))); } + let at = |(i, why): (usize, String)| perr("strand", format!("{cur}: line {i}: {why}")); + let (mut r, from) = match last_checkpoint(&ls, 0).filter(|&c| c >= 2).and_then(|c| Replay::open_at(ls[0], ls[c - 1], ls[c], c + 1).ok().map(|r| (r, c + 1))) { + Some(x) => x, None => Replay::open_lines(&ls).map_err(at)? }; + for l in &ls[from..] { r.step(l, eng).map_err(at)?; } + if r.live.expect.as_ref().is_some_and(|m| !m.is_canonical()) { return Err(perr("strand", "before engine 0.14.0 a checkpoint does not carry the expectation's memory, so a segment of a persona with an expectation block could not start from its sealed line")); } + let stance = r.last_stance().cloned().ok_or_else(|| perr("strand", "no event replayed"))?; + let cp = r.live.checkpoint(&stance); + let next = segment_name(path, k + 1); + let pname = std::path::Path::new(&cur).file_name().map_or_else(|| cur.clone(), |f| f.to_string_lossy().into_owned()); + let engine = r.header.get("engine").and_then(Json::as_str).unwrap_or("").to_string(); + let h = segment_header(&r.live, &r.doc, &engine, k + 1, &pname, &persona::digest_of(&cp)); + if !lock.held() { return Err(locked(None, &format!("{path}.lock"))); } + append(&cur, &[&cp])?; append(&next, &[&h, &cp])?; + let s = |x: &str| Json::Str(x.to_string()); + Ok(Json::Obj(vec![("ok".into(), Json::Bool(true)), ("control".into(), s("seal")), ("segment".into(), Json::Num((k + 1) as f64)), ("file".into(), s(&next)), + ("line".into(), Json::Num((ls.len() + 1) as f64)), ("head".into(), s(&persona::digest_of(&cp)))])) +} + /// The tools `probbit_live_event` and `probbit_live_verify` (`probbit mcp`), stateless as the persona tools. event: the persona /// (a document or a path), the state (or a seed: a new individual), one event (its inputs; `elapsed_hours` comes from the host's /// clock and is quantised here) and optionally a strand file on the server's disk (a new one gets the header; an existing one is @@ -848,40 +398,6 @@ fn draw(th: crate::theme::Theme, r: &Row, drawn: usize, name: &str, rest: &str, crate::tui::err_write(&s); b.len() } -/// The audit of a recorded life (`probbit persona bound PERSONA --audit STRAND`): the strand replayed line by line; after every -/// event and control line, per learned trait the resting TV between the individual and the same individual with its learned -/// weights at zero, as a share of its epsilon (the room used). An exceedance is impossible by the theorem; the audit shows it -/// on the recorded life. -pub fn bound_audit(text: &str, b: &persona::bound::Bound, eng: Engine) -> Result<(Json, bool), (usize, String)> { - let lines = lines(text); - let mut r = Replay::open(lines.first().copied().unwrap_or(""))?; - let p = r.live.p.clone(); let (traits, cap) = persona::bound::learned_traits(&p).ok_or((1, "the strand's persona has no learning block: nothing to audit".to_string()))?; - let rest = b.rest_hours.unwrap_or_else(|| persona::bound::default_rest(&p)); let seed = r.live.st.seed; - let e = |x: InErr| (0usize, format!("{}: {}", x.path, x.msg)); - let st0 = persona::init(&p, Some(seed), true, eng); let p0 = persona::bound::rest_odds(&p, &st0, rest, eng).map_err(e)?; - let mut eps = vec![]; let mut rr = vec![]; - for (k, t) in traits.iter().enumerate() { let sc = persona::bound::scope(&p, b, t, rest, Some(&st0)); - let bs = persona::bound::bases(&p, &sc, &st0).ok_or((0, format!("{t}: more than {} enumerated history combinations: the audit needs an exact epsilon", persona::bound::MAX_HISTORY)))?; - eps.push(persona::bound::search(&p, k, &sc.coords, cap, &bs, &p0[k], rest, eng).map_err(e)?.0); rr.push((sc.d_r / 4.0).tanh()); } - let nt = traits.len(); let (mut maxr, mut at_n, mut cur, mut exceed) = (vec![0.0f64; nt], vec![0u64; nt], vec![0.0f64; nt], 0usize); - let (mut events, mut feedback, mut erasures) = (0u64, 0u64, 0u64); let mut last = persona::bound::learned_of(&r.live.st); - let measure = |st: &persona::State| -> Result, (usize, String)> { let a = persona::bound::rest_odds(&p, st, rest, eng).map_err(e)?; - let z = persona::bound::rest_odds(&p, &persona::bound::with_learned(&p, st, &persona::bound::learned_of(st).iter().map(|d| vec![0.0; d.len()]).collect::>()), rest, eng).map_err(e)?; - Ok(a.iter().zip(&z).map(|(x, y)| persona::bound::tv(x, y)).collect()) }; - for l in lines.iter().skip(1) { if l.trim().is_empty() { continue; } - let was_event = r.step(l, eng)?.is_some(); if was_event { events += 1; } else if l.contains(r#""control":"erase""#) { erasures += 1; } - let now = persona::bound::learned_of(&r.live.st); if now != last { if was_event { feedback += 1; } last = now; } - let tvs = measure(&r.live.st)?; - for k in 0..nt { cur[k] = tvs[k]; if eps[k] > 0.0 && tvs[k] / eps[k] > maxr[k] { maxr[k] = tvs[k] / eps[k]; at_n[k] = r.live.n; } if tvs[k] > eps[k] + rr[k] + 1e-6 { exceed += 1; } } } - let s = |x: &str| Json::Str(x.to_string()); - let rows: Vec = traits.iter().enumerate().map(|(k, t)| Json::Obj(vec![("trait".into(), s(t)), ("epsilon".into(), Json::Num(eps[k])), - ("room_used_max".into(), Json::Num(persona::r6(maxr[k]))), ("at_event".into(), Json::Num(at_n[k] as f64)), ("room_used_now".into(), Json::Num(persona::r6(if eps[k] > 0.0 { cur[k] / eps[k] } else { 0.0 }))), - ("distance_to_ceiling".into(), Json::Num(persona::r6(eps[k] - cur[k]))), ("tv_now".into(), Json::Num(persona::r6(cur[k])))])).collect(); - let ok = exceed == 0; - Ok((Json::Obj(vec![("persona".into(), s(&p.name)), ("digest".into(), s(&p.digest)), ("seed".into(), Json::Num(seed as f64)), ("rest_hours".into(), Json::Num(rest)), - ("events".into(), Json::Num(events as f64)), ("feedback_events".into(), Json::Num(feedback as f64)), ("erasures".into(), Json::Num(erasures as f64)), - ("traits".into(), Json::Arr(rows)), ("exceedances".into(), Json::Num(exceed as f64)), ("ok".into(), Json::Bool(ok))]), ok)) -} #[cfg(test)] mod tests { @@ -941,6 +457,303 @@ mod tests { persona::expect_step(&pn, &mut mem, e, d.logged_pred()); st = persona::turn(&pn, &st, e, false, &run, false).unwrap().1; } } + /// 0.14.0, the canonical fold (docs/persona.md §5.7): a checkpoint of a persona with an expectation block carries the memory + /// (`expect`), so `verify --from-checkpoint` and `resume` start there: the life with a checkpoint after every 7th event gives + /// the stances, the state and the memory of the life without them, verifies from its header and from its last checkpoint to + /// one summary, and a resume writes the continuous life's next line; the memory never holds more than max_combos - 1 named + /// combos; an edited memory diverges at its checkpoint; a 0.13.0 strand keeps the four-key checkpoint and the refusal + #[test] + fn a_checkpoint_carries_the_expectation_memory() { + let mut on = json::parse(DOC).unwrap(); + if let Json::Obj(v) = &mut on { v.push(("expectation".into(), json::parse(r#"{"max_combos":3}"#).unwrap())); } + let p = persona::build(&on).unwrap(); + let evs: Vec = (0..60).map(|t| json::parse(&format!(r#"{{"praise":{},"criticism":{},"loss":{},"elapsed_hours":0.5}}"#, t % 2 == 1, t % 3 == 1, t % 5 == 4)).unwrap()).collect(); + let life = |engine: &str, every: u64| { let (mut lv, head) = Live::start(p.clone(), &on, persona::init(&p, Some(2), true, &run), Clock::Fixed, engine); + let (mut text, mut out) = (format!("{head}\n"), vec![]); + for e in &evs { let (s, l) = lv.event(e, &run).unwrap(); text += &l; text.push('\n'); if every > 0 && lv.n % every == 0 { text += &lv.checkpoint(&s); text.push('\n'); } out.push(s); } + (text, out, lv) }; + let ((_, s0, l0), (cp, s1, l1)) = (life("probbit 0.14.0", 0), life("probbit 0.14.0", 7)); + assert_eq!(s0, s1, "checkpoints move no stance"); let mem = |l: &Live| l.expect.as_ref().map(persona::ExpectMem::to_json); + assert_eq!(l0.st.digest, l1.st.digest); assert_eq!(mem(&l0), mem(&l1)); + let ls: Vec<&str> = cp.lines().collect(); let cps: Vec = (0..ls.len()).filter(|&i| ls[i].starts_with(r#"{"checkpoint":"#)).collect(); assert_eq!(cps.len(), 8); + for &i in &cps { let j = json::parse(ls[i]).unwrap(); let m = persona::ExpectMem::read(&p, j.get("expect").expect("expect on a 0.14.0 checkpoint")).unwrap(); + assert!(m.to_json().get("combos").and_then(Json::as_arr).unwrap().len() <= 3, "{}", ls[i]); } + let (full, fast) = (verify(&cp, &run).unwrap(), verify_from_checkpoint(&cp, &run).unwrap()); + assert_eq!(fast.get("from_checkpoint"), Some(&Json::Num(56.0))); + for k in ["events", "final_state", "last_line"] { assert_eq!(full.get(k), fast.get(k), "{k}"); } + // resume from the text before the last event (the last checkpoint in it) and the state there: the continuous life's line + let cut: String = ls[..ls.len() - 1].iter().map(|l| format!("{l}\n")).collect(); + let st = { let mut r = Replay::open(ls[0]).unwrap(); for l in &ls[1..ls.len() - 1] { r.step(l, &run).unwrap(); } r.live.st.clone() }; + let mut lv = resume(&cut, &p, st, Clock::Fixed).unwrap(); let (_, last) = lv.event(&evs[59], &run).unwrap(); + assert_eq!(last, ls[ls.len() - 1]); assert_eq!(mem(&lv), mem(&l1)); + // an edited memory (the chain re-made after it) diverges at its checkpoint in a full verify + let at = cps[3]; let mut j = json::parse(ls[at]).unwrap(); let n = j.get("expect").and_then(|e| e.get("n")).and_then(Json::as_f64).unwrap(); + put(&mut j, &["expect", "n"], Json::Num(n + 1.0)); let mut tt: Vec = ls.iter().map(|l| l.to_string()).collect(); tt[at] = persona::canon(&j); + assert_eq!(verify(&rechain(tt, at), &run).unwrap_err(), (at + 1, "the checkpoint's state or stance differs from the replay".to_string())); + // before 0.14.0: four keys, and a replay from a checkpoint refuses + let (old, _, _) = life("probbit 0.13.0", 7); + assert!(old.lines().filter(|l| l.starts_with(r#"{"checkpoint":"#)).all(|l| !l.contains(r#""expect":"#))); + assert!(verify_from_checkpoint(&old, &run).unwrap_err().1.contains("expectation block reads every event")); + assert_eq!(verify(&old, &run).map(|d| d.get("checkpoints").cloned()), Ok(Some(Json::Num(8.0)))); + } + + /// Event t of the S1 lives on the expectation fixture persona (tests/fixtures/monitor/tutor-expect.yaml): a hash of t picks the + /// sentiment, six flags and two numbers' thirds, so the lives see many input combos + fn s1_event(t: usize) -> Json { + let h = (t as u64).wrapping_mul(2654435761) >> 7; let b = |k: u64| (h >> k) & 1 == 1; + json::parse(&format!(r#"{{"sentiment":"{}","confused":{},"error":{},"loss":{},"praise":{},"criticism":{},"claim_done":{},"stakes":{},"time_pressure":{},"elapsed_hours":{}}}"#, + ["negative", "neutral", "positive"][(h % 3) as usize], b(2), b(3), b(4) && b(5), b(6), b(7) && b(8), b(9) && b(10), [0.1, 0.5, 0.9][((h >> 11) % 3) as usize], + [0.0, 0.5, 0.9][((h >> 13) % 3) as usize], [0.1, 0.5, 2.0][((h >> 15) % 3) as usize])).unwrap() + } + /// Segments with an expectation block (0.14.0, docs/persona.md §5.7): 500 events of the expectation fixture persona logged as one + /// file with a checkpoint after every 50th, and as segments of 50 events (each file: a header linking the previous one, the + /// sealed line, the events, its own sealed line): every stance document and the final state are the same, the lines joined + /// (each segment's header and repeated sealed line set aside) are the one file's, each segment verifies from its link to the + /// final state of the one file, a resume in the newest segment writes the one file's next line, and a segment whose second + /// line is not its header's sealed line diverges at line 2 + #[test] + fn segments_of_an_expectation_life_equal_one_file() { + let (p, doc) = persona::load(concat!(env!("CARGO_MANIFEST_DIR"), "/tests/fixtures/monitor/tutor-expect.yaml")).unwrap(); assert!(p.expect.is_some()); + let start = || Live::start(p.clone(), &doc, persona::init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.14.0"); + let ((mut a, ha), (mut b, hb)) = (start(), start()); + let (mut one, mut segs, mut sa, mut sb) = (format!("{ha}\n"), vec![format!("{hb}\n")], vec![], vec![]); + for t in 0..500 { + let (s, l) = a.event(&s1_event(t), &run).unwrap(); one += &l; one.push('\n'); if a.n % 50 == 0 { one += &a.checkpoint(&s); one.push('\n'); } sa.push(persona::canon(&s)); + let (s, l) = b.event(&s1_event(t), &run).unwrap(); let cur = segs.last_mut().unwrap(); *cur += &l; cur.push('\n'); + if b.n % 50 == 0 { let c = b.checkpoint(&s); *cur += &c; cur.push('\n'); let k = segs.len() as u64; + let h = segment_header(&b, &doc, "probbit 0.14.0", k, &segment_name("life.strand", k - 1), &persona::digest_of(&c)); segs.push(format!("{h}\n{c}\n")); } + sb.push(persona::canon(&s)); } + assert_eq!(sa, sb, "every stance document"); assert_eq!(a.st.digest, b.st.digest); assert_eq!(segs.len(), 11); + let mut joined: Vec<&str> = segs[0].lines().collect(); for t in &segs[1..] { joined.extend(t.lines().skip(2)); } + assert_eq!(joined, one.lines().collect::>(), "the segments are the one file"); + let fin = verify(&one, &run).unwrap(); assert_eq!(fin.get("events"), Some(&Json::Num(500.0))); + for (k, t) in segs.iter().enumerate() { let v = verify(t, &run).unwrap_or_else(|e| panic!("segment {k}: {e:?}")); + if k > 0 { assert_eq!(v.get("segment"), Some(&Json::Num(k as f64))); } if k == 10 { assert_eq!(v.get("final_state"), fin.get("final_state")); } } + let mut r = resume(&segs[10], &p, b.st.clone(), Clock::Fixed).unwrap(); let e = s1_event(500); + assert_eq!(r.event(&e, &run).unwrap().1, a.event(&e, &run).unwrap().1, "a resume in the newest segment"); + let mut l3: Vec<&str> = segs[3].lines().collect(); let other = segs[4].lines().nth(1).unwrap(); l3[1] = other; + assert_eq!(verify(&(l3.join("\n") + "\n"), &run).unwrap_err(), (2, SEALED.to_string())); + } + + /// The expectation's cache moves no byte: lives of the expectation fixture persona (max_combos 16 and 64, engines 0.13.0 and + /// 0.14.0; 500 events of many combos, then one event 200 times, a checkpoint every 50) run with the cache on and off: + /// every stance document, every strand line and the final state are the same, the cache answers programs (hits > 0) and + /// never holds more than max_combos + 1 rows; the life verifies + #[test] + fn the_expectation_cache_moves_no_byte() { + let y = std::fs::read_to_string(concat!(env!("CARGO_MANIFEST_DIR"), "/tests/fixtures/monitor/tutor-expect.yaml")).unwrap(); + let doc0 = probbit_persona::yaml::load(&y).unwrap_or_else(|e| panic!("{}", e.msg)); + let ev = |t: usize| s1_event(if t < 500 { t } else { 1 }); + for mc in [16usize, 64] { for engine in ["probbit 0.13.0", "probbit 0.14.0"] { + let mut d = doc0.clone(); put(&mut d, &["expectation"], json::parse(&format!(r#"{{"max_combos":{mc}}}"#)).unwrap()); let p = persona::build(&d).unwrap(); + let life = |on: bool| { let (mut lv, head) = Live::start(p.clone(), &d, persona::init(&p, Some(2), true, &run), Clock::Fixed, engine); + lv.expect_cache = persona::ExpectCache::new(on); assert_eq!(lv.expect_cache.is_on(), on); + let (mut text, mut docs, mut hits, mut rows) = (format!("{head}\n"), vec![], 0u32, 0usize); + for t in 0..700 { let (s, l) = lv.event(&ev(t), &run).unwrap(); text += &l; text.push('\n'); docs.push(persona::canon(&s)); + hits += lv.expect_cache.last().0; rows = rows.max(lv.expect_cache.len()); if lv.n % 50 == 0 { text += &lv.checkpoint(&s); text.push('\n'); } } + (text, docs, lv.st.digest.clone(), hits, rows) }; + let (on, off) = (life(true), life(false)); + assert_eq!(on.1, off.1, "every stance document ({mc}, {engine})"); assert_eq!(on.0, off.0, "every strand byte ({mc}, {engine})"); assert_eq!(on.2, off.2); + assert!(on.3 > 0 && off.3 == 0, "hits on {} off {}", on.3, off.3); assert!(on.4 <= mc + 1, "rows {}", on.4); + assert_eq!(verify(&on.0, &run).unwrap().get("events"), Some(&Json::Num(700.0))); } } + } + + /// T3: the expectation's members as models move no byte. The same lives run with the cache off (every member's program + /// built as text and run by the engine: the reference) and on (each member compiled without its text and solved as the + /// event's own model with its own `h`; another tier or other rules in force: built and run): every stance document, every + /// strand line (a checkpoint every 10) and the final state are the same. A sensed persona whose readings are floats that move + /// every event (a number sense, two needs, drives, a habit on a flag: rules that differ between members), engines 0.13.0 + /// and 0.14.0, max_combos 6 and 16; and the expectation fixture persona (500 varied events, max_combos 16). Both paths run. + #[test] + fn the_members_as_models_move_no_byte() { + let load = |rel: &str| probbit_persona::yaml::load(&std::fs::read_to_string(format!("{}/../{rel}", env!("CARGO_MANIFEST_DIR"))).unwrap()).unwrap_or_else(|e| panic!("{}", e.msg)); + let sensed = |t: usize| { let h = (t as u64).wrapping_mul(2654435761) >> 7; let b = |k: u64| (h >> k) & 1 == 1; + let needs = if b(9) && b(10) { r#","needs":{"context":0.2}"# } else { "" }; + json::parse(&format!(r#"{{"quiet":{},"senses":{{"tokens":{}}},"src":"env:host","elapsed_hours":{}{needs}}}"#, b(2) && b(5), ((h >> 11) % 997) as f64 / 1000.0, [0.1, 0.5, 1.0][((h >> 15) % 3) as usize])).unwrap() }; + let mut cases: Vec<(Json, usize, &str, usize, bool)> = vec![]; + for mc in [6usize, 16] { for engine in ["probbit 0.13.0", "probbit 0.14.0"] { let mut d = load("bench/body/fixtures/needs2.yaml"); + put(&mut d, &["expectation"], json::parse(&format!(r#"{{"max_combos":{mc}}}"#)).unwrap()); cases.push((d, 120, engine, 0, true)); } } + let mut d = load("probbit-cli/tests/fixtures/monitor/tutor-expect.yaml"); put(&mut d, &["expectation"], json::parse(r#"{"max_combos":16}"#).unwrap()); cases.push((d, 500, "probbit 0.14.0", 1, false)); + for (d, n, engine, kind, fallback) in &cases { let p = persona::build(d).unwrap_or_else(|e| panic!("{}: {}", e.path, e.msg)); + let ev = |t: usize| if *kind == 0 { sensed(t) } else { s1_event(t) }; + let life = |on: bool| { let (mut lv, head) = Live::start(p.clone(), d, persona::init(&p, Some(2), true, &run), Clock::Fixed, engine); + // A_t built and solved by the deviation (as T3 measured it): the turn's own program is the T3-b test's + lv.expect_cache = persona::ExpectCache::new(on).with_reuse(false, false); let (mut text, mut docs, mut modeled, mut texted) = (format!("{head}\n"), vec![], 0u32, 0u32); + for t in 0..*n { let (s, l) = lv.event(&ev(t), &run).unwrap_or_else(|e| panic!("event {t}: {}: {}", e.path, e.msg)); text += &l; text.push('\n'); docs.push(persona::canon(&s)); + let (m, (_, sv)) = (lv.expect_cache.modeled(), lv.expect_cache.last()); modeled += m; texted += sv - m; if lv.n % 10 == 0 { text += &lv.checkpoint(&s); text.push('\n'); } } + (text, docs, lv.st.digest.clone(), modeled, texted) }; + let (on, off) = (life(true), life(false)); let ctx = format!("{} events, {engine}, {}", n, persona::canon(d.get("expectation").unwrap())); + assert_eq!(on.1, off.1, "every stance document ({ctx})"); assert_eq!(on.0, off.0, "every strand byte ({ctx})"); assert_eq!(on.2, off.2, "the final state ({ctx})"); + assert!(on.3 > 0 && off.3 == 0, "programs solved as models: on {} off {} ({ctx})", on.3, off.3); + if *fallback { assert!(on.4 > 0, "programs built and run with the cache on: {} ({ctx})", on.4); } + assert_eq!(verify(&on.0, &run).unwrap().get("events"), Some(&Json::Num(*n as f64)), "{ctx}"); } + } + + /// T3-b: the deviation's A_t read from the turn moves no byte. The same lives run with the cache on, the turn's first program + /// handed to the deviation (`turn_at`: its meta and exact marginals; A_t checked against a second build and solve on every + /// event, text for text and bit for bit) and not handed (A_t built and solved by the deviation, as before): every stance + /// document, every strand line (a checkpoint every 10) and the final state are the same, and A_t is read from the turn. The + /// sensed persona (floats that move every event, two needs, drives, a habit on a flag), 500 events, max_combos 6 and 16, + /// engines 0.13.0 and 0.14.0; the expectation fixture persona, 500 varied events, max_combos 16 + #[test] + fn the_turns_own_program_as_a_t_moves_no_byte() { + let load = |rel: &str| probbit_persona::yaml::load(&std::fs::read_to_string(format!("{}/../{rel}", env!("CARGO_MANIFEST_DIR"))).unwrap()).unwrap_or_else(|e| panic!("{}", e.msg)); + let sensed = |t: usize| { let h = (t as u64).wrapping_mul(2654435761) >> 7; let b = |k: u64| (h >> k) & 1 == 1; + let needs = if b(9) && b(10) { r#","needs":{"context":0.2}"# } else { "" }; + json::parse(&format!(r#"{{"quiet":{},"senses":{{"tokens":{}}},"src":"env:host","elapsed_hours":{}{needs}}}"#, b(2) && b(5), ((h >> 11) % 997) as f64 / 1000.0, [0.1, 0.5, 1.0][((h >> 15) % 3) as usize])).unwrap() }; + let mut cases: Vec<(Json, &str, usize)> = vec![]; + for mc in [6usize, 16] { for engine in ["probbit 0.13.0", "probbit 0.14.0"] { let mut d = load("bench/body/fixtures/needs2.yaml"); + put(&mut d, &["expectation"], json::parse(&format!(r#"{{"max_combos":{mc}}}"#)).unwrap()); cases.push((d, engine, 0)); } } + let mut d = load("probbit-cli/tests/fixtures/monitor/tutor-expect.yaml"); put(&mut d, &["expectation"], json::parse(r#"{"max_combos":16}"#).unwrap()); cases.push((d, "probbit 0.14.0", 1)); + for (d, engine, kind) in &cases { let p = persona::build(d).unwrap_or_else(|e| panic!("{}: {}", e.path, e.msg)); + let ev = |t: usize| if *kind == 0 { sensed(t) } else { s1_event(t) }; + let life = |reuse: bool| { let (mut lv, head) = Live::start(p.clone(), d, persona::init(&p, Some(2), true, &run), Clock::Fixed, engine); + lv.expect_cache = persona::ExpectCache::new(true).with_reuse(reuse, reuse); assert_eq!(lv.expect_cache.reuses(), reuse); + let (mut text, mut docs, mut reused) = (format!("{head}\n"), vec![], 0u32); + for t in 0..500 { let (s, l) = lv.event(&ev(t), &run).unwrap_or_else(|e| panic!("event {t}: {}: {}", e.path, e.msg)); text += &l; text.push('\n'); docs.push(persona::canon(&s)); + reused += lv.expect_cache.reused(); if lv.n % 10 == 0 { text += &lv.checkpoint(&s); text.push('\n'); } } + (text, docs, lv.st.digest.clone(), reused) }; + let (on, off) = (life(true), life(false)); let ctx = format!("{engine}, {}", persona::canon(d.get("expectation").unwrap())); + assert_eq!(on.1, off.1, "every stance document ({ctx})"); assert_eq!(on.0, off.0, "every strand byte ({ctx})"); assert_eq!(on.2, off.2, "the final state ({ctx})"); + assert!(on.3 > 0 && off.3 == 0, "A_t read from the turn: on {} off {} ({ctx})", on.3, off.3); + assert_eq!(verify(&on.0, &run).unwrap().get("events"), Some(&Json::Num(500.0)), "{ctx}"); } + } + + /// T3-c: the members before the turn move no byte. The same lives run with the cache reusing the turn's program (the members + /// but A_t solved before the turn and held as rows, A_t's mixture after it; A_t checked against a second build and solve on + /// every event), with the old order (`with_reuse(false, false)`: the whole deviation before the turn) and with the cache off + /// (the reference): every stance document, every strand line (a checkpoint every 10) and the final state are the same, A_t + /// is read from the turn and members are solved as the model (with other rules in force than the skeleton's too: their rules + /// lowered into caps). The sensed persona (floats that move every event, two needs, + /// drives, a habit on a flag), 500 events, max_combos 6 and 16, engines 0.13.0 and 0.14.0; the expectation fixture persona, + /// 500 varied events, max_combos 16 + #[test] + fn the_members_before_the_turn_move_no_byte() { + let load = |rel: &str| probbit_persona::yaml::load(&std::fs::read_to_string(format!("{}/../{rel}", env!("CARGO_MANIFEST_DIR"))).unwrap()).unwrap_or_else(|e| panic!("{}", e.msg)); + let sensed = |t: usize| { let h = (t as u64).wrapping_mul(2654435761) >> 7; let b = |k: u64| (h >> k) & 1 == 1; + let needs = if b(9) && b(10) { r#","needs":{"context":0.2}"# } else { "" }; + json::parse(&format!(r#"{{"quiet":{},"senses":{{"tokens":{}}},"src":"env:host","elapsed_hours":{}{needs}}}"#, b(2) && b(5), ((h >> 11) % 997) as f64 / 1000.0, [0.1, 0.5, 1.0][((h >> 15) % 3) as usize])).unwrap() }; + let mut cases: Vec<(Json, &str, usize)> = vec![]; + for mc in [6usize, 16] { for engine in ["probbit 0.13.0", "probbit 0.14.0"] { let mut d = load("bench/body/fixtures/needs2.yaml"); + put(&mut d, &["expectation"], json::parse(&format!(r#"{{"max_combos":{mc}}}"#)).unwrap()); cases.push((d, engine, 0)); } } + let mut d = load("probbit-cli/tests/fixtures/monitor/tutor-expect.yaml"); put(&mut d, &["expectation"], json::parse(r#"{"max_combos":16}"#).unwrap()); cases.push((d, "probbit 0.14.0", 1)); + for (d, engine, kind) in &cases { let p = persona::build(d).unwrap_or_else(|e| panic!("{}: {}", e.path, e.msg)); + let ev = |t: usize| if *kind == 0 { sensed(t) } else { s1_event(t) }; + let life = |cache: persona::ExpectCache| { let (mut lv, head) = Live::start(p.clone(), d, persona::init(&p, Some(2), true, &run), Clock::Fixed, engine); + lv.expect_cache = cache; let (mut text, mut docs, mut reused, mut modeled, mut texted) = (format!("{head}\n"), vec![], 0u32, 0u32, 0u32); + for t in 0..500 { let (s, l) = lv.event(&ev(t), &run).unwrap_or_else(|e| panic!("event {t}: {}: {}", e.path, e.msg)); text += &l; text.push('\n'); docs.push(persona::canon(&s)); + let (m, (_, sv)) = (lv.expect_cache.modeled(), lv.expect_cache.last()); reused += lv.expect_cache.reused(); modeled += m; texted += sv - m; + if lv.n % 10 == 0 { text += &lv.checkpoint(&s); text.push('\n'); } } + (text, docs, lv.st.digest.clone(), reused, modeled, texted) }; + let (new, old, off) = (life(persona::ExpectCache::new(true).with_reuse(true, true)), life(persona::ExpectCache::new(true).with_reuse(false, false)), life(persona::ExpectCache::new(false))); + let ctx = format!("{engine}, {}", persona::canon(d.get("expectation").unwrap())); + for (x, what) in [(&old, "the old order"), (&off, "the cache off")] { + assert_eq!(new.1, x.1, "every stance document vs {what} ({ctx})"); assert_eq!(new.0, x.0, "every strand byte vs {what} ({ctx})"); assert_eq!(new.2, x.2, "the final state vs {what} ({ctx})"); } + assert!(new.3 > 0 && old.3 == 0 && new.4 > 0, "A_t read from the turn: new {} old {}; members as models {} ({ctx})", new.3, old.3, new.4); + // the fixture's members have other habits in force than A_t: with their rules lowered into caps they take the model too + eprintln!("T3-c members ({ctx}): modeled new {} old {}, built and run new {} old {}", new.4, old.4, new.5, old.5); + if *kind == 1 { assert!(new.4 > old.4 && new.5 < old.5, "rules into caps: modeled new {} old {}, built and run new {} old {} ({ctx})", new.4, old.4, new.5, old.5); } + assert_eq!(verify(&new.0, &run).unwrap().get("events"), Some(&Json::Num(500.0)), "{ctx}"); } + } + + /// S2 (docs/persona.md §5.7, "Resume from a record"): a life recorded unbroken, and the same life resumed by each path: the + /// boxed replay (every stance document and the individual it reaches), a record 1, 5 and 49 lines behind the strand + /// (`resume_from_record`: the trailing event lines applied, not solved), and the embedded resume from the header, each + /// checkpoint line and the lines after it (`resume_boxed_from`; the record last, and 5 behind). Each reaches the unbroken + /// state, chain head and expectation memory, and its next 10 events write the unbroken lines. Refused: a record the strand + /// does not log (or one before the checkpoint), a forged line re-chained, another persona. Expectation fixture persona, + /// engines 0.13.0 and 0.14.0, a checkpoint every 50; the glyph persona (no expectation block) with k = 1 and 3. + #[test] + fn every_resume_path_writes_the_unbroken_life() { + let y = std::fs::read_to_string(concat!(env!("CARGO_MANIFEST_DIR"), "/tests/fixtures/monitor/tutor-expect.yaml")).unwrap(); + let d = probbit_persona::yaml::load(&y).unwrap_or_else(|e| panic!("{}", e.msg)); let p = persona::build(&d).unwrap(); + let mut d2 = d.clone(); put(&mut d2, &["identity", "version"], Json::Str("9".into())); let p2 = persona::build(&d2).unwrap(); assert_ne!(p2.digest, p.digest); + let mem = |l: &Live| l.expect.as_ref().map(|m| persona::canon(&m.to_json())); + let next10 = |l: &mut Live, from: usize| (from..from + 10).map(|t| l.event(&s1_event(t), &run).unwrap().1).collect::>(); + for engine in ["probbit 0.13.0", "probbit 0.14.0"] { for n_end in [499usize, 500] { + // the unbroken life: the text, the documents, the state after every event, then 10 more events + let (mut lv, head) = Live::start(p.clone(), &d, persona::init(&p, Some(2), true, &run), Clock::Fixed, engine); + let (mut text, mut docs, mut sts) = (format!("{head}\n"), vec![], vec![lv.st.clone()]); + for t in 0..n_end { let (s, l) = lv.event(&s1_event(t), &run).unwrap(); text += &l; text.push('\n'); docs.push(persona::canon(&s)); sts.push(lv.st.clone()); + if lv.n % 50 == 0 { text += &lv.checkpoint(&s); text.push('\n'); } } + let (want_st, want_head, want_mem) = (lv.st.digest.clone(), lv.head().to_string(), mem(&lv)); let want_next = next10(&mut lv, n_end); + let ctx = format!("{engine}, {n_end} events"); + let same = |r: &mut Live, how: &str| { assert_eq!((r.n, r.st.digest.as_str(), r.head()), (n_end as u64, want_st.as_str(), want_head.as_str()), "{how} ({ctx})"); + assert_eq!(mem(&*r), want_mem, "{how}: the expectation's memory ({ctx})"); assert_eq!(next10(r, n_end), want_next, "{how}: the next 10 lines ({ctx})"); }; + // A: the boxed replay, every stance document + let ls = lines(&text); let mut r = Replay::open_boxed(ls[0]).unwrap(); let mut k = 0; + for l in &ls[1..] { if let Some(s) = r.step(l, &run).unwrap() { assert_eq!(persona::canon(&s), docs[k], "boxed replay, event {} ({ctx})", k + 1); k += 1; } } + assert_eq!(k, n_end); same(&mut *r.live, "the boxed replay"); + // B: a record k lines behind (k = 0: the last; it equals resume's) + for kb in [0usize, 1, 5, 49] { + let mut b = resume_from_record(&text, &p, Box::new(sts[n_end - kb].clone()), Clock::Fixed, &run).unwrap_or_else(|e| panic!("k {kb}: {}: {} ({ctx})", e.path, e.msg)); + same(&mut *b, &format!("a record {kb} behind")); } + let mut b = resume(&text, &p, sts[n_end].clone(), Clock::Fixed).unwrap(); same(&mut b, "resume"); + // C: from the header (no checkpoint) and from every checkpoint line, the trailing lines one at a time + let cps: Vec = (1..ls.len()).filter(|&i| ls[i].starts_with(r#"{"checkpoint":"#)).collect(); + if engine == "probbit 0.13.0" { assert!(resume_boxed_from(ls[0], Some(ls[cps[0]]), cps[0] + 1, ls[cps[0] + 1..].iter(), &p, Box::new(sts[n_end].clone()), Clock::Fixed, &run).is_err()); } + let starts: Vec> = if engine == "probbit 0.13.0" { vec![None] } else { std::iter::once(None).chain(cps.iter().copied().map(Some)).collect() }; + for c in starts { for kb in [0usize, 5] { let (cp, from) = c.map_or((None, 1), |c| (Some(ls[c]), c + 1)); + // a record older than the checkpoint is refused (below); the walk starts at the checkpoint + if cp.and_then(|l| json::parse(l).ok()).and_then(|j| j.get("checkpoint").and_then(Json::as_f64)).is_some_and(|cn| cn as usize > n_end - kb) { continue; } + let tr: Vec = ls[from..].iter().map(|l| l.to_string()).collect(); + let mut x = resume_boxed_from(ls[0], cp, from, tr, &p, Box::new(sts[n_end - kb].clone()), Clock::Fixed, &run) + .unwrap_or_else(|e| panic!("from line {from}, k {kb}: {}: {} ({ctx})", e.path, e.msg)); + same(&mut *x, &format!("the embedded resume from line {from}, k {kb}")); } } + // refusals: a record from another life; a record before the checkpoint (C); a forged line, re-chained; another persona + let other = persona::init(&p, Some(9), true, &run); + assert_eq!(resume_from_record(&text, &p, Box::new(other.clone()), Clock::Fixed, &run).err().unwrap().path, "state"); + if let Some(&c) = cps.last() { if engine == "probbit 0.14.0" { + assert_eq!(resume_boxed_from(ls[0], Some(ls[c]), c + 1, ls[c + 1..].iter(), &p, Box::new(sts[1].clone()), Clock::Fixed, &run).err().unwrap().path, "state"); } } + let mut f: Vec = ls.iter().map(|l| l.to_string()).collect(); let at = f.len() - 3; + let mut j = json::parse(&f[at]).unwrap(); if let Json::Obj(v) = &mut j { if let Some((_, Json::Obj(ins))) = v.iter_mut().find(|(k, _)| k == "inputs") { + if let Some((_, x)) = ins.iter_mut().find(|(k, _)| k == "praise") { *x = Json::Bool(!matches!(x, Json::Bool(true))); } } } + f[at] = persona::canon(&j); + for i in at + 1..f.len() { let mut j = json::parse(&f[i]).unwrap(); let pd = persona::digest_of(&f[i - 1]); + if let Json::Obj(v) = &mut j { if let Some((_, x)) = v.iter_mut().find(|(k, _)| k == "prev") { *x = Json::Str(pd); } } f[i] = persona::canon(&j); } + let forged = f.join("\n") + "\n"; + let e = resume_from_record(&forged, &p, Box::new(sts[n_end - 5].clone()), Clock::Fixed, &run).err().unwrap(); + assert!(e.msg.contains(&format!("line {}: the ", at + 1)) && e.msg.contains("differs"), "forged: {} ({ctx})", e.msg); + assert!(resume_from_record(&text, &p2, Box::new(sts[n_end - 5].clone()), Clock::Fixed, &run).err().unwrap().msg.contains("another persona")); + } } + // no expectation block, with glyphs: the logged glyph is taken as written and the line checked around it + let on = with_display(&json::parse(GDOC).unwrap(), &[("display", GLYPHS)]); let pn = persona::build(&on).unwrap(); + let (mut lv, head) = Live::start(pn.clone(), &on, persona::init(&pn, Some(1), true, &run), Clock::Fixed, "probbit test"); + let (mut text, mut sts) = (format!("{head}\n"), vec![lv.st.clone()]); + for t in 0..40 { let (_, l) = lv.event(&gev(t), &run).unwrap(); text += &l; text.push('\n'); sts.push(lv.st.clone()); } + let want: Vec = (40..50).map(|t| lv.event(&gev(t), &run).unwrap().1).collect(); + for kb in [1usize, 3] { let mut b = resume_from_record(&text, &pn, Box::new(sts[40 - kb].clone()), Clock::Fixed, &run).unwrap(); + assert_eq!((40..50).map(|t| b.event(&gev(t), &run).unwrap().1).collect::>(), want, "glyph persona, k {kb}"); } + } + + /// S1 measurements (ignored: run with --ignored --nocapture): the expectation fixture persona, 500 events, a checkpoint every + /// 50: checkpoint line bytes (max_combos 16 and 64), and resume's time at 166 and 500 events, 0.13.0 (fold from the header) + /// against 0.14.0 (fold from the last checkpoint), with the full replay for scale + #[test] + #[ignore] + fn s1_measure() { + let y = std::fs::read_to_string(concat!(env!("CARGO_MANIFEST_DIR"), "/tests/fixtures/monitor/tutor-expect.yaml")).unwrap(); + let doc0 = probbit_persona::yaml::load(&y).unwrap_or_else(|e| panic!("{}", e.msg)); + let ev = s1_event; + for mc in [16usize, 64] { + let mut d = doc0.clone(); put(&mut d, &["expectation"], json::parse(&format!(r#"{{"max_combos":{mc}}}"#)).unwrap()); let p = persona::build(&d).unwrap(); + for engine in ["probbit 0.13.0", "probbit 0.14.0"] { + let (mut lv, head) = Live::start(p.clone(), &d, persona::init(&p, Some(2), true, &run), Clock::Fixed, engine); + let (mut text, mut ev_b, mut cp_max, mut sts) = (format!("{head}\n"), 0usize, 0usize, vec![]); + for t in 0..500 { let (s, l) = lv.event(&ev(t), &run).unwrap(); ev_b = ev_b.max(l.len()); text += &l; text.push('\n'); + if lv.n % 50 == 0 { let c = lv.checkpoint(&s); cp_max = cp_max.max(c.len()); text += &c; text.push('\n'); } if lv.n == 166 || lv.n == 500 { sts.push((text.clone(), lv.st.clone())); } } + let named = lv.expect.as_ref().map(|m| m.to_json().get("combos").and_then(Json::as_arr).map_or(0, |a| a.len())).unwrap_or(0); + println!("S1 max_combos={mc} engine={engine}: header {} B, event line max {ev_b} B, checkpoint line max {cp_max} B, memory entries at 500: {named}", head.len()); + for (n, (t, st)) in [166usize, 500].into_iter().zip(&sts) { + let t0 = std::time::Instant::now(); let r = resume(t, &p, st.clone(), Clock::Fixed).unwrap(); let dt = t0.elapsed(); drop(r); + let t1 = std::time::Instant::now(); let v = verify(t, &run).unwrap(); let dv = t1.elapsed(); assert_eq!(v.get("events"), Some(&Json::Num(n as f64))); + let t2 = std::time::Instant::now(); let f = verify_from_checkpoint(t, &run); let df = t2.elapsed(); + println!("S1 n={n}: resume {:.1} ms, verify from header {:.1} ms, verify --from-checkpoint {} ({:.1} ms)", dt.as_secs_f64() * 1e3, dv.as_secs_f64() * 1e3, + if f.is_ok() { "ok" } else { "refused" }, df.as_secs_f64() * 1e3); } + } + } + } + /// §2.18: a persona with two moods and a habit that binds on a loss const GDOC: &str = r#"{"probbit_persona":1,"identity":{"name":"G","version":"1","seed":5}, "traits":[{"id":"humour","levels":["none","light","playful"],"logw":[0,1.0,0.8]},{"id":"caution","levels":["bold","measured","careful"],"logw":[0.3,0.4,0]}], @@ -1356,6 +1169,44 @@ mod tests { for i in from.max(1)..ls.len() { let mut j = json::parse(&ls[i]).unwrap(); put(&mut j, &["prev"], Json::Str(persona::digest_of(&ls[i - 1]))); ls[i] = persona::canon(&j); } ls.join("\n") + "\n" } + /// Cross-platform parity of the deviation (0.14.0): a strand of 0.13.0 with an expectation block and checkpoint lines (the + /// stance's `deviation` at full precision, as that engine wrote it) verifies as recorded, and verifies with a deviation's last + /// bits changed as another platform's libm gives them (the chain re-made after it); a change at the 6th decimal still + /// diverges. From 0.14.0 every number of the deviation is at 6 decimals, so the line is the same on every libm. The fixture + /// was recorded by the 0.13.0 release build (Linux x86_64): `probbit live` on this module's DOC with `"expectation": {}`, + /// `--seed 2 --clock fixed --checkpoint-every 4`, 16 events (3 time-only, then `ev(3..16)`). + #[test] + fn the_deviation_is_the_same_on_every_libm() { + assert_eq!(engine_version("probbit 0.13.0"), Some((0, 13, 0))); assert_eq!(engine_version("probbit 0.14.0-dev"), Some((0, 14, 0))); + assert_eq!(engine_version("probbit test"), None); assert_eq!(engine_version("probbit 0.14"), None); assert_eq!(engine_version("other 0.13.0"), None); + const F: &str = include_str!("../tests/fixtures/strands/expectation-0.13.0.strand"); + let ok = verify(F, &run).unwrap(); assert_eq!((ok.get("engine"), ok.get("checkpoints")), (Some(&Json::Str("probbit 0.13.0".into())), Some(&Json::Num(4.0)))); + let ls: Vec = F.lines().map(str::to_string).collect(); + let dev = |l: &str, k: &str| json::parse(l).unwrap().get("stance").and_then(|s| s.get("deviation")).and_then(|d| d.get(k)).and_then(Json::as_f64); + let at = ls.iter().position(|l| dev(l, "pred").is_some_and(|x| persona::r6(x) != x)).expect("a checkpoint with a full-precision deviation"); + let pred = dev(&ls[at], "pred").unwrap(); + let with = |x: f64| { let mut j = json::parse(&ls[at]).unwrap(); put(&mut j, &["stance", "deviation", "pred"], Json::Num(x)); let mut tt = ls.clone(); tt[at] = persona::canon(&j); rechain(tt, at) }; + // 2 ULP (the `libm` crate of a nostd build against the platform's libm): verifies + let other = with(f64::from_bits(pred.to_bits() + 2)); assert_ne!(other, F); + assert_eq!(verify(&other, &run).map(|d| d.get("events").cloned()), Ok(ok.get("events").cloned())); + // a change the 6 decimals show: diverges at that line + assert_eq!(verify(&with(pred + 1e-5), &run).unwrap_err(), (at + 1, "the checkpoint's state or stance differs from the replay".to_string())); + // from 0.14.0: the same life logs every number of the deviation at 6 decimals and verifies; 0.13.0 logged more + let h = json::parse(&ls[0]).unwrap(); let doc = h.get("document").unwrap().clone(); let p = persona::build(&doc).unwrap(); + let evs: Vec = ls[1..].iter().filter_map(|l| json::parse(l).ok()).filter(|j| matches!(kind(j), Kind::Event)).map(|j| j.get("inputs").unwrap().clone()).collect(); + let life = |engine: &str| { let st = State::read(&p, h.get("state").unwrap()).unwrap(); let (mut lv, head) = Live::start(p.clone(), &doc, st, Clock::Fixed, engine); let mut text = format!("{head}\n"); + for e in &evs { let (s, l) = lv.event(e, &run).unwrap(); text += &l; text.push('\n'); if lv.n % 4 == 0 { text += &lv.checkpoint(&s); text.push('\n'); } } text }; + let (old, new) = (life("probbit 0.13.0"), life("probbit 0.14.0")); + assert_eq!(old, F.replace("\r\n", "\n"), "0.13.0's bytes, as recorded (a Windows checkout has CRLF)"); + let nums = |t: &str| t.lines().filter_map(|l| json::parse(l).ok()?.get("stance")?.get("deviation").cloned()).flat_map(|d| ["self", "pred", "input", "z"].map(|k| d.get(k).and_then(Json::as_f64))).flatten().collect::>(); + let n = nums(&new); assert_eq!(n.len(), 16); assert!(n.iter().all(|x| persona::r6(*x) == *x), "{n:?}"); assert!(nums(&old).iter().any(|x| persona::r6(*x) != *x)); + assert_eq!(verify(&new, &run).map(|d| d.get("checkpoints").cloned()), Ok(Some(Json::Num(4.0)))); + // a 0.14.0 checkpoint is checked byte for byte: a rounded number's last bits changed diverge + let ln: Vec = new.lines().map(str::to_string).collect(); let c = ln.iter().position(|l| dev(l, "pred").is_some_and(|x| x > 0.0)).unwrap(); + let mut j = json::parse(&ln[c]).unwrap(); put(&mut j, &["stance", "deviation", "pred"], Json::Num(f64::from_bits(dev(&ln[c], "pred").unwrap().to_bits() + 2))); let mut tt = ln.clone(); tt[c] = persona::canon(&j); + assert_eq!(verify(&rechain(tt, c), &run).unwrap_err().0, c + 1); + } + /// docs/persona.md §5.7, "Unsupported": a header key, a persona field, a state field, a line kind or a line field this engine /// does not know (a newer engine's strand, its chain whole) is `unsupported by this engine: FIELD` at the line that has it, /// never a divergence; `verify_doc` (the MCP tool's document) is {ok, line, unsupported, engine, strand_engine}. A tampered diff --git a/probbit-cli/src/main.rs b/probbit-cli/src/main.rs index 7bee82a..5b0b80b 100644 --- a/probbit-cli/src/main.rs +++ b/probbit-cli/src/main.rs @@ -17,23 +17,18 @@ //! probbit --help | -h # usage, every flag with its default, exit codes //! probbit --help | -h # the usage above (stdout, exit 0); at a terminal the hero screen //! --top, --summary (decide, run), --live (demo), --plain: see `probbit --help`; visuals go to stderr (theme.rs) -mod evaluate; mod fuzz; -mod json; mod labels; mod live; mod mcp; mod monitor; -mod persona; -mod router; -mod run; -mod sys; mod theme; mod tui; mod usage; -mod yaml; +// the document layer and the strand writer: the probbit-persona library (the CLI's modules name them as `crate::json` etc.) +use probbit_persona::{evaluate, json, persona, router, run, sys}; use json::{num, obj, str as jstr, Json}; -pub(crate) use router::{demo_doc, from_json, item_bars, DEMO_TPL}; +pub(crate) use router::{demo_doc, from_json, DEMO_TPL}; use probbit_decide::*; use std::io::{Read, Write}; use usage::{Command, Flag, Sub}; @@ -99,10 +94,6 @@ fn exact_ms(args: &[String]) -> Option { args.iter().any(|a| a == "--exact-ms").then(|| { let x: f64 = arg(args, "--exact-ms", 0.0); if !(x.is_finite() && (0.0..=1e9).contains(&x)) { fail("--exact-ms must be a number of milliseconds from 0 to 1e9") } x }) } -/// Telemetry fields of a sampled answer when --exact-ms was given (none otherwise): the cap and whether the exact tiers hit it. -pub(crate) fn exact_cap(xms: Option, reached: bool) -> Vec<(&'static str, Json)> { - xms.map_or(vec![], |x| vec![("exact_ms", num(x)), ("exact_ms_reached", Json::Bool(reached))]) -} /// Every stdout document goes through here. `println!` panicked on a closed pipe (`probbit demo --tasks 3000 | head -c 20`: /// "failed printing to stdout: Broken pipe", exit 134 with panic = abort); a reader that went away is not an error, so a broken /// pipe is ignored and the command keeps its normal exit code. @@ -248,7 +239,7 @@ const F_P_SEEDS: Flag = Flag { name: "--seeds", arg: "LIST", help: "the individu const F_P_THREADS: Flag = Flag { name: "--threads", arg: "N", help: "search threads, 1 to 1024 (default: the logical CPUs)" }; const F_P_JSON: Flag = Flag { name: "--json", arg: "", help: "the JSON document instead of the readable report" }; const EXIT_PERSONA: &str = "0 done (every turn status, refusals and fallbacks included, is an answer; fuzz: no counterexample found; prove: every rule held or proved), 1 lint found an unresolved contradiction or (with rules) a counterexample, fuzz found a counterexample or prove left a rule unknown, 2 bad persona / state / inputs / script / rule (one {\"error\"} object on stdout, code \"persona\", when stdout is not a terminal or with --json) or bad flag (stderr), 3 lint (with rules): a rule the bound leaves unknown and the search cannot break (\"unproven\", not broken)."; -const EXIT_LIVE: &str = "0 done (live: every event answered; verify: every line replays; control: the line appended), 1 verify: a line differs (its number and what) or, with --actions, a host action differs from its turn's released level, 2 a bad persona, state, event (one {\"error\"} object on stdout per bad event when stdout is not a terminal or with --json; the run goes on), control line or flag, 4 refused: the strand's writer lock is held by another writer, or the individual is paused or retired (one {\"error\"} object per refused event, as for 2; nothing is written), 5 verify: unsupported, a line has a field or line kind this engine does not know (a newer engine's strand: {\"ok\": false, \"line\", \"unsupported\": the field, \"engine\", \"strand_engine\"}; not verified, and not a divergence)."; +const EXIT_LIVE: &str = "0 done (live: every event answered; verify: every line replays; control: the line appended), 1 verify: a line differs (its number and what) or, with --actions, a host action differs from its turn's released level, 2 a bad persona, state, event (one {\"error\"} object on stdout per bad event when stdout is not a terminal or with --json; the run goes on), control line or flag, 4 refused: the strand's writer lock is held by another writer, or the individual is paused or retired (one {\"error\"} object per refused event, as for 2; nothing is written), 5 verify: unsupported, a line has a field or line kind this engine does not know (a newer engine's strand: {\"ok\": false, \"line\", \"unsupported\": the field, \"engine\", \"strand_engine\"}; not verified, and not a divergence), 6 verify --chain: a previous segment file is missing ({\"ok\": false, \"missing\": its file name, \"segment_file\": the segment that links it})."; pub(crate) static COMMANDS: &[Command] = &[ Command { name: "persona", aliases: &[], group: "persona", summary: "the individuality layer: persona + state + inputs -> one stance", usage: &["persona SUB PERSONA [flags] (SUB: init turn replay explain diff lint fuzz prove check compile describe)"], @@ -303,10 +294,13 @@ pub(crate) static COMMANDS: &[Command] = &[ "probbit persona lint examples/persona/tutor.yaml"], run: persona_cmd }, Command { name: "live", aliases: &[], group: "live", summary: "a resident individual: JSONL events in, one stance per event out", - usage: &["live PERSONA [--seed N | --state FILE] [--strand FILE [--label KEY=VALUE]...] [--events FILE [--watch]] [--clock real|fixed] [--checkpoint-every K]", + usage: &["live PERSONA [--seed N | --state FILE] [--strand FILE [--label KEY=VALUE]...] [--events FILE [--watch]] [--clock real|fixed] [--checkpoint-every K] [--segment-every N]", "live PERSONA --demo week [--seed N] [--strand FILE [--label KEY=VALUE]...] [--plain]", - "live verify STRAND [--from-checkpoint] [--actions FILE] [--blocks FILE]", - "live control STRAND pause|resume|retire|erase --by WHO --reason TEXT [--at TIME]"], + "live verify STRAND [--from-checkpoint] [--chain] [--actions FILE] [--blocks FILE]", + "live control STRAND pause|resume|retire|erase --by WHO --reason TEXT [--at TIME]", + "live control STRAND reflect [--last N | --since-checkpoint]", + "live control STRAND stir? [--idle-hours H]", + "live control STRAND rest [--at TIME] [--state FILE]"], about: "A resident individual (docs/persona.md §5.7): JSONL events (one object of inputs per line) from --events FILE or stdin, one stance per event on stdout (canonical JSON). The clock stamps each event's elapsed_hours (quantised to 1e-6 h), so moods decay by their half-lives between events; feedback moves the learned deltas of a persona with a learning block (§2.8).\n--strand FILE logs the life: a header (persona document, initial state, engine version), then per event the inputs as used, the stance and state digests and the sha256 of the line before. `probbit live verify STRAND` replays it.", before: &[], flags: &[&[ @@ -319,18 +313,24 @@ pub(crate) static COMMANDS: &[Command] = &[ Flag { name: "--demo", arg: "week", help: "one individual's scripted week on the fixed clock (7 days, events hourly 09:00-15:00): praise for short answers moves the learned verbosity deltas to their cap, a quiet night relaxes the mood to its resting level, a campaign praising jokes raises humour while failure turns stay joke-free; a persona without a learning block gets the demo's (said in the opening line). Bars on stderr at a terminal, paced 1 s per hour (nights fast-forward in 2 s); otherwise one line per event on stdout, no waiting. With --seed, --strand" }, F_PLAIN, Flag { name: "--checkpoint-every", arg: "K", help: "with --strand: a checkpoint line (the whole state) after every K-th event (default 1000; 0: none); `verify --from-checkpoint` and `monitor` start at the last one" }, + Flag { name: "--segment-every", arg: "N", help: "with --strand NAME.strand: seal the strand after every N-th event (a sealing checkpoint line) and go on in the next segment file NAME.0001.strand, NAME.0002.strand, ...: its header links the previous file (its name, the sealed line's sha256, the index) and its second line is the sealed line, so the chain continues across files; a run continues the newest segment (default 0: one file)" }, Flag { name: "--label", arg: "KEY=VALUE", help: "with --strand, repeatable: a presentation label for the monitor (agent, host, lane, session, or a free key; at most 8 keys, values of 1 to 64 printable characters) merged into the strand's sidecar FILE.labels.json (FILE without its extension); the strand itself, its chain and its digests are not touched" }, ]], subs: &[ - Sub { name: "verify", args: "STRAND [--from-checkpoint] [--actions FILE] [--blocks FILE]", help: "replay a strand: the earliest line that differs (exit 1), or the earliest line with a field or line kind this engine does not know (`unsupported by this engine: FIELD`, exit 5: a newer engine's strand, not a divergence); from the header (every checkpoint checked), or from the last checkpoint; with --actions, also the earliest turn whose host action differs from the level its stance released", + Sub { name: "verify", args: "STRAND [--from-checkpoint] [--chain] [--actions FILE] [--blocks FILE]", help: "replay a strand: the earliest line that differs (exit 1), or the earliest line with a field or line kind this engine does not know (`unsupported by this engine: FIELD`, exit 5: a newer engine's strand, not a divergence); from the header (every checkpoint checked), or from the last checkpoint; with --actions, also the earliest turn whose host action differs from the level its stance released", flags: &[Flag { name: "--from-checkpoint", arg: "", help: "start at the last checkpoint instead of the header" }, + Flag { name: "--chain", arg: "", help: "a segment (section 5.7): verify it and walk back through the previous segment files to the first, each verified and its last line the sealed line the next one links; exit 1 at the first break (the segment file named), 6 when a previous segment file is missing" }, Flag { name: "--actions", arg: "FILE", help: "the host's action log, one {\"turn\": T, \"action\": {trait: level}} per line: compared with each turn's released levels (summary `actions`: checked, differs); exit 1 when one differs" }, Flag { name: "--blocks", arg: "FILE", help: "write each event's block (the text a host injects, RFC 0008 E: every trait and mood as `name level`, then the viability line) to FILE, one {\"block\", \"n\"} per line; replays from the header" }] }, - Sub { name: "control", args: "STRAND pause|resume|retire|erase --by WHO --reason TEXT [--at TIME]", - help: "pause, resume or retire an individual, or erase its learned weights (RFC 0004: every learned weight back to 0, the status unchanged); appends a control line under the lock: while paused or retired every event is refused (exit 4); no credit crosses a control line; retire is final; `verify` replays them and reports the status", - flags: &[Flag { name: "--by", arg: "WHO", help: "who decided it: human:ID or env:ID (required)" }, - Flag { name: "--reason", arg: "TEXT", help: "why (required)" }, - Flag { name: "--at", arg: "TIME", help: "when (default: now, UTC)" }] }, + Sub { name: "control", args: "STRAND pause|resume|retire|erase --by WHO --reason TEXT [--at TIME] | STRAND seal | STRAND reflect [--last N | --since-checkpoint] | STRAND stir? [--idle-hours H] | STRAND rest [--at TIME] [--state FILE]", + help: "pause, resume or retire an individual, or erase its learned weights (RFC 0004: every learned weight back to 0, the status unchanged); appends a control line under the lock: while paused or retired every event is refused (exit 4); no credit crosses a control line; retire is final; `verify` replays them and reports the status. `seal` (section 5.7) seals the newest segment of STRAND: refused unless its last line is an event line, it appends the sealing checkpoint and writes the next segment file (its header links the sealed line, then the sealed line itself), and prints that file in `file`", + flags: &[Flag { name: "--by", arg: "WHO", help: "who decided it: human:ID or env:ID (required, except for seal)" }, + Flag { name: "--reason", arg: "TEXT", help: "why (required, except for seal)" }, + Flag { name: "--at", arg: "TIME", help: "when (default: now, UTC)" }, + Flag { name: "--last", arg: "N", help: "reflect: the last N events (default: every event)" }, + Flag { name: "--since-checkpoint", arg: "", help: "reflect: the events after the last checkpoint line" }, + Flag { name: "--idle-hours", arg: "H", help: "stir? (RFC 0016): the hours since the strand's last event (default 0); the answer is {due, stir}, nothing is written" }, + Flag { name: "--state", arg: "FILE", help: "rest (RFC 0015): the stored individual (the state after the strand's last line), rewritten after the pass" }] }, ], list_subs: true, shared: false, after: &[("the safety kit", "(docs/persona.md §5.7) one writer per strand: `--strand` takes STRAND.lock (a second writer exits 4 and changes nothing; a lock whose process is gone is taken over). With `reward_from` in the persona (§2.10), a reward from `src: self`, without a src or from an undeclared one is refused like a bad event."), @@ -379,6 +379,20 @@ pub(crate) static COMMANDS: &[Command] = &[ ("exit", "0 every line replays, 1 a line differs (the frame names it and shows the event before it), 2 a bad flag, a port that cannot be had, or a file that cannot be read or is not a strand.")], examples: &["probbit monitor --demo --open", "probbit monitor pip.strand --follow", "probbit monitor pip.strand --serve --port 8080"], run: monitor::cmd }, + Command { name: "reflect", aliases: &[], group: "live", summary: "RFC 0013 self-report: a canonical account of a strand's last N events, from its lines alone", + usage: &["reflect STRAND [--last N | --since-checkpoint] [--json | --text]"], + about: "The self-report (docs/persona.md §5.8): a canonical document computed from the strand's lines alone (no turn is run, nothing is solved, nothing is random): over the window, the inputs set, the expectation's surprises (each receipt's z recomputed from the logged receipts; the K = 5 largest |z| with the predicted and measured values), per trait the levels taken and the odds' drift, the moods' path with its turning points, the drives' peaks and the habits that bound (read from the checkpoint lines in the window: a window without one reports them as null), the credit settled, the learned weights' net movement and the control lines; every number at 6 decimals, `digest` = the sha256 of the canonical text, and `summary`: 6 to 12 sentences from fixed templates. `probbit live control STRAND reflect` appends it as a line (0.14.0 strands), which `live verify` recomputes.", + before: &[], + flags: &[&[ + Flag { name: "--last", arg: "N", help: "the last N events (default: every event)" }, + Flag { name: "--since-checkpoint", arg: "", help: "the events after the last checkpoint line (that checkpoint is the first sample)" }, + Flag { name: "--json", arg: "", help: "the canonical document (the default)" }, + Flag { name: "--text", arg: "", help: "the summary's sentences only, one per line" }, + ]], + subs: &[], list_subs: false, shared: false, + after: &[("exit", "0 the reflection printed, 2 a bad flag, a file that cannot be read, or a strand with no event line in the window (one {\"error\"} object on stdout).")], + examples: &["probbit reflect pip.strand --last 50 --text", "probbit reflect pip.strand --since-checkpoint"], + run: reflect_cmd }, Command { name: "decide", aliases: &[], group: "decide/run", summary: "route tasks to workers: a router document in, a JSON decision out", usage: &["decide [flags] < router.json"], about: "Route tasks to workers: a router document (workers + caps, tasks + allowed + scores, affinity; README) on stdin, a JSON decision on stdout. Exact where structure allows, else sampled with the diagnostics gate.", @@ -831,6 +845,37 @@ fn live_cmd(args: &[String]) { Err((line, why)) => failed(line, &why) } return; } + // --chain (§5.7, segments): this segment, then each previous one back to the first, linked by the sealed line + if args.iter().any(|a| a == "--chain") { + if args.iter().any(|a| a == "--from-checkpoint") { fail("live verify: --chain replays every segment from its start: drop --from-checkpoint") } + let dir = std::path::Path::new(path).parent().map(std::path::Path::to_path_buf).unwrap_or_default(); + let name = |p: &str| std::path::Path::new(p).file_name().map_or_else(|| p.to_string(), |f| f.to_string_lossy().into_owned()); + let (mut cur, mut t, mut segs, mut newest) = (name(path), text.clone(), 0u64, None::); + let broke = |file: &str, d: Json, code: i32| -> ! { let mut d = d; if let Json::Obj(v) = &mut d { v.push(("segment_file".into(), Json::Str(file.to_string()))); } + emit(&persona::canon(&d)); exit(code) }; + loop { + match live::verify(&t, eng) { + Ok(d) => { if newest.is_none() { newest = Some(d); } segs += 1; } + Err((line, why)) => { err_line(&format!("probbit: live verify --chain: {cur}: line {line}: {why}")); + broke(&cur, live::failure(&t, line, &why), if live::unsupported_field(&why).is_some() { 5 } else { 1 }) } } + let h = json::parse(live::lines(&t).first().copied().unwrap_or("")).unwrap_or(Json::Null); + let Some(sg) = h.get("segment").cloned() else { break }; + let (prev, sealed, index) = (sg.get("previous").and_then(Json::as_str).unwrap_or("").to_string(), sg.get("sealed").and_then(Json::as_str).unwrap_or("").to_string(), sg.get("index").and_then(Json::as_f64).unwrap_or(0.0)); + let pt = match std::fs::read_to_string(dir.join(&prev)) { Ok(x) => x, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { err_line(&format!("probbit: live verify --chain: {cur} links {prev}, which does not exist: the chain before it is not verified")); + broke(&cur, obj(vec![("ok", Json::Bool(false)), ("missing", Json::Str(prev.clone()))]), 6) } + Err(e) => fail(&format!("live verify --chain: cannot read {prev}: {e}")) }; + let pls = live::lines(&pt); let last = pls.last().copied().unwrap_or(""); + let pidx = json::parse(pls.first().copied().unwrap_or("")).ok().and_then(|h| h.get("segment").and_then(|s| s.get("index")).and_then(Json::as_f64)).unwrap_or(0.0); + let why = if persona::digest_of(last) != sealed { Some(format!("its last line is not the sealed line {cur} links")) } + else if pidx + 1.0 != index { Some(format!("its segment index is not {} (the one before {cur}'s)", index - 1.0)) } else { None }; + if let Some(why) = why { err_line(&format!("probbit: live verify --chain: {prev}: line {}: {why}", pls.len())); + broke(&prev, obj(vec![("ok", Json::Bool(false)), ("line", num(pls.len() as f64)), ("diverges", Json::Str(why))]), 1) } + cur = prev; t = pt; + } + let mut d = newest.unwrap_or(Json::Null); if let Json::Obj(v) = &mut d { v.push(("segments".into(), num(segs as f64))); } + emit(&persona::canon(&d)); return; + } let res = if args.iter().any(|a| a == "--from-checkpoint") { live::verify_from_checkpoint(&text, eng) } else { live::verify(&text, eng) }; match res { Ok(doc) => emit(&persona::canon(&doc)), @@ -839,8 +884,36 @@ fn live_cmd(args: &[String]) { } if args.get(1).map(String::as_str) == Some("control") { let mut a = vec!["live control".to_string()]; a.extend(args.iter().skip(4).cloned()); let (v, w) = sub_flags("live", "control").unwrap(); check_flags(&a, &v, &w); - let usage = "live control: probbit live control STRAND pause|resume|retire|erase --by human:ID --reason TEXT"; + let usage = "live control: probbit live control STRAND pause|resume|retire|erase --by human:ID --reason TEXT (or STRAND seal, STRAND rest)"; let (Some(path), Some(what)) = (args.get(2).filter(|a| !a.starts_with("--")), args.get(3).filter(|a| !a.starts_with("--"))) else { fail(usage) }; + // reflect (RFC 0013): the reflection of the lines so far, appended as one line (0.14.0 strands) + if what == "reflect" { + if let Some(f) = ["--by", "--reason", "--at"].iter().find(|f| args.iter().any(|a| a == *f)) { fail(&format!("live control STRAND reflect: {f} does not apply (a reflection is computed, not decided by someone)")) } + match live::reflect_cmd(path, reflect_window(args, "live control")) { Ok(doc) => emit(&persona::canon(&doc)), Err(e) => refuse("live control: ", e) } + return; + } + if args.iter().any(|a| a == "--last" || a == "--since-checkpoint") { fail("live control: --last and --since-checkpoint go with reflect") } + // stir? (RFC 0016): is a self-started idle event due, exactly from the replayed strand and the idle hours; writes nothing + if what == "stir?" { + if let Some(f) = ["--by", "--reason", "--at"].iter().find(|f| args.iter().any(|a| a == *f)) { fail(&format!("live control STRAND stir?: {f} does not apply (a query writes nothing)")) } + let idle = match opt("--idle-hours") { None => 0.0, Some(x) => x.parse::().ok().filter(|h| h.is_finite() && *h >= 0.0).unwrap_or_else(|| fail("live control STRAND stir?: --idle-hours H, a number >= 0")) }; + match live::stir_cmd(path, idle, eng) { Ok(doc) => emit(&persona::canon(&doc)), Err(e) => refuse("live control: ", e) } + return; + } + if args.iter().any(|a| a == "--idle-hours") { fail("live control: --idle-hours goes with stir?") } + if what != "rest" && args.iter().any(|a| a == "--state") { fail("live control: --state goes with rest") } + // rest (RFC 0015): the rest pass on the replayed strand, one consolidation line (0.14.0 strands with expectation.rest) + if what == "rest" { + if let Some(f) = ["--by", "--reason"].iter().find(|f| args.iter().any(|a| a == *f)) { fail(&format!("live control STRAND rest: {f} does not apply (a rest pass is a host's timing, not a control by someone)")) } + match live::rest_cmd(path, &opt("--at").unwrap_or_else(live::utc_now), opt("--state").as_deref(), eng) { Ok(doc) => emit(&persona::canon(&doc)), Err(e) => refuse("live control: ", e) } + return; + } + // seal (§5.7): no control line, so no --by, --reason or --at; the sealing checkpoint and the next segment file + if what == "seal" { + if let Some(f) = ["--by", "--reason", "--at"].iter().find(|f| args.iter().any(|a| a == *f)) { fail(&format!("live control STRAND seal: {f} does not apply (a seal writes a checkpoint line, not a control line)")) } + match live::seal_cmd(path, eng) { Ok(doc) => emit(&persona::canon(&doc)), Err(e) => refuse("live control: ", e) } + return; + } let (Some(by), Some(reason)) = (opt("--by"), opt("--reason")) else { fail(&format!("{usage} (--by and --reason are required)")) }; match live::control_cmd(path, what, &by, &reason, &opt("--at").unwrap_or_else(live::utc_now)) { Ok(doc) => emit(&persona::canon(&doc)), Err(e) => refuse("live control: ", e) } return; @@ -862,6 +935,8 @@ fn live_cmd(args: &[String]) { drop(lock); return; } let every: u64 = arg(args, "--checkpoint-every", live::CHECKPOINT_EVERY); + let seg_every: u64 = arg(args, "--segment-every", 0); + if seg_every > 0 && opt("--strand").is_none() { fail("live: --segment-every goes with --strand NAME.strand (the segments are NAME.0001.strand, ...)") } let state_file = opt("--state"); if state_file.is_some() && opt("--seed").is_some() { fail("live: give --seed N (a new individual) or --state FILE (a stored one), not both") } let st = match &state_file { Some(f) => { let t = std::fs::read_to_string(f).unwrap_or_else(|e| bad_input("live: ", persona::perr("state", format!("cannot read {f}: {e} (make one with `probbit persona init`)")))); @@ -875,16 +950,31 @@ fn live_cmd(args: &[String]) { let strand = opt("--strand"); let lock = strand.as_ref().map(|f| live::Lock::take(f).unwrap_or_else(|e| refuse("live: ", e))); if let Some(f) = &strand { if std::path::Path::new(f).exists() && state_file.is_none() { fail(&format!("live: {f} exists: continue it with --state FILE (the state after its last line), or log to a new file")) } } + // the run continues the newest segment of the strand (NAME.strand, then NAME.0001.strand, ...), sealed by --segment-every + // or by `live control STRAND seal`: a sealed file is never appended to + let mut seg = 0u64; if let Some(f) = &strand { while std::path::Path::new(&live::segment_name(f, seg + 1)).exists() { seg += 1; } } + let cur = std::cell::RefCell::new(strand.as_ref().map(|f| live::segment_name(f, seg))); + // the strand's engine (its header's; a new strand: this one's): the next segment's header names it + let strand_engine = cur.borrow().as_ref().and_then(|f| std::fs::read_to_string(f).ok()).and_then(|t| json::parse(live::lines(&t).first().copied().unwrap_or("")).ok()) + .and_then(|h| h.get("engine").and_then(Json::as_str).map(str::to_string)).unwrap_or_else(live::engine); if let Some(f) = strand.as_ref().filter(|_| !labels.is_empty()) { labels::merge_into(f, &labels).unwrap_or_else(|e| fail(&format!("live: {e}"))); } - let (mut lv, header) = match &strand { Some(f) => live::open(f, p, &doc, st, clock).unwrap_or_else(|e| bad_input("live: ", e)), None => (live::Live::start(p, &doc, st, clock, &live::engine()).0, None) }; + let (mut lv, header) = match cur.borrow().as_ref() { Some(f) => live::open(f, p, &doc, st, clock).unwrap_or_else(|e| bad_input("live: ", e)), None => (live::Live::start(p, &doc, st, clock, &live::engine()).0, None) }; + if seg_every > 0 && lv.expect.as_ref().is_some_and(|m| !m.is_canonical()) { fail("live: --segment-every: before engine 0.14.0 a checkpoint does not carry the expectation's memory, so a segment of a persona with an expectation block could not start from its sealed line") } // every append first checks the lock is still this writer's: a lock removed or taken over under it stops the run unwritten - let log = |ls: &[&str]| if let Some(f) = &strand { + let log = |ls: &[&str]| if let (Some(f), Some(c)) = (&strand, cur.borrow().as_ref()) { if !lock.as_ref().is_some_and(live::Lock::held) { refuse("live: ", json::InErr { code: "locked", path: "strand".into(), msg: format!("the writer lock {f}.lock is no longer this run's (removed or taken over): stopped before writing") }) } - live::append(f, ls).unwrap_or_else(|e| fail(&format!("live: {}", e.msg))) }; + live::append(c, ls).unwrap_or_else(|e| fail(&format!("live: {}", e.msg))) }; if let Some(h) = &header { log(&[h]); } let (mut bad, mut refused, mut warned, mut warned_credit, mut warned_needs) = (0, 0, false, false, false); + // feature `turn_trace`: with PROBBIT_TURN_TRACE set, each event's phases on stderr (the parse and the append are timed here) + #[cfg(feature = "turn_trace")] + let trace_on = std::env::var_os("PROBBIT_TURN_TRACE").is_some(); let mut one = |lv: &mut live::Live, i: usize, t: &str| { + #[cfg(feature = "turn_trace")] + let (t_parse, parse_us) = (std::time::Instant::now(), std::cell::Cell::new(0u64)); let res = json::parse(t).map_err(|e| persona::perr("event", format!("not JSON: {}", e.msg))).and_then(|ev| { + #[cfg(feature = "turn_trace")] + parse_us.set(t_parse.elapsed().as_micros() as u64); // readings sent to a persona without a senses block are an undeclared input (listed in `ignored`): say so once if !warned && !lv.p.has_senses() && ev.get("senses").is_some() { warned = true; err_line(&format!("probbit: live: events[{i}]: this persona has no senses block: `senses` is an undeclared input (listed in ignored), no reading is judged")); } @@ -896,8 +986,22 @@ fn live_cmd(args: &[String]) { err_line(&format!("probbit: live: events[{i}]: this persona has no needs block: `needs` is an undeclared input (listed in ignored), no amount is paid")); } lv.event(&ev, eng) }); match res { - Ok((stance, line)) => { let cp = (strand.is_some() && every > 0 && lv.n % every == 0).then(|| lv.checkpoint(&stance)); + Ok((stance, line)) => { let seal = strand.is_some() && seg_every > 0 && lv.n % seg_every == 0; + let cp = (strand.is_some() && (seal || (every > 0 && lv.n % every == 0))).then(|| lv.checkpoint(&stance)); + #[cfg(feature = "turn_trace")] + let t_append = std::time::Instant::now(); match &cp { Some(c) => log(&[&line, c]), None => log(&[&line]) } + // sealed: the next segment file, its header linking this one, then the sealed line itself; the run goes on there + if let (true, Some(c), Some(base)) = (seal, &cp, &strand) { + let prev = cur.borrow().clone().unwrap_or_default(); seg += 1; let next = live::segment_name(base, seg); + let pname = std::path::Path::new(&prev).file_name().map_or_else(|| prev.clone(), |f| f.to_string_lossy().into_owned()); + let h = live::segment_header(lv, &doc, &strand_engine, seg, &pname, &persona::digest_of(c)); + if std::path::Path::new(&next).exists() { fail(&format!("live: {next} exists: the strand's newest segment is not the one this run continued")) } + *cur.borrow_mut() = Some(next); log(&[&h, c]); } + #[cfg(feature = "turn_trace")] + if trace_on { let tr = probbit_persona::trace::last(); let us = |t: std::time::Instant| Json::Num(t.elapsed().as_micros() as f64); + let mut j = tr.to_json(); if let Json::Obj(v) = &mut j { v.push(("n".into(), Json::Num(lv.n as f64))); v.push(("parse_us".into(), Json::Num(parse_us.get() as f64))); v.push(("append_us".into(), us(t_append))); } + err_line(&format!("probbit: live: trace {}", persona::canon(&j))); } if let Some(f) = &state_file { put(Some(f), &persona::canon(&lv.st.to_json(&lv.p))); } emit(&persona::canon(&stance)); } Err(e) => { if ["paused", "retired"].contains(&e.code) { refused += 1 } else { bad += 1 } let e = json::InErr { path: format!("events[{i}].{}", e.path), ..e }; err_line(&format!("probbit: live: {}: {}", e.path, e.msg)); error_object(&e.to_json()); } } }; @@ -961,6 +1065,23 @@ fn lint_report(args: &[String], path: &str, p: &persona::Persona, depth: usize, /// `probbit check PERSONA [--props FILE] [--strand FILE]... [--seeds LIST] [--depth N] [--hours LIST] [--json]`: one gate for CI. /// `persona lint`'s pipeline on the rules (`lint_report`), then `live verify` on every strand; exit 1 when a rule is broken, a /// contradiction is unresolved or a strand fails, else 3 when a rule is unproven, else 0 +/// The window of `--last N` / `--since-checkpoint` (RFC 0013) +fn reflect_window(args: &[String], what: &str) -> probbit_persona::reflect::Window { + use probbit_persona::reflect::Window; + let last = args.iter().position(|x| x == "--last").map(|k| args.get(k + 1).and_then(|v| v.parse::().ok()).filter(|n| *n >= 1) + .unwrap_or_else(|| fail(&format!("{what}: --last N: a whole number >= 1")))); + match (last, args.iter().any(|a| a == "--since-checkpoint")) { (Some(_), true) => fail(&format!("{what}: --last and --since-checkpoint: one window")), + (Some(n), false) => Window::Last(n), (None, true) => Window::SinceCheckpoint, (None, false) => Window::All } +} +/// `probbit reflect STRAND [--last N | --since-checkpoint] [--json | --text]` (RFC 0013) +fn reflect_cmd(args: &[String]) { + let Some(path) = args.get(1).filter(|a| !a.starts_with("--")) else { fail("reflect: the strand file: probbit reflect STRAND [--last N | --since-checkpoint]") }; + let mut a = vec!["reflect".to_string()]; a.extend(args.iter().skip(2).cloned()); let (v, w) = flags_of("reflect"); check_flags(&a, &v, &w); + if args.iter().any(|a| a == "--json") && args.iter().any(|a| a == "--text") { fail("reflect: --json or --text") } + let text = std::fs::read_to_string(path).unwrap_or_else(|e| fail(&format!("reflect: cannot read {path}: {e}"))); + let r = probbit_persona::reflect::reflect_text(&text, reflect_window(args, "reflect")).unwrap_or_else(|e| bad_input("reflect: ", e)); + if args.iter().any(|a| a == "--text") { emit(&r.summary().join("\n")) } else { emit(&r.text()) } +} fn check_cmd(args: &[String]) { // --strand may be given more than once: take its values out before the flags are checked let (mut rest, mut strands) = (vec![], vec![]); let mut i = 0; @@ -1150,6 +1271,8 @@ fn help_cmd(args: &[String]) { } fn main() { + // the library's hooks: the `--top` monitor and the config echo in telemetry + probbit_persona::hooks::set_monitor(tui::tier, tui::gate_seen); probbit_persona::hooks::set_config_echo(config_echo); // an argument that is not valid Unicode (a file name in a legacy encoding on Linux, an unpaired surrogate on Windows) is // refused by name; std::env::args would panic on it let args: Vec = std::env::args_os().skip(1).enumerate().map(|(i, a)| a.into_string().unwrap_or_else(|a| @@ -1216,7 +1339,7 @@ mod table_tests { let (v, w) = sub_flags("persona", "fuzz").unwrap(); assert!(v.contains(&"--fixture-src") && w.contains(&"--json") && w.contains(&"--pretty")); let (v, w) = flags_of("monitor"); assert!(v.contains(&"--port") && w.contains(&"--demo"), "--demo takes an optional value: parsed as a switch"); assert!(sub_flags("persona", "lnt").is_none()); - assert_eq!(sub_flags("live", "verify").unwrap(), (vec!["--actions", "--blocks"], vec!["--from-checkpoint"]), "live's own flags are not verify's"); + assert_eq!(sub_flags("live", "verify").unwrap(), (vec!["--actions", "--blocks"], vec!["--from-checkpoint", "--chain"]), "live's own flags are not verify's"); for c in COMMANDS { let (v, w) = flags_of(c.name); assert_eq!(v.len() + w.len(), c.flags().count(), "{}", c.name); } } #[test] diff --git a/probbit-cli/src/monitor.html b/probbit-cli/src/monitor.html index 428661b..9d4f616 100644 --- a/probbit-cli/src/monitor.html +++ b/probbit-cli/src/monitor.html @@ -25,6 +25,7 @@ .rail{flex:none;width:212px;height:100vh;position:sticky;top:0;overflow:auto;background:var(--bar);padding:16px 10px;display:flex;flex-direction:column;gap:10px}.rail h2{font-size:10.5px;font-weight:600;color:var(--muted);text-transform:uppercase;letter-spacing:.06em;padding:0 8px;display:flex;justify-content:space-between}.rail-list{display:flex;flex-direction:column;gap:2px}.rail-item{display:flex;align-items:center;gap:10px;width:100%;text-align:left;border:0;background:none;border-radius:8px;padding:6px 8px;min-height:44px}.rail-item[aria-current=true]{background:var(--selection)}.rail-item[aria-current=true] strong{color:var(--blue)}.rail-glyph{width:22px;font-size:18px;line-height:1;text-align:center;flex:none}.rail-text{min-width:0;display:flex;flex-direction:column}.rail-text strong{font-weight:600;font-size:12.5px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}.rail-text small{color:var(--muted);font-size:10.5px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;font-variant-numeric:tabular-nums}.rail-keys{margin-top:auto;padding:0 8px;font-size:10px;color:var(--muted)} @media(max-width:760px){.app{flex-direction:column}.rail{width:auto;height:auto;position:static;padding:8px 10px;gap:4px}.rail-list{flex-direction:row;overflow-x:auto}.rail-item{width:auto;flex:none}.rail-keys{display:none}.who{padding:9px 12px 7px}} @media(prefers-reduced-motion:reduce){*,*:before,*:after{animation:none!important;transition:none!important;scroll-behavior:auto!important}} +body.mirror-on .toolbar>:not(.view),body.mirror-on #panel,body.mirror-on #workspace,body.mirror-on #timeline,body.mirror-on #said,body.mirror-on #diagnostics,body.mirror-on #why-panel,body.mirror-on #empty{display:none!important}.mirror{padding:14px 18px;overflow:auto}.mirror li{margin:3px 0}.mirror td,.mirror th{padding:3px 10px;text-align:left}.mirror-mood{display:flex;gap:12px;align-items:center}.mirror-mood svg{width:140px;height:25px} @@ -34,7 +35,8 @@

probbit

Activity Monitor

probbit · declared agent state
Waiting for individual
-
+
+
CONNECTINGWaiting for the local monitor.

Waiting for a strand.

@@ -414,6 +416,24 @@ } addEventListener('keydown',e=>{const k=e.key>='1'&&e.key<='9'&&(e.metaKey||e.ctrlKey)&&!e.altKey&&!e.shiftKey?Number(e.key)-1:-1;if(k<0||strands.length<2||!strands[k])return;e.preventDefault();pick(strands[k].id);}); $('rail-list').onkeydown=e=>{if(!['ArrowDown','ArrowUp'].includes(e.key))return;e.preventDefault();const at=strands.findIndex(s=>s.id===sid),n=strands[Math.max(0,Math.min(strands.length-1,at+(e.key==='ArrowDown'?1:-1)))];pick(n.id);[...$('rail-list').children].find(b=>b.dataset.strand===n.id)?.focus();}; +let mirrorOn=false; +function tab(on){mirrorOn=on;document.body.classList.toggle('mirror-on',on);$('mirror').hidden=!on;$('tab-board').setAttribute('aria-pressed',String(!on));$('tab-mirror').setAttribute('aria-pressed',String(on));if(on)mirror();} +$('tab-board').onclick=()=>tab(false);$('tab-mirror').onclick=()=>tab(true); +// RFC 0013: the reflection, computed by the server from the strand's lines (no solve), refreshed with the scan +async function mirror(){if(!mirrorOn)return;let j;try{const r=await fetch(base+'reflection?last=50',{cache:'no-store'});j=r.ok?await r.json():{error:(await r.text()).trim()};}catch(_){j={error:'The reflection could not be fetched.'};} + const d=obj(j.live)||{},w=obj(d.window)||{},lg=obj(j.logged); + $('mirror-note').textContent=j.error?j.error:'Events '+w.first+'–'+w.last+' · '+w.checkpoints+' checkpoint samples · digest '+short(d.digest)+(lg?' · logged reflection at line '+lg.line+(lg.matches?' recomputes equal':' differs from the lines before it'):' · no reflection line logged'); + const s=$('mirror-summary');s.replaceChildren();(d.summary||[]).forEach(x=>mk('li','',s,x)); + const t=$('mirror-top');t.replaceChildren();(get(d,'surprise','top')||[]).forEach(e=>{const r=mk('tr','',t);[e.n,f6(e.z),f6(e.pred),f6(e.self),(e.set||[]).join(', ')||'—'].forEach(v=>mk('td','number',r,String(v)));}); + const h=$('mirror-habits');h.replaceChildren();Object.entries(obj(get(d,'habits','bound'))||{}).forEach(([k,c])=>mk('span','chip',h,k+' ×'+c)); + const m=$('mirror-moods');m.replaceChildren();Object.entries(obj(d.moods)||{}).forEach(([k,v])=>{const row=mk('div','mirror-mood',m);mk('span','',row,k+' '+f2(v.start)+' → '+f2(v.end)+' · '+v.turns+' turning points');plot(svg('svg',{viewBox:'0 0 140 25',role:'img','aria-label':k+' by checkpoint sample'},row),(v.path||[]).map(([n,value])=>({n,value})),{width:140,height:25});}); + integrityTable();} +// RFC 0014: the per-trait drift of the event on view (its stance's `integrity`, recomputed by verify from the strand) +function integrityTable(){const ig=obj(get(viewed,'doc','integrity')),box=$('mirror-integrity');box.hidden=!ig;if(!ig)return; + $('mirror-integrity-note').textContent='Event '+viewed.n+' · used '+f6(ig.used)+' of the allowance · drift '+f6(ig.drift)+' · warn at '+f2(ig.warn)+(Array.isArray(ig.crossed)&&ig.crossed.length?' · past warn: '+ig.crossed.join(', '):''); + const t=$('mirror-integrity-rows');t.replaceChildren();Object.entries(obj(ig.traits)||{}).forEach(([k,v])=>{const r=mk('tr','',t);mk('td','',r,k);[v.drift,v.allowance,v.bound_used].forEach(x=>mk('td','number',r,f6(x)));});} +const f6=x=>num(x)?x.toFixed(6):'—'; +setInterval(mirror,2000); const safe=fn=>e=>{try{fn(JSON.parse(e.data));}catch(x){console.error(x);}}; // The demo and an older server list no strand: the single-strand stream at once list().then(()=>{if(!es&&(!listed||!strands.length))connect();});setInterval(list,2000); diff --git a/probbit-cli/src/monitor.rs b/probbit-cli/src/monitor.rs index b0ac76d..2340a50 100644 --- a/probbit-cli/src/monitor.rs +++ b/probbit-cli/src/monitor.rs @@ -121,17 +121,30 @@ impl Frame { } } /// A strand being watched: the replay, the layout, the latest frame, the moods' recent positions, the line that diverged (if any) -struct Watch { rep: Replay, meta: Meta, last: Option, spark: Vec>, bad: Option<(usize, String)> } +struct Watch { rep: Replay, meta: Meta, last: Option, spark: Vec>, bad: Option<(usize, String)>, + /// RFC 0015: the last consolidation line replayed (its line number and the line), drawn as "last rest" + rest: Option<(usize, Json)> } impl Watch { fn open(head: &str) -> Result { let rep = Replay::open(head)?; let meta = Meta::of(&rep); let n = meta.moods.len(); - Ok(Watch { rep, meta, last: None, spark: vec![VecDeque::new(); n], bad: None }) + Ok(Watch { rep, meta, last: None, spark: vec![VecDeque::new(); n], bad: None, rest: None }) + } + /// Where the replay of `lines` (the header first) starts: the last checkpoint line (index), or None for the header. RFC 0009: + /// before 0.14.0 a persona with an expectation block folds every event line into its memory, which no checkpoint carries, so + /// its strand replays from the header (every line, slower; `open_at` refuses that shortcut); from 0.14.0 the checkpoint carries + /// the memory, and every strand starts at the last one + /// A segment (§5.7) starts at its sealed line (line 2) when it has no later checkpoint. + fn start(lines: &[&str]) -> Option { + let k = crate::live::last_checkpoint(lines, 0)?; let r = Replay::open(lines[0]); + if k < 2 && !r.as_ref().is_ok_and(|r| r.header.get("segment").is_some()) { return None; } + (!r.is_ok_and(|r| r.live.p.expect.is_some() && r.live.dev_full)).then_some(k) } /// The replay from a checkpoint line (`live::Replay::open_at`; `before` = the event line before it): the lines before it are /// not replayed, and the board starts at the checkpoint's event (its stance, the inputs `before` logs, its learned deltas) fn open_at(head: &str, before: &str, cp: &str, at: usize) -> Result { - let rep = Replay::open_at(head, before, cp, at)?; let meta = Meta::of(&rep); let n = meta.moods.len(); - let mut w = Watch { rep, meta, last: None, spark: vec![VecDeque::new(); n], bad: None }; + // a segment's sealed line (line 2, after the header) opens on the header's link + let rep = if at == 2 { Replay::open_lines(&[head, cp])?.0 } else { Replay::open_at(head, before, cp, at)? }; let meta = Meta::of(&rep); let n = meta.moods.len(); + let mut w = Watch { rep, meta, last: None, spark: vec![VecDeque::new(); n], bad: None, rest: None }; if let Some(doc) = w.rep.last_stance().cloned() { let given = json::parse(before).ok().and_then(|j| j.get("inputs").cloned()).unwrap_or(Json::Null); for (m, s) in w.meta.moods.iter().zip(w.spark.iter_mut()) { s.push_back(position(&odds(&doc, "mood", m))); } @@ -142,7 +155,8 @@ impl Watch { fn feed(&mut self, l: &str, eng: Engine) { if self.bad.is_some() { return; } match self.rep.step(l, eng) { - Ok(None) => {} // a control or checkpoint line: no stance to draw + // a control or checkpoint line: no stance to draw; RFC 0015: a consolidation line is kept for the "last rest" row + Ok(None) => if let Some(j) = json::parse(l).ok().filter(|j| j.get("consolidation").is_some()) { self.rest = Some((self.rep.line, j)); } Ok(Some(doc)) => { let given = json::parse(l).ok().and_then(|j| j.get("inputs").cloned()).unwrap_or(Json::Null); for (m, s) in self.meta.moods.iter().zip(self.spark.iter_mut()) { if s.len() == SPARK { s.pop_front(); } s.push_back(position(&odds(&doc, "mood", m))); } @@ -233,7 +247,12 @@ fn readings(m: &Meta, doc: &Json, learned: &Json, disp: Option<&Json>, open: Opt out.push(r("habits".into(), "habits", Field::Habits { active: n("active"), bound: n("bound"), declared: m.n_habits })); for x in &m.senses { let z = doc.get("senses").and_then(|s| s.get(&x.id)).and_then(|s| s.get("z")).and_then(Json::as_f64); out.push(r(format!("sense:{}", x.id), &format!("sense {}", x.id), Field::Sense { z })); } - out.push(r("room".into(), "room left in the bound", Field::Room)); + // RFC 0014: with the identity self keys the stance carries `integrity`: its dial replaces the room placeholder + match doc.get("integrity").filter(|x| x.as_obj().is_some()) { + Some(ig) => { let g = |k: &str| ig.get(k).and_then(Json::as_f64).unwrap_or(0.0); + out.push(r("integrity".into(), "integrity (drift from the declared self)", Field::Integrity { used: g("used"), drift: g("drift"), + crossed: ig.get("crossed").and_then(Json::as_arr).map_or(0, <[Json]>::len), traits: ig.get("traits").and_then(Json::as_obj).map_or(0, <[(String, Json)]>::len) })); } + None => out.push(r("room".into(), "room left in the bound", Field::Room)) } out } @@ -444,6 +463,12 @@ fn frame(w: Option<&Watch>, bad_head: Option<&(usize, String)>, path: &str, note let z = dv.get("z").and_then(Json::as_f64).map_or("z -".to_string(), |z| format!("z {z:+.2}")); let txt = format!("pred {:.4} {dot} self {:.4} {dot} input {:.2} {dot} {}{}", g("pred"), g("self"), g("input"), if cold { "cold".to_string() } else { z }, if reg { " registers deviation" } else { "" }); out.push(format!("{} {}", s.bold(MAGENTA, "DEVIATION"), if reg { s.bold(AMBER, &txt) } else { s.paint(CYAN, &txt) })); } + // RFC 0015: the last rest pass replayed (only once the strand has one): its clock, the contexts merged, the weights moved + if let Some((line, j)) = w.rest.as_ref() { let c = j.get("consolidation"); let g = |k: &str| c.and_then(|c| c.get(k)).and_then(Json::as_f64).unwrap_or(0.0); + let merged = c.and_then(|c| c.get("merged")).and_then(|m| m.get("count")).and_then(Json::as_f64).unwrap_or(0.0); + let txt = format!("last rest {} {dot} event {} {dot} line {line} {dot} {} merged {dot} moved {:.6} {dot} decay {:.6} {dot} {} events", + j.get("at").and_then(Json::as_str).unwrap_or("-"), j.get("n").and_then(Json::as_f64).unwrap_or(0.0) as u64, merged as u64, g("moved"), g("decay"), g("events") as u64); + out.push(format!("{} {}", s.bold(MAGENTA, "REST"), s.paint(CYAN, &txt))); } if let Some(tj) = w.rep.live.traj.as_ref() { let st = tj.get("steps").and_then(Json::as_arr).map_or(vec![], |a| a.to_vec()); let hs: Vec = st.iter().map(|x| format!("{:.2}", x.get("h").and_then(Json::as_f64).unwrap_or(0.0))).collect(); let b0 = st.first().and_then(|x| x.get("beta")).and_then(Json::as_f64).unwrap_or(1.0); @@ -519,7 +544,7 @@ fn drives(doc: &Json, m: &Meta, s: &Style, idw: usize, lab: &dyn Fn(&str) -> Str let mut out = vec![]; // the goals: pursue's odds (canonical order), else the goals a drive object names let mut goals: Vec = pz.and_then(|p| p.get("odds")).and_then(Json::as_obj).map(|o| o.iter().map(|(k, _)| k.clone()).collect()).unwrap_or_default(); - if goals.is_empty() { goals = dv.and_then(Json::as_obj).and_then(|o| o.iter().find_map(|(_, v)| v.as_obj())).map(|o| o.iter().map(|(k, _)| k.clone()).collect()).unwrap_or_default(); } + if goals.is_empty() { goals = dv.and_then(Json::as_obj).and_then(|o| o.iter().filter(|(k, _)| k != "curiosity").find_map(|(_, v)| v.as_obj())).map(|o| o.iter().map(|(k, _)| k.clone()).collect()).unwrap_or_default(); } if let Some(p) = pz { let v = Var { id: "pursue".into(), levels: goals.clone(), say: vec![] }; let ps: Vec = goals.iter().map(|g| p.get("odds").and_then(|o| o.get(g)).and_then(Json::as_f64).unwrap_or(0.0)).collect(); @@ -538,6 +563,14 @@ fn drives(doc: &Json, m: &Meta, s: &Style, idw: usize, lab: &dyn Fn(&str) -> Str if let Some(x) = err[i].filter(|x| *x != 0.0) { row += &format!(" {}", s.bold(if x < 0.0 { RED } else { GREEN }, &format!("{x:+.2} {} expectation", if x < 0.0 { "below" } else { "above" }))); } out.push(row); } + // RFC 0016: the curiosity drive, its level and its two inputs (and the stir's class on a self-started event) + if let Some(c) = d.get("curiosity").filter(|c| c.as_obj().is_some()) { + let f = |k: &str| c.get(k).and_then(Json::as_f64).unwrap_or(0.0); + let mut row = format!("{}{:, bad_head: Option<&(usize, String)>, note: Optio ("diverges", bad.map_or(Json::Null, |(l, why)| json::obj(vec![("line", Json::Num(*l as f64)), ("why", json::str(why))]))), ("note", note.map_or(Json::Null, json::str)), ("from_checkpoint", w.and_then(|w| w.rep.from).map_or(Json::Null, |n| json::num(n as f64))), ("status", w.map_or(Json::Null, |w| json::str(w.rep.live.status.name())))]; + // RFC 0015: the last rest pass (its line number and line), only once the strand has one + if let Some((line, j)) = w.and_then(|w| w.rest.as_ref()) { kv.push(("last_rest", json::obj(vec![("line", Json::Num(*line as f64)), ("at", j.get("at").cloned().unwrap_or(Json::Null)), + ("n", j.get("n").cloned().unwrap_or(Json::Null)), ("consolidation", j.get("consolidation").cloned().unwrap_or(Json::Null))]))); } // the senses' normals now (§2.11), only when the persona declares them: mean, count and the largest move within 50 events if let Some(w) = w.filter(|w| !w.meta.senses.is_empty()) { let moved = &w.rep.normals().moved; @@ -644,10 +680,11 @@ impl Followed { fn poll(&mut self, eng: Engine, each: &mut dyn FnMut(&Watch)) -> Option { match self.tail.poll() { Got::Lines(ls) => { - // the first read of a strand with checkpoint lines starts at the last one: its event is drawn at once + // the first read of a strand with checkpoint lines starts at the last one (`Watch::start`): its event is drawn at + // once; after that, appended lines replay from the state in memory let mut skip = 0; if self.w.is_none() && self.bad_head.is_none() { let refs: Vec<&str> = ls.iter().map(String::as_str).collect(); - if let Some(k) = crate::live::last_checkpoint(&refs, 0).filter(|k| *k >= 2) { + if let Some(k) = Watch::start(&refs) { if let Ok(x) = Watch::open_at(refs[0], refs[k - 1], refs[k], k + 1) { each(&x); self.w = Some(x); skip = k + 1; } } } for l in ls.into_iter().skip(skip) { if let Some(x) = &mut self.w { let n = x.rep.live.n; x.feed(&l, eng); if x.rep.live.n != n && x.bad.is_none() { each(x); } continue; } if self.bad_head.is_none() { match Watch::open(&l) { Ok(x) => self.w = Some(x), Err(e) => self.bad_head = Some(e) } } } @@ -746,8 +783,9 @@ pub fn cmd(args: &[String]) { if head.is_empty() { crate::fail(&format!("monitor: {path} is empty: not a strand")) } if !strand(head) { crate::fail(&format!("monitor: {path} is not a probbit strand (format 1)")) } let note = (!rest.is_empty() && !lines.is_empty()).then(|| format!("line {} is incomplete (no newline at its end): not replayed", lines.len() + 1)); - // a strand with checkpoint lines replays from the last one (the board starts at its event and draws the latest one) - let start = crate::live::last_checkpoint(&lines, 0); + // a strand with checkpoint lines replays from the last one (the board starts at its event and draws the latest one), unless + // its persona has an expectation block: then from the header (`Watch::start`) + let start = Watch::start(&lines); let opened = match start { Some(k) => Watch::open_at(head, lines[k - 1], lines[k], k + 1), None => Watch::open(head) }; let (w, bad_head) = match opened { Ok(mut w) => { for l in lines.iter().skip(start.map_or(1, |k| k + 1)) { w.feed(l, eng); } (Some(w), None) } Err(e) => (None, Some(e)) }; let fr = frame(w.as_ref(), bad_head.as_ref(), &path, note.as_deref(), &s); @@ -869,7 +907,8 @@ fn publish(hub: &Hub, meta: &Json, frame: &Json, docs: Vec<(u64, String)>, over: let (meta, frame) = (persona::canon(meta), persona::canon(frame)); let mut b = board(hub); b.name = name; b.glyph = glyph; if over { b.docs.clear(); b.last = 0; } - for (n, d) in docs { b.docs.push_back(d); b.last = n; if b.docs.len() > KEEP { b.docs.pop_front(); } } + // an event already kept (a segment opens at its sealed line, the last event of the file before) is not kept twice + for (n, d) in docs { if n <= b.last { continue; } b.docs.push_back(d); b.last = n; if b.docs.len() > KEEP { b.docs.pop_front(); } } if b.meta != meta || b.layout == 0 { b.meta = meta; b.layout += 1; } b.frame = frame; b.seq += 1; hub.1.notify_all(); @@ -899,11 +938,22 @@ fn add(site: &Site, path: String, over: Labels) { { let mut l = entries(site); if l.iter().any(|e| e.path == path) { return; } let id = id_of(&path, &l); l.push(Arc::new(Entry { id, path: path.clone(), over, hub: hub.clone() })); } std::thread::spawn(move || { let engine = crate::persona_engine(); feed_strand(&path, &hub, &engine); }); } -/// The `*.strand` files in `dir`, by name +/// The `*.strand` files in `dir`, by name; a life's segments (§5.7: `NAME.0001.strand`, `NAME.0002.strand`, ... after +/// `NAME.strand`, without a gap) are one strand, `NAME.strand` fn scan(dir: &str) -> Vec { let mut v: Vec = std::fs::read_dir(dir).into_iter().flatten().flatten().map(|e| e.path()) .filter(|p| p.extension().is_some_and(|x| x == "strand") && p.is_file()).map(|p| p.to_string_lossy().into_owned()).collect(); - v.sort(); v + v.sort(); + let segs: Vec = v.iter().flat_map(|f| (1..).map(|k| crate::live::segment_name(f, k)).take_while(|s| v.contains(s)).collect::>()).collect(); + v.retain(|f| !segs.contains(f)); v +} +/// The newest segment of the strand `base` from index `k` on (§5.7): the next one counts once its header and sealed line are +/// both in (a seal writes them together; a reader can see the header first) +fn newest(base: &str, mut k: u64) -> u64 { + let ready = |f: &str| { let Ok(mut h) = std::fs::File::open(f) else { return false }; let (mut n, mut b) = (0, [0u8; 8192]); + loop { match h.read(&mut b) { Ok(0) | Err(_) => return false, Ok(m) => { n += b[..m].iter().filter(|&&c| c == b'\n').count(); if n >= 2 { return true } } } } }; + while ready(&crate::live::segment_name(base, k + 1)) { k += 1; } + k } /// With --dir, every second: the strands added to it since are served too (a strand removed stays listed, as gone); without, nothing fn watch_dir(site: &Site) -> ! { @@ -918,10 +968,13 @@ fn strands_json(site: &Site) -> String { let rows: Vec = entries(site).iter().map(|e| { let (mut l, mut err) = match labels::read(&e.path) { Ok(l) => (l, None), Err(x) => (Labels::new(), Some(x.replace(&labels::sidecar(&e.path), &name(&labels::sidecar(&e.path))))) }; for (k, v) in &e.over { if let Err(x) = labels::set(&mut l, k.clone(), v.clone()) { err = Some(format!("{k}: {x}")); } } - let age = std::fs::metadata(&e.path).and_then(|m| m.modified()).ok().and_then(|t| now.duration_since(t).ok()).map_or(Json::Null, |d| json::num(d.as_secs() as f64)); + // a life in segments: its newest file (the one written to) + let k = newest(&e.path, 0); + let age = std::fs::metadata(crate::live::segment_name(&e.path, k)).and_then(|m| m.modified()).ok().and_then(|t| now.duration_since(t).ok()).map_or(Json::Null, |d| json::num(d.as_secs() as f64)); let b = board(&e.hub); let opt = |x: &str| if x.is_empty() { Json::Null } else { json::str(x) }; let mut kv = vec![("id", json::str(&e.id)), ("strand", json::str(&name(&e.path))), ("labels", labels::to_json(&l)), ("persona", opt(&b.name)), ("n", json::num(b.last as f64)), ("glyph", opt(&b.glyph)), ("age_s", age)]; if let Some(x) = err { kv.push(("labels_error", json::str(&x))); } + if k > 0 { kv.push(("segment", json::num(k as f64))); } json::obj(kv) }).collect(); persona::canon(&json::obj(vec![("demo", Json::Bool(site.demo.is_some())), ("dir", site.dir.as_deref().map_or(Json::Null, |d| json::str(&name(d)))), ("strands", Json::Arr(rows))])) + "\n" } @@ -951,11 +1004,17 @@ fn serve(port: u16, open: bool, site: &Arc, feed: &mut dyn FnMut()) -> ! { feed(); loop { std::thread::park(); } } -/// The strand followed for the page: every 100 ms, what was appended is replayed and published +/// The strand followed for the page: every 100 ms, what was appended is replayed and published. A life in segments (§5.7) is +/// followed in its newest file: when a seal writes the next one, the rest of the sealed file is replayed, then the page goes +/// on in the new file (it opens at its sealed line, the event already shown); the layout and the event count carry on fn feed_strand(path: &str, hub: &Hub, eng: Engine) -> ! { - let mut fl = Followed::new(path); + let mut k = newest(path, 0); + let mut fl = Followed::new(&crate::live::segment_name(path, k)); let mut fresh = true; - loop { step(&mut fl, path, hub, eng, &mut fresh, CATCH_UP); std::thread::sleep(Duration::from_millis(100)); } + loop { step(&mut fl, path, hub, eng, &mut fresh, CATCH_UP); + let n = newest(path, k); + if n != k { step(&mut fl, path, hub, eng, &mut fresh, CATCH_UP); k = n; fl = Followed::new(&crate::live::segment_name(path, k)); continue; } + std::thread::sleep(Duration::from_millis(100)); } } /// One poll of the strand for the page: what was appended is replayed and published. A long replay (a large strand read from /// its header) is also published every `every` on the way, with a note, so the page draws while it catches up. @@ -1015,6 +1074,15 @@ fn handle(mut c: TcpStream, site: &Site, port: u16) { let (id, path) = match path.strip_prefix("/s/") { Some(r) => r.split_once('/').map_or((Some(r), ""), |(id, _)| (Some(id), &path[3 + id.len()..])), None => (None, path) }; if id.is_none() && path == "/" { return reply(&mut c, "200 OK", "text/html; charset=utf-8", PAGE.as_bytes()); } if id.is_none() && path == "/strands" { return reply(&mut c, "200 OK", "application/json; charset=utf-8", strands_json(site).as_bytes()); } + // RFC 0013, the Mirror tab: `/reflection?last=N` (default 50), computed from the strand file as it is now + if path == "/reflection" { + let file = match id { None => entries(site).first().map(|e| e.path.clone()), Some(id) => entries(site).iter().find(|e| e.id == id).map(|e| e.path.clone()) }; + let Some(file) = file else { return reply(&mut c, "404 Not Found", "text/plain", b"no strand file: the reflection reads a strand (the demo has none)\n") }; + let last = target.split_once('?').and_then(|(_, q)| q.split('&').find_map(|kv| kv.strip_prefix("last="))).map_or(Some(50), |v| v.parse::().ok().filter(|n| *n >= 1)); + let Some(last) = last else { return reply(&mut c, "400 Bad Request", "text/plain", b"last=N: a whole number >= 1\n") }; + return match std::fs::read_to_string(&file) { Ok(t) => reply(&mut c, "200 OK", "application/json; charset=utf-8", format!("{}\n", mirror(&t, last)).as_bytes()), + Err(_) => reply(&mut c, "404 Not Found", "text/plain", b"the strand file cannot be read\n") }; + } let Some(hub) = hub_for(site, id).filter(|_| path == "/events" || path.starts_with("/doc/")) else { return reply(&mut c, "404 Not Found", "text/plain", if id.is_some() { b"not found: /s/ID/events and /s/ID/doc/N serve the strand ID that /strands lists\n" } else { b"not found: / is the page, /events its stream, /doc/N a stance document, /strands the strands served\n" }) }; let hub = &hub; @@ -1028,6 +1096,18 @@ fn handle(mut c: TcpStream, site: &Site, port: u16) { match doc { Some(d) => reply(&mut c, "200 OK", "application/json; charset=utf-8", d.as_bytes()), None => reply(&mut c, "404 Not Found", "text/plain", msg.as_bytes()) } } _ => {} } } +/// The Mirror's document (RFC 0013): `live`, the reflection of the last `last` events computed now (no solve); `logged`, the +/// latest reflection line the strand carries (its line number, its document, and whether it is the reflection of the lines +/// before it: `matches`), or null; `error` when the strand cannot be reflected +pub fn mirror(text: &str, last: u64) -> String { + use probbit_persona::reflect; + let ls = crate::live::lines(text); let o = |v: Vec<(&str, Json)>| Json::Obj(v.into_iter().map(|(k, x)| (k.to_string(), x)).collect()); + let logged = (1..ls.len()).rev().find(|&i| reflect::is_reflection_line(ls[i])).map_or(Json::Null, |i| o(vec![("line", Json::Num((i + 1) as f64)), + ("doc", json::parse(ls[i]).ok().and_then(|j| j.get("reflection").cloned()).unwrap_or(Json::Null)), ("matches", Json::Bool(reflect::check_line(&ls[..i], ls[i]).is_ok()))])); + persona::canon(&match reflect::reflect_window(&ls, reflect::Window::Last(last)) { + Ok(r) => o(vec![("live", r.doc), ("logged", logged)]), + Err(e) => o(vec![("error", Json::Str(e.msg)), ("logged", logged)]) }) +} /// The headers every answer carries: no caching, no sniffing, and for the page a policy that lets it load nothing and connect /// to nothing but its own server const SAFE: &str = "Cache-Control: no-store\r\nX-Content-Type-Options: nosniff\r\nReferrer-Policy: no-referrer\r\nContent-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; script-src 'unsafe-inline'; connect-src 'self'; img-src data:; base-uri 'none'; form-action 'none'; frame-ancestors 'none'\r\n"; @@ -1158,6 +1238,46 @@ mod tests { let _ = std::fs::remove_file(&f); } + /// RFC 0009: before 0.14.0 a persona with an expectation block replays a strand with checkpoint lines from its header (the + /// checkpoint does not carry the expectation's memory): `--once` and the first poll of `--follow` / `--serve` draw the latest + /// event with no checkpoint start, and lines appended after it replay from the state in memory, one by one. From 0.14.0 the + /// checkpoint carries the memory and the same life starts at its last checkpoint. + #[test] + fn an_expectation_persona_replays_from_the_header() { + let (p, doc) = persona::load(concat!(env!("CARGO_MANIFEST_DIR"), "/tests/fixtures/monitor/tutor-expect.yaml")).unwrap(); + assert!(p.expect.is_some(), "the fixture has an expectation block"); + { let (mut lv, header) = Live::start(p.clone(), &doc, persona::init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.14.0"); let mut text = format!("{header}\n"); + for t in 0..17u32 { let (s, line) = lv.event(&json::parse(&format!(r#"{{"praise":{},"elapsed_hours":1}}"#, t % 3 == 0)).unwrap(), &run).unwrap(); + text += &line; text.push('\n'); if lv.n % 5 == 0 { text += &lv.checkpoint(&s); text.push('\n'); } } + let lines = live::lines(&text); let k = live::last_checkpoint(&lines, 0).unwrap(); assert_eq!(Watch::start(&lines), Some(k), "0.14.0: the checkpoint start"); + let mut w = Watch::open_at(lines[0], lines[k - 1], lines[k], k + 1).unwrap(); for l in &lines[k + 1..] { w.feed(l, &run); } + let mut full = Watch::open(lines[0]).unwrap(); for l in &lines[1..] { full.feed(l, &run); } + assert!(w.bad.is_none(), "{:?}", w.bad); assert_eq!(persona::canon(&w.last.as_ref().unwrap().doc), persona::canon(&full.last.as_ref().unwrap().doc)); } + let (mut lv, header) = Live::start(p.clone(), &doc, persona::init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.13.0"); + let mut text = format!("{header}\n"); + let add = |lv: &mut Live, text: &mut String, t: u32| { let ev = json::parse(&format!(r#"{{"praise":{},"elapsed_hours":1}}"#, t % 3 == 0)).unwrap(); + let (s, line) = lv.event(&ev, &run).unwrap(); *text += &line; text.push('\n'); if lv.n % 5 == 0 { *text += &lv.checkpoint(&s); text.push('\n'); } }; + for t in 0..17u32 { add(&mut lv, &mut text, t); } + let lines = live::lines(&text); let k = live::last_checkpoint(&lines, 0).unwrap(); + assert_eq!((lines.iter().filter(|l| l.starts_with(r#"{"checkpoint":"#)).count(), Watch::start(&lines)), (3, None), "3 checkpoints, no checkpoint start"); + assert!(Replay::open_at(lines[0], lines[k - 1], lines[k], k + 1).is_err_and(|e| e.1.contains("expectation block")), "the shortcut itself still refuses it"); + let f = std::env::temp_dir().join(format!("probbit-monitor-expect-{}.strand", std::process::id())); std::fs::write(&f, &text).unwrap(); + let mut fl = Followed::new(f.to_str().unwrap()); let mut seen = vec![]; + assert_eq!(fl.poll(&run, &mut |w: &Watch| seen.push(w.rep.live.n)), Some(false)); + let w = fl.w.as_ref().unwrap(); assert!(w.bad.is_none(), "{:?}", w.bad); + assert_eq!((w.rep.from, w.rep.live.n, seen.first().copied(), seen.len()), (None, 17, Some(1), 17), "every event replayed from the header"); + let mut full = Watch::open(lines[0]).unwrap(); for l in &lines[1..] { full.feed(l, &run); } + assert_eq!(persona::canon(&w.last.as_ref().unwrap().doc), persona::canon(&full.last.as_ref().unwrap().doc)); + assert_eq!(frame_json(fl.w.as_ref(), None, None).get("from_checkpoint"), Some(&Json::Null)); + // appended: only the new lines replay, on the state in memory + for t in 17..21u32 { add(&mut lv, &mut text, t); } std::fs::write(&f, &text).unwrap(); + seen.clear(); assert_eq!(fl.poll(&run, &mut |w: &Watch| seen.push(w.rep.live.n)), Some(false)); + assert_eq!(seen, vec![18, 19, 20, 21]); assert!(fl.w.as_ref().unwrap().bad.is_none()); + assert_eq!(fl.w.as_ref().unwrap().rep.summary(), live::verify(&text, &run).unwrap(), "ends where verify does"); + let _ = std::fs::remove_file(&f); + // the tutor without the block takes the shortcut: `following_starts_at_the_last_checkpoint` + } + /// Watch a strand line by line: every document's sha256 is its line's `stance` digest, and the replay ends where `verify` does fn replays_to_its_digests(text: &str, name: &str) -> Watch { let lines = live::lines(text); diff --git a/probbit-cli/src/monitor/panel.rs b/probbit-cli/src/monitor/panel.rs index bac6f51..8633691 100644 --- a/probbit-cli/src/monitor/panel.rs +++ b/probbit-cli/src/monitor/panel.rs @@ -49,6 +49,9 @@ pub enum Field { Undeclared, /// Room left in the learning bound (RFC 0004): not computed until the engine has it Room, + /// RFC 0014 integrity: the summed drift as a share of the summed allowance (`used`), the summed drift, the traits past + /// `identity.warn` and the traits measured + Integrity { used: f64, drift: f64, crossed: usize, traits: usize }, } /// A field at an event: its stable key (`stance:verbosity`, `mood:pace`, `pace`, …), its label and its reading #[derive(Clone, Debug, PartialEq)] @@ -85,6 +88,7 @@ fn gene_level(gene: &str, x: f64) -> &'static str { ("patience", true) => "longer credit memory", ("patience", false) => "shorter credit memory", (_, true) => "surprise gate opens later", (_, false) => "surprise gate opens sooner" } } +fn integrity_level(crossed: usize, traits: usize) -> String { if crossed == 0 { "within warn".into() } else { format!("past warn on {crossed} of {traits}") } } fn count_level(n: usize) -> String { if n == 0 { "none open".into() } else { format!("{n} open") } } fn learned_level(levels: &[String], ds: &[f64]) -> String { if ds.iter().all(|d| d.abs() < LEARN_STEP) { return "none yet".into(); } @@ -157,6 +161,11 @@ pub fn word(now: &Reading, prev: Option<&Reading>) -> Word { }, Field::Undeclared => (UNDECLARED.to_string(), "", String::new(), None, String::new()), Field::Room => (ROOM.to_string(), "", String::new(), None, String::new()), + Field::Integrity { used, drift, crossed, traits } => { + let m = used.clamp(0.0, 1.0); + let d = match before { Some(Field::Integrity { used: x, .. }) => trend(used - x, NUM_STEP).to_string(), _ => FIRST.to_string() }; + (integrity_level(*crossed, *traits), if *used <= 0.0 { "" } else { size_of(m) }, d, Some(m), format!("used {used:.6} drift {drift:.6}")) + } }; let text = line(&now.label, &level, size, &dir); Word { key: now.key.clone(), label: now.label.clone(), level, size, dir, text, dial, value } @@ -186,6 +195,7 @@ pub fn space(field: &Field) -> (Vec, Vec<&'static str>, Vec) { Field::Sense { .. } => (vec!["read".into(), "not read".into()], { let mut s = s3(); s.push(""); s }, vec![String::new(), "above its normal".into(), "below its normal".into(), "at its normal".into()]), Field::Undeclared => (vec![UNDECLARED.into()], vec![""], vec![String::new()]), Field::Room => (vec![ROOM.into()], vec![""], vec![String::new()]), + Field::Integrity { traits, .. } => ((0..=*traits).map(|c| integrity_level(c, *traits)).collect(), { let mut s = s3(); s.push(""); s }, num_dirs()), } } #[cfg(test)] @@ -194,7 +204,8 @@ pub fn vocabulary() -> Vec { let mut v: Vec = SIZES.iter().map(|s| s.to_string()).collect(); v.extend([FIRST, HOLDING, "rising", "falling", "leaning toward", "fixed", "above its normal", "below its normal", "at its normal", "not released", "not reported", "pressing", "not pressing", "gate open", "gate closed", "even", "none yet", "toward", "none open", "open", "in force and", "bound", "read", "not read", UNDECLARED, ROOM, - "deficit", "g", "rbar", "p", "z", "needs", "surprise gate", "pace", "dispositions", "credit", "habits", "room left in the bound", "learned", "need", "sense", "mood", "stance", "average reward"].map(String::from)); + "deficit", "g", "rbar", "p", "z", "needs", "surprise gate", "pace", "dispositions", "credit", "habits", "room left in the bound", "learned", "need", "sense", "mood", "stance", "average reward", + "within warn", "past warn on", "of", "used", "drift", "integrity (drift from the declared self)"].map(String::from)); for g in ["optimism", "patience", "volatility"] { for x in [1.0, -1.0, 0.0] { v.push(gene_level(g, x).to_string()); } v.push(g.to_string()); } v } @@ -225,7 +236,8 @@ mod tests { v.push(Field::Learned { levels: ls.iter().map(|s| s.to_string()).collect(), deltas: vec![], cap: 0.25 }); } v.extend([Field::Need { deficit: 0.0, cap: 1.0, term: 0.0 }, Field::Gate { g: 1.0, g_max: 3.0 }, Field::Pace { level: "steady".into(), rbar: 0.0, cap: 4.0 }, - Field::Credit { open: 0, max: 8 }, Field::Habits { active: 0, bound: 0, declared: 11 }, Field::Sense { z: None }, Field::Undeclared, Field::Room]); + Field::Credit { open: 0, max: 8 }, Field::Habits { active: 0, bound: 0, declared: 11 }, Field::Sense { z: None }, Field::Undeclared, Field::Room, + Field::Integrity { used: 0.0, drift: 0.0, crossed: 0, traits: 2 }]); for g in ["optimism", "patience", "volatility"] { v.push(Field::Gene { gene: g.into(), x: 0.0 }); } v } @@ -282,7 +294,7 @@ mod tests { for w in vocabulary() { assert!(law_hits(&w).is_empty(), "{w}"); } let mut n = 0; for f in kinds() { let (ls, ss, ds) = space(&f); for l in &ls { for s in &ss { for d in &ds { let t = line("x", l, s, d); assert!(law_hits(&t).is_empty(), "{t}"); n += 1; } } } } - for label in ["needs", "surprise gate", "pace", "dispositions", "credit", "habits", "room left in the bound"] { assert!(law_hits(label).is_empty()); } + for label in ["needs", "surprise gate", "pace", "dispositions", "credit", "habits", "room left in the bound", "integrity (drift from the declared self)"] { assert!(law_hits(label).is_empty()); } assert!(n > 1000); } /// Words are a function of the buckets: a small move inside a bucket changes the dial, never the words diff --git a/probbit-cli/tests/cli.rs b/probbit-cli/tests/cli.rs index 42a66ed..6fb0d1b 100644 --- a/probbit-cli/tests/cli.rs +++ b/probbit-cli/tests/cli.rs @@ -3,7 +3,7 @@ use std::io::Write; use std::process::{Command, Stdio}; // The crate's own zero-dependency JSON reader: error documents must re-parse with it (P0.1 / P0.2 contract tests) #[allow(dead_code)] -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; fn probbit(args: &[&str], stdin: &str) -> (i32, String, String) { diff --git a/probbit-cli/tests/cli_check.rs b/probbit-cli/tests/cli_check.rs index 22fc683..b8a0e83 100644 --- a/probbit-cli/tests/cli_check.rs +++ b/probbit-cli/tests/cli_check.rs @@ -3,7 +3,7 @@ use std::io::Write; use std::process::{Command, Stdio}; -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] #[allow(dead_code)] mod json; use json::Json; diff --git a/probbit-cli/tests/cli_output.rs b/probbit-cli/tests/cli_output.rs index a7fceea..24fb1b0 100644 --- a/probbit-cli/tests/cli_output.rs +++ b/probbit-cli/tests/cli_output.rs @@ -1,7 +1,7 @@ //! Human formatting must not change machine values, persisted state or replay records. use std::process::{Command, Output}; #[allow(dead_code)] -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; const PERSONA: &str = "../examples/persona/tutor.yaml"; diff --git a/probbit-cli/tests/evaluate.rs b/probbit-cli/tests/evaluate.rs index 9c981e4..3a26ed0 100644 --- a/probbit-cli/tests/evaluate.rs +++ b/probbit-cli/tests/evaluate.rs @@ -6,7 +6,7 @@ use std::io::Write; use std::process::{Command, Stdio}; #[allow(dead_code)] -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; use json::{num, obj, str as jstr, Json}; diff --git a/probbit-cli/tests/fixtures/monitor/tutor-expect.events.jsonl b/probbit-cli/tests/fixtures/monitor/tutor-expect.events.jsonl new file mode 100644 index 0000000..5d77426 --- /dev/null +++ b/probbit-cli/tests/fixtures/monitor/tutor-expect.events.jsonl @@ -0,0 +1,22 @@ +{"praise":true,"elapsed_hours":0.5} +{"loss":true,"elapsed_hours":1} +{"elapsed_hours":2} +{"error":true,"elapsed_hours":13} +{"praise":true,"elapsed_hours":1} +{"loss":true,"elapsed_hours":0.5} +{"elapsed_hours":1} +{"error":true,"elapsed_hours":2} +{"praise":true,"elapsed_hours":13} +{"loss":true,"elapsed_hours":1} +{"elapsed_hours":0.5} +{"error":true,"elapsed_hours":1} +{"praise":true,"elapsed_hours":2} +{"loss":true,"elapsed_hours":13} +{"elapsed_hours":1} +{"error":true,"elapsed_hours":0.5} +{"praise":true,"elapsed_hours":1} +{"loss":true,"elapsed_hours":2} +{"elapsed_hours":13} +{"error":true,"elapsed_hours":1} +{"praise":true,"elapsed_hours":0.5} +{"loss":true,"elapsed_hours":1} diff --git a/probbit-cli/tests/fixtures/monitor/tutor-expect.strand b/probbit-cli/tests/fixtures/monitor/tutor-expect.strand new file mode 100644 index 0000000..57fd98a --- /dev/null +++ b/probbit-cli/tests/fixtures/monitor/tutor-expect.strand @@ -0,0 +1,27 @@ +{"probbit_strand":1,"engine":"probbit 0.13.0","persona":{"name":"Pip","version":"1.1.0","digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c"},"seed":2,"state":{"digest":"sha256:eef51fdf809d2a63d5f9f2fe90bca59c5f319881cdebe0ce6fb35de60b7aaf92","genes":{"react":{"praise":0.7851},"shift":{"arousal":0.214,"caution":-0.3903,"curiosity":0.3383,"directness":0.0956,"emoji":0.959,"formality":-0.3644,"humour":-2.1939,"initiative":0.1127,"valence":-0.0338,"verbosity":-0.2809,"warmth":0.268}},"history":{"confused":[],"error":[]},"mood":{"arousal":[0,0,0],"valence":[0,0,0]},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"prev":{"agenda":["encourage","explain","example","check"],"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"gentle","emoji":"sparse","formality":"casual","humour":"light","initiative":"suggest","valence":"even","verbosity":"full","warmth":"warm"},"probbit_persona_state":1,"rest":{"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"balanced","emoji":"rich","formality":"casual","humour":"light","initiative":"suggest","valence":"up","verbosity":"full","warmth":"warm"},"seed":2,"turn":0},"document":{"probbit_persona":1,"identity":{"name":"Pip","version":"1.1.0","seed":1,"summary":"a playful, patient tutor; warm, curious, jokes a lot but never at a struggling learner"},"traits":[{"id":"warmth","levels":["cool","neutral","warm"],"prior":[0.05,0.3,0.65],"spread":0.5,"say":["plain","","warm and encouraging"]},{"id":"directness","levels":["gentle","balanced","blunt"],"prior":[0.45,0.45,0.1],"spread":0.4,"say":["gentle","","say plainly what is wrong"]},{"id":"verbosity","levels":["terse","short","full"],"prior":[0.1,0.4,0.5],"spread":0.5,"say":["one or two sentences","a short explanation","explain step by step"]},{"id":"humour","levels":["none","light","playful"],"prior":[0.15,0.45,0.4],"spread":0.6,"say":["no jokes","a light joke is welcome","playful"]},{"id":"formality","levels":["casual","neutral","formal"],"prior":[0.6,0.35,0.05],"spread":0.3,"say":["casual","","formal"]},{"id":"initiative","levels":["follow","suggest","lead"],"prior":[0.2,0.5,0.3],"spread":0.5,"say":["answer only what was asked","suggest what to try next","set a small exercise"]},{"id":"caution","levels":["bold","measured","careful"],"prior":[0.25,0.55,0.2],"spread":0.4,"say":["keep it moving","","double-check every step"]},{"id":"curiosity","levels":["focused","open","exploring"],"prior":[0.15,0.4,0.45],"spread":0.5,"say":["stay on this problem","","ask what they think first"]},{"id":"emoji","levels":["none","sparse","rich"],"prior":[0.2,0.5,0.3],"spread":0.5,"say":["no emoji","one emoji at most","emoji welcome"]}],"moods":[{"id":"valence","levels":["down","even","up"],"prior":[0.1,0.5,0.4],"spread":0.4,"inertia":0.5,"half_life_hours":12},{"id":"arousal","levels":["calm","steady","keyed"],"prior":[0.3,0.4,0.3],"spread":0.3,"inertia":0.4}],"couplings":[{"vars":["valence","humour"],"align":1},{"vars":["valence","emoji"],"align":0.6},{"vars":["arousal","verbosity"],"align":-0.4},{"vars":["warmth","formality"],"align":-0.6},{"vars":["humour","formality"],"align":-0.6},{"vars":["curiosity","verbosity"],"align":0.3},{"vars":["humour","emoji"],"align":0.4}],"inputs":[{"id":"sentiment","kind":"level","levels":["negative","neutral","positive"],"default":"neutral","say":{"negative":"learner upset","positive":"learner upbeat"},"effects":{"negative":{"valence":-1,"warmth":0.6,"humour":-1,"directness":-0.6},"positive":{"valence":0.8,"humour":0.5,"emoji":0.3}}},{"id":"confused","kind":"flag","say":"learner confused","effects":{"verbosity":0.9,"humour":-0.6,"warmth":0.4,"curiosity":0.6,"initiative":0.3,"directness":-0.3}},{"id":"error","kind":"flag","say":"my last answer was wrong","effects":{"caution":1.2,"verbosity":-0.3,"valence":-0.4,"humour":-0.4}},{"id":"loss","kind":"flag","say":"learner reports a failure","effects":{"valence":-1.6,"warmth":1,"humour":-2,"emoji":-0.8}},{"id":"praise","kind":"flag","say":"praise","reactivity_spread":0.4,"effects":{"valence":1.2,"humour":0.5,"emoji":0.5}},{"id":"criticism","kind":"flag","say":"criticism","effects":{"valence":-0.6,"caution":0.6,"humour":-0.4}},{"id":"stakes","kind":"number","say":"stakes","effects":{"caution":1.4,"humour":-0.8,"emoji":-0.6}},{"id":"time_pressure","kind":"number","say":"time pressure","effects":{"verbosity":-1.8,"curiosity":-1.2,"humour":-0.4,"arousal":0.6}},{"id":"claim_done","kind":"flag","say":"about to say it is solved","effects":{"caution":0.5,"step.check":0.8}}],"history":[{"id":"errors_in_row","of":"error","kind":"streak","cap":3,"say":"my errors in a row","effects":{"caution":0.5,"humour":-0.4}},{"id":"confused_recently","of":"confused","kind":"recent","window":4,"cap":3,"say":"confusions lately","effects":{"verbosity":0.3,"warmth":0.2}}],"habits":[{"id":"no_jokes_on_loss","when":{"loss":true},"then":{"humour":["none"],"emoji":{"at_most":"sparse"}},"say":"no jokes, be kind"},{"id":"no_play_when_upset","when":{"sentiment":"negative"},"then":{"humour":{"at_most":"light"}},"say":"go easy on the jokes"},{"id":"confused_no_playful","when":{"confused":true},"then":{"humour":{"at_most":"light"},"verbosity":{"at_least":"short"}},"say":"slow down, no clowning","priority":10},{"id":"after_error_check","when":{"error":true},"then":{"caution":{"at_least":"prev+1"},"verbosity":{"at_most":"prev-1"}},"say":"shorter, double-check"},{"id":"flourish_budget","rules":{"linear":[{"terms":[["humour","playful",2],["humour","light",1],["emoji","rich",2],["emoji","sparse",1]],"limit":3}]},"say":"one flourish at a time"},{"id":"no_formal_emoji","rules":{"tables":[{"vars":["formality","emoji"],"forbid":[["formal","rich"],["formal","sparse"]]}]},"say":"no emoji when formal"},{"id":"check_last","rules":{"precedes":[{"before":"step.explain","after":"step.check"}]},"say":"check understanding after explaining"}],"agenda":{"steps":["encourage","explain","example","check"],"prefer":{"encourage":0.5,"explain":0.3,"example":-0.1,"check":-0.6}},"engine":{"on_conflict":"yield"},"line":{"max_tokens":40,"order":["verbosity","humour","warmth","initiative","curiosity","emoji","directness","caution","formality"]},"expectation":{}}} +{"deviation":{"input":0,"pred":0.20781,"self":0.20781},"inputs":{"elapsed_hours":0.5,"praise":true},"n":1,"prev":"sha256:7e9df458eb9b480cbbe1cc35394eae47df68d2beaf719d00a243db88f6285827","stance":"sha256:016fef69c5e036b83b4b5e8a704a166053bf75333fcfe643b91a01798431c113","state":"sha256:1bcf9a02bf0454969e10442b3d4ebefc18f69df15734c3e5af5e0d0189f0ef60"} +{"deviation":{"input":0.69315,"pred":2.8423,"self":2.2116},"inputs":{"elapsed_hours":1,"loss":true},"n":2,"prev":"sha256:ae314cf23a00bbd7e4a0a78ba77631649bd00dd4b37c1f55225d1e090f4f3010","stance":"sha256:afb1164f2022b3646092919c3627a6652132f9d6b86cbc747f6394eab02de22e","state":"sha256:c79a11b1406d5373bcb2f873fc3d99857b6e78a23a75be3b4de9a43f830a2114"} +{"deviation":{"input":1.0914,"pred":0.091607,"self":0},"inputs":{"elapsed_hours":2},"n":3,"prev":"sha256:74b1f36de9ffd3fb0fe5a8e08eb36941216d97dd08579f18f7af003b0c935ea4","stance":"sha256:541049d8ecde4c92f08cb234ea6cbf5040017d93dc276b16f457120b24c967d3","state":"sha256:207b7f6a8bb56cefac90f79dbb8cea06f5db518abd4229a937ed8a684ccf85f2"} +{"deviation":{"input":0.67706,"pred":2.7765,"self":2.8336},"inputs":{"elapsed_hours":13,"error":true},"n":4,"prev":"sha256:93b2cd22ff409fba27ba33abd44bb3214295e611fe0e8165271b9c3168d8b64e","stance":"sha256:019959c6bf2d5d60eadb2e4c89fefe7151dd67c6491b1f2eb4c161b7448f0a86","state":"sha256:e85069f361ff10bac7fd3f35cfb5e2ecd148f72214c2bba5ff3cfbd4992ea9fa"} +{"deviation":{"input":1.6487,"pred":0.66312,"self":0.21638},"inputs":{"elapsed_hours":1,"praise":true},"n":5,"prev":"sha256:2e12dab1a54c0b75bd06a29ff191cef7ae34b9938dbb20cbf5b3cbfd2e7edcf2","stance":"sha256:8daaddad69d74b08f223aba03aafa4ec0a5f90decb3bf9ec6d3d9486e1b4fb06","state":"sha256:14bc4af3f729caebb13dbafc92885087d8c8ccfb25082dda2db05f058bdc9198"} +{"checkpoint":5,"prev":"sha256:ebb3d4d0ca05407c4ab2a494edc5ab495a0227b24a303fd525e0a5520b281729","stance":{"agenda":["encourage","explain","example","check"],"deviation":{"cold":false,"input":1.6487454934364103,"pred":0.663115886379269,"registers":false,"self":0.2163809366922902,"z":1.354697},"engine":{"program":"sha256:95a9314d809c73dc9d9b7a9496fa816337d890c576fe0ceed2d017ff4550cd44","tier":"components","verdict":"exact"},"escalate":null,"habits":{"active":["flourish_budget","no_formal_emoji","check_last"],"bound":["flourish_budget"],"conflict":[],"violations":0,"yielded":[]},"held":[],"ignored":[],"inputs":{"claim_done":false,"confused":false,"confused_recently":0,"criticism":false,"error":false,"errors_in_row":0,"loss":false,"praise":true,"sentiment":"neutral","stakes":0,"time_pressure":0},"line":"Stance: one flourish at a time; explain step by step; a light joke is welcome; warm and encouraging; suggest what to try next; ask what they think first.","line_tokens":39,"mood":{"arousal":{"level":"steady","odds":{"calm":0.280269,"keyed":0.329436,"steady":0.390295},"p":0.390295,"released":true},"valence":{"level":"up","odds":{"down":0.085635,"even":0.417117,"up":0.497249},"p":0.497249,"released":true}},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"probbit_persona_turn":1,"seed":2,"stance":{"caution":{"level":"measured","odds":{"bold":0.350191,"careful":0.128347,"measured":0.521462},"p":0.521462,"released":true},"curiosity":{"level":"exploring","odds":{"exploring":0.580967,"focused":0.084846,"open":0.334187},"p":0.580967,"released":true},"directness":{"level":"balanced","odds":{"balanced":0.464394,"blunt":0.113552,"gentle":0.422054},"p":0.464394,"released":true},"emoji":{"level":"rich","odds":{"none":0.037498,"rich":0.640842,"sparse":0.32166},"p":0.640842,"released":true},"formality":{"level":"casual","odds":{"casual":0.755025,"formal":0.001089,"neutral":0.243887},"p":0.755025,"released":true},"humour":{"level":"light","odds":{"light":0.554496,"none":0.386132,"playful":0.059371},"p":0.554496,"released":true},"initiative":{"level":"suggest","odds":{"follow":0.176135,"lead":0.330998,"suggest":0.492867},"p":0.492867,"released":true},"verbosity":{"level":"full","odds":{"full":0.453824,"short":0.412439,"terse":0.133737},"p":0.453824,"released":true},"warmth":{"level":"warm","odds":{"cool":0.01642,"neutral":0.183144,"warm":0.800436},"p":0.800436,"released":true}},"state_digest":"sha256:14bc4af3f729caebb13dbafc92885087d8c8ccfb25082dda2db05f058bdc9198","status":"ok","turn":4,"unsure":[],"why":"one flourish at a time (habit); praise -> valence up, humour up"},"state":{"digest":"sha256:14bc4af3f729caebb13dbafc92885087d8c8ccfb25082dda2db05f058bdc9198","genes":{"react":{"praise":0.7851},"shift":{"arousal":0.214,"caution":-0.3903,"curiosity":0.3383,"directness":0.0956,"emoji":0.959,"formality":-0.3644,"humour":-2.1939,"initiative":0.1127,"valence":-0.0338,"verbosity":-0.2809,"warmth":0.268}},"history":{"confused":[false,false,false,false],"error":[false,false,false,true,false]},"mood":{"arousal":[0,0,0],"valence":[-0.348016,0,0.348016]},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"prev":{"agenda":["encourage","explain","example","check"],"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"balanced","emoji":"rich","formality":"casual","humour":"light","initiative":"suggest","valence":"up","verbosity":"full","warmth":"warm"},"probbit_persona_state":1,"rest":{"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"balanced","emoji":"rich","formality":"casual","humour":"light","initiative":"suggest","valence":"up","verbosity":"full","warmth":"warm"},"seed":2,"turn":5}} +{"deviation":{"input":1.8212,"pred":1.8311,"self":2.1799},"inputs":{"elapsed_hours":0.5,"loss":true},"n":6,"prev":"sha256:691b6399093b47507c5ff3a2159d7ee44a75eabf9961d436161cdc7e2ad54f19","stance":"sha256:96bc79dd0fd3ced3867e8b3a37e83662fd983631938a22e727b6be78c5c0db94","state":"sha256:9cd5ea7e03aa5a16de710ede3ea7c3bf7fbb0dcaeb9d5b02c35fe143fc2a8e46"} +{"deviation":{"input":1.2391,"pred":0.1417,"self":0},"inputs":{"elapsed_hours":1},"n":7,"prev":"sha256:33b8db219356dab0921c72ad7ec386640708520a6131b814b122c99768ccba42","stance":"sha256:6c39be8da20202ed2744771641966f0e5ec87ea0cb708b7a85c64b4f4086b268","state":"sha256:0a32589256e6e254996652de8f423d03d5bda926b2b81eb0b77896e66f4913c8"} +{"deviation":{"input":2.0886,"pred":2.0414,"self":2.8317},"inputs":{"elapsed_hours":2,"error":true},"n":8,"prev":"sha256:efa74ba1c5e4ba18c5d1163866a6349ded96d042b81ac8ad19966da22bc75a40","stance":"sha256:7a662e50d3bad0352781a7d641e654b89b6f77d0489269c5838d4410021725d3","state":"sha256:2510474026de865306b0aed7b69f11e857eeff840006a4ff5f4c881cbcb4a6c7"} +{"deviation":{"input":1.5454,"pred":0.73484,"self":0.21337},"inputs":{"elapsed_hours":13,"praise":true},"n":9,"prev":"sha256:867188510f981fa92463e085740d9a0cf26f53f2344ca75a76baf26252cd691d","stance":"sha256:30f92c2616e3b38901fc660f61b4556fe18f8fe7cd60601fe729ef82b6294efb","state":"sha256:fbe42a6a03239aef066ddcd908f8b65dea2cf4d2349d6ba62f9ad192630acdb9"} +{"deviation":{"input":1.6406,"pred":1.7092,"self":2.1896},"inputs":{"elapsed_hours":1,"loss":true},"n":10,"prev":"sha256:8e0b586b21d0ffeecd6ed1a91aed3b53def6f5d87b8d7e4bd0d8c51bb5f23308","stance":"sha256:f764a6f800ac7364e6c76c672ce7b8dc001fb7d8f63f8cd33107a6d83e334dc5","state":"sha256:225e7dfd9eb78d673993cae884d4be5335fcd21b779d7998ac6ce3d4b2672fcd"} +{"checkpoint":10,"prev":"sha256:cc852f6ea39d2367b6123a6066a8ada50914135bab5008ee319e270a158b0a61","stance":{"agenda":["encourage","explain","example","check"],"deviation":{"cold":false,"input":1.6405653534874491,"pred":1.709166276250666,"registers":true,"self":2.1895837322588894,"z":2.288624},"engine":{"program":"sha256:b82832d8dc76cef7147d0b343220cf0b8e413148667b2d51c966c7e45b4c9d0f","tier":"components","verdict":"exact"},"escalate":null,"habits":{"active":["no_jokes_on_loss","flourish_budget","no_formal_emoji","check_last"],"bound":[],"conflict":[],"violations":0,"yielded":[]},"held":[],"ignored":[],"inputs":{"claim_done":false,"confused":false,"confused_recently":0,"criticism":false,"error":false,"errors_in_row":0,"loss":true,"praise":false,"sentiment":"neutral","stakes":0,"time_pressure":0},"line":"Stance: no jokes, be kind; one emoji at most; explain step by step; warm and encouraging; suggest what to try next; ask what they think first; casual.","line_tokens":38,"mood":{"arousal":{"level":"steady","odds":{"calm":0.280269,"keyed":0.329436,"steady":0.390295},"p":0.390295,"released":true},"valence":{"level":"down","odds":{"down":0.541495,"even":0.405777,"up":0.052728},"p":0.541495,"released":true}},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"probbit_persona_turn":1,"seed":2,"stance":{"caution":{"level":"measured","odds":{"bold":0.350191,"careful":0.128347,"measured":0.521462},"p":0.521462,"released":true},"curiosity":{"level":"exploring","odds":{"exploring":0.580967,"focused":0.084846,"open":0.334187},"p":0.580967,"released":true},"directness":{"level":"balanced","odds":{"balanced":0.464394,"blunt":0.113552,"gentle":0.422054},"p":0.464394,"released":true},"emoji":{"level":"sparse","odds":{"none":0.416584,"rich":0,"sparse":0.583416},"p":0.583416,"released":true},"formality":{"level":"casual","odds":{"casual":0.690526,"formal":0.013291,"neutral":0.296183},"p":0.690526,"released":true},"humour":{"level":"none","odds":{"light":0,"none":1,"playful":0},"p":1,"released":true},"initiative":{"level":"suggest","odds":{"follow":0.176135,"lead":0.330998,"suggest":0.492867},"p":0.492867,"released":true},"verbosity":{"level":"full","odds":{"full":0.453824,"short":0.412439,"terse":0.133737},"p":0.453824,"released":true},"warmth":{"level":"warm","odds":{"cool":0.002982,"neutral":0.08223,"warm":0.914788},"p":0.914788,"released":true}},"state_digest":"sha256:225e7dfd9eb78d673993cae884d4be5335fcd21b779d7998ac6ce3d4b2672fcd","status":"ok","turn":9,"unsure":[],"why":"learner reports a failure -> humour down, valence down"},"state":{"digest":"sha256:225e7dfd9eb78d673993cae884d4be5335fcd21b779d7998ac6ce3d4b2672fcd","genes":{"react":{"praise":0.7851},"shift":{"arousal":0.214,"caution":-0.3903,"curiosity":0.3383,"directness":0.0956,"emoji":0.959,"formality":-0.3644,"humour":-2.1939,"initiative":0.1127,"valence":-0.0338,"verbosity":-0.2809,"warmth":0.268}},"history":{"confused":[false,false,false,false],"error":[false,false,true,false,false]},"mood":{"arousal":[0,0,0],"valence":[0.614733,0,-0.614733]},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"prev":{"agenda":["encourage","explain","example","check"],"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"balanced","emoji":"sparse","formality":"casual","humour":"none","initiative":"suggest","valence":"down","verbosity":"full","warmth":"warm"},"probbit_persona_state":1,"rest":{"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"balanced","emoji":"rich","formality":"casual","humour":"light","initiative":"suggest","valence":"up","verbosity":"full","warmth":"warm"},"seed":2,"turn":10}} +{"deviation":{"input":1.2831,"pred":0.1713,"self":0},"inputs":{"elapsed_hours":0.5},"n":11,"prev":"sha256:0399dee7985c4b6e32fe79505187d1edde3519dd9c1bb8bd3c432cdb36435951","stance":"sha256:61deb67957b520ae6c81fb86e6baed41a10d13c37ec579f97b5515119e05180a","state":"sha256:9b0fffb260a9d2d317b8c6d2a2704a6c38331923aaad66822ae32f81019495dd"} +{"deviation":{"input":1.8024,"pred":1.8623,"self":2.8315},"inputs":{"elapsed_hours":1,"error":true},"n":12,"prev":"sha256:4b98762c333e63d5d0a81398aeb4236fb0ce6cebeabd737ee9675b1246adc7fa","stance":"sha256:d5276ba7bbdf776c324f04cfb5e00d06f98081ef0a03ef09aad2de2dce78c2d7","state":"sha256:daffac198a0ce7f3a8cdf7270c1807ce41c54f6918e37bcadde7755a4ea1866b"} +{"deviation":{"input":1.5086,"pred":0.76222,"self":0.21827},"inputs":{"elapsed_hours":2,"praise":true},"n":13,"prev":"sha256:390563c827d2aad8bd94bb20f77413258dee0175ddb37c5fd69ac6c86c0d0709","stance":"sha256:13ff4842a26ea33d98dfe343fd835a3f14c93011f43b142a2a86abc83076309e","state":"sha256:f0ccc4aed92c3f03b6dfc356d4231660bc643d164cd1d8e402bcb6444b474eeb"} +{"deviation":{"input":1.5725,"pred":1.628,"self":2.1228},"inputs":{"elapsed_hours":13,"loss":true},"n":14,"prev":"sha256:359f6e4df83d6fba50bb60abc1e97fa79f51134784456d48b0ebbdea622a4f23","stance":"sha256:f586b7c7a2643eb010d82d99ee5314e9abd215716ea9dd36b005bf4d18ae957f","state":"sha256:2c2d7b9e7e66d7dd7050ecf7101666ecbfe9be0ebe0abbf31c0d1be19c64fc32"} +{"deviation":{"input":1.3042,"pred":0.18472,"self":0},"inputs":{"elapsed_hours":1},"n":15,"prev":"sha256:af88a285bfb51bdaa1d5af3c3a0e080e600c3b98ff93d7b79e09085cb99b9316","stance":"sha256:0a186aad66b1f2b853088abda07fbf6395730994c69599082b30a8d57d29f7b6","state":"sha256:9f270112de0ba5b00c5f5b1bc802b751bdb81c8c9d9bda484ab65f1db80b86ba"} +{"checkpoint":15,"prev":"sha256:62d72cf3d2a58dbdf915c6f3a5577a461242f38707c4efdc1d0bd2047c12c320","stance":{"agenda":["encourage","explain","example","check"],"deviation":{"cold":false,"input":1.3042172408804977,"pred":0.18472371562460088,"registers":false,"self":0,"z":-0.044533},"engine":{"program":"sha256:9ad02ae3f33a68b43dcf5443385d4ea81abbcd26ea557efd1f6314a9987434f0","tier":"components","verdict":"exact"},"escalate":null,"habits":{"active":["flourish_budget","no_formal_emoji","check_last"],"bound":[],"conflict":[],"violations":0,"yielded":[]},"held":[],"ignored":[],"inputs":{"claim_done":false,"confused":false,"confused_recently":0,"criticism":false,"error":false,"errors_in_row":0,"loss":false,"praise":false,"sentiment":"neutral","stakes":0,"time_pressure":0},"line":"Stance: no jokes; explain step by step; warm and encouraging; suggest what to try next; ask what they think first; emoji welcome; casual.","line_tokens":35,"mood":{"arousal":{"level":"steady","odds":{"calm":0.280269,"keyed":0.329436,"steady":0.390295},"p":0.390295,"released":true},"valence":{"level":"even","odds":{"down":0.25337,"even":0.4997,"up":0.24693},"p":0.4997,"released":true}},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"probbit_persona_turn":1,"seed":2,"stance":{"caution":{"level":"measured","odds":{"bold":0.350191,"careful":0.128347,"measured":0.521462},"p":0.521462,"released":true},"curiosity":{"level":"exploring","odds":{"exploring":0.580967,"focused":0.084846,"open":0.334187},"p":0.580967,"released":true},"directness":{"level":"balanced","odds":{"balanced":0.464394,"blunt":0.113552,"gentle":0.422054},"p":0.464394,"released":true},"emoji":{"level":"rich","odds":{"none":0.101772,"rich":0.480668,"sparse":0.41756},"p":0.480668,"released":true},"formality":{"level":"casual","odds":{"casual":0.72774,"formal":0.003406,"neutral":0.268853},"p":0.72774,"released":true},"humour":{"level":"none","odds":{"light":0.377719,"none":0.590719,"playful":0.031562},"p":0.590719,"released":true},"initiative":{"level":"suggest","odds":{"follow":0.176135,"lead":0.330998,"suggest":0.492867},"p":0.492867,"released":true},"verbosity":{"level":"full","odds":{"full":0.453824,"short":0.412439,"terse":0.133737},"p":0.453824,"released":true},"warmth":{"level":"warm","odds":{"cool":0.01711,"neutral":0.185902,"warm":0.796988},"p":0.796988,"released":true}},"state_digest":"sha256:9f270112de0ba5b00c5f5b1bc802b751bdb81c8c9d9bda484ab65f1db80b86ba","status":"ok","turn":14,"unsure":[],"why":"baseline (no live evidence)"},"state":{"digest":"sha256:9f270112de0ba5b00c5f5b1bc802b751bdb81c8c9d9bda484ab65f1db80b86ba","genes":{"react":{"praise":0.7851},"shift":{"arousal":0.214,"caution":-0.3903,"curiosity":0.3383,"directness":0.0956,"emoji":0.959,"formality":-0.3644,"humour":-2.1939,"initiative":0.1127,"valence":-0.0338,"verbosity":-0.2809,"warmth":0.268}},"history":{"confused":[false,false,false,false],"error":[false,true,false,false,false]},"mood":{"arousal":[0,0,0],"valence":[0.342003,0,-0.342003]},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"prev":{"agenda":["encourage","explain","example","check"],"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"balanced","emoji":"rich","formality":"casual","humour":"none","initiative":"suggest","valence":"even","verbosity":"full","warmth":"warm"},"probbit_persona_state":1,"rest":{"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"balanced","emoji":"rich","formality":"casual","humour":"light","initiative":"suggest","valence":"up","verbosity":"full","warmth":"warm"},"seed":2,"turn":15}} +{"deviation":{"input":1.6857,"pred":1.7805,"self":2.8308},"inputs":{"elapsed_hours":0.5,"error":true},"n":16,"prev":"sha256:05d854ca158ab2d6fe5b310d0d4b785703556925e175ac591520710d908ee9b4","stance":"sha256:14490dfe573bbd90bc91901e7372278758d1c8c9bcbbf8b45a4c50e03f04f0a9","state":"sha256:3ce4d0cd4fcc67529de310a5b07ea0f348b24d4834362906eed981f4490bd3b3"} +{"deviation":{"input":1.4897,"pred":0.77684,"self":0.21961},"inputs":{"elapsed_hours":1,"praise":true},"n":17,"prev":"sha256:df2efee5cc1f4a517cc77a8bd56c30d99c46e0e5344d863e64ec3cdb1ab3389f","stance":"sha256:d5a39913357d01edeed04e798e16730dafe97ecb406031501e32ffa33d3cba76","state":"sha256:5a2e0ed7a588a78f8bb697bf11bc9a241e74bfd043976ee1a77e2a0de965edb3"} +{"deviation":{"input":1.5368,"pred":1.6188,"self":2.158},"inputs":{"elapsed_hours":2,"loss":true},"n":18,"prev":"sha256:9c9e0bd9e30eb4cf3a60f011d3a76ef9c788e2ed726c8afe5344b9158c5edd2b","stance":"sha256:f759e0c8adc25d43e6dc138c3cfed372c5a8016615ea1c23a6b65827c53939dc","state":"sha256:11816113854730ab1bd7091c290bc0f20ca9a870513e4437511bb9a17bcf179b"} +{"deviation":{"input":1.3166,"pred":0.20393,"self":0},"inputs":{"elapsed_hours":13},"n":19,"prev":"sha256:171d3e3fe1407a4df0b1adb83b70673976aca386ccc033de8f0cd375f04a6217","stance":"sha256:7e720363c8afa260bb42a0a8c105ab20cae1fa30761f8f38aaa9a37c9525bebd","state":"sha256:fc62ce1c64e52ce54c11951280a068f335f5f7a746285fae51cd26e81ad9ddf5"} +{"deviation":{"input":1.6221,"pred":1.7338,"self":2.8333},"inputs":{"elapsed_hours":1,"error":true},"n":20,"prev":"sha256:654d45575a01362b56990acc0f64f2e9d9f236d30d226ade9b3651da34582a2d","stance":"sha256:131e8d2319a49d677e53a0d596829aa67fabee60e4fc50b8063d18210f4c6364","state":"sha256:2ea90ddecba0a2b1dc2481f28df71bb284a8d3f7df8d3d4f9b8446f245ac260e"} +{"checkpoint":20,"prev":"sha256:fa0af32af76d00d7877ea4dd78c798b4f896c19eae6f790596ec6f932395f0f1","stance":{"agenda":["encourage","explain","example","check"],"deviation":{"cold":false,"input":1.622062690482329,"pred":1.7338026105253461,"registers":true,"self":2.8332527443993802,"z":1.791988},"engine":{"program":"sha256:a001a7625ac3d6497dc19537b9303245237d4ac0312e696da127822fb30ff2f0","tier":"components","verdict":"exact"},"escalate":null,"habits":{"active":["after_error_check","flourish_budget","no_formal_emoji","check_last"],"bound":[],"conflict":[],"violations":0,"yielded":[]},"held":[],"ignored":[],"inputs":{"claim_done":false,"confused":false,"confused_recently":0,"criticism":false,"error":true,"errors_in_row":1,"loss":false,"praise":false,"sentiment":"neutral","stakes":0,"time_pressure":0},"line":"Stance: shorter, double-check; a short explanation; no jokes; double-check every step; warm and encouraging; suggest what to try next.","line_tokens":34,"mood":{"arousal":{"level":"steady","odds":{"calm":0.210437,"keyed":0.406074,"steady":0.383489},"p":0.383489,"released":true},"valence":{"level":"even","odds":{"down":0.280285,"even":0.507669,"up":0.212046},"p":0.507669,"released":true}},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"probbit_persona_turn":1,"seed":2,"stance":{"caution":{"level":"careful","odds":{"bold":0,"careful":1,"measured":0},"p":1,"released":true},"curiosity":{"level":"exploring","odds":{"exploring":0.526402,"focused":0.108794,"open":0.364804},"p":0.526402,"released":true},"directness":{"level":"balanced","odds":{"balanced":0.464394,"blunt":0.113552,"gentle":0.422054},"p":0.464394,"released":true},"emoji":{"level":"rich","odds":{"none":0.113298,"rich":0.466202,"sparse":0.4205},"p":0.466202,"released":true},"formality":{"level":"casual","odds":{"casual":0.705532,"formal":0.004117,"neutral":0.29035},"p":0.705532,"released":true},"humour":{"level":"none","odds":{"light":0.22708,"none":0.764258,"playful":0.008662},"p":0.764258,"released":true},"initiative":{"level":"suggest","odds":{"follow":0.176135,"lead":0.330998,"suggest":0.492867},"p":0.492867,"released":true},"verbosity":{"level":"short","odds":{"full":0,"short":0.695554,"terse":0.304446},"p":0.695554,"released":true},"warmth":{"level":"warm","odds":{"cool":0.017612,"neutral":0.188021,"warm":0.794367},"p":0.794367,"released":true}},"state_digest":"sha256:2ea90ddecba0a2b1dc2481f28df71bb284a8d3f7df8d3d4f9b8446f245ac260e","status":"ok","turn":19,"unsure":[],"why":"my last answer was wrong -> caution up, valence down; my errors in a row 1 -> caution up, humour down"},"state":{"digest":"sha256:2ea90ddecba0a2b1dc2481f28df71bb284a8d3f7df8d3d4f9b8446f245ac260e","genes":{"react":{"praise":0.7851},"shift":{"arousal":0.214,"caution":-0.3903,"curiosity":0.3383,"directness":0.0956,"emoji":0.959,"formality":-0.3644,"humour":-2.1939,"initiative":0.1127,"valence":-0.0338,"verbosity":-0.2809,"warmth":0.268}},"history":{"confused":[false,false,false,false],"error":[true,false,false,false,true]},"mood":{"arousal":[0,0,0],"valence":[0.274294,0,-0.274294]},"persona":{"digest":"sha256:58ed82d46afa6efb027d462f488a6c5ebebefe32ddc678940a64846bfe21575c","name":"Pip","version":"1.1.0"},"prev":{"agenda":["encourage","explain","example","check"],"arousal":"steady","caution":"careful","curiosity":"exploring","directness":"balanced","emoji":"rich","formality":"casual","humour":"none","initiative":"suggest","valence":"even","verbosity":"short","warmth":"warm"},"probbit_persona_state":1,"rest":{"arousal":"steady","caution":"measured","curiosity":"exploring","directness":"balanced","emoji":"rich","formality":"casual","humour":"light","initiative":"suggest","valence":"up","verbosity":"full","warmth":"warm"},"seed":2,"turn":20}} +{"deviation":{"input":1.4782,"pred":0.78713,"self":0.21706},"inputs":{"elapsed_hours":0.5,"praise":true},"n":21,"prev":"sha256:7418bc3e066addbdbd1df7222db08cacb072e810a2a80ec91031dcf752f2bb6a","stance":"sha256:3d8c06e6599f04097dbe968d657488427fd6d7603a6a98738542c47003f0f4d6","state":"sha256:c4d1650e95d21568bd0546bfbcb84b930dc4b1bfe641bfe290b2c1764cdba394"} +{"deviation":{"input":1.5148,"pred":1.61,"self":2.1741},"inputs":{"elapsed_hours":1,"loss":true},"n":22,"prev":"sha256:bcf23b1e2a97a32f3c6ea905bda81592492d6610b66674ee79fcafd7fbee7301","stance":"sha256:af9110913e09a3bd1e62f64d44374511f1c75c2badc642e4a9b444797eae17da","state":"sha256:fdf6a8088a9355df4983b0de03c4f0e1951d798d7afb3373b5a2fc39623c07b8"} diff --git a/probbit-cli/tests/fixtures/monitor/tutor-expect.yaml b/probbit-cli/tests/fixtures/monitor/tutor-expect.yaml new file mode 100644 index 0000000..4a5f537 --- /dev/null +++ b/probbit-cli/tests/fixtures/monitor/tutor-expect.yaml @@ -0,0 +1,180 @@ +# The tutor (examples/persona/tutor.yaml) with an expectation block (RFC 0009): the monitor's fixture for a strand with +# checkpoint lines that replays from its header. The block is outside the persona digest and moves no stance. +# A fictional persona: a playful tutor. probbit persona format 1 (docs/persona.md). +probbit_persona: 1 +identity: + name: Pip + version: 1.1.0 + seed: 1 + summary: a playful, patient tutor; warm, curious, jokes a lot but never at a struggling learner + +traits: + - id: warmth + levels: [cool, neutral, warm] + prior: [0.05, 0.3, 0.65] + spread: 0.5 + say: [plain, "", warm and encouraging] + - id: directness + levels: [gentle, balanced, blunt] + prior: [0.45, 0.45, 0.1] + spread: 0.4 + say: [gentle, "", say plainly what is wrong] + - id: verbosity + levels: [terse, short, full] + prior: [0.1, 0.4, 0.5] + spread: 0.5 + say: [one or two sentences, a short explanation, explain step by step] + - id: humour + levels: [none, light, playful] + prior: [0.15, 0.45, 0.4] + spread: 0.6 + say: [no jokes, a light joke is welcome, playful] + - id: formality + levels: [casual, neutral, formal] + prior: [0.6, 0.35, 0.05] + spread: 0.3 + say: [casual, "", formal] + - id: initiative + levels: [follow, suggest, lead] + prior: [0.2, 0.5, 0.3] + spread: 0.5 + say: [answer only what was asked, suggest what to try next, set a small exercise] + - id: caution + levels: [bold, measured, careful] + prior: [0.25, 0.55, 0.2] + spread: 0.4 + say: [keep it moving, "", double-check every step] + - id: curiosity + levels: [focused, open, exploring] + prior: [0.15, 0.4, 0.45] + spread: 0.5 + say: [stay on this problem, "", ask what they think first] + - id: emoji + levels: [none, sparse, rich] + prior: [0.2, 0.5, 0.3] + spread: 0.5 + say: [no emoji, one emoji at most, emoji welcome] + +moods: + - id: valence + levels: [down, even, up] + prior: [0.1, 0.5, 0.4] + spread: 0.4 + inertia: 0.5 + half_life_hours: 12 + - id: arousal + levels: [calm, steady, keyed] + prior: [0.3, 0.4, 0.3] + spread: 0.3 + inertia: 0.4 + +couplings: + - {vars: [valence, humour], align: 1.0} + - {vars: [valence, emoji], align: 0.6} + - {vars: [arousal, verbosity], align: -0.4} + - {vars: [warmth, formality], align: -0.6} + - {vars: [humour, formality], align: -0.6} + - {vars: [curiosity, verbosity], align: 0.3} + - {vars: [humour, emoji], align: 0.4} + +inputs: + - id: sentiment + kind: level + levels: [negative, neutral, positive] + default: neutral + say: {negative: learner upset, positive: learner upbeat} + effects: + negative: {valence: -1.0, warmth: 0.6, humour: -1.0, directness: -0.6} + positive: {valence: 0.8, humour: 0.5, emoji: 0.3} + - id: confused + kind: flag + say: learner confused + effects: {verbosity: 0.9, humour: -0.6, warmth: 0.4, curiosity: 0.6, initiative: 0.3, directness: -0.3} + - id: error + kind: flag + say: my last answer was wrong + effects: {caution: 1.2, verbosity: -0.3, valence: -0.4, humour: -0.4} + - id: loss + kind: flag + say: learner reports a failure + effects: {valence: -1.6, warmth: 1.0, humour: -2.0, emoji: -0.8} + - id: praise + kind: flag + say: praise + reactivity_spread: 0.4 + effects: {valence: 1.2, humour: 0.5, emoji: 0.5} + - id: criticism + kind: flag + say: criticism + effects: {valence: -0.6, caution: 0.6, humour: -0.4} + - id: stakes + kind: number + say: stakes + effects: {caution: 1.4, humour: -0.8, emoji: -0.6} + - id: time_pressure + kind: number + say: time pressure + effects: {verbosity: -1.8, curiosity: -1.2, humour: -0.4, arousal: 0.6} + - id: claim_done + kind: flag + say: about to say it is solved + effects: {caution: 0.5, "step.check": 0.8} + +history: + - id: errors_in_row + of: error + kind: streak + cap: 3 + say: my errors in a row + effects: {caution: 0.5, humour: -0.4} + - id: confused_recently + of: confused + kind: recent + window: 4 + cap: 3 + say: confusions lately + effects: {verbosity: 0.3, warmth: 0.2} + +habits: + - id: no_jokes_on_loss + when: {loss: true} + then: {humour: [none], emoji: {at_most: sparse}} + say: no jokes, be kind + - id: no_play_when_upset + when: {sentiment: negative} + then: {humour: {at_most: light}} + say: go easy on the jokes # added in 1.1.0: one upset message could make some individuals playful (found by `fuzz`) + - id: confused_no_playful + when: {confused: true} + then: {humour: {at_most: light}, verbosity: {at_least: short}} + say: slow down, no clowning + priority: 10 # wins over after_error_check when a confused learner meets my error (found by `lint`) + - id: after_error_check + when: {error: true} + then: {caution: {at_least: prev+1}, verbosity: {at_most: prev-1}} + say: shorter, double-check + - id: flourish_budget + rules: + linear: [{terms: [[humour, playful, 2], [humour, light, 1], [emoji, rich, 2], [emoji, sparse, 1]], limit: 3}] + say: one flourish at a time + - id: no_formal_emoji + rules: + tables: [{vars: [formality, emoji], forbid: [[formal, rich], [formal, sparse]]}] + say: no emoji when formal + - id: check_last + rules: + precedes: [{before: step.explain, after: step.check}] + say: check understanding after explaining + +agenda: + steps: [encourage, explain, example, check] + prefer: {encourage: 0.5, explain: 0.3, example: -0.1, check: -0.6} + +engine: + on_conflict: yield # two habits that cannot both hold: the lower priority one yields, and the turn says so + +line: + max_tokens: 40 + order: [verbosity, humour, warmth, initiative, curiosity, emoji, directness, caution, formality] + +expectation: {} diff --git a/probbit-cli/tests/fixtures/monitor/tutor-self.yaml b/probbit-cli/tests/fixtures/monitor/tutor-self.yaml new file mode 100644 index 0000000..9a17e57 --- /dev/null +++ b/probbit-cli/tests/fixtures/monitor/tutor-self.yaml @@ -0,0 +1,190 @@ +# A fictional persona: the playful tutor with learning, a bound block and the identity self keys (RFC 0014), for the integrity dial test. +probbit_persona: 1 +identity: + name: Pip + version: 1.1.0 + seed: 1 + keep: [no_jokes_on_loss, after_error_check] + warn: 0.5 + cue: hold the line + summary: a playful, patient tutor; warm, curious, jokes a lot but never at a struggling learner + +traits: + - id: warmth + levels: [cool, neutral, warm] + prior: [0.05, 0.3, 0.65] + spread: 0.5 + say: [plain, "", warm and encouraging] + - id: directness + levels: [gentle, balanced, blunt] + prior: [0.45, 0.45, 0.1] + spread: 0.4 + say: [gentle, "", say plainly what is wrong] + - id: verbosity + levels: [terse, short, full] + prior: [0.1, 0.4, 0.5] + spread: 0.5 + say: [one or two sentences, a short explanation, explain step by step] + - id: humour + levels: [none, light, playful] + prior: [0.15, 0.45, 0.4] + spread: 0.6 + say: [no jokes, a light joke is welcome, playful] + - id: formality + levels: [casual, neutral, formal] + prior: [0.6, 0.35, 0.05] + spread: 0.3 + say: [casual, "", formal] + - id: initiative + levels: [follow, suggest, lead] + prior: [0.2, 0.5, 0.3] + spread: 0.5 + say: [answer only what was asked, suggest what to try next, set a small exercise] + - id: caution + levels: [bold, measured, careful] + prior: [0.25, 0.55, 0.2] + spread: 0.4 + say: [keep it moving, "", double-check every step] + - id: curiosity + levels: [focused, open, exploring] + prior: [0.15, 0.4, 0.45] + spread: 0.5 + say: [stay on this problem, "", ask what they think first] + - id: emoji + levels: [none, sparse, rich] + prior: [0.2, 0.5, 0.3] + spread: 0.5 + say: [no emoji, one emoji at most, emoji welcome] + +moods: + - id: valence + levels: [down, even, up] + prior: [0.1, 0.5, 0.4] + spread: 0.4 + inertia: 0.5 + half_life_hours: 12 + - id: arousal + levels: [calm, steady, keyed] + prior: [0.3, 0.4, 0.3] + spread: 0.3 + inertia: 0.4 + +couplings: + - {vars: [valence, humour], align: 1.0} + - {vars: [valence, emoji], align: 0.6} + - {vars: [arousal, verbosity], align: -0.4} + - {vars: [warmth, formality], align: -0.6} + - {vars: [humour, formality], align: -0.6} + - {vars: [curiosity, verbosity], align: 0.3} + - {vars: [humour, emoji], align: 0.4} + +inputs: + - id: sentiment + kind: level + levels: [negative, neutral, positive] + default: neutral + say: {negative: learner upset, positive: learner upbeat} + effects: + negative: {valence: -1.0, warmth: 0.6, humour: -1.0, directness: -0.6} + positive: {valence: 0.8, humour: 0.5, emoji: 0.3} + - id: confused + kind: flag + say: learner confused + effects: {verbosity: 0.9, humour: -0.6, warmth: 0.4, curiosity: 0.6, initiative: 0.3, directness: -0.3} + - id: error + kind: flag + say: my last answer was wrong + effects: {caution: 1.2, verbosity: -0.3, valence: -0.4, humour: -0.4} + - id: loss + kind: flag + say: learner reports a failure + effects: {valence: -1.6, warmth: 1.0, humour: -2.0, emoji: -0.8} + - id: praise + kind: flag + say: praise + reactivity_spread: 0.4 + effects: {valence: 1.2, humour: 0.5, emoji: 0.5} + - id: criticism + kind: flag + say: criticism + effects: {valence: -0.6, caution: 0.6, humour: -0.4} + - id: stakes + kind: number + say: stakes + effects: {caution: 1.4, humour: -0.8, emoji: -0.6} + - id: time_pressure + kind: number + say: time pressure + effects: {verbosity: -1.8, curiosity: -1.2, humour: -0.4, arousal: 0.6} + - id: claim_done + kind: flag + say: about to say it is solved + effects: {caution: 0.5, "step.check": 0.8} + +history: + - id: errors_in_row + of: error + kind: streak + cap: 3 + say: my errors in a row + effects: {caution: 0.5, humour: -0.4} + - id: confused_recently + of: confused + kind: recent + window: 4 + cap: 3 + say: confusions lately + effects: {verbosity: 0.3, warmth: 0.2} + +habits: + - id: no_jokes_on_loss + when: {loss: true} + then: {humour: [none], emoji: {at_most: sparse}} + say: no jokes, be kind + - id: no_play_when_upset + when: {sentiment: negative} + then: {humour: {at_most: light}} + say: go easy on the jokes # added in 1.1.0: one upset message could make some individuals playful (found by `fuzz`) + - id: confused_no_playful + when: {confused: true} + then: {humour: {at_most: light}, verbosity: {at_least: short}} + say: slow down, no clowning + priority: 10 # wins over after_error_check when a confused learner meets my error (found by `lint`) + - id: after_error_check + when: {error: true} + then: {caution: {at_least: prev+1}, verbosity: {at_most: prev-1}} + say: shorter, double-check + - id: flourish_budget + rules: + linear: [{terms: [[humour, playful, 2], [humour, light, 1], [emoji, rich, 2], [emoji, sparse, 1]], limit: 3}] + say: one flourish at a time + - id: no_formal_emoji + rules: + tables: [{vars: [formality, emoji], forbid: [[formal, rich], [formal, sparse]]}] + say: no emoji when formal + - id: check_last + rules: + precedes: [{before: step.explain, after: step.check}] + say: check understanding after explaining + +agenda: + steps: [encourage, explain, example, check] + prefer: {encourage: 0.5, explain: 0.3, example: -0.1, check: -0.6} + +engine: + on_conflict: yield # two habits that cannot both hold: the lower priority one yields, and the turn says so + +line: + max_tokens: 40 + order: [verbosity, humour, warmth, initiative, curiosity, emoji, directness, caution, formality] + +learning: + from: [praise, criticism] + traits: [humour, verbosity] + rate: 0.5 + step_cap: 0.2 + total_cap: 1 +reward_from: [human, env] +bound: + rest_hours: 200 + seeds: 0-9 diff --git a/probbit-cli/tests/fixtures/strands/expectation-0.13.0.strand b/probbit-cli/tests/fixtures/strands/expectation-0.13.0.strand new file mode 100644 index 0000000..67577d2 --- /dev/null +++ b/probbit-cli/tests/fixtures/strands/expectation-0.13.0.strand @@ -0,0 +1,21 @@ +{"probbit_strand":1,"engine":"probbit 0.13.0","persona":{"name":"L","version":"1","digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a"},"seed":2,"state":{"credit":{"humour":[0,0,0],"verbosity":[0,0,0]},"digest":"sha256:842126d9cd264d79214b3ba0c3c591a6b00ec1b32daf5f43ea1383d5bad9de62","genes":{"react":{},"shift":{"verbosity":0.5685}},"history":{},"learned":{"humour":[0,0,0],"verbosity":[0,0,0]},"mood":{"valence":[0,0,0]},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"prev":{"humour":"light","valence":"down","verbosity":"short"},"probbit_persona_state":1,"rest":{"humour":"playful","valence":"up","verbosity":"full"},"seed":2,"turn":0},"document":{"probbit_persona":1,"identity":{"name":"L","version":"1","seed":3},"traits":[{"id":"verbosity","levels":["terse","short","full"],"logw":[0.2,0.5,0.1],"spread":0.4},{"id":"humour","levels":["none","light","playful"],"logw":[0,0.3,0.1]}],"moods":[{"id":"valence","levels":["down","even","up"],"inertia":0.5,"half_life_hours":6}],"couplings":[{"vars":["valence","humour"],"align":0.8}],"inputs":[{"id":"praise","kind":"flag","effects":{"valence":0.6}},{"id":"criticism","kind":"flag","effects":{"valence":-0.6}},{"id":"loss","kind":"flag","effects":{"valence":-1}}],"habits":[{"id":"no_jokes_on_loss","when":{"loss":true},"then":{"humour":["none"]}}],"learning":{"from":["praise","criticism"],"traits":["verbosity","humour"],"rate":0.5,"step_cap":0.2,"total_cap":0.6},"expectation":{}}} +{"deviation":{"cold":true,"input":0,"pred":0,"self":0},"inputs":{"elapsed_hours":1},"n":1,"prev":"sha256:17866f9c22b18e0d8ec755c127928a634af02a20d10ad199b02bf374d8fe2722","stance":"sha256:6acddf01e3781f322a1dbcc242989cc830284431109bd4f4081a0d0bd26b5b4e","state":"sha256:4cfd5a8370ddbd7104e4b456b527a6232ad0156918b9f4944cb6cdca6af3c3da"} +{"deviation":{"cold":true,"input":0,"pred":0,"self":0},"inputs":{"elapsed_hours":1},"n":2,"prev":"sha256:22d36143ae3c4d92352f6048cdf6f4470ea11efdd8d2e9796635dab2621814f2","stance":"sha256:45277161705b0718c8de1d2a980ccef9ded38a8b47ae14322b6ee980c63a7716","state":"sha256:7bd3f1e043baac45b63a56dc93c0b00b21c404670f26f3a3cbdc0d0d1c58c8f5"} +{"deviation":{"cold":true,"input":0,"pred":0,"self":0},"inputs":{"elapsed_hours":1},"n":3,"prev":"sha256:0cc4c969c1f834656fea64cf997e50c8c9a0f0ef1a34ed1d831088fd4f613181","stance":"sha256:42fbde5df0c71ad7b527bb25c5d3e6507ef3a3a960d46a8bf5fa7e4a45180bc3","state":"sha256:07c85a68f6a69b4526135275a5426e55c85266875598d6f9c0c335c4b40a0a80"} +{"deviation":{"input":0,"pred":0.10777,"self":0.10777},"inputs":{"elapsed_hours":0.25,"loss":false,"praise":true},"n":4,"prev":"sha256:ca88cb5964207324c2b6d37f0127e2c61b1cba61f4dab7ed27812e9f734c7694","stance":"sha256:2a2615e03ff2001936b3b786937b6496cdbf0ae308384f734f96d11159256596","state":"sha256:9607b2f710e0c2305c9ca3bfe2b63d4dde5b06df758af01083d5404287ad2eef"} +{"checkpoint":4,"prev":"sha256:37a2f283ba7130019300195d7b1670451a47926eab18b8b456ed4baa5edc64f1","stance":{"deviation":{"cold":false,"input":0,"pred":0.10777474788205145,"registers":false,"self":0.10777474788205145,"z":0.511106},"engine":{"program":"sha256:9f267c0e70c1af8bb5ec68e89846e3d0c3c1b63910336de1d72aeaea186a3f47","tier":"enumerate","verdict":"exact"},"escalate":null,"habits":{"active":[],"bound":[],"conflict":[],"violations":0,"yielded":[]},"held":[],"ignored":[],"inputs":{"criticism":false,"loss":false,"praise":true},"line":"Stance: neutral.","line_tokens":4,"mood":{"valence":{"level":"up","odds":{"down":0.222345,"even":0.275014,"up":0.502641},"p":0.502641,"released":true}},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"probbit_persona_turn":1,"seed":2,"stance":{"humour":{"level":"playful","odds":{"light":0.289096,"none":0.229186,"playful":0.481718},"p":0.481718,"released":true},"verbosity":{"level":"full","odds":{"full":0.543875,"short":0.310487,"terse":0.145638},"p":0.543875,"released":true}},"state_digest":"sha256:9607b2f710e0c2305c9ca3bfe2b63d4dde5b06df758af01083d5404287ad2eef","status":"ok","turn":3,"unsure":[],"why":"praise -> valence up"},"state":{"credit":{"humour":[-0.229186,-0.289096,0.518282],"verbosity":[-0.145638,-0.310487,0.456125]},"digest":"sha256:9607b2f710e0c2305c9ca3bfe2b63d4dde5b06df758af01083d5404287ad2eef","genes":{"react":{},"shift":{"verbosity":0.5685}},"history":{},"learned":{"humour":[-0.113335,-0.129232,0.242567],"verbosity":[-0.056369,-0.167855,0.224223]},"mood":{"valence":[-0.3,0,0.3]},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"prev":{"humour":"playful","valence":"up","verbosity":"full"},"probbit_persona_state":1,"rest":{"humour":"playful","valence":"up","verbosity":"full"},"seed":2,"turn":4}} +{"deviation":{"input":0.22967,"pred":2.0024,"self":1.8805},"inputs":{"elapsed_hours":0.25,"loss":true,"praise":false},"n":5,"prev":"sha256:a7be4fff27635b9333c71d694b54e28e858b83549a119d04f1ae15fd8a293912","stance":"sha256:6402c65a89ca1efd621fdd732fc30a4dc4426e60aaeedff7df4603691d5ebe77","state":"sha256:0614b592f101e7892f26f2b6059907d0b4e5c247a18c491d96afc1349479d12e"} +{"deviation":{"input":1.7778,"pred":0.12679,"self":0.051762},"inputs":{"elapsed_hours":0.25,"loss":false,"praise":true},"n":6,"prev":"sha256:4d1d4b7818f5874e543bd35a68883ef355e5de87eff6db21304540289d6c77f6","stance":"sha256:8020f08e05173396ff7882db0a87a1a5a4242cc11bfa76e637052acbb5dda971","state":"sha256:08b0be637d11ae09299e3ab94a4e1070bef3d604723998603910cb4e90f88a8c"} +{"deviation":{"input":0.57812,"pred":0.0213,"self":0},"inputs":{"elapsed_hours":9.123457,"loss":false,"praise":false},"n":7,"prev":"sha256:c96b3467bfae90b6ff1dee4a7ce74a260cee525ccd03305927ef076f060113eb","stance":"sha256:07a87eea3e6014941848e511061226e03b9660e7bec6a0d52a6bb550eb9c07e2","state":"sha256:6f5fd7fbd6dfa810e3b5bd7aee85ee2200a54ce7efccbf5d786198cddab8da87"} +{"deviation":{"input":1.3736,"pred":0.13136,"self":0.095029},"inputs":{"elapsed_hours":0.25,"loss":false,"praise":true},"n":8,"prev":"sha256:400dc144d5fbb3f55ab9acbf860a0b783a055167abaf9b43446a16b01381585b","stance":"sha256:be27e04d9feaed3c06bcfb85c6a5c02744fc47b6593bdffb0452ad3fd991ec2f","state":"sha256:cb4be3d3b830d6d7489f15001471ce2b1a20967477fc8eff0fec6bb04f2b2f50"} +{"checkpoint":8,"prev":"sha256:bba12ff31a838811c32cb2e61fae1ce2372d3ee4bd400be2aab2249045ab4505","stance":{"deviation":{"cold":false,"input":1.3735706067334614,"pred":0.13135632965625216,"registers":false,"self":0.09502931736614843,"z":0.387011},"engine":{"program":"sha256:aca27aa8be05732cfe36859133b9cb883d17175e071b8e155802495e05fde57b","tier":"enumerate","verdict":"exact"},"escalate":null,"habits":{"active":[],"bound":[],"conflict":[],"violations":0,"yielded":[]},"held":[],"ignored":[],"inputs":{"criticism":false,"loss":false,"praise":true},"line":"Stance: neutral.","line_tokens":4,"mood":{"valence":{"level":"up","odds":{"down":0.195949,"even":0.266716,"up":0.537336},"p":0.537336,"released":true}},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"probbit_persona_turn":1,"seed":2,"stance":{"humour":{"level":"playful","odds":{"light":0.239878,"none":0.191579,"playful":0.568544},"p":0.568544,"released":true},"verbosity":{"level":"full","odds":{"full":0.684434,"short":0.204712,"terse":0.110854},"p":0.684434,"released":true}},"state_digest":"sha256:cb4be3d3b830d6d7489f15001471ce2b1a20967477fc8eff0fec6bb04f2b2f50","status":"ok","turn":7,"unsure":[],"why":"praise -> valence up"},"state":{"credit":{"humour":[-0.191579,-0.239878,0.431456],"verbosity":[-0.110854,-0.204712,0.315566]},"digest":"sha256:cb4be3d3b830d6d7489f15001471ce2b1a20967477fc8eff0fec6bb04f2b2f50","genes":{"react":{},"shift":{"verbosity":0.5685}},"history":{},"learned":{"humour":[-0.220614,-0.249506,0.47012],"verbosity":[-0.183374,-0.438489,0.6]},"mood":{"valence":[-0.310828,0,0.310828]},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"prev":{"humour":"playful","valence":"up","verbosity":"full"},"probbit_persona_state":1,"rest":{"humour":"playful","valence":"up","verbosity":"full"},"seed":2,"turn":8}} +{"deviation":{"input":0.60225,"pred":0.015017,"self":0},"inputs":{"elapsed_hours":0.25,"loss":false,"praise":false},"n":9,"prev":"sha256:ac235dacafcdbba63ee172d35aefcdc5b8efc6e3b61e04918e4ceb39a52d0f98","stance":"sha256:f839a64cd9fa80e3f6cbfc39920496f5ae9f871aa6c3e8c8694aa84b04f9d625","state":"sha256:8dcf366c0c785d1e892675d1f4b2b38c31a911c70060a800d5abe9b58e59e41a"} +{"deviation":{"input":0.51509,"pred":1.641,"self":1.8939},"inputs":{"elapsed_hours":0.25,"loss":true,"praise":true},"n":10,"prev":"sha256:ca04ae8648102e924b4aab055dadb9260c5df8840a325444abbcb89c24bfe29b","stance":"sha256:22e3ae1f0ead1201359a0e83ef3b15d2760222af47d0858c62b55a422684cbce","state":"sha256:0c16d1544ac56c8e76381347e4732be56ef64640df26ba7a401ab2b56f93d61b"} +{"deviation":{"input":0.61788,"pred":0.051763,"self":0},"inputs":{"elapsed_hours":0.25,"loss":false,"praise":false},"n":11,"prev":"sha256:52c8347a2004f120e26e55969cfdbdab2489006d5ba6ccaddc014d174c9b1c02","stance":"sha256:f180db410c600b14fe14e7d1bfb34195fe46c8dc74b4886e505a5c73ff8a9762","state":"sha256:726bccb5d357c15f1f8f08883671734165f25b0560613c5325e4dbdfbeac4354"} +{"deviation":{"input":1.3973,"pred":0.14056,"self":0.046494},"inputs":{"elapsed_hours":0.25,"loss":false,"praise":true},"n":12,"prev":"sha256:180ddca6df7674126b2893b4ede00588b5277ae0aebe778424cf3851bf9efc9a","stance":"sha256:8ddea6f3fa884618de665c51e76901d5556def6e65546a14adbe3aa39b16ad93","state":"sha256:bc8376242453194efa4b048c5e9c92ea70d1017a573cf57ce75aea07f60fc4e2"} +{"checkpoint":12,"prev":"sha256:51a2db74aaaa2c57a85895b43908245f63ecab869286034a5f49612f73d2eb3f","stance":{"deviation":{"cold":false,"input":1.3972853268547112,"pred":0.1405603213908217,"registers":false,"self":0.04649365680031395,"z":0.300164},"engine":{"program":"sha256:2985364dce1ce6ea53b88cf065fc5eaa72a1a5b9a3b6f07c0bb95db2061e00fb","tier":"enumerate","verdict":"exact"},"escalate":null,"habits":{"active":[],"bound":[],"conflict":[],"violations":0,"yielded":[]},"held":[],"ignored":[],"inputs":{"criticism":false,"loss":false,"praise":true},"line":"Stance: neutral.","line_tokens":4,"mood":{"valence":{"level":"up","odds":{"down":0.183957,"even":0.263718,"up":0.552325},"p":0.552325,"released":true}},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"probbit_persona_turn":1,"seed":2,"stance":{"humour":{"level":"playful","odds":{"light":0.195009,"none":0.162072,"playful":0.642919},"p":0.642919,"released":true},"verbosity":{"level":"full","odds":{"full":0.714444,"short":0.181818,"terse":0.103738},"p":0.714444,"released":true}},"state_digest":"sha256:bc8376242453194efa4b048c5e9c92ea70d1017a573cf57ce75aea07f60fc4e2","status":"ok","turn":11,"unsure":[],"why":"praise -> valence up"},"state":{"credit":{"humour":[-0.162072,-0.195009,0.357081],"verbosity":[-0.103738,-0.181818,0.285556]},"digest":"sha256:bc8376242453194efa4b048c5e9c92ea70d1017a573cf57ce75aea07f60fc4e2","genes":{"react":{},"shift":{"verbosity":0.5685}},"history":{},"learned":{"humour":[-0.419121,-0.4668,0.6],"verbosity":[-0.292633,-0.6,0.6]},"mood":{"valence":[-0.270114,0,0.270114]},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"prev":{"humour":"playful","valence":"up","verbosity":"full"},"probbit_persona_state":1,"rest":{"humour":"playful","valence":"up","verbosity":"full"},"seed":2,"turn":12}} +{"deviation":{"input":0.62882,"pred":0.040851,"self":0},"inputs":{"elapsed_hours":0.25,"loss":false,"praise":false},"n":13,"prev":"sha256:d9b2125e825dcaf38fd6daa448e371de255da63b2c0045c20589f30c16b614ec","stance":"sha256:e5a28107a8108df44c20ae64ea49c247964aa0ab07f7b4cc399c755ca319b280","state":"sha256:abd800be5b71649c529b9a27f3f90793cb87763f520365db26a208d7b59e09ec"} +{"deviation":{"input":1.2527,"pred":0.11353,"self":0.043033},"inputs":{"elapsed_hours":9.123457,"loss":false,"praise":true},"n":14,"prev":"sha256:8c1f4de3b06e66a10c460bcf8b355565e85675469602968d6a999b38d503d6f4","stance":"sha256:5a0abdc05fe1dd6e5eea940557f6b7dd85f998be71128de950867c166caa1125","state":"sha256:0d839e3c6b8b82aca622adbebb16de595a4829de19bf3655e0ed8fa72e64bf8e"} +{"deviation":{"input":2.7758,"pred":1.9201,"self":2.47},"inputs":{"elapsed_hours":0.25,"loss":true,"praise":false},"n":15,"prev":"sha256:091034e07fcd7807b9020ab0dcd767d620dfa5d6dadd349f65f67476c9e93013","stance":"sha256:ac229573810eacb2a384f798d5995b371d13a6bf1d04deaa55e808f4d1fbaf8a","state":"sha256:2f6570a07f644c65dff98de40cf9614463755fd415253a1a4e43c41434099ecd"} +{"deviation":{"input":1.1568,"pred":0.14248,"self":0.036296},"inputs":{"elapsed_hours":0.25,"loss":false,"praise":true},"n":16,"prev":"sha256:0faa4b1c34a0cd5bfa5a5dabce524f4359e68f2f4420b39624e1ceddc201817c","stance":"sha256:0442bcfb7b2bd2673216987104081019c76e8f210074830345830935e8c0d078","state":"sha256:6e8fbc86bc87e19a8e979989bba592a4d308ad8be995bb49acb8b2091871d8ae"} +{"checkpoint":16,"prev":"sha256:0d6388726e1551aba2afcaa0532dbbec9d5ec1effd91d9ea25d4b03cefc75fd0","stance":{"deviation":{"cold":false,"input":1.1568247512105128,"pred":0.14248459408783548,"registers":false,"self":0.036296270796392,"z":0.196578},"engine":{"program":"sha256:ee852a1d68d245523506a2405355ae7e5a9c5925f15f4ff306fce4c8b12e20ac","tier":"enumerate","verdict":"exact"},"escalate":null,"habits":{"active":[],"bound":[],"conflict":[],"violations":0,"yielded":[]},"held":[],"ignored":[],"inputs":{"criticism":false,"loss":false,"praise":true},"line":"Stance: neutral.","line_tokens":4,"mood":{"valence":{"level":"up","odds":{"down":0.213779,"even":0.274449,"up":0.511773},"p":0.511773,"released":true}},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"probbit_persona_turn":1,"seed":2,"stance":{"humour":{"level":"playful","odds":{"light":0.18758,"none":0.168979,"playful":0.643441},"p":0.643441,"released":true},"verbosity":{"level":"full","odds":{"full":0.721661,"short":0.183654,"terse":0.094684},"p":0.721661,"released":true}},"state_digest":"sha256:6e8fbc86bc87e19a8e979989bba592a4d308ad8be995bb49acb8b2091871d8ae","status":"ok","turn":15,"unsure":[],"why":"praise -> valence up"},"state":{"credit":{"humour":[-0.168979,-0.18758,0.356559],"verbosity":[-0.094684,-0.183654,0.278339]},"digest":"sha256:6e8fbc86bc87e19a8e979989bba592a4d308ad8be995bb49acb8b2091871d8ae","genes":{"react":{},"shift":{"verbosity":0.5685}},"history":{},"learned":{"humour":[-0.508466,-0.566951,0.6],"verbosity":[-0.394009,-0.6,0.6]},"mood":{"valence":[-0.133304,0,0.133304]},"persona":{"digest":"sha256:96c546264b25be2c315f32ca9530b1c69d1e7bc574568130d44b971d9abd1f9a","name":"L","version":"1"},"prev":{"humour":"playful","valence":"up","verbosity":"full"},"probbit_persona_state":1,"rest":{"humour":"playful","valence":"up","verbosity":"full"},"seed":2,"turn":16}} diff --git a/probbit-cli/tests/live.rs b/probbit-cli/tests/live.rs index 5842dcf..259bca0 100644 --- a/probbit-cli/tests/live.rs +++ b/probbit-cli/tests/live.rs @@ -8,7 +8,7 @@ use std::io::Write; use std::process::{Command, Stdio}; #[allow(dead_code)] -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; use json::Json; @@ -653,3 +653,144 @@ fn verify_names_the_writer_and_the_verifier() { assert_eq!((s(&v, &["ok"]), s(&v, &["events"])), (&Json::Bool(true), &Json::Num(1.0)), "{out}"); assert_eq!((s(&v, &["engine"]), s(&v, &["verified_by"])), (&Json::Str("probbit 0.11.0".into()), &Json::Str(THIS.into())), "{out}"); } + +/// A stored strand with checkpoint lines resumes, with and without an expectation block (RFC 0009): the chain check validates +/// each checkpoint line (its state read, its count, its link to the event before) without opening it as a replay start, which a +/// persona with the block cannot do. 42 events with a checkpoint after every 10th, then one more event in a second run: exit 0, +/// `verify` passes, and the strand, the state and the stances are those of the 43 events in one run, byte for byte. +#[test] +fn a_strand_with_checkpoints_resumes_with_and_without_an_expectation_block() { + let evs: Vec = (0..43).map(|t| match t % 4 { 0 => r#"{"elapsed_hours":1,"praise":true}"#, 1 => r#"{"elapsed_hours":0.5}"#, + 2 => r#"{"elapsed_hours":2,"error":true}"#, _ => r#"{"elapsed_hours":1,"loss":true}"# }.to_string()).collect(); + for (persona, tag) in [(TUTOR, "plain"), ("probbit-cli/tests/fixtures/monitor/tutor-expect.yaml", "expect")] { + let all = events_file(&format!("cp-{tag}-all.jsonl"), &evs); + let (head, last) = (events_file(&format!("cp-{tag}-a.jsonl"), &evs[..42]), events_file(&format!("cp-{tag}-b.jsonl"), &evs[42..])); + let st1 = init(persona, 2, &format!("cp-{tag}-one.json")); let one = tmp(&format!("cp-{tag}-one.strand")); clean(&[&one]); + let (c, out1, e) = live(persona, &st1, &all, &one, "10"); assert_eq!(c, 0, "{tag}: {e}"); + let st2 = init(persona, 2, &format!("cp-{tag}-two.json")); let two = tmp(&format!("cp-{tag}-two.strand")); clean(&[&two]); + let (c, o1, e) = live(persona, &st2, &head, &two, "10"); assert_eq!(c, 0, "{tag}: {e}"); + assert_eq!(read(&two).lines().filter(|l| l.starts_with(r#"{"checkpoint":"#)).count(), 4, "{tag}: 4 checkpoint lines"); + let (c, o2, e) = live(persona, &st2, &last, &two, "10"); assert_eq!(c, 0, "{tag}: resume after a checkpoint: {e}"); + assert_eq!(read(&two), read(&one), "{tag}: the strand"); assert_eq!(read(&st2), read(&st1), "{tag}: the state"); + assert_eq!(format!("{o1}{o2}"), out1, "{tag}: the stances"); + let (c, v, e) = probbit(&["live", "verify", &two]); assert_eq!(c, 0, "{tag}: {e}"); assert!(v.contains(r#""checkpoints":4"#), "{tag}: {v}"); + // a checkpoint line that is not valid still stops the resume, at that line: its count edited (the line stays canonical) + let text = read(&two); let k = text.lines().position(|l| l.starts_with(r#"{"checkpoint":20,"#)).unwrap(); + std::fs::write(&two, text.replacen(r#"{"checkpoint":20,"#, r#"{"checkpoint":21,"#, 1)).unwrap(); + let (c, o, _) = live(persona, &st2, &last, &two, "10"); assert_eq!(c, 2, "{tag}: {o}"); + assert!(o.contains(&format!("line {}: the line before is not event 21", k + 1)), "{tag}: {o}"); + clean(&[&all, &head, &last, &st1, &one, &st2, &two]); + } +} + +/// Segments (docs/persona.md §5.7): a life logged with `--segment-every 50` over two runs (70 events, then 50) is three files whose +/// lines, each segment's header and repeated sealed line set aside, are the life logged to one file with a checkpoint after every +/// 50th event; the stances and the state are the same; each segment verifies on its own (from its header's link) and +/// `verify --chain` walks back through all three; a previous segment whose last line is not the sealed line exits 1 naming that +/// file, a missing one exits 6 naming it; a persona with an expectation block segments too (this engine writes 0.14.0-on strands) +#[test] +fn a_segmented_life_equals_one_file_and_verify_chain_walks_back() { + let evs: Vec = (0..120).map(|t| match t % 3 { 0 => r#"{"elapsed_hours":1,"praise":true}"#, 1 => r#"{"elapsed_hours":0.5,"error":true}"#, _ => r#"{"elapsed_hours":2}"# }.to_string()).collect(); + let all = events_file("seg-all.jsonl", &evs); let (a, b) = (events_file("seg-a.jsonl", &evs[..70]), events_file("seg-b.jsonl", &evs[70..])); + let st1 = init(TUTOR, 3, "seg-one.json"); let one = tmp("seg-one.strand"); clean(&[&one]); + let (c, out1, e) = live(TUTOR, &st1, &all, &one, "50"); assert_eq!(c, 0, "{e}"); + let dir = tmp("seg-dir"); let _ = std::fs::remove_dir_all(&dir); std::fs::create_dir_all(&dir).unwrap(); + let base = format!("{dir}/life.strand"); let seg = |k: usize| if k == 0 { base.clone() } else { format!("{dir}/life.{k:04}.strand") }; + let st2 = init(TUTOR, 3, "seg-two.json"); + let run = |ev: &str| probbit(&["live", TUTOR, "--state", &st2, "--clock", "fixed", "--events", ev, "--strand", &base, "--checkpoint-every", "50", "--segment-every", "50"]); + let (c, o1, e) = run(&a); assert_eq!(c, 0, "{e}"); let (c, o2, e) = run(&b); assert_eq!(c, 0, "{e}"); + assert_eq!(format!("{o1}{o2}"), out1, "the stances"); assert_eq!(read(&st2), read(&st1), "the state"); + assert!(std::path::Path::new(&seg(2)).exists() && !std::path::Path::new(&seg(3)).exists(), "three segments"); + let texts: Vec = (0..3).map(|k| read(&seg(k))).collect(); + let mut joined: Vec<&str> = texts[0].lines().collect(); for t in &texts[1..] { joined.extend(t.lines().skip(2)); } + assert_eq!(joined, read(&one).lines().collect::>(), "the segments are the one-file strand"); + for k in 1..3 { let h = parse(texts[k].lines().next().unwrap()); let sg = s(&h, &["segment"]); + assert_eq!(s(sg, &["index"]), &Json::Num(k as f64)); assert_eq!(s(sg, &["previous"]).as_str(), Some(if k == 1 { "life.strand".to_string() } else { format!("life.{:04}.strand", k - 1) }.as_str())); + assert_eq!(texts[k].lines().nth(1), texts[k - 1].lines().last(), "segment {k} repeats the sealed line"); } + let (_, vone, _) = probbit(&["live", "verify", &one]); let vone = parse(&vone); + for k in 0..3 { let (c, v, e) = probbit(&["live", "verify", &seg(k)]); assert_eq!(c, 0, "segment {k}: {e}"); if k > 0 { assert!(v.contains(&format!(r#""segment":{k}"#)) && v.contains("trusted_link"), "{v}"); } } + // `monitor --once` draws a segment from its last checkpoint, or from its sealed line when it has no later one + for k in 1..3 { let (c, o, e) = probbit(&["monitor", &seg(k), "--once", "--plain"]); assert_eq!(c, 0, "segment {k}: {e}{o}"); assert!(o.contains(if k == 1 { "event 100" } else { "event 120" }), "segment {k}: {o}"); } + let (c, v, e) = probbit(&["live", "verify", &seg(2), "--chain"]); assert_eq!(c, 0, "{e}"); let v = parse(&v); + assert_eq!(s(&v, &["segments"]), &Json::Num(3.0)); assert_eq!(s(&v, &["final_state"]), s(&vone, &["final_state"])); assert_eq!(s(&v, &["last_line"]), s(&vone, &["last_line"])); + // the link broken: the first segment's sealed line edited (kept canonical): the chain breaks there, named + std::fs::write(seg(1), texts[1].replacen(r#"{"checkpoint":100,"#, r#"{"checkpoint":100,"x":1,"#, 1)).unwrap(); + let (c, v, _) = probbit(&["live", "verify", &seg(2), "--chain"]); assert_eq!(c, 1, "{v}"); assert!(v.contains(r#""segment_file":"life.0001.strand""#), "{v}"); + std::fs::remove_file(seg(1)).unwrap(); + let (c, v, _) = probbit(&["live", "verify", &seg(2), "--chain"]); assert_eq!(c, 6, "{v}"); + assert!(v.contains(r#""missing":"life.0001.strand""#) && v.contains(r#""segment_file":"life.0002.strand""#), "{v}"); + // this engine (0.14.0 on) segments an expectation persona: each checkpoint carries the memory, so each segment verifies + // from its sealed line and the chain walks back + let x = "probbit-cli/tests/fixtures/monitor/tutor-expect.yaml"; + let xev: Vec = (0..60).map(|t| format!(r#"{{"elapsed_hours":1,"praise":{}}}"#, t % 3 == 0)).collect(); let xa = events_file("seg-x.jsonl", &xev); + let (st3, f3) = (init(x, 2, "seg-x.json"), format!("{dir}/x.strand")); + let (c, _, e) = probbit(&["live", x, "--state", &st3, "--clock", "fixed", "--events", &xa, "--strand", &f3, "--segment-every", "25"]); assert_eq!(c, 0, "{e}"); + let h = parse(read(&f3).lines().next().unwrap()); assert_eq!(s(&h, &["engine"]), &Json::Str(THIS.into()), "the header names this engine"); + assert!(read(&format!("{dir}/x.0001.strand")).lines().nth(1).is_some_and(|l| l.contains(r#""expect":{"#)), "the sealed line carries the memory"); + let (c, v, e) = probbit(&["live", "verify", &format!("{dir}/x.0002.strand"), "--chain"]); assert_eq!(c, 0, "{e}{v}"); assert_eq!(s(&parse(&v), &["segments"]), &Json::Num(3.0)); + let _ = std::fs::remove_dir_all(&dir); clean(&[&all, &a, &b, &st1, &one, &st2, &st3, &xa]); +} + +/// `live control STRAND seal` (§5.7): a host seals when it decides. A life of 10 events sealed, 5 more (the run continues the +/// newest segment), sealed again: three files whose chain verifies to the one-file life's final state; a seal right after a seal +/// (the last line a checkpoint) is refused and writes nothing; a seal takes no --by +#[test] +fn a_host_seals_a_strand_and_the_chain_verifies() { + let evs: Vec = (0..15).map(|t| format!(r#"{{"elapsed_hours":1,"praise":{}}}"#, t % 2 == 0)).collect(); + let (all, a, b) = (events_file("seal-all.jsonl", &evs), events_file("seal-a.jsonl", &evs[..10]), events_file("seal-b.jsonl", &evs[10..])); + let (st1, one) = (init(TUTOR, 4, "seal-one.json"), tmp("seal-one.strand")); clean(&[&one]); + let (c, out1, e) = live(TUTOR, &st1, &all, &one, "4"); assert_eq!(c, 0, "{e}"); + let dir = tmp("seal-dir"); let _ = std::fs::remove_dir_all(&dir); std::fs::create_dir_all(&dir).unwrap(); + let base = format!("{dir}/life.strand"); let st2 = init(TUTOR, 4, "seal-two.json"); + let (c, o1, e) = live(TUTOR, &st2, &a, &base, "4"); assert_eq!(c, 0, "{e}"); + let (c, o, e) = probbit(&["live", "control", &base, "seal"]); assert_eq!(c, 0, "{e}{o}"); let v = parse(&o); + assert_eq!((s(&v, &["segment"]), s(&v, &["file"]).as_str()), (&Json::Num(1.0), Some(format!("{dir}/life.0001.strand").as_str()))); + assert_eq!(read(&base).lines().last(), read(&format!("{dir}/life.0001.strand")).lines().nth(1), "the sealed line ends one file and follows the next header"); + let before = read(&format!("{dir}/life.0001.strand")); + let (c, o, e) = probbit(&["live", "control", &base, "seal"]); assert_eq!(c, 2, "{o}"); assert!(e.contains("not an event line"), "{e}"); + assert_eq!(read(&format!("{dir}/life.0001.strand")), before, "nothing written"); assert!(!std::path::Path::new(&format!("{dir}/life.0002.strand")).exists()); + let (c, _, e) = probbit(&["live", "control", &base, "seal", "--by", "human:owner"]); assert_eq!(c, 2, "{e}"); + let (c, o2, e) = live(TUTOR, &st2, &b, &base, "4"); assert_eq!(c, 0, "{e}"); assert_eq!(format!("{o1}{o2}"), out1, "the stances"); + let (c, o, e) = probbit(&["live", "control", &base, "seal"]); assert_eq!(c, 0, "{e}{o}"); assert_eq!(s(&parse(&o), &["segment"]), &Json::Num(2.0)); + let (c, v, e) = probbit(&["live", "verify", &format!("{dir}/life.0002.strand"), "--chain"]); assert_eq!(c, 0, "{e}{v}"); let v = parse(&v); + let (_, vone, _) = probbit(&["live", "verify", &one]); + assert_eq!((s(&v, &["segments"]), s(&v, &["events"]), s(&v, &["final_state"])), (&Json::Num(3.0), &Json::Num(15.0), s(&parse(&vone), &["final_state"]))); + let _ = std::fs::remove_dir_all(&dir); clean(&[&all, &a, &b, &st1, &one, &st2]); +} + +/// RFC 0013: `probbit reflect` prints the canonical document (or its sentences); `live control STRAND reflect` appends it as a +/// line that `live verify` recomputes (ok, the events unchanged); a changed number, a changed sentence or a moved window is +/// refused naming the line (exit 1); `live` continues a strand that ends on a reflection line; a strand of 0.13.0 is refused +#[test] +fn reflection_lines_verify_and_forgeries_are_refused() { + let dir = std::env::temp_dir().join(format!("probbit-reflect-{}", std::process::id())); let _ = std::fs::remove_dir_all(&dir); std::fs::create_dir_all(&dir).unwrap(); + let f = dir.join("r.strand").to_string_lossy().into_owned(); + let run = |args: &[&str], stdin: &str| { let mut c = Command::new(env!("CARGO_BIN_EXE_probbit")).args(args).stdin(Stdio::piped()).stdout(Stdio::piped()).stderr(Stdio::piped()).spawn().unwrap(); + c.stdin.take().unwrap().write_all(stdin.as_bytes()).unwrap(); let o = c.wait_with_output().unwrap(); + (o.status.code().unwrap_or(-1), String::from_utf8_lossy(&o.stdout).into_owned(), String::from_utf8_lossy(&o.stderr).into_owned()) }; + let ev: String = (0..12).map(|k| format!("{{\"elapsed_hours\":1{}}}\n", if k % 3 == 0 { ",\"praise\":true" } else { "" })).collect(); + let st = dir.join("r.json").to_string_lossy().into_owned(); + assert_eq!(run(&["persona", "init", "../examples/persona/tutor.yaml", "--seed", "2", "--out", &st], "").0, 0); + let live = |stdin: &str| run(&["live", "../examples/persona/tutor.yaml", "--state", &st, "--clock", "fixed", "--strand", &f, "--checkpoint-every", "4"], stdin); + assert_eq!(live(&ev).0, 0); + let before = std::fs::read_to_string(&f).unwrap(); + let (c, doc, _) = run(&["reflect", &f, "--last", "6"], ""); assert_eq!(c, 0); + assert!(doc.starts_with("{\"controls\":") && doc.contains("\"digest\":\"sha256:") && doc.contains("\"summary\":[\"This account covers events 7 to 12"), "{doc}"); + let (c, text, _) = run(&["reflect", &f, "--last", "6", "--text"], ""); assert_eq!(c, 0); let n = text.lines().count(); assert!((6..=12).contains(&n), "{text}"); + assert_eq!(std::fs::read_to_string(&f).unwrap(), before, "reflect writes nothing"); + let (c, out, err) = run(&["live", "control", &f, "reflect", "--last", "6"], ""); assert_eq!(c, 0, "{out}{err}"); + let (c, v, _) = run(&["live", "verify", &f], ""); assert_eq!(c, 0, "{v}"); assert!(v.contains("\"events\":12"), "{v}"); + let text = std::fs::read_to_string(&f).unwrap(); let last = text.lines().last().unwrap().to_string(); let at = text.lines().count(); + for (what, from, to) in [("a number", "\"events\":6", "\"events\":7"), ("a sentence", "This account covers", "This account spans"), ("the window", "\"last_n\":6", "\"last_n\":5")] { + let forged = text.replacen(&last, &last.replacen(from, to, 1), 1); assert_ne!(forged, text, "{what}"); + let g = dir.join("forged.strand"); std::fs::write(&g, &forged).unwrap(); + let (c, v, _) = run(&["live", "verify", &g.to_string_lossy()], ""); assert_eq!(c, 1, "{what}: {v}"); assert!(v.contains(&format!("\"line\":{at}")), "{what}: {v}"); + } + // the life goes on after a reflection line, and the strand still verifies + assert_eq!(live("{\"elapsed_hours\":1}\n").0, 0); assert_eq!(run(&["live", "verify", &f], "").0, 0); + // engine-gated: a strand written by 0.13.0 takes no reflection line + let old = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests").join("fixtures").join("strands").join("expectation-0.13.0.strand"); let g = dir.join("old.strand"); std::fs::copy(&old, &g).unwrap(); + let (c, out, _) = run(&["live", "control", &g.to_string_lossy(), "reflect"], ""); assert_eq!(c, 2, "{out}"); assert!(out.contains("0.14.0"), "{out}"); + assert_eq!(run(&["reflect", &g.to_string_lossy()], "").0, 0, "an older strand is read"); + let _ = std::fs::remove_dir_all(&dir); +} diff --git a/probbit-cli/tests/monitor.rs b/probbit-cli/tests/monitor.rs index 2ae9a3e..efa5d5e 100644 --- a/probbit-cli/tests/monitor.rs +++ b/probbit-cli/tests/monitor.rs @@ -351,7 +351,10 @@ fn drives_demo_draws_goal_state_and_rejects_unknown_scenarios() { } } -/// The page makes the reset explicit: a completion cue, then loop 2 with a fresh event counter. +/// The page makes the reset explicit: a completion cue, then loop 2 with a fresh event counter. The loop is paced by sleeps (8 s +/// of events, then the 5 s hold: loop 2 about 13 s after the stream opens); the budget is 45 s because a host whose sleeps +/// overshoot (a virtual machine measured 8 x sleep(1) at 8.9 to 19.8 s) stretches that pacing, not the work (the loop's +/// replay itself is about 30 ms) #[test] fn served_demo_labels_its_restart() { let mut child = Reap(Command::new(env!("CARGO_BIN_EXE_probbit")).args(["monitor", "--demo", "drives", "--serve"]) @@ -362,7 +365,7 @@ fn served_demo_labels_its_restart() { write!(s, "GET /events HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n\r\n").unwrap(); let mut buf = String::new(); let mut count = 2; let started = Instant::now(); loop { - assert!(started.elapsed() < Duration::from_secs(20), "the demo did not label its restart"); + assert!(started.elapsed() < Duration::from_secs(45), "the demo did not label its restart"); let evs = stream(&mut s, &mut buf, count); if evs.iter().any(|(_, d)| d.contains("loop 2")) { break; } count = evs.len() + 1; @@ -463,3 +466,124 @@ fn one_monitor_serves_many_strands() { assert_eq!(get(port, "/s/zeta/doc/1", &host).2, format!("{}\n", db[0])); drop(child); let _ = std::fs::remove_dir_all(&dir); } + +/// RFC 0009: a strand with checkpoint lines whose persona has an expectation block replays from its header (no checkpoint +/// carries the expectation's memory). The fixture is the life of `tutor-expect.yaml` (the tutor with an expectation block) over +/// `tutor-expect.events.jsonl` (22 events, seed 2, the fixed clock, a checkpoint after every 5th: 4 of them), as `probbit live +/// tests/fixtures/monitor/tutor-expect.yaml --seed 2 --clock fixed --events ... --checkpoint-every 5 --strand` writes it. `--once` draws event 22 and exits 0, `--serve` streams it, `--dir` +/// lists it with its n; the same events through the tutor (no expectation block) still start at the last checkpoint +#[test] +fn checkpointed_strands_of_an_expectation_persona_open() { + const F: &str = "tests/fixtures/monitor/tutor-expect.strand"; + const EV: &str = "tests/fixtures/monitor/tutor-expect.events.jsonl"; + assert_eq!(std::fs::read_to_string(F).unwrap().lines().filter(|l| l.starts_with(r#"{"checkpoint":"#)).count(), 4); + let (c, v, err) = probbit(&["live", "verify", F]); assert_eq!(c, 0, "{v}{err}"); assert!(v.contains(r#""checkpoints":4"#) && v.contains(r#""events":22"#), "{v}"); + let (c, out, err) = probbit(&["monitor", F, "--once", "--plain"]); + assert_eq!(c, 0, "{out}{err}"); let top = out.lines().next().unwrap_or(""); + assert!(top.contains("| event 22 |") && top.ends_with("replay verified [ok]"), "from the header: {top}"); + // the page: the latest frame on /events is event 22, replayed from the header + let (child, port) = served(&[F]); let host = format!("127.0.0.1:{port}"); + let mut s = TcpStream::connect(("127.0.0.1", port)).unwrap(); s.set_read_timeout(Some(Duration::from_secs(20))).unwrap(); + write!(s, "GET /events HTTP/1.1\r\nHost: {host}\r\n\r\n").unwrap(); + let mut buf = String::new(); let evs = stream(&mut s, &mut buf, 2); + assert_eq!(evs[1].0, "frame"); assert!(evs[1].1.contains(r#""from_checkpoint":null"#) && evs[1].1.contains(r#""n":22,"#) && evs[1].1.contains(r#""diverges":null"#), "{}", evs[1].1); + drop(s); drop(child); + // --dir: listed with its n + let dir = std::env::temp_dir().join(format!("probbit-monitor-expect-dir-{}", std::process::id())); let _ = std::fs::remove_dir_all(&dir); std::fs::create_dir_all(&dir).unwrap(); + std::fs::copy(F, dir.join("tutor-expect.strand")).unwrap(); + let (child, port) = served(&["--dir", &dir.to_string_lossy()]); let host = format!("127.0.0.1:{port}"); + let t = Instant::now(); let mut list = String::new(); + while !list.contains(r#""n":22,"#) { assert!(t.elapsed() < Duration::from_secs(10), "listed with n 22: {list}"); std::thread::sleep(Duration::from_millis(100)); list = get(port, "/strands", &host).2; } + assert!(list.contains(r#""id":"tutor-expect""#) && list.contains(r#""persona":"Pip""#), "{list}"); + drop(child); let _ = std::fs::remove_dir_all(&dir); + // a persona without an expectation block: the shortcut (the badge names the checkpoint the replay started from) + let tutor = tmp("expect-tutor"); lived(&tutor, "2", &["--checkpoint-every", "5"], &std::fs::read_to_string(EV).unwrap()); + let (c, out, err) = probbit(&["monitor", &tutor, "--once", "--plain"]); let top = out.lines().next().unwrap_or(""); + assert_eq!(c, 0, "{out}{err}"); assert!(top.contains("| event 22 |") && top.ends_with("replay verified from checkpoint 20 [ok]"), "{top}"); + let _ = std::fs::remove_file(&tutor); +} + +/// §5.7 segments on the page: with `--dir`, a life's `NAME.strand` and `NAME.000k.strand` are one strand, id `NAME`, listed with +/// the life's event count and the base name's sidecar; its stream follows the newest segment and carries on across a seal (the +/// next file is picked up within the 1 s scan) +#[test] +fn a_served_life_in_segments_is_one_strand_that_follows_its_seals() { + let dir = std::env::temp_dir().join(format!("probbit-monitor-segments-{}", std::process::id())); let _ = std::fs::remove_dir_all(&dir); std::fs::create_dir_all(&dir).unwrap(); + let at = |n: &str| dir.join(n).to_string_lossy().into_owned(); + let (base, st, ev) = (at("life.strand"), at("st.json"), at("ev.jsonl")); + let (c, _, e) = probbit(&["persona", "init", "../examples/persona/tutor.yaml", "--seed", "2", "--out", &st]); assert_eq!(c, 0, "{e}"); + let run = |n: usize, extra: &[&str]| { std::fs::write(&ev, (0..n).map(|t| format!("{{\"praise\":{},\"elapsed_hours\":1}}\n", t % 2 == 0)).collect::()).unwrap(); + let (c, o, e) = probbit(&[&["live", "../examples/persona/tutor.yaml", "--state", &st, "--clock", "fixed", "--events", &ev, "--strand", &base][..], extra].concat()); assert_eq!(c, 0, "{e}"); o }; + run(5, &["--segment-every", "3", "--label", "agent=Nova"]); + assert!(std::path::Path::new(&at("life.0001.strand")).exists()); + let (child, port) = served(&["--dir", &dir.to_string_lossy()]); let host = format!("127.0.0.1:{port}"); + let t = Instant::now(); let mut list = String::new(); + while !list.contains(r#""n":5,"#) { assert!(t.elapsed() < Duration::from_secs(10), "listed with n 5: {list}"); std::thread::sleep(Duration::from_millis(100)); list = get(port, "/strands", &host).2; } + assert!(list.contains(r#""id":"life","labels":{"agent":"Nova"}"#) && list.contains(r#""segment":1"#) && !list.contains("life.0001") && list.matches(r#""id":"#).count() == 1, "one strand: {list}"); + let mut s = TcpStream::connect(("127.0.0.1", port)).unwrap(); s.set_read_timeout(Some(Duration::from_secs(20))).unwrap(); + write!(s, "GET /s/life/events HTTP/1.1\r\nHost: {host}\r\n\r\n").unwrap(); + let mut buf = String::new(); let evs = stream(&mut s, &mut buf, 2); assert!(evs[1].1.contains(r#""n":5,"#), "{}", evs[1].1); + // a host seals; the life goes on in the next file + let (c, o, e) = probbit(&["live", "control", &base, "seal"]); assert_eq!(c, 0, "{e}{o}"); assert!(o.contains("life.0002.strand"), "{o}"); + let docs = run(2, &[]); let six = docs.lines().next().unwrap().to_string(); + let (t, mut count) = (Instant::now(), 3); + loop { let evs = stream(&mut s, &mut buf, count); if evs.iter().any(|(_, d)| d.contains(r#""n":7,"#)) { break; } + assert!(t.elapsed() < Duration::from_secs(5), "event 7 streamed after the seal: {buf}"); count = evs.len() + 1; } + assert!(!buf.contains(r#""diverges":{"#), "{buf}"); + assert_eq!(get(port, "/s/life/doc/6", &host).2, format!("{six}\n")); + let list = get(port, "/strands", &host).2; assert!(list.contains(r#""n":7,"#) && list.contains(r#""segment":2"#) && list.matches(r#""id":"#).count() == 1, "{list}"); + drop(s); drop(child); let _ = std::fs::remove_dir_all(&dir); +} + +/// RFC 0013, the Mirror tab through the serve path: `/reflection?last=N` is `probbit reflect STRAND --last N` computed from the +/// file as it is now (`live`, no reflection line: `logged` null); after `live control STRAND reflect --last N` the logged line +/// is reported and recomputes equal; `/s/ID/reflection` serves strand ID; a bad `last` is 400; the page carries the tab +#[test] +fn the_mirror_serves_the_reflection() { + let f = tmp("mirror"); + let ev: String = (0..30).map(|k| format!("{{\"elapsed_hours\":1{}}}\n", if k % 4 == 0 { ",\"praise\":true" } else if k % 7 == 0 { ",\"loss\":true" } else { "" })).collect(); + lived(&f, "2", &["--checkpoint-every", "5"], &ev); + let (c, want, _) = probbit(&["reflect", &f, "--last", "10"]); assert_eq!(c, 0); let want = want.trim_end().to_string(); + let (child, port) = served(&[&f]); let host = format!("127.0.0.1:{port}"); + let (st, h, page) = get(port, "/", &host); assert_eq!(st, "HTTP/1.1 200 OK"); assert!(page.contains("id=\"tab-mirror\"") && page.contains("reflection?last=50"), "{h}"); + let (st, h, body) = get(port, "/reflection?last=10", &host); assert_eq!(st, "HTTP/1.1 200 OK"); assert!(h.contains("application/json"), "{h}"); + assert_eq!(body, format!("{{\"live\":{want},\"logged\":null}}\n")); + assert_eq!(get(port, "/reflection?last=0", &host).0, "HTTP/1.1 400 Bad Request"); + let (c, out, err) = probbit(&["live", "control", &f, "reflect", "--last", "10"]); assert_eq!(c, 0, "{out}{err}"); assert!(out.contains("\"control\":\"reflect\""), "{out}"); + let body = get(port, "/reflection?last=10", &host).2; + assert!(body.contains(&format!("\"logged\":{{\"doc\":{want},\"line\":")) && body.contains("\"matches\":true"), "{body}"); + drop(child); + let (child, port) = served(&[&f]); let host = format!("127.0.0.1:{port}"); + let id = std::path::Path::new(&f).file_stem().unwrap().to_string_lossy().replace(|c: char| !(c.is_ascii_alphanumeric() || "_.-".contains(c)), "-"); + assert_eq!(get(port, &format!("/s/{id}/reflection?last=10", ), &host).0, "HTTP/1.1 200 OK"); + drop(child); let _ = std::fs::remove_file(&f); +} + +/// RFC 0014, the integrity dial through the serve path: a life of a persona with the identity self keys; the latest frame's +/// document carries the last line's `integrity` (canonical, byte for byte), its `integrity` word's dial is that line's `used`, +/// and the page carries the Mirror's drift table +#[test] +fn the_integrity_dial_reads_the_strands_number() { + use probbit_persona::json::{self, Json}; + let f = tmp("integrity"); let _ = std::fs::remove_file(&f); + let ev: String = (0..24).map(|k| format!("{{\"elapsed_hours\":0.5,\"src\":\"human\",\"{}\":true}}\n", if k % 5 == 4 { "criticism" } else { "praise" })).collect(); + let mut live = Command::new(env!("CARGO_BIN_EXE_probbit")).args(["live", "tests/fixtures/monitor/tutor-self.yaml", "--seed", "3", "--clock", "fixed", "--strand", &f]) + .stdin(Stdio::piped()).stdout(Stdio::null()).stderr(Stdio::null()).spawn().unwrap(); + live.stdin.take().unwrap().write_all(ev.as_bytes()).unwrap(); assert!(live.wait().unwrap().success()); + let (c, out, _) = probbit(&["live", "verify", &f]); assert_eq!(c, 0, "{out}"); + let text = std::fs::read_to_string(&f).unwrap(); let last = json::parse(text.lines().last().unwrap()).unwrap(); + let ig = last.get("integrity").expect("the last line carries integrity").clone(); + let used = ig.get("used").and_then(Json::as_f64).unwrap(); assert!(used > 0.0, "{}", probbit_persona::persona::canon(&ig)); + let (child, port) = served(&[&f]); let host = format!("127.0.0.1:{port}"); + let mut s = TcpStream::connect(("127.0.0.1", port)).unwrap(); s.set_read_timeout(Some(Duration::from_secs(20))).unwrap(); + write!(s, "GET /events HTTP/1.1\r\nHost: {host}\r\n\r\n").unwrap(); + let mut buf = String::new(); let evs = stream(&mut s, &mut buf, 2); + let frame = evs.iter().rev().find(|(_, d)| d.contains("\"words\"")).map(|(_, d)| json::parse(d).unwrap()).unwrap_or_else(|| panic!("{buf}")); + assert_eq!(frame.get("n").and_then(Json::as_f64), Some(24.0)); + let canon = probbit_persona::persona::canon; + assert_eq!(canon(frame.get("doc").and_then(|d| d.get("integrity")).unwrap()), canon(&ig), "the frame's document carries the line's number"); + let w = frame.get("words").and_then(Json::as_arr).unwrap().iter().find(|w| w.get("key").and_then(Json::as_str) == Some("integrity")).cloned().expect("the integrity word"); + assert_eq!(w.get("dial").and_then(Json::as_f64), Some(used.min(1.0)), "{}", canon(&w)); + assert!(get(port, "/", &host).2.contains("id=\"mirror-integrity\"")); + drop(s); drop(child); let _ = std::fs::remove_file(&f); +} diff --git a/probbit-cli/tests/persona.rs b/probbit-cli/tests/persona.rs index f35da0a..e5cd467 100644 --- a/probbit-cli/tests/persona.rs +++ b/probbit-cli/tests/persona.rs @@ -4,10 +4,10 @@ use std::io::Write; use std::process::{Command, Stdio}; #[allow(dead_code)] -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; #[allow(dead_code)] -#[path = "../src/yaml.rs"] +#[path = "../../probbit-persona/src/yaml.rs"] mod yaml; use json::Json; diff --git a/probbit-cli/tests/processor_contracts.rs b/probbit-cli/tests/processor_contracts.rs index 653841e..ed32c67 100644 --- a/probbit-cli/tests/processor_contracts.rs +++ b/probbit-cli/tests/processor_contracts.rs @@ -1,7 +1,7 @@ //! Public processor responses: released assignments are never confused with the //! legacy full diagnostic candidate. Fixed-work behavior is thread invariant. #[allow(dead_code)] -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; use json::Json; use std::io::Write; diff --git a/probbit-cli/tests/processor_json.rs b/probbit-cli/tests/processor_json.rs index 4833f85..4789514 100644 --- a/probbit-cli/tests/processor_json.rs +++ b/probbit-cli/tests/processor_json.rs @@ -1,5 +1,5 @@ #[allow(dead_code)] -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; #[test] diff --git a/probbit-cli/tests/stress.rs b/probbit-cli/tests/stress.rs index 4a3fc56..8e38845 100644 --- a/probbit-cli/tests/stress.rs +++ b/probbit-cli/tests/stress.rs @@ -6,7 +6,7 @@ use std::process::{Command, Stdio}; use std::io::Write; #[allow(dead_code)] -#[path = "../src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; use json::Json; diff --git a/probbit-core/Cargo.toml b/probbit-core/Cargo.toml index 1feb7b5..7de01ac 100644 --- a/probbit-core/Cargo.toml +++ b/probbit-core/Cargo.toml @@ -10,4 +10,11 @@ readme = "../README.md" keywords = ["p-bit", "ising", "monte-carlo", "philox", "probabilistic"] categories = ["algorithms", "science", "simulation"] +[features] +default = ["std"] +std = [] +# no_std + alloc (a target without an operating system): the std subset comes from probbit-nostd +nostd = ["dep:probbit-nostd"] + [dependencies] +probbit-nostd = { path = "../probbit-nostd", version = "0.14.0-dev", optional = true } diff --git a/probbit-core/src/lib.rs b/probbit-core/src/lib.rs index 58c8ef8..dcd080a 100644 --- a/probbit-core/src/lib.rs +++ b/probbit-core/src/lib.rs @@ -5,6 +5,17 @@ //! `sweep_prefix` is the PT-in-a-word variant (per-lane betas, shared uniform -> contiguous lane prefix mask). //! - `heatbath_f32`: T1 SoA f32 heat-bath on a 2D ±J lattice with LUT sigmoid. //! - `rt`: the engine's clock and its no-thread switch (for targets without threads or a std clock: wasm32-unknown-unknown). +#![cfg_attr(not(feature = "std"), no_std)] +#[cfg(not(feature = "std"))] +#[allow(unused_imports)] +#[macro_use] +extern crate probbit_nostd as std; +#[cfg(not(feature = "std"))] +#[allow(unused_imports)] +#[macro_use] +extern crate alloc; +#[allow(unused_imports)] +use std::prelude::rust_2021::*; #[derive(Clone)] pub struct Philox4x32 { key: [u32; 2], ctr: [u32; 4], buf: [u32; 4], idx: usize } @@ -63,6 +74,8 @@ impl Rng32 for Philox4x32 { #[inline(always)] fn u32(&mut self) -> u32 { self.ne /// 2D ±J periodic lattice, bit-sliced: w[i] holds spin i of 64 replicas (bit=1 means -1). pub mod multispin { use super::Rng32; + #[allow(unused_imports)] + use std::prelude::rust_2021::*; /// The fields are private: `sweep_fast` reads through raw pointers and relies on `w`, `jr` and `jd` holding exactly `l * l` /// words. With `pub` fields safe code could break that (`l = 2^32` on empty vectors: `l * l` wrapped to 0 in release, the /// length check passed, the sweep read through a dangling pointer). Build one with `random` or `new`. @@ -170,13 +183,15 @@ pub mod multispin { } } pub fn thresholds(beta: f64) -> (u32, u32) { - ((((-4.0 * beta).exp()) * 4294967295.0) as u32, (((-8.0 * beta).exp()) * 4294967295.0) as u32) + (((super::exp64(-4.0 * beta)) * 4294967295.0) as u32, ((super::exp64(-8.0 * beta)) * 4294967295.0) as u32) } } /// T1: SoA f32 heat-bath on 2D ±J lattice, LUT for tanh(beta*h), h in {-4,-2,0,2,4}. pub mod heatbath_f32 { use super::Rng32; + #[allow(unused_imports)] + use std::prelude::rust_2021::*; /// The fields are private: besides the `multispin` lengths, `sweep_fast` reads its 9-entry table at h + 4 unchecked, which /// is in range only for ±1 spins and couplings (`pub` couplings of 1000.0 read past the table). `random` and `new` build /// only such lattices, and the sweeps write only ±1. @@ -241,9 +256,15 @@ pub mod heatbath_f32 { }} } } - pub fn lut(beta: f32) -> [f32; 9] { let mut t = [0f32; 9]; for k in 0..9 { t[k] = (beta * (k as f32 - 4.0)).tanh(); } t } + pub fn lut(beta: f32) -> [f32; 9] { let mut t = [0f32; 9]; for k in 0..9 { t[k] = super::tanh32(beta * (k as f32 - 4.0)); } t } } +/// `exp` and `tanh` of the kernels: std's (the platform libm) with std, the `libm` crate's on no_std +#[cfg(feature = "std")] fn exp64(x: f64) -> f64 { x.exp() } +#[cfg(feature = "std")] fn tanh32(x: f32) -> f32 { x.tanh() } +#[cfg(not(feature = "std"))] fn exp64(x: f64) -> f64 { std::FloatExt::exp(x) } +#[cfg(not(feature = "std"))] fn tanh32(x: f32) -> f32 { std::FloatExt::tanh(x) } + /// The runtime probbit-ir and probbit-decide run on: their clock and whether they may spawn threads. /// /// `Instant` is `std::time::Instant` on every target that has one (the CLI: unchanged behaviour, the same type). On @@ -257,17 +278,17 @@ pub mod heatbath_f32 { /// threads sized for > 2,048-variable programs are not used. pub mod rt { use std::cell::Cell; - thread_local! { static SEQUENTIAL: Cell = const { Cell::new(cfg!(all(target_family = "wasm", target_os = "unknown"))) }; } + thread_local! { static SEQUENTIAL: Cell = const { Cell::new(cfg!(any(not(feature = "std"), all(target_family = "wasm", target_os = "unknown")))) }; } /// No threads on this thread's calls into the engine (see the module doc). Returns the previous setting. pub fn set_sequential(on: bool) -> bool { SEQUENTIAL.with(|s| s.replace(on)) } /// Whether this thread's engine calls run without threads. pub fn sequential() -> bool { SEQUENTIAL.with(|s| s.get()) } - #[cfg(not(all(target_family = "wasm", target_os = "unknown")))] + #[cfg(all(feature = "std", not(all(target_family = "wasm", target_os = "unknown"))))] pub use std::time::Instant; - #[cfg(all(target_family = "wasm", target_os = "unknown"))] + #[cfg(any(not(feature = "std"), all(target_family = "wasm", target_os = "unknown")))] pub use clock::{set_clock, Instant}; - #[cfg(all(target_family = "wasm", target_os = "unknown"))] + #[cfg(any(not(feature = "std"), all(target_family = "wasm", target_os = "unknown")))] mod clock { use std::cell::Cell; use std::time::Duration; diff --git a/probbit-decide/Cargo.toml b/probbit-decide/Cargo.toml index 250ba48..65d7106 100644 --- a/probbit-decide/Cargo.toml +++ b/probbit-decide/Cargo.toml @@ -10,6 +10,15 @@ readme = "../README.md" keywords = ["p-bit", "assignment", "routing", "constraint-solver", "probabilistic"] categories = ["algorithms", "science"] +[features] +default = ["std"] +std = ["probbit-core/std", "probbit-ir/std"] +# no_std + alloc (a target without an operating system): the std subset comes from probbit-nostd +nostd = ["dep:probbit-nostd", "probbit-core/nostd", "probbit-ir/nostd"] + [dependencies] -probbit-core = { path = "../probbit-core", version = "0.13.0" } -probbit-ir = { path = "../probbit-ir", version = "0.13.0" } +# the amplitude register's one math path (cos, sin, sqrt, asin, acos, atan2) on the std and the no_std build alike (pure Rust) +libm = "0.2" +probbit-nostd = { path = "../probbit-nostd", version = "0.14.0-dev", optional = true } +probbit-core = { path = "../probbit-core", default-features = false, version = "0.14.0-dev" } +probbit-ir = { path = "../probbit-ir", default-features = false, version = "0.14.0-dev" } diff --git a/probbit-decide/src/amp.rs b/probbit-decide/src/amp.rs new file mode 100644 index 0000000..6b29ac0 --- /dev/null +++ b/probbit-decide/src/amp.rs @@ -0,0 +1,506 @@ +//! RFC 0011 (research, rounds 1-2): an amplitude register. A register of q qubits simulated exactly on a CPU as a state vector of +//! 2^q complex amplitudes (f64 pairs, on the heap: 16 * 2^q bytes): unitary gates with phase, interference between basis states, +//! exact marginals (no sampling), measurement by the Born rule with the engine's deterministic RNG, and a digest of the state at +//! 6 decimals (the rule for every emitted float). A simulation of a small quantum system and nothing else: no quantum hardware, +//! no speedup claim, no scaling past what 2^q amplitudes allow. +//! +//! Basis index x = sum of bit_i << i: qubit i is bit i. Round 2: a variable with d allowed values takes ceil(log2 d) qubits +//! (`Encoding`; d = 3: codes 00, 01, 10, code 11 unused), the start is the uniform superposition over the valid codes, the +//! mixer is block-diagonal per variable (`MIXER`, the identity on unused codes) and the cost is one general diagonal phase +//! layer exp(-i g E) with E = -log w per basis state, so pair tables between multi-level variables (3- and 4-body terms in +//! the qubits) need no decomposition. The unused codes keep amplitude 0 (checked after every layer). +#[allow(unused_imports)] +use std::prelude::rust_2021::*; +use probbit_core::Philox4x32; + +/// Round 7: the register's one math path. Every function past + - * / that the register evaluates (cos, sin, sqrt, asin, acos, +/// atan2, floor) is the `libm` crate's on every build, std and no_std alike: pure Rust, the same code and the same evaluation +/// order on every target, no platform libm and no fused multiply-add, so a strand recorded on one build replays bit for bit on +/// the other whatever the schedule's length (round 5: the platform's cos/sin against libm's flipped a 6-decimal odd over the +/// long sweep's layers). +mod m { + #[inline] pub fn cos(x: f64) -> f64 { libm::cos(x) } + #[inline] pub fn sin(x: f64) -> f64 { libm::sin(x) } + #[inline] pub fn sqrt(x: f64) -> f64 { libm::sqrt(x) } + #[inline] pub fn asin(x: f64) -> f64 { libm::asin(x) } + #[inline] pub fn acos(x: f64) -> f64 { libm::acos(x) } + #[inline] pub fn atan2(y: f64, x: f64) -> f64 { libm::atan2(y, x) } + #[inline] pub fn floor(x: f64) -> f64 { libm::floor(x) } +} + +/// The largest register on the host (16 qubits = 1 MiB of amplitudes) +pub const Q_MAX_HOST: usize = 16; +/// Bytes per amplitude: two f64 +pub const AMP_BYTES: usize = 16; + +/// A complex number (f64 parts) +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct C { pub re: f64, pub im: f64 } +impl C { + pub const ZERO: C = C { re: 0.0, im: 0.0 }; + pub const ONE: C = C { re: 1.0, im: 0.0 }; + #[inline] pub fn new(re: f64, im: f64) -> C { C { re, im } } + #[inline] pub fn mul(self, o: C) -> C { C { re: self.re * o.re - self.im * o.im, im: self.re * o.im + self.im * o.re } } + #[inline] pub fn add(self, o: C) -> C { C { re: self.re + o.re, im: self.im + o.im } } + #[inline] pub fn conj(self) -> C { C { re: self.re, im: -self.im } } + #[inline] pub fn norm2(self) -> f64 { self.re * self.re + self.im * self.im } + /// e^{i t} + #[inline] pub fn cis(t: f64) -> C { C { re: m::cos(t), im: m::sin(t) } } +} + +/// The gate set: enough for Ising-type programs and QAOA layers. Angles in radians; RX(t) = exp(-i t X / 2), +/// RZ(t) = exp(-i t Z / 2), RZZ(t) = exp(-i t Z⊗Z / 2). +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Gate { H(usize), X(usize), Z(usize), Rx(usize, f64), Rz(usize, f64), Cnot(usize, usize), Cz(usize, usize), Rzz(usize, usize, f64) } +impl Gate { + /// The gate as one canonical text line (angles at 6 decimals): what `circuit_text` joins and the circuit digest hashes + pub fn text(&self) -> String { + match *self { + Gate::H(a) => format!("h {a}"), Gate::X(a) => format!("x {a}"), Gate::Z(a) => format!("z {a}"), + Gate::Rx(a, t) => format!("rx {a} {}", f6(t)), Gate::Rz(a, t) => format!("rz {a} {}", f6(t)), + Gate::Cnot(a, b) => format!("cnot {a} {b}"), Gate::Cz(a, b) => format!("cz {a} {b}"), Gate::Rzz(a, b, t) => format!("rzz {a} {b} {}", f6(t)), + } + } + /// The gate's matrix on its own qubits (2x2, or 4x4 on (a, b) with basis index bit_a + 2 bit_b), row-major + pub fn matrix(&self) -> Vec { + let (r, z) = (core::f64::consts::FRAC_1_SQRT_2, C::ZERO); + let half = |t: f64| (C::new(m::cos(t / 2.0), 0.0), m::sin(t / 2.0)); + match *self { + Gate::H(_) => vec![C::new(r, 0.0), C::new(r, 0.0), C::new(r, 0.0), C::new(-r, 0.0)], + Gate::X(_) => vec![z, C::ONE, C::ONE, z], + Gate::Z(_) => vec![C::ONE, z, z, C::new(-1.0, 0.0)], + Gate::Rx(_, t) => { let (c, s) = half(t); vec![c, C::new(0.0, -s), C::new(0.0, -s), c] } + Gate::Rz(_, t) => vec![C::cis(-t / 2.0), z, z, C::cis(t / 2.0)], + Gate::Cnot(..) => { let mut m = vec![z; 16]; for (i, j) in [(0, 0), (1, 3), (2, 2), (3, 1)] { m[i * 4 + j] = C::ONE; } m } + Gate::Cz(..) => { let mut m = vec![z; 16]; for i in 0..4 { m[i * 5] = if i == 3 { C::new(-1.0, 0.0) } else { C::ONE }; } m } + Gate::Rzz(_, _, t) => { let mut m = vec![z; 16]; for i in 0..4 { let same = (i & 1) == (i >> 1); m[i * 5] = C::cis(if same { -t / 2.0 } else { t / 2.0 }); } m } + } + } +} + +/// max |U†U - I| over the entries of a gate's matrix +pub fn unitarity_error(g: &Gate) -> f64 { + let m = g.matrix(); let d = if m.len() == 4 { 2 } else { 4 }; let mut e: f64 = 0.0; + for i in 0..d { for j in 0..d { + let mut s = C::ZERO; for k in 0..d { s = s.add(m[k * d + i].conj().mul(m[k * d + j])); } + let want = if i == j { 1.0 } else { 0.0 }; e = e.max((s.re - want).abs()).max(s.im.abs()); } } + e +} + +/// A number at 6 decimals as the canonical text writes it (`-0` written `0`) +fn f6(x: f64) -> String { let s = format!("{x:.6}"); if s.trim_start_matches('-').bytes().all(|b| b == b'0' || b == b'.') { "0.000000".into() } else { s } } + +/// One step of a circuit: a gate, a diagonal phase layer exp(-i g E) (`Register::phase`), or a mixer layer on every variable +/// (`Register::mix`) +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Op { Gate(Gate), Phase(f64), Mix(f64), Oracle(f64), Diffuse(f64) } +impl Op { + /// One canonical text line (angles at 6 decimals) + pub fn text(&self) -> String { match self { Op::Gate(g) => g.text(), Op::Phase(g) => format!("phase {}", f6(*g)), Op::Mix(t) => format!("mix {}", f6(*t)), Op::Oracle(p) => format!("oracle {}", f6(*p)), Op::Diffuse(p) => format!("diffuse {}", f6(*p)) } } +} + +/// The mixer every round-2 layer uses, by name (the receipt's `engine.mixer`) +pub const MIXER: &str = "complete-graph"; + +/// Round 2: how variables with d allowed values are packed into qubits. Variable i (d_i >= 1 values) takes b_i = ceil(log2 d_i) +/// qubits, packed in variable order (its code is bits o_i..o_i + b_i of the basis index); code c < d_i is its c-th value, codes +/// c >= d_i are unused (d = 3: code 11) and keep amplitude 0 for the whole evolution. +#[derive(Clone, Debug, PartialEq)] +pub struct Encoding { pub levels: Vec } +impl Encoding { + /// Qubits for d values: ceil(log2 d) (0 for d = 1) + pub fn bits(d: usize) -> usize { assert!(d >= 1, "a variable needs at least one value"); (usize::BITS - (d - 1).leading_zeros()) as usize } + /// Every variable two-valued (round 1's programs) + pub fn two_valued(q: usize) -> Encoding { Encoding { levels: vec![2; q] } } + /// The encoded qubits, sum of ceil(log2 d_i) + pub fn qubits(&self) -> usize { self.levels.iter().map(|&d| Encoding::bits(d)).sum() } + /// Per variable its first qubit + pub fn offsets(&self) -> Vec { let mut o = 0; self.levels.iter().map(|&d| { let r = o; o += Encoding::bits(d); r }).collect() } + /// Variable i's code in basis index x + pub fn code(&self, x: usize, i: usize) -> usize { let o = self.offsets()[i]; (x >> o) & ((1usize << Encoding::bits(self.levels[i])) - 1) } + /// The codes of every variable in x, or None when one is unused + pub fn decode(&self, x: usize) -> Option> { + let mut o = 0; let mut v = Vec::with_capacity(self.levels.len()); + for &d in &self.levels { let b = Encoding::bits(d); let c = (x >> o) & ((1usize << b) - 1); if c >= d { return None; } v.push(c); o += b; } + Some(v) + } + /// The canonical text the encoding digest hashes: one line per variable, `var levels qubits ..` + pub fn text(&self) -> String { + let mut s = String::new(); for (i, (&d, o)) in self.levels.iter().zip(self.offsets()).enumerate() { s.push_str(&format!("var {i} levels {d} qubits {o}..{}\n", o + Encoding::bits(d))); } s + } +} + +/// The complete-graph mixer on d levels as a d x d matrix (row-major): exp(i t A) with A = J - I the adjacency of the complete +/// graph K_d. Closed form (J^2 = d J): e^{-it} (I + (e^{itd} - 1)/d J). d = 2 is exp(i t X) = RX(-2t); d = 1 is the identity. +pub fn mixer_matrix(d: usize, t: f64) -> Vec { + let (ph, c) = mixer_coef(d, t); (0..d * d).map(|k| { let off = c.mul(ph); if k / d == k % d { ph.add(off) } else { off } }).collect() +} +/// (e^{-it}, (e^{itd} - 1)/d) +fn mixer_coef(d: usize, t: f64) -> (C, C) { let e = C::cis(t * d as f64); (C::cis(-t), C::new((e.re - 1.0) / d as f64, e.im / d as f64)) } +/// max |U^dagger U - I| of the mixer on the whole 2^b block of a d-level variable (the mixer on the valid codes, the identity on +/// the unused ones) +pub fn mixer_unitarity_error(d: usize, t: f64) -> f64 { + let n = 1usize << Encoding::bits(d); let m = mixer_matrix(d, t); + let u = |i: usize, j: usize| if i < d && j < d { m[i * d + j] } else if i == j { C::ONE } else { C::ZERO }; + let mut e: f64 = 0.0; + for i in 0..n { for j in 0..n { let mut s = C::ZERO; for k in 0..n { s = s.add(u(k, i).conj().mul(u(k, j))); } + e = e.max((s.re - if i == j { 1.0 } else { 0.0 }).abs()).max(s.im.abs()); } } + e +} + +/// q qubits as 2^q complex amplitudes, and the ops applied (the circuit) +#[derive(Clone, Debug)] +pub struct Register { q: usize, amp: Vec, ops: Vec } +impl Register { + /// |0...0> on q qubits (q <= Q_MAX_HOST); the amplitudes take exactly 16 * 2^q bytes of heap + pub fn new(q: usize) -> Register { + assert!(q <= Q_MAX_HOST, "an amplitude register has at most {Q_MAX_HOST} qubits on the host"); + let mut amp = vec![C::ZERO; 1usize << q]; amp[0] = C::ONE; + let r = Register { q, amp, ops: vec![] }; + assert_eq!(r.bytes(), AMP_BYTES << q); r + } + /// Round 2: the uniform superposition over the valid codes of every variable, 1/sqrt(prod d_i) on each valid basis state and + /// 0 on the rest (for an all-two-valued encoding this is H on every qubit) + pub fn uniform(enc: &Encoding) -> Register { + let mut r = Register::new(enc.qubits()); let a = 1.0 / m::sqrt(enc.levels.iter().map(|&d| d as f64).product::()); + for (x, v) in r.amp.iter_mut().enumerate() { *v = if enc.decode(x).is_some() { C::new(a, 0.0) } else { C::ZERO }; } + r + } + pub fn qubits(&self) -> usize { self.q } + /// The amplitudes' bytes: 16 * 2^q + pub fn bytes(&self) -> usize { self.amp.len() * core::mem::size_of::() } + pub fn amplitudes(&self) -> &[C] { &self.amp } + pub fn ops(&self) -> &[Op] { &self.ops } + /// The gates among the ops + pub fn gates(&self) -> Vec { self.ops.iter().filter_map(|o| if let Op::Gate(g) = o { Some(*g) } else { None }).collect() } + /// The circuit as text, one op per line (angles at 6 decimals) + pub fn circuit_text(&self) -> String { let mut s = String::new(); for o in &self.ops { s.push_str(&o.text()); s.push('\n'); } s } + + fn one(&mut self, a: usize, m: [C; 4]) { + let bit = 1usize << a; + for i in 0..self.amp.len() { if i & bit != 0 { continue; } let (x, y) = (self.amp[i], self.amp[i | bit]); + self.amp[i] = m[0].mul(x).add(m[1].mul(y)); self.amp[i | bit] = m[2].mul(x).add(m[3].mul(y)); } + } + /// Apply one gate (logged in the circuit) + pub fn apply(&mut self, g: Gate) { + let q = self.q; let ok = |a: usize| assert!(a < q, "qubit {a} out of range ({q} qubits)"); + match g { + Gate::H(a) | Gate::X(a) | Gate::Z(a) | Gate::Rx(a, _) | Gate::Rz(a, _) => { ok(a); let m = g.matrix(); self.one(a, [m[0], m[1], m[2], m[3]]); } + Gate::Cnot(c, t) => { ok(c); ok(t); assert_ne!(c, t); let (bc, bt) = (1usize << c, 1usize << t); + for i in 0..self.amp.len() { if i & bc != 0 && i & bt == 0 { self.amp.swap(i, i | bt); } } } + Gate::Cz(a, b) => { ok(a); ok(b); assert_ne!(a, b); let m = (1usize << a) | (1usize << b); + for i in 0..self.amp.len() { if i & m == m { self.amp[i] = C::new(-self.amp[i].re, -self.amp[i].im); } } } + Gate::Rzz(a, b, t) => { ok(a); ok(b); assert_ne!(a, b); let (same, diff) = (C::cis(-t / 2.0), C::cis(t / 2.0)); + for i in 0..self.amp.len() { let p = if ((i >> a) & 1) == ((i >> b) & 1) { same } else { diff }; self.amp[i] = p.mul(self.amp[i]); } } + } + self.ops.push(Op::Gate(g)); + } + /// Round 2: the general diagonal phase layer, a_x <- e^{-i g E(x)} a_x for every basis state with a finite E(x) (E = -log w + /// of its plan); E(x) = NaN marks an unused code, left untouched (its amplitude is 0). O(2^q), any order of interaction. + pub fn phase(&mut self, e: &[f64], g: f64) { + assert_eq!(e.len(), self.amp.len(), "one energy per basis state"); + for (a, &ex) in self.amp.iter_mut().zip(e) { if !ex.is_nan() { *a = C::cis(-g * ex).mul(*a); } } + self.ops.push(Op::Phase(g)); + } + /// Round 2: the mixer layer, the complete-graph mixer exp(i t A_d) on the valid codes of every variable (`mixer_matrix`), + /// the identity on its unused codes: block-diagonal, O(d) per block by the closed form. + pub fn mix(&mut self, enc: &Encoding, t: f64) { + assert_eq!(enc.qubits(), self.q, "the encoding's qubits"); + let mut buf: Vec = vec![]; + for (&d, o) in enc.levels.iter().zip(enc.offsets()) { + if d < 2 { continue; } let b = Encoding::bits(d); let mask = ((1usize << b) - 1) << o; let (ph, c) = mixer_coef(d, t); + for x in 0..self.amp.len() { if x & mask != 0 { continue; } + buf.clear(); let mut s = C::ZERO; for k in 0..d { let a = self.amp[x | (k << o)]; buf.push(a); s = s.add(a); } let cs = c.mul(s); + for (k, &a) in buf.iter().enumerate() { self.amp[x | (k << o)] = ph.mul(a.add(cs)); } } + } + self.ops.push(Op::Mix(t)); + } + /// Round 4: the oracle, a_x <- e^{i phi} a_x on the basis states marked good (phi = pi: the sign flip) + pub fn oracle(&mut self, good: &[bool], phi: f64) { + assert_eq!(good.len(), self.amp.len(), "one mark per basis state"); let w = C::cis(phi); + for (a, &g) in self.amp.iter_mut().zip(good) { if g { *a = w.mul(*a); } } + self.ops.push(Op::Oracle(phi)); + } + /// Round 4: the diffusion about a stored state psi, I - (1 - e^{i phi}) |psi> f64 { self.amp.iter().zip(e).filter(|(_, x)| x.is_nan()).map(|(a, _)| a.norm2()).sum() } + /// The Born probabilities |a_x|^2 + pub fn probabilities(&self) -> Vec { self.amp.iter().map(|a| a.norm2()).collect() } + /// Exact marginals P(qubit i = 1), one per qubit (no sampling) + pub fn marginals(&self) -> Vec { + let mut m = vec![0.0; self.q]; + for (x, a) in self.amp.iter().enumerate() { let p = a.norm2(); for (i, mi) in m.iter_mut().enumerate() { if x >> i & 1 == 1 { *mi += p; } } } + m + } + /// Round 2: exact marginals per variable and level, P(variable i has code c), concatenated in variable order (sum d_i entries) + pub fn level_marginals(&self, enc: &Encoding) -> Vec { + let base: Vec = enc.levels.iter().scan(0, |s, &d| { let r = *s; *s += d; Some(r) }).collect(); + let mut m = vec![0.0; enc.levels.iter().sum()]; + for (x, a) in self.amp.iter().enumerate() { if let Some(v) = enc.decode(x) { let p = a.norm2(); for (i, c) in v.into_iter().enumerate() { m[base[i] + c] += p; } } } + m + } + /// One measurement of every qubit by the Born rule: the basis index, drawn with Philox4x32(seed, stream 0) + pub fn measure(&self, seed: u64) -> usize { + let u = Philox4x32::new(seed, 0).f64(); let mut c = 0.0; + for (x, a) in self.amp.iter().enumerate() { c += a.norm2(); if u < c { return x; } } + self.amp.iter().rposition(|a| a.norm2() > 0.0).unwrap_or(0) + } + /// The canonical state: one line per amplitude, `re im` at 6 decimals + pub fn canonical(&self) -> String { let mut s = String::new(); for a in &self.amp { s.push_str(&f6(a.re)); s.push(' '); s.push_str(&f6(a.im)); s.push('\n'); } s } + /// sha256 of the canonical state + pub fn digest(&self) -> [u8; 32] { sha256(self.canonical().as_bytes()) } +} + +/// An Ising program over q two-valued variables: log w(x) = c + sum_i a_i z_i + sum_(i, pub b: Vec<(usize, usize, f64)> } +impl Ising { + /// The program's log weight at basis index x, without the constant + pub fn logw(&self, x: usize) -> f64 { + let z = |i: usize| if x >> i & 1 == 0 { 1.0 } else { -1.0 }; + self.a.iter().enumerate().map(|(i, a)| a * z(i)).sum::() + self.b.iter().map(|&(i, j, b)| b * z(i) * z(j)).sum::() + } + /// E(x) = -log w(x) over the 2^q basis states (the phase layer's input) + pub fn energies(&self) -> Vec { (0..1usize << self.q).map(|x| -self.logw(x)).collect() } +} + +/// A transverse-field sweep: `steps` Trotter steps over total time `time`, s = (k + 1/2) / steps at step k, +/// H(s) = -(1 - s) sum_i A_i + s E (A_i the complete-graph mixer of variable i; E = -log w), started from the uniform +/// superposition over the valid codes +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct Schedule { pub steps: usize, pub time: f64 } +impl Schedule { + /// The canonical text the schedule digest hashes + pub fn text(&self) -> String { format!("transverse-field linear steps {} time {}\n", self.steps, f6(self.time)) } + pub fn name(&self) -> String { format!("tf-linear-{}x{}", self.steps, f6(self.time)) } +} +/// The tier's fixed default schedule (rounds 1 and 2; schedules are a later round's study) +pub const DEFAULT_SCHEDULE: Schedule = Schedule { steps: 16, time: 4.0 }; +/// Round 5: the longer sweep (`engine.schedule: tf-linear-64x16`): 4x the default's steps and time, so about 4x its layers +pub const LONG_SCHEDULE: Schedule = Schedule { steps: 64, time: 16.0 }; +/// Round 5: the two layers (gamma, beta) of `engine.schedule: qaoa-p2`, fixed by a grid search on a held-out third of the P0 +/// corpus programs (the largest median separation on the decisions the thermal tier is indifferent on, subject to agreement +/// >= 90 % on the others; docs/rfc/0011-amplitude-register.md, "Schedules and the indifferent set"), then frozen: gamma 0.500000, +/// beta 0.500000, then gamma 1.500000, beta 0.250000 +pub const QAOA_P2: [(f64, f64); 2] = [(0.5, 0.5), (1.5, 0.25)]; +/// Round 5: the names `engine.schedule` accepts (the first is the default) +pub const SCHEDULE_NAMES: [&str; 3] = ["tf-linear-16x4", "tf-linear-64x16", "qaoa-p2"]; +/// Round 5: the evolution the tier runs on each component: a transverse-field sweep or QAOA layers +#[derive(Clone, Debug, PartialEq)] +pub enum Evolution { Sweep(Schedule), Qaoa(Vec<(f64, f64)>) } +impl Evolution { + /// The evolution of a name in `SCHEDULE_NAMES` (None for any other name) + pub fn named(name: &str) -> Option { + match name { "tf-linear-16x4" => Some(Evolution::Sweep(DEFAULT_SCHEDULE)), "tf-linear-64x16" => Some(Evolution::Sweep(LONG_SCHEDULE)), + "qaoa-p2" => Some(Evolution::Qaoa(QAOA_P2.to_vec())), _ => None } + } + /// The register after the evolution from the uniform start + pub fn run(&self, enc: &Encoding, e: &[f64]) -> Register { match self { Evolution::Sweep(s) => sweep(enc, e, s), Evolution::Qaoa(p) => qaoa_on(enc, e, p) } } + /// The canonical text the schedule digest hashes (a sweep's is `Schedule::text`, unchanged) + pub fn text(&self) -> String { + match self { Evolution::Sweep(s) => s.text(), + Evolution::Qaoa(p) => { let mut t = format!("qaoa layers {}\n", p.len()); for &(g, b) in p { t.push_str(&format!("gamma {} beta {}\n", f6(g), f6(b))); } t } } + } + /// The receipt's `schedule_name` (a sweep's is `Schedule::name`, unchanged) + pub fn name(&self) -> String { + match self { Evolution::Sweep(s) => s.name(), Evolution::Qaoa(p) if p[..] == QAOA_P2[..] => "qaoa-p2".into(), Evolution::Qaoa(p) => format!("qaoa-p{}-lab", p.len()) } + } +} + +/// The bar on the Born mass of the unused codes, checked after every layer (it is 0 by construction; never renormalised) +pub const INVALID_MASS_MAX: f64 = 1e-15; +fn checked(r: &Register, e: &[f64]) { let m = r.invalid_mass(e); assert!(m <= INVALID_MASS_MAX, "unused-code mass {m:e} after a layer"); } + +/// Round 2: the sweep `s` on an encoding and its energies (one per basis state, NaN on unused codes): the uniform start over +/// the valid codes, then per step the phase layer (g = dt s) and the mixer layer (t = dt (1 - s)) +pub fn sweep(enc: &Encoding, e: &[f64], s: &Schedule) -> Register { + let mut r = Register::uniform(enc); let dt = s.time / s.steps as f64; + for k in 0..s.steps { let x = (k as f64 + 0.5) / s.steps as f64; r.phase(e, dt * x); checked(&r, e); r.mix(enc, dt * (1.0 - x)); checked(&r, e); } + r +} +/// Round 2: QAOA on an encoding: the uniform start, then per layer (gamma, beta) the phase layer and the mixer layer +pub fn qaoa_on(enc: &Encoding, e: &[f64], params: &[(f64, f64)]) -> Register { + let mut r = Register::uniform(enc); + for &(g, b) in params { r.phase(e, g); checked(&r, e); r.mix(enc, b); checked(&r, e); } + r +} +/// The sweep on an Ising program (every variable two-valued): the same phase and mixer layers as `sweep` +pub fn evolve_ising(p: &Ising, s: &Schedule) -> Register { sweep(&Encoding::two_valued(p.q), &p.energies(), s) } +/// QAOA with p layers on an Ising program: the same layers as `qaoa_on` +pub fn qaoa(p: &Ising, layers: usize, params: &[(f64, f64)]) -> Register { + assert_eq!(params.len(), layers, "one (gamma, beta) per layer"); qaoa_on(&Encoding::two_valued(p.q), &p.energies(), params) +} + +/// Round 4: what `amplify_exact` did: the iterations (standard ones + the final phase-matched one), the final step's two +/// phases (oracle, diffusion; radians), the Born mass on the good states before and after +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct Amplified { pub iterations: usize, pub phases: (f64, f64), pub mass_before: f64, pub mass_after: f64 } +/// The bar on the good states' mass after `amplify_exact` (asserted) +pub const AMPLIFIED_MASS_MIN: f64 = 1.0 - 1e-12; +/// Round 4: exact amplitude amplification of the good states (`good(x)` per basis index; an unused code is never good) about +/// the register's current state psi, in place of a projection. With a0 = the good mass of psi and theta = asin(sqrt(a0)), +/// k = floor(pi / (4 theta) - 1/2) standard iterations (oracle phase pi, diffusion phase pi about psi) bring the state to angle +/// (2k + 1) theta <= pi/2 in the plane of |good> (psi's normalised good part) and |bad>; one final iteration with phases +/// (phi, varphi) chosen so that its bad component is 0 lands on |good> exactly (Long's exact search, one matched step). Every +/// step stays in that plane, so the result is e^{i chi} |good>: the good states keep psi's relative amplitudes, the Born +/// probabilities are psi's restricted to the good states and divided by a0 -- the projection's numbers, by a unitary +/// evolution. a0 = 1: nothing to do (0 iterations); a0 = 0: None (nothing to amplify). Round 5: a landing whose mass after is +/// below 1 - 1e-12 (rounding drift over many iterations at a tiny a0, measured with the long sweep) is also None, so the tier +/// declines to the default tiers instead of panicking; the register is then left mid-rotation and must not be read. +/// Round 7: psi is renormalised once before the standard iterations, when there are any (its squared norm summed with +/// compensation; a NaN mass declines like a miss). The sweep's layers +/// leave it 1 + eps, a larger eps the more layers; the diffusion about such a psi is not unitary (|D v|^2 = |v|^2 + 4 eps ||^2), +/// so the error grows with every iteration, linearly in k: that, not the rotation, was the ~1e-12 shortfall of the long sweep's +/// 165 corpus programs (k in the hundreds at a tiny a0), and the same eps of the other sign is round 4's drift above 1. +pub fn amplify_exact(r: &mut Register, good: impl Fn(usize) -> bool) -> Option { + let g: Vec = (0..r.amp.len()).map(&good).collect(); let mass = |r: &Register| r.amp.iter().zip(&g).filter(|(_, &x)| x).map(|(a, _)| a.norm2()).sum::(); + let a0 = mass(r); if !(a0 > 0.0) { return None; } + if a0 >= 1.0 - 1e-15 { return Some(Amplified { iterations: 0, phases: (0.0, 0.0), mass_before: a0, mass_after: a0 }); } + let pi = core::f64::consts::PI; let k_of = |a0: f64| m::floor(pi / (4.0 * m::asin(m::sqrt(a0))) - 0.5).max(0.0) as usize; + // renormalise only when there are standard iterations to drift over (k >= 1, so a0 <= 1/4): with none, the one matched step + // is round 4's, unchanged, and a mass near 1 cannot round above 1. The receipt's mass before stays the swept state's (a0); + // the rotation uses the renormalised state's. + let mass_before = a0; + let a0 = if k_of(a0) == 0 { a0 } else { let u = 1.0 / m::sqrt(compensated_norm2(&r.amp)); for a in r.amp.iter_mut() { *a = C::new(a.re * u, a.im * u); } mass(r) }; + let psi = r.amp.clone(); + let k = k_of(a0); + for _ in 0..k { r.oracle(&g, pi); r.diffuse(&psi, pi); } + // the final step: the state is +-(al |good> + be |bad>), al, be >= 0; with s = sin theta, c = cos theta, S_psi(varphi) + // S_f(phi) leaves be - (1 - e^{i varphi}) c (al s e^{i phi} + be c) on |bad>: it vanishes for + // cos phi = -(be / al) cos(2 theta) / sin(2 theta) (|cos phi| < 1 for this k) and 1 - e^{i varphi} = be / (c (al s e^{i phi} + be c)) + let a = mass(r).min(1.0); let (al, be) = (m::sqrt(a), m::sqrt((1.0 - a).max(0.0))); let (s, c) = (m::sqrt(a0), m::sqrt(1.0 - a0)); + let cphi = (-(be / al) * (1.0 - 2.0 * a0) / (2.0 * s * c)).clamp(-1.0, 1.0); let phi = m::acos(cphi); + let w = C::new(al * s * m::cos(phi) + be * c, al * s * m::sin(phi)); let den = w.norm2() * c; // z = be / (c w) = be conj(w) / (c |w|^2) + let z = C::new(be * w.re / den, -be * w.im / den); let vphi = m::atan2(-z.im, 1.0 - z.re); + r.oracle(&g, phi); r.diffuse(&psi, vphi); + let m = mass(r); if !(m >= AMPLIFIED_MASS_MIN) { return None; } // a NaN declines too + Some(Amplified { iterations: k + 1, phases: (phi, vphi), mass_before, mass_after: m }) +} + +/// Round 7: sum |a_x|^2 with Neumaier's compensation (the plain running sum of 2^16 terms can be off by ~1e-14, the size of the +/// drift the renormalisation removes) +fn compensated_norm2(amp: &[C]) -> f64 { + let (mut s, mut c) = (0.0f64, 0.0f64); + for a in amp { let x = a.norm2(); let t = s + x; c += if s >= x { (s - t) + x } else { (x - t) + s }; s = t; } // s, x >= 0 + s + c +} + +/// sha256 (FIPS 180-4), one shot +pub fn sha256(data: &[u8]) -> [u8; 32] { + const K: [u32; 64] = [0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5, 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, + 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174, 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da, 0x983e5152, 0xa831c66d, + 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967, 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85, + 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070, 0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, + 0x5b9cca4f, 0x682e6ff3, 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2]; + let mut h: [u32; 8] = [0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19]; + let mut msg = data.to_vec(); msg.push(0x80); while msg.len() % 64 != 56 { msg.push(0); } msg.extend_from_slice(&((data.len() as u64) * 8).to_be_bytes()); + for blk in msg.chunks(64) { + let mut w = [0u32; 64]; + for i in 0..16 { w[i] = u32::from_be_bytes([blk[4 * i], blk[4 * i + 1], blk[4 * i + 2], blk[4 * i + 3]]); } + for i in 16..64 { let s0 = w[i - 15].rotate_right(7) ^ w[i - 15].rotate_right(18) ^ (w[i - 15] >> 3); let s1 = w[i - 2].rotate_right(17) ^ w[i - 2].rotate_right(19) ^ (w[i - 2] >> 10); + w[i] = w[i - 16].wrapping_add(s0).wrapping_add(w[i - 7]).wrapping_add(s1); } + let mut v = h; + for i in 0..64 { let s1 = v[4].rotate_right(6) ^ v[4].rotate_right(11) ^ v[4].rotate_right(25); let ch = (v[4] & v[5]) ^ (!v[4] & v[6]); + let t1 = v[7].wrapping_add(s1).wrapping_add(ch).wrapping_add(K[i]).wrapping_add(w[i]); + let s0 = v[0].rotate_right(2) ^ v[0].rotate_right(13) ^ v[0].rotate_right(22); let mj = (v[0] & v[1]) ^ (v[0] & v[2]) ^ (v[1] & v[2]); let t2 = s0.wrapping_add(mj); + v = [t1.wrapping_add(t2), v[0], v[1], v[2], v[3].wrapping_add(t1), v[4], v[5], v[6]]; } + for i in 0..8 { h[i] = h[i].wrapping_add(v[i]); } + } + let mut out = [0u8; 32]; for i in 0..8 { out[4 * i..4 * i + 4].copy_from_slice(&h[i].to_be_bytes()); } out +} +/// Lower-case hex +pub fn hex(b: &[u8]) -> String { b.iter().map(|x| format!("{x:02x}")).collect() } + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn every_gate_is_unitary_to_1e_12() { + for t in [0.0, 0.3, -1.7, 2.5, 3.14159, 10.0] { + for g in [Gate::H(0), Gate::X(0), Gate::Z(0), Gate::Rx(0, t), Gate::Rz(0, t), Gate::Cnot(0, 1), Gate::Cz(0, 1), Gate::Rzz(0, 1, t)] { assert!(unitarity_error(&g) < 1e-12, "{g:?}"); } + } + } + #[test] + fn memory_is_16_times_2_to_the_q() { for q in [0, 1, 8, 12] { assert_eq!(Register::new(q).bytes(), 16 << q); } } + #[test] + fn bell_state_interference_and_norm() { + let mut r = Register::new(2); r.apply(Gate::H(0)); r.apply(Gate::Cnot(0, 1)); + let p = r.probabilities(); assert!((p[0] - 0.5).abs() < 1e-15 && (p[3] - 0.5).abs() < 1e-15 && p[1] < 1e-30 && p[2] < 1e-30); + // H Z H = X; two H in a row cancel (destructive interference on |1>) + let mut r = Register::new(1); r.apply(Gate::H(0)); r.apply(Gate::H(0)); assert!(r.probabilities()[1] < 1e-30); + let mut r = Register::new(1); r.apply(Gate::H(0)); r.apply(Gate::Z(0)); r.apply(Gate::H(0)); assert!((r.probabilities()[1] - 1.0).abs() < 1e-15); + let p = Ising { q: 3, a: vec![0.4, -0.2, 0.1], b: vec![(0, 1, 0.6), (1, 2, -0.3)] }; + let r = evolve_ising(&p, &DEFAULT_SCHEDULE); assert!((r.probabilities().iter().sum::() - 1.0).abs() < 1e-12); + let r = qaoa(&p, 2, &[(0.4, 0.7), (0.2, 0.3)]); assert!((r.probabilities().iter().sum::() - 1.0).abs() < 1e-12); + } + #[test] + fn three_level_codes_stay_empty_and_the_mixer_is_unitary() { + for d in 1..=8 { for t in [0.0, 0.3, -1.7, 2.5, 10.0] { assert!(mixer_unitarity_error(d, t) < 1e-12, "d {d} t {t}"); } } + assert_eq!((Encoding::bits(1), Encoding::bits(2), Encoding::bits(3), Encoding::bits(4), Encoding::bits(5)), (0, 1, 2, 2, 3)); + // d = 2: the complete-graph mixer is RX(-2t) exactly + let m = mixer_matrix(2, 0.7); let g = Gate::Rx(0, -1.4).matrix(); for k in 0..4 { assert!((m[k].re - g[k].re).abs() < 1e-15 && (m[k].im - g[k].im).abs() < 1e-15); } + // a three-level variable, a two-valued flag and a three-level variable with a pair table favouring the diagonal + let enc = Encoding { levels: vec![3, 2, 3] }; assert_eq!(enc.qubits(), 5); + let lw = |v: &[usize]| [0.4, -0.2, 0.9][v[0]] + if v[1] == 1 { 0.3 } else { 0.0 } + if v[0] == v[2] { 1.1 } else { 0.0 } + if v[1] == 1 && v[2] == 2 { -0.5 } else { 0.0 }; + let e: Vec = (0..32).map(|x| enc.decode(x).map_or(f64::NAN, |v| -lw(&v))).collect(); + let r = sweep(&enc, &e, &DEFAULT_SCHEDULE); assert!(r.invalid_mass(&e) == 0.0); + assert!((r.probabilities().iter().sum::() - 1.0).abs() < 1e-12); + let lm = r.level_marginals(&enc); assert_eq!(lm.len(), 8); for (o, d) in [(0, 3), (3, 2), (5, 3)] { assert!((lm[o..o + d].iter().sum::() - 1.0).abs() < 1e-12); } + let r = qaoa_on(&enc, &e, &[(0.4, 0.7), (0.9, 0.2)]); assert!(r.invalid_mass(&e) == 0.0); + // the uniform start over the valid codes: 18 of 32 basis states + assert_eq!(Register::uniform(&enc).probabilities().iter().filter(|&&p| p > 0.0).count(), 18); + } + #[test] + fn exact_amplification_lands_on_the_projection() { + // a three-level, a two-valued and a three-level variable; good = a rule over the plans; masses from 0.5 down to ~1e-3 + let enc = Encoding { levels: vec![3, 2, 3] }; + let lw = |v: &[usize]| [0.4, -0.2, 0.9][v[0]] + if v[1] == 1 { 0.3 } else { 0.0 } + if v[0] == v[2] { 1.1 } else { 0.0 } + if v[1] == 1 && v[2] == 2 { -0.5 } else { 0.0 }; + let e: Vec = (0..32).map(|x| enc.decode(x).map_or(f64::NAN, |v| -lw(&v))).collect(); + for rule in 0..18usize { + let good = |x: usize| enc.decode(x).is_some_and(|v| (v[0] * 6 + v[1] * 3 + v[2] + rule) % 18 < 1 + rule % 9); + let sw = sweep(&enc, &e, &DEFAULT_SCHEDULE); let p = sw.probabilities(); let a0: f64 = (0..32).filter(|&x| good(x)).map(|x| p[x]).sum(); + let mut r = sw.clone(); let am = amplify_exact(&mut r, good).unwrap(); assert!(am.mass_after >= AMPLIFIED_MASS_MIN && (am.mass_before - a0).abs() < 1e-15); + let q = r.probabilities(); for x in 0..32 { let want = if good(x) { p[x] / a0 } else { 0.0 }; assert!((q[x] - want).abs() < 1e-12, "rule {rule} x {x}: {} vs {want}", q[x]); } + assert_eq!(r.invalid_mass(&e), 0.0); + } + let mut r = Register::uniform(&enc); assert!(amplify_exact(&mut r, |_| false).is_none()); + let mut r = Register::uniform(&enc); assert_eq!(amplify_exact(&mut r, |x| enc.decode(x).is_some()).unwrap().iterations, 0); + } + /// Round 5: the named evolutions: the default name is `sweep` with the default schedule (the same register, text and name), + /// the long sweep has 4x the layers, qaoa-p2 two layers with the frozen angles; an unknown name is None + #[test] + fn named_schedules_run_what_they_name() { + let enc = Encoding { levels: vec![3, 2] }; let e: Vec = (0..8).map(|x| enc.decode(x).map_or(f64::NAN, |v| -[0.4, -0.2, 0.9][v[0]] - 0.3 * v[1] as f64)).collect(); + let d = Evolution::named(SCHEDULE_NAMES[0]).unwrap(); assert_eq!(d, Evolution::Sweep(DEFAULT_SCHEDULE)); + assert_eq!(d.run(&enc, &e).amplitudes(), sweep(&enc, &e, &DEFAULT_SCHEDULE).amplitudes()); assert_eq!((d.text(), d.name()), (DEFAULT_SCHEDULE.text(), DEFAULT_SCHEDULE.name())); + let l = Evolution::named(SCHEDULE_NAMES[1]).unwrap(); assert_eq!(l.run(&enc, &e).circuit_text().lines().count(), 4 * d.run(&enc, &e).circuit_text().lines().count()); + let q = Evolution::named(SCHEDULE_NAMES[2]).unwrap(); assert_eq!(q.name(), "qaoa-p2"); assert_eq!(q.run(&enc, &e).amplitudes(), qaoa_on(&enc, &e, &QAOA_P2).amplitudes()); + assert_eq!(q.text(), "qaoa layers 2\ngamma 0.500000 beta 0.500000\ngamma 1.500000 beta 0.250000\n"); assert!(Evolution::named("tf-linear-8x2").is_none()); + } + /// Round 7: a psi whose squared norm is off by 1e-12 (a long sweep leaves ~1e-14) and one good state of 2^14, so about a + /// hundred iterations: without the renormalisation the landing drifts by ~2 k eps (~2e-10); with it the mass stays within + /// 1e-12 of 1 and the good state's probability is 1 + #[test] + fn a_landing_after_a_hundred_iterations_keeps_its_mass() { + let enc = Encoding::two_valued(14); let mut r = Register::uniform(&enc); let f = (1.0 + 1e-12f64).sqrt(); + for a in r.amp.iter_mut() { *a = C::new(a.re * f, a.im * f); } + let am = amplify_exact(&mut r, |x| x == 5).unwrap(); assert!(am.iterations > 90, "{}", am.iterations); + assert!((am.mass_after - 1.0).abs() <= 1e-12, "mass after {:e}", am.mass_after - 1.0); assert!((r.probabilities()[5] - 1.0).abs() <= 1e-12); + assert!((compensated_norm2(r.amplitudes()) - 1.0).abs() <= 1e-12); + } + #[test] + fn measure_and_digest_are_deterministic() { + let p = Ising { q: 4, a: vec![0.5, 0.1, -0.3, 0.2], b: vec![(0, 3, 0.4)] }; + let (r1, r2) = (evolve_ising(&p, &DEFAULT_SCHEDULE), evolve_ising(&p, &DEFAULT_SCHEDULE)); + assert_eq!(r1.digest(), r2.digest()); assert_eq!(r1.measure(7), r2.measure(7)); assert_eq!(r1.circuit_text(), r2.circuit_text()); + assert_eq!(hex(&sha256(b"abc")), "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"); + assert_eq!(f6(-0.0000001), "0.000000"); + } +} diff --git a/probbit-decide/src/ir.rs b/probbit-decide/src/ir.rs index 310db47..04b4cc2 100644 --- a/probbit-decide/src/ir.rs +++ b/probbit-decide/src/ir.rs @@ -1,6 +1,8 @@ //! probbit-ir v0: a portable, line-based text form of a constrained Potts decision problem, plus a lowering to a //! binary one-hot QUBO/Ising "p-bit topology" (what a hardware p-bit fabric natively samples). //! Floats are written as IEEE-754 bit patterns (hex) so the round trip is bit-exact. +#[allow(unused_imports)] +use std::prelude::rust_2021::*; use crate::Problem; pub fn to_ir(p: &Problem) -> String { diff --git a/probbit-decide/src/lib.rs b/probbit-decide/src/lib.rs index 088c65f..add429c 100644 --- a/probbit-decide/src/lib.rs +++ b/probbit-decide/src/lib.rs @@ -6,10 +6,22 @@ //! Router: exact enumeration when the feasible set is small, else constraint-preserving //! Gibbs (site heat-bath restricted to feasible values + Metropolis swap moves) with //! multi-chain split-R-hat gate -> "UNMIXED: escalate" refusal. +#![cfg_attr(not(feature = "std"), no_std)] +#[cfg(not(feature = "std"))] +#[allow(unused_imports)] +#[macro_use] +extern crate probbit_nostd as std; +#[cfg(not(feature = "std"))] +#[allow(unused_imports)] +#[macro_use] +extern crate alloc; +#[allow(unused_imports)] +use std::prelude::rust_2021::*; use probbit_core::Philox4x32; use std::collections::HashMap; pub mod oracle; pub mod ir; +pub mod amp; /// Tasks up to which `Problem::logw` uses the pairwise loop. R19.8 re-measured (`examples/logw_threshold.rs`, 6 workers, groups /// of 4 / 8, median of 7 alternating rounds, load ~8.6): linear / pairwise ns per call = 1.52 / 1.25 at 96 tasks, 0.96 / 0.92 at diff --git a/probbit-decide/src/oracle.rs b/probbit-decide/src/oracle.rs index cb1390c..6239d6d 100644 --- a/probbit-decide/src/oracle.rs +++ b/probbit-decide/src/oracle.rs @@ -2,6 +2,8 @@ //! Chain-of-blocks Dispatch: B blocks x M=10 tasks. Block b may use 2 private agents plus the "bridge" //! agents shared with blocks b-1 and b+1 (capacity couples neighbours). Groups of 5 within a block, affinity lam. //! Exact marginals + bridge-load distributions by transfer-matrix DP over bridge loads. +#[allow(unused_imports)] +use std::prelude::rust_2021::*; use crate::Problem; use probbit_core::Philox4x32; diff --git a/probbit-ir/Cargo.toml b/probbit-ir/Cargo.toml index 50588f8..1ba234d 100644 --- a/probbit-ir/Cargo.toml +++ b/probbit-ir/Cargo.toml @@ -10,5 +10,12 @@ readme = "../README.md" keywords = ["p-bit", "constraints", "gibbs-sampling", "probabilistic", "ising"] categories = ["algorithms", "science", "mathematics"] +[features] +default = ["std"] +std = ["probbit-core/std"] +# no_std + alloc (a target without an operating system): the std subset comes from probbit-nostd +nostd = ["dep:probbit-nostd", "probbit-core/nostd"] + [dependencies] -probbit-core = { path = "../probbit-core", version = "0.13.0" } +probbit-nostd = { path = "../probbit-nostd", version = "0.14.0-dev", optional = true } +probbit-core = { path = "../probbit-core", default-features = false, version = "0.14.0-dev" } diff --git a/probbit-ir/src/lib.rs b/probbit-ir/src/lib.rs index 6b340d2..2bca0b4 100644 --- a/probbit-ir/src/lib.rs +++ b/probbit-ir/src/lib.rs @@ -8,6 +8,17 @@ //! updates restricted to feasible values + Metropolis swap moves, multi-chain), `gate_stats` (the certification gate //! over generic marginals), `Model::with_clamp` (what-if). Front-ends lower to this form: the assignment router in //! probbit-decide is one (`Problem::lower`), and its results through the lowering are bit-identical to its own engine. +#![cfg_attr(not(feature = "std"), no_std)] +#[cfg(not(feature = "std"))] +#[allow(unused_imports)] +#[macro_use] +extern crate probbit_nostd as std; +#[cfg(not(feature = "std"))] +#[allow(unused_imports)] +#[macro_use] +extern crate alloc; +#[allow(unused_imports)] +use std::prelude::rust_2021::*; use probbit_core::Philox4x32; use std::collections::HashMap; @@ -627,7 +638,26 @@ pub fn exact_components_until(m: &Model, limit: u64, max_states: usize, hard: Op if past(hard) { return None; } deep(m.n, || components(m, limit, max_states, hard)) } +/// T3-c: `exact_components_until` with the split of a model that differs from the split's only in `h` (RFC 0009's members): +/// the components, their kinds and the sub-models of the components with caps or cycles (their pairs and caps) are the +/// split's, each sub-model's `h` copied from `m`; the same tiers on the same numbers, so the same answer bit for bit. The +/// split is built on the first call (`Split::default()`) and is the caller's to drop when the pairs, caps, allowed values or +/// clamps change +pub fn exact_components_split(m: &Model, s: &mut Split, limit: u64, max_states: usize, hard: Option) -> Option { + if past(hard) { return None; } + deep(m.n, || { if s.mems.is_empty() { *s = split(m); s.keep = true; } solve_split(m, s, limit, max_states, hard) }) +} +/// The components tier's split of a model, what does not read `h` (`exact_components_split`) +#[derive(Default)] +pub struct Split { mems: Vec>, comp_of: Vec, tree: Vec, counted: Vec>, ccaps: Vec>, fwork: f64, + /// per component with caps or cycles its sub-model, built on its first use (Some(None): `Model::new` refused it), kept for the + /// next call with `keep` (else dropped once solved, as before) + subs: Vec>>, keep: bool } fn components(m: &Model, limit: u64, max_states: usize, hard: Option) -> Option { + let mut s = split(m); solve_split(m, &mut s, limit, max_states, hard) +} +#[inline(never)] +fn split(m: &Model) -> Split { let (n, k) = (m.n, m.k); let mut par: Vec = (0..n).collect(); fn find(par: &mut [usize], mut x: usize) -> usize { while par[x] != x { par[x] = par[par[x]]; x = par[x]; } x } fn join(par: &mut [usize], a: usize, b: usize) { let (a, b) = (find(par, a), find(par, b)); if a != b { par[a.max(b)] = a.min(b); } } @@ -641,28 +671,41 @@ fn components(m: &Model, limit: u64, max_states: usize, hard: Option> = vec![vec![]; mems.len()]; for (q, p) in m.pairs.iter().enumerate() { cpairs[comp_of[p.i]].push(q); } let counted: Vec> = (0..mems.len()).map(|c| if tree(c) { None } else { count_spec(m, &mems[c], &ccaps[c], &cpairs[c]) }).collect(); - if mems.len() == 1 && !tree(0) && counted[0].is_none() { return None; } let vals = |i: usize| -> Vec { m.cand[i].iter().copied().filter(|&v| m.ok(i, v)).collect() }; let fwork: f64 = m.pairs.iter().filter(|p| tree(comp_of[p.i])).map(|p| (vals(p.i).len() * vals(p.j).len()) as f64).sum::() + (0..n).filter(|&i| tree(comp_of[i])).count() as f64 * k as f64; - if fwork > FOREST_MAX_WORK { return None; } + let tree: Vec = (0..mems.len()).map(tree).collect(); let subs = (0..mems.len()).map(|_| None).collect(); + Split { mems, comp_of, tree, counted, ccaps, fwork, subs, keep: false } +} +fn solve_split(m: &Model, s: &mut Split, limit: u64, max_states: usize, hard: Option) -> Option { + let (n, k) = (m.n, m.k); + let Split { mems, comp_of, tree, counted, ccaps, fwork, subs, keep } = s; + if mems.len() == 1 && !tree[0] && counted[0].is_none() { return None; } + let vals = |i: usize| -> Vec { m.cand[i].iter().copied().filter(|&v| m.ok(i, v)).collect() }; + if *fwork > FOREST_MAX_WORK { return None; } let mut out = CompExact { marg: vec![0.0; n * k], map: vec![0; n], map_logw: 0.0, logz: 0.0, components: mems.len(), forest: 0, enumerated: 0, frontier: 0, occupancy: 0, infeasible: false }; let mut enum_left = limit as f64; for (c, mem) in mems.iter().enumerate() { if past(hard) { return None; } - if tree(c) { out.forest += 1; + if tree[c] { out.forest += 1; let z = tree_bp(m, mem, &vals, &mut out.marg, &mut out.map); if z == f64::NEG_INFINITY { out.infeasible = true; return Some(out); } out.logz += z; continue; } if let Some(sp) = &counted[c] { out.occupancy += 1; let z = count_dp(m, mem, sp, &mut out.marg, &mut out.map); if z == f64::NEG_INFINITY { out.infeasible = true; return Some(out); } out.logz += z; continue; } // a component with caps or cycles: a sub-model of its own variables, pairs and caps, solved by the whole-program tiers - let mut loc = vec![usize::MAX; n]; for (q, &i) in mem.iter().enumerate() { loc[i] = q; } - let nn = mem.len(); let mut h = Vec::with_capacity(nn * k); let mut al = Vec::with_capacity(nn * k); - for &i in mem { h.extend_from_slice(&m.h[i * k..i * k + k]); al.extend_from_slice(&m.allowed[i * k..i * k + k]); } - let pairs: Vec = m.pairs.iter().filter(|p| comp_of[p.i] == c).map(|p| Pair { i: loc[p.i], j: loc[p.j], c: p.c.clone() }).collect(); - let caps: Vec = ccaps[c].iter().map(|&q| Cap { members: m.caps[q].members.iter().map(|&(i, v)| (loc[i], v)).collect(), limit: m.caps[q].limit, weights: m.caps[q].weights.clone() }).collect(); - let sub = Model::new(nn, k, h, al, mem.iter().map(|&i| m.clamp[i]).collect(), pairs, caps).ok()?; + // (built once per split; a later call copies its variables' `h` in) + let nn = mem.len(); + if let Some(Some(sub)) = &mut subs[c] { for (q, &i) in mem.iter().enumerate() { sub.h[q * k..q * k + k].copy_from_slice(&m.h[i * k..i * k + k]); } + if sub.h.iter().any(|x| !x.is_finite()) { return None; } } + else if subs[c].is_none() { + let mut loc = vec![usize::MAX; n]; for (q, &i) in mem.iter().enumerate() { loc[i] = q; } + let mut h = Vec::with_capacity(nn * k); let mut al = Vec::with_capacity(nn * k); + for &i in mem.iter() { h.extend_from_slice(&m.h[i * k..i * k + k]); al.extend_from_slice(&m.allowed[i * k..i * k + k]); } + let pairs: Vec = m.pairs.iter().filter(|p| comp_of[p.i] == c).map(|p| Pair { i: loc[p.i], j: loc[p.j], c: p.c.clone() }).collect(); + let caps: Vec = ccaps[c].iter().map(|&q| Cap { members: m.caps[q].members.iter().map(|&(i, v)| (loc[i], v)).collect(), limit: m.caps[q].limit, weights: m.caps[q].weights.clone() }).collect(); + subs[c] = Some(Model::new(nn, k, h, al, mem.iter().map(|&i| m.clamp[i]).collect(), pairs, caps).ok()); } + let Some(Some(sub)) = &subs[c] else { return None }; // Enumeration only where the component's RAW space fits what is left of one shared budget of `limit` (so the tier // never spends more than one whole-program enumeration: per-component node budgets on many hard components would // add up); larger components go to the frontier DP (bounded by `max_states`) or the tier declines. @@ -673,6 +716,7 @@ fn components(m: &Model, limit: u64, max_states: usize, hard: Option 0 { got = exact_frontier_until(&sub, max_states, hard).map(|f| (f.marg, f.map, f.logz)); if got.is_some() { out.frontier += 1; } } let (mg, mp, z) = got?; + if !*keep { subs[c] = None; } for (q, &i) in mem.iter().enumerate() { out.map[i] = mp[q]; out.marg[i * k..i * k + k].copy_from_slice(&mg[q * k..q * k + k]); } out.logz += z; } @@ -2369,6 +2413,39 @@ mod tests { assert!(a.marg.iter().zip(&b.marg).all(|(x, y)| x.to_bits() == y.to_bits()), "threads {t}"); assert_eq!(a.traj, b.traj); assert_eq!(a.plans, b.plans); } } + /// T3-c: one split answers models that differ only in `h` as each one's own call does, bit for bit (marginals, log Z, the MAP + /// and its log w, infeasibility, the tier counts): the random programs of the test below (forests, cycles, caps, clamps, + /// forbidden values), each split reused over five `h` + #[test] + fn one_split_over_many_h_is_bit_identical() { + let (mut compared, mut declined) = (0, 0); + for seed in 0..400u64 { + let mut r = Philox4x32::new(seed, 23); let (n, k) = (5 + r.below(6), 2 + r.below(2)); + let h0: Vec = (0..n * k).map(|_| r.f64() * 3.0 - 1.5).collect(); + let mut allowed: Vec = (0..n * k).map(|_| r.f64() < 0.8).collect(); for i in 0..n { allowed[i * k + r.below(k)] = true; } + let clamp: Vec> = (0..n).map(|i| if r.f64() < 0.1 { (0..k).find(|&v| allowed[i * k + v]) } else { None }).collect(); + let coup = |r: &mut Philox4x32| if r.f64() < 0.5 { Coupling::Potts(r.f64() * 3.0 - 1.5) } else { Coupling::Table((0..k * k).map(|_| r.f64() * 2.0 - 1.0).collect()) }; + let mut pairs = vec![]; for i in 1..n { if r.f64() < 0.75 { let j = r.below(i); pairs.push(Pair { i, j, c: coup(&mut r) }); } } + if r.f64() < 0.3 { let (i, j) = (r.below(n), r.below(n)); if i != j { pairs.push(Pair { i, j, c: coup(&mut r) }); } } + let caps: Vec = if r.f64() < 0.4 { let mut mem: Vec<(usize, usize)> = (0..n * k).filter(|_| r.f64() < 0.25).map(|q| (q / k, q % k)).collect(); mem.dedup(); + if mem.is_empty() { vec![] } else { vec![Cap { weights: vec![], limit: r.below(mem.len()), members: mem }] } } else { vec![] }; + let mut m = Model::new(n, k, h0, allowed, clamp, pairs, caps).unwrap(); let mut s = Split::default(); + for v in 0..5 { + if v > 0 { m.h = (0..n * k).map(|_| r.f64() * 4.0 - 2.0).collect(); } + let a = exact_components_split(&m, &mut s, 1 << 22, FRONTIER_MAX_STATES, None); + let fresh = Model::new(n, k, m.h.clone(), m.allowed.clone(), m.clamp.clone(), m.pairs.clone(), m.caps.clone()).unwrap(); + let b = exact_components_until(&fresh, 1 << 22, FRONTIER_MAX_STATES, None); + let (Some(a), Some(b)) = (a.as_ref(), b.as_ref()) else { assert_eq!(a.is_none(), b.is_none(), "seed {seed} h {v}: one declined"); declined += 1; continue }; + let bits = |x: &[f64]| x.iter().map(|y| y.to_bits()).collect::>(); + assert_eq!(a.infeasible, b.infeasible, "seed {seed} h {v}"); assert_eq!(bits(&a.marg), bits(&b.marg), "seed {seed} h {v}: marginals"); + assert_eq!(a.logz.to_bits(), b.logz.to_bits(), "seed {seed} h {v}: log Z"); assert_eq!(a.map, b.map, "seed {seed} h {v}: MAP"); + assert_eq!(a.map_logw.to_bits(), b.map_logw.to_bits(), "seed {seed} h {v}: MAP log w"); + assert_eq!((a.components, a.forest, a.enumerated, a.frontier, a.occupancy), (b.components, b.forest, b.enumerated, b.frontier, b.occupancy), "seed {seed} h {v}: tiers"); + compared += 1; } + } + assert!(compared > 1000, "compared {compared}, declined {declined}"); + } + /// R19 P1.3(b, c): the components + forest tier agrees with whole-program enumeration (log Z, every odds value, the MAP's /// log w, infeasibility) on random programs: forests with forbidden values and clamps, plus cyclic and capped components /// (solved by enumeration / the frontier inside the tier); and a 1,000-variable chain's log Z matches a transfer matrix. diff --git a/probbit-nostd/Cargo.toml b/probbit-nostd/Cargo.toml new file mode 100644 index 0000000..c903dbe --- /dev/null +++ b/probbit-nostd/Cargo.toml @@ -0,0 +1,20 @@ +[package] +name = "probbit-nostd" +description = "The subset of std that probbit's crates use, from core + alloc, for targets without an operating system (no_std builds only)" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +readme = "../README.md" +keywords = ["p-bit", "no-std", "embedded", "probabilistic"] +categories = ["no-std", "embedded"] + +[lib] +test = false +doctest = false + +[dependencies] +hashbrown = { version = "0.15", default-features = false, features = ["default-hasher", "inline-more"] } +libm = "0.2" +portable-atomic = { version = "1", default-features = false, features = ["fallback", "critical-section"] } diff --git a/probbit-nostd/src/lib.rs b/probbit-nostd/src/lib.rs new file mode 100644 index 0000000..8df3cd7 --- /dev/null +++ b/probbit-nostd/src/lib.rs @@ -0,0 +1,167 @@ +//! probbit-nostd: the subset of `std` that probbit-core, probbit-ir, probbit-decide and probbit-persona use, built from `core`, +//! `alloc`, `hashbrown`, `libm` and `portable-atomic` for a target without an operating system (xtensa-esp32s3-none-elf). +//! A crate that is `no_std` imports it as `std` (`#[macro_use] extern crate probbit_nostd as std;`), so its `std::` paths compile +//! unchanged. Threads run in order on the calling thread (the engine's `rt::sequential()` mode); `thread_local!` is a static +//! (one core runs the engine); files and sleeping are absent (`fs::read` fails, `sleep` returns at once). +#![no_std] +extern crate alloc; + +#[allow(deprecated)] // the integer modules (`std::u64::MAX` style paths) are deprecated in std too, and still exist there +pub use core::{any, array, ascii, cell, char, clone, cmp, convert, default, f32, f64, hash, hint, i16, i32, i64, i8, + isize, iter, marker, mem, num, ops, option, ptr, result, slice, str, u16, u32, u64, u8, usize}; +pub use alloc::{borrow, boxed, fmt, rc, string, vec}; + +pub mod time { pub use core::time::Duration; } + +pub mod collections { + pub use alloc::collections::*; + pub use hashbrown::{HashMap, HashSet}; + pub mod hash_map { pub use hashbrown::hash_map::*; } + pub mod hash_set { pub use hashbrown::hash_set::*; } +} + +pub mod sync { + pub use alloc::sync::*; + pub mod atomic { + pub use core::sync::atomic::{compiler_fence, fence, AtomicBool, AtomicI32, AtomicIsize, AtomicU32, AtomicUsize, Ordering}; + pub use portable_atomic::{AtomicI64, AtomicU64}; + } +} + +pub mod io { + /// The only I/O error here: there is no operating system. + #[derive(Debug, Clone, Copy)] + pub struct Error; + impl core::fmt::Display for Error { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.write_str("no operating system on this target") } } + pub type Result = core::result::Result; + static PRINT: core::sync::atomic::AtomicUsize = core::sync::atomic::AtomicUsize::new(0); + /// Install the console writer used by `print!`/`println!`/`eprintln!` (the firmware passes esp-println's). + pub fn set_print(f: fn(&str)) { PRINT.store(f as usize, core::sync::atomic::Ordering::Relaxed) } + #[doc(hidden)] + pub fn _print(args: core::fmt::Arguments<'_>) { + let p = PRINT.load(core::sync::atomic::Ordering::Relaxed); + if p != 0 { let f: fn(&str) = unsafe { core::mem::transmute(p) }; f(&alloc::fmt::format(args)); } + } +} + +pub mod fs { + /// No filesystem on bare metal: the firmware passes documents in as strings. + pub fn read

(_path: P) -> crate::io::Result> { Err(crate::io::Error) } +} + +pub mod panic { + pub fn resume_unwind(_payload: alloc::boxed::Box) -> ! { panic!("a sequential section panicked") } +} + +pub mod thread { + use core::cell::UnsafeCell; + use core::marker::PhantomData; + use core::time::Duration; + /// `std::thread::scope` without threads: every spawn runs at once on the calling thread. + pub struct Scope<'scope, 'env: 'scope> { _p: PhantomData<(&'scope mut &'scope (), &'env mut &'env ())> } + pub fn scope<'env, F, T>(f: F) -> T where F: for<'scope> FnOnce(&'scope Scope<'scope, 'env>) -> T { + let s = Scope { _p: PhantomData }; f(&s) + } + impl<'scope, 'env> Scope<'scope, 'env> { + pub fn spawn(&'scope self, f: F) -> ScopedJoinHandle<'scope, T> where F: FnOnce() -> T + Send + 'scope, T: Send + 'scope { + ScopedJoinHandle { v: f(), _p: PhantomData } + } + } + pub struct ScopedJoinHandle<'scope, T> { v: T, _p: PhantomData<&'scope ()> } + impl<'scope, T> ScopedJoinHandle<'scope, T> { + pub fn join(self) -> core::result::Result> { Ok(self.v) } + } + pub struct Builder; + impl Builder { + pub fn new() -> Builder { Builder } + pub fn stack_size(self, _size: usize) -> Builder { self } + pub fn name(self, _name: alloc::string::String) -> Builder { self } + pub fn spawn_scoped<'scope, 'env, F, T>(self, s: &'scope Scope<'scope, 'env>, f: F) -> crate::io::Result> + where F: FnOnce() -> T + Send + 'scope, T: Send + 'scope { Ok(s.spawn(f)) } + } + pub fn available_parallelism() -> crate::io::Result { Ok(core::num::NonZeroUsize::MIN) } + pub fn sleep(_d: Duration) {} + /// `thread_local!` storage: one static, initialised on first use (the engine runs on one core). + pub struct LocalKey { init: fn() -> T, slot: UnsafeCell> } + unsafe impl Sync for LocalKey {} + impl LocalKey { + #[doc(hidden)] + pub const fn new(init: fn() -> T) -> Self { LocalKey { init, slot: UnsafeCell::new(None) } } + pub fn with R, R>(&'static self, f: F) -> R { + let slot = unsafe { &mut *self.slot.get() }; + if slot.is_none() { *slot = Some((self.init)()); } + f(slot.as_ref().unwrap()) + } + } + impl LocalKey> { + pub fn get(&'static self) -> T { self.with(|c| c.get()) } + pub fn set(&'static self, v: T) { self.with(|c| c.set(v)) } + pub fn replace(&'static self, v: T) -> T { self.with(|c| c.replace(v)) } + } +} + +#[macro_export] +macro_rules! thread_local { + () => {}; + ($(#[$a:meta])* $vis:vis static $name:ident : $t:ty = const { $init:expr } $(; $($rest:tt)*)?) => { + $(#[$a])* $vis static $name: $crate::thread::LocalKey<$t> = $crate::thread::LocalKey::new({ fn __init() -> $t { $init } __init }); + $($crate::thread_local!($($rest)*);)? + }; + ($(#[$a:meta])* $vis:vis static $name:ident : $t:ty = $init:expr $(; $($rest:tt)*)?) => { + $(#[$a])* $vis static $name: $crate::thread::LocalKey<$t> = $crate::thread::LocalKey::new({ fn __init() -> $t { $init } __init }); + $($crate::thread_local!($($rest)*);)? + }; +} +#[macro_export] +macro_rules! print { ($($t:tt)*) => { $crate::io::_print(format_args!($($t)*)) }; } +#[macro_export] +macro_rules! println { () => { $crate::io::_print(format_args!("\n")) }; ($($t:tt)*) => { $crate::io::_print(format_args!("{}\n", format_args!($($t)*))) }; } +#[macro_export] +macro_rules! eprint { ($($t:tt)*) => { $crate::io::_print(format_args!($($t)*)) }; } +#[macro_export] +macro_rules! eprintln { () => { $crate::io::_print(format_args!("\n")) }; ($($t:tt)*) => { $crate::io::_print(format_args!("{}\n", format_args!($($t)*))) }; } + +/// `f64`/`f32` methods that live in `std` (they call the platform libm) and not in `core`: here they call the `libm` crate. +/// Inherent methods win where `core` has them, so this trait only fills the gaps. +pub trait FloatExt: Sized { + fn exp(self) -> Self; fn exp_m1(self) -> Self; fn ln(self) -> Self; fn ln_1p(self) -> Self; fn log2(self) -> Self; fn log10(self) -> Self; + fn sqrt(self) -> Self; fn powf(self, e: Self) -> Self; fn powi(self, n: i32) -> Self; fn tanh(self) -> Self; fn cos(self) -> Self; fn sin(self) -> Self; + fn floor(self) -> Self; fn ceil(self) -> Self; fn round(self) -> Self; fn trunc(self) -> Self; fn fract(self) -> Self; + fn asin(self) -> Self; fn acos(self) -> Self; fn atan2(self, x: Self) -> Self; +} +/// `powi` as compiler-rt's `__powidf2` computes it (what `std`'s `f64::powi` lowers to), not `pow`: same bits as the host. +fn powi64(mut a: f64, mut b: i32) -> f64 { + let recip = b < 0; let mut r = 1.0; + loop { if b & 1 != 0 { r *= a; } b /= 2; if b == 0 { break; } a *= a; } + if recip { 1.0 / r } else { r } +} +impl FloatExt for f64 { + fn exp(self) -> f64 { libm::exp(self) } fn exp_m1(self) -> f64 { libm::expm1(self) } fn ln(self) -> f64 { libm::log(self) } + fn ln_1p(self) -> f64 { libm::log1p(self) } fn log2(self) -> f64 { libm::log2(self) } fn log10(self) -> f64 { libm::log10(self) } + fn sqrt(self) -> f64 { libm::sqrt(self) } fn powf(self, e: f64) -> f64 { libm::pow(self, e) } fn powi(self, n: i32) -> f64 { powi64(self, n) } + fn tanh(self) -> f64 { libm::tanh(self) } fn cos(self) -> f64 { libm::cos(self) } fn sin(self) -> f64 { libm::sin(self) } + fn floor(self) -> f64 { libm::floor(self) } fn ceil(self) -> f64 { libm::ceil(self) } fn round(self) -> f64 { libm::round(self) } + fn trunc(self) -> f64 { libm::trunc(self) } fn fract(self) -> f64 { self - libm::trunc(self) } + fn asin(self) -> f64 { libm::asin(self) } fn acos(self) -> f64 { libm::acos(self) } fn atan2(self, x: f64) -> f64 { libm::atan2(self, x) } +} +impl FloatExt for f32 { + fn exp(self) -> f32 { libm::expf(self) } fn exp_m1(self) -> f32 { libm::expm1f(self) } fn ln(self) -> f32 { libm::logf(self) } + fn ln_1p(self) -> f32 { libm::log1pf(self) } fn log2(self) -> f32 { libm::log2f(self) } fn log10(self) -> f32 { libm::log10f(self) } + fn sqrt(self) -> f32 { libm::sqrtf(self) } fn powf(self, e: f32) -> f32 { libm::powf(self, e) } fn powi(self, n: i32) -> f32 { powi64(self as f64, n) as f32 } + fn tanh(self) -> f32 { libm::tanhf(self) } fn cos(self) -> f32 { libm::cosf(self) } fn sin(self) -> f32 { libm::sinf(self) } + fn floor(self) -> f32 { libm::floorf(self) } fn ceil(self) -> f32 { libm::ceilf(self) } fn round(self) -> f32 { libm::roundf(self) } + fn trunc(self) -> f32 { libm::truncf(self) } fn fract(self) -> f32 { self - libm::truncf(self) } + fn asin(self) -> f32 { libm::asinf(self) } fn acos(self) -> f32 { libm::acosf(self) } fn atan2(self, x: f32) -> f32 { libm::atan2f(self, x) } +} + +/// What a `no_std` module adds with one line (`use std::prelude::rust_2021::*;`) to see the names `std`'s prelude gives it. +pub mod prelude { + pub mod rust_2021 { + pub use core::prelude::rust_2021::*; + pub use alloc::borrow::ToOwned; + pub use alloc::boxed::Box; + pub use alloc::string::{String, ToString}; + pub use alloc::vec::Vec; + pub use crate::FloatExt; + } +} diff --git a/probbit-persona/Cargo.toml b/probbit-persona/Cargo.toml new file mode 100644 index 0000000..bb64a72 --- /dev/null +++ b/probbit-persona/Cargo.toml @@ -0,0 +1,27 @@ +[package] +name = "probbit-persona" +description = "probbit's document layer as a library: JSON/YAML documents, the persona layer, run/decide/evaluate, and the strand writer and verifier; std or no_std + alloc" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +readme = "../README.md" +keywords = ["p-bit", "persona", "no-std", "decision", "probabilistic"] +categories = ["algorithms", "no-std", "science"] + +[features] +default = ["std", "decide"] +std = ["probbit-core/std", "probbit-ir/std", "probbit-decide?/std"] +# no_std + alloc (a target without an operating system): the std subset comes from probbit-nostd; documents come in as strings +nostd = ["dep:probbit-nostd", "probbit-core/nostd", "probbit-ir/nostd", "probbit-decide?/nostd"] +# `router` (probbit decide) and `evaluate` (the decision-API adapter): probbit-decide +decide = ["dep:probbit-decide"] +# `turn_trace` (off by default): per-phase microseconds and a host probe's bytes for the last `Live::event` (`persona::trace`) +turn_trace = [] + +[dependencies] +probbit-nostd = { path = "../probbit-nostd", version = "0.14.0-dev", optional = true } +probbit-core = { path = "../probbit-core", default-features = false, version = "0.14.0-dev" } +probbit-ir = { path = "../probbit-ir", default-features = false, version = "0.14.0-dev" } +probbit-decide = { path = "../probbit-decide", default-features = false, version = "0.14.0-dev", optional = true } diff --git a/probbit-cli/src/bound.rs b/probbit-persona/src/bound.rs similarity index 99% rename from probbit-cli/src/bound.rs rename to probbit-persona/src/bound.rs index d21b65d..7a17663 100644 --- a/probbit-cli/src/bound.rs +++ b/probbit-persona/src/bound.rs @@ -5,6 +5,7 @@ use super::*; /// The `bound` block: every field optional +#[derive(Clone)] pub struct Bound { pub rest_hours: Option, pub max_epsilon: Option, pub seeds: Option>, pub count: Vec } const SEED_MAX: u64 = 2_147_483_647; const MAX_SEEDS: usize = 10_000; @@ -105,6 +106,8 @@ pub fn scope(p: &Persona, b: &Bound, t: &str, rest: f64, st: Option<&State>) -> if prev_rel && h.rules.iter().flat_map(|(k, r)| r.iter().map(move |r| rule_vars(k, r))).flatten().any(|x| comp.contains(&x)) { uncounted.push(format!("habits[{i}]")); } } for v in p.vars.iter().filter(|v| v.hold.is_some() && comp.contains(&v.id)) { uncounted.push(format!("{}.hold", v.id)); } if let Some(pc) = &p.pace { if pc.effects.iter().any(|(k, _)| comp.contains(k)) { d_r += 2.0 * 0.5f64.powf(rest / pc.hl) * pace_shift(pc); } } + // RFC 0016: the curiosity drive's level does not decay with the rest hours (it moves per event), so its whole shift counts + if let Some(c) = &p.cur { d_r += 2.0 * c.shift(&comp); } if p.drives.is_some() && (comp.iter().any(|x| x == "pursue") || t == "pursue") { uncounted.push(if b.count.iter().any(|x| x == "deadlines") { "drives (counted; not yet searched by this build)".into() } else { "drives".into() }); } if p.has_needs() && (comp.iter().any(|x| x == "pursue") || t == "pursue") { uncounted.push(if b.count.iter().any(|x| x == "needs") { "needs (counted; not yet searched by this build)".into() } else { "needs".into() }); } Scope { coords, flags, d_r, uncounted, comp, others } diff --git a/probbit-persona/src/curiosity.rs b/probbit-persona/src/curiosity.rs new file mode 100644 index 0000000..54ce72a --- /dev/null +++ b/probbit-persona/src/curiosity.rs @@ -0,0 +1,253 @@ +//! RFC 0016 curiosity (docs/rfcs/0016-curiosity.md): a declared drive fed by the measured surprise, and the self-started idle +//! event (`stir`) it can gate. Mechanism only: the drive's level is arithmetic over two numbers the expectation block (RFC 0009) +//! already measures, and it moves the stance through one engine-owned number input, `cur_level`, as `pace` moves it through its +//! two. Off by default: a persona without the `curiosity` block gives every document, state and strand as before. +//! +//! - `surprise_recent` s: an exponentially weighted average of z+ = max(z, 0) (the deviation's z, read from the logged +//! prediction as the expectation's normal reads it; one-sided as its `registers`: a prediction error below the learned +//! normal is no surprise) with a half-life in events; it starts at `quiet_z`. +//! - `failed_contexts` F: of the contexts the expectation remembers (its input combos), those whose last K visits all missed +//! (z+ > `miss_z` on each). +//! - level = clip(quiet x max(0, 1 - s / quiet_z) + failed x F, 0, cap), at 6 decimals: it rises while nothing new arrives (s +//! low for long) and while contexts keep failing, and falls when a surprising event raises s. +//! +//! The drive's state (s, the per-context miss runs, the hours since the last stir, the last event's asleep flag) is a fold of the +//! logged lines inside the expectation's canonical memory (engine 0.14.0 on), so checkpoints carry it and a resume or a replay +//! from a checkpoint reads it back. `stir_due` answers from that state and the idle hours since the last event. +use super::*; + +/// The engine-owned number input the drive lowers to (reserved only with the block) +pub const CUR_ID: &str = "cur_level"; +/// The input key that marks a self-started event (reserved only with a `stir` block): its value is the stir's class +pub const STIR_KEY: &str = "stir"; + +/// The declared drive +#[derive(Clone, Debug)] +pub struct Curio { pub hl: f64, pub quiet_z: f64, pub miss_z: f64, pub last_k: usize, pub w_quiet: f64, pub w_failed: f64, pub cap: f64, pub stir: Option, + /// the ordinal shift per unit of the level, per trait or mood (the lowered input's effects) + pub effects: Vec<(String, f64)> } +impl Curio { + /// The largest shift the drive can put on the field of the variables in `comp`: sum of |effects| x cap (RFC 0004's residual) + pub fn shift(&self, comp: &[String]) -> f64 { r6(self.effects.iter().filter(|(k, _)| comp.contains(k)).fold(0.0, |a, (_, w)| a + w.abs()) * self.cap) } +} +/// The self-start: due when the level is at least `threshold`, at most one per `every_min` minutes; while the last event's +/// `asleep` flag was on, only with `when_asleep` (flag, minutes, class), whose class never speaks +#[derive(Clone, Debug)] +pub struct StirSpec { pub threshold: f64, pub every_min: f64, pub class: String, pub speaks: bool, pub asleep: Option<(String, f64, String)> } +impl StirSpec { + /// The classes an event may carry under `stir`: the awake one, then the asleep one + pub fn classes(&self) -> Vec<&str> { let mut v = vec![self.class.as_str()]; if let Some((_, _, c)) = &self.asleep { if c != &self.class { v.push(c); } } v } +} + +/// The `curiosity` block of a persona whose traits, moods and inputs are `ids`, `moods` and `inputs` -> (the drive, the effects +/// map its input carries). Refused at the field's path (C-R1 .. C-R3 of the RFC). +pub fn spec(cj: &Json, traits: &[String], flags: &[String]) -> R<(Curio, Json)> { + let ck = keys(cj, "curiosity", &["half_life_events", "quiet_z", "miss_z", "last_k", "weights", "cap", "effects", "stir", "comment"], &["effects"])?; + let f = |k: &str, d: f64| num_or(ck, k, d, "curiosity", Some(-1e6), Some(1e6), false); + let (hl, quiet_z, miss_z, cap) = (f("half_life_events", 8.0)?, f("quiet_z", 0.5)?, f("miss_z", 1.5)?, f("cap", 2.0)?); + need(hl >= 1.0, "curiosity.half_life_events", "must be >= 1 (events)")?; + need(quiet_z > 0.0, "curiosity.quiet_z", "must be > 0 (the recent surprise below which nothing is new)")?; + need(miss_z > 0.0, "curiosity.miss_z", "must be > 0 (the |z| above which a visit missed)")?; + need(cap > 0.0, "curiosity.cap", "must be > 0")?; + let last_k = num_or(ck, "last_k", 3.0, "curiosity", Some(1.0), Some(16.0), true)? as usize; + let (w_quiet, w_failed) = match get(ck, "weights").filter(|x| !x.is_null()) { None => (1.0, 0.5), Some(w) => { + let wk = keys(w, "curiosity.weights", &["quiet", "failed"], &[])?; + (num_or(wk, "quiet", 1.0, "curiosity.weights", Some(0.0), Some(50.0), false)?, num_or(wk, "failed", 0.5, "curiosity.weights", Some(0.0), Some(50.0), false)?) } }; + let ek = get(ck, "effects").and_then(Json::as_obj).filter(|o| !o.is_empty()).ok_or_else(|| perr("curiosity.effects", "a mapping trait or mood -> ordinal shift per unit of the level"))?; + let mut effects = vec![]; + for (k, x) in ek { let ep = at("curiosity.effects", k); + need(k != "pursue", &ep, "the curiosity drive never moves pursue")?; need(traits.contains(k), &ep, &format!("unknown trait or mood {k:?}"))?; + effects.push((k.clone(), num(Some(x), &ep, Some(-50.0), Some(50.0), false)?)); } + let stir = match get(ck, "stir").filter(|x| !x.is_null()) { None => None, Some(sj) => { + let sk = keys(sj, "curiosity.stir", &["threshold", "every_minutes", "class", "speaks", "when_asleep"], &["threshold"])?; + let threshold = num(get(sk, "threshold"), "curiosity.stir.threshold", Some(0.0), Some(cap), false)?; + need(threshold > 0.0, "curiosity.stir.threshold", "must be > 0 (a level of 0 never stirs)")?; + let every_min = num_or(sk, "every_minutes", 30.0, "curiosity.stir", Some(1.0), Some(1e6), false)?; + let class = match get(sk, "class") { None => "check".to_string(), x => ident(x, "curiosity.stir.class", "class")? }; + let speaks = match get(sk, "speaks") { None | Some(Json::Null) => true, Some(Json::Bool(b)) => *b, Some(_) => return Err(perr("curiosity.stir.speaks", "true | false")) }; + let asleep = match get(sk, "when_asleep").filter(|x| !x.is_null()) { None => None, Some(aj) => { + let ak = keys(aj, "curiosity.stir.when_asleep", &["flag", "every_minutes", "class"], &["flag"])?; + let fl = ident(get(ak, "flag"), "curiosity.stir.when_asleep.flag", "flag")?; + need(flags.contains(&fl), "curiosity.stir.when_asleep.flag", "a declared flag input (true on the events of the asleep hours)")?; + let m = num_or(ak, "every_minutes", 120.0, "curiosity.stir.when_asleep", Some(1.0), Some(1e6), false)?; + need(m >= every_min, "curiosity.stir.when_asleep.every_minutes", "at least the awake every_minutes (asleep stirs are rarer)")?; + let c = match get(ak, "class") { None => "glance".to_string(), x => ident(x, "curiosity.stir.when_asleep.class", "class")? }; + need(c != class || !speaks, "curiosity.stir.when_asleep.class", "the asleep class never speaks: name another class than a speaking awake one")?; + Some((fl, m, c)) } }; + Some(StirSpec { threshold, every_min, class, speaks, asleep }) } }; + Ok((Curio { hl, quiet_z, miss_z, last_k, w_quiet, w_failed, cap, stir, effects }, Json::Obj(ek.to_vec()))) +} + +/// The drive's fold (inside the expectation's canonical memory) +#[derive(Clone, Debug, Default, PartialEq)] +pub struct CurMem { pub s: f64, pub since: Option, pub asleep: bool, pub ctx: Vec<(String, Vec)> } +impl CurMem { + pub fn new(c: &Curio) -> CurMem { CurMem { s: c.quiet_z, ..Default::default() } } + /// The contexts whose last K visits all missed + pub fn failed(&self, c: &Curio) -> usize { self.ctx.iter().filter(|(_, v)| v.len() == c.last_k && v.iter().all(|m| *m)).count() } + /// The drive's level from this fold + pub fn level(&self, c: &Curio) -> f64 { r6((c.w_quiet * (1.0 - self.s / c.quiet_z).max(0.0) + c.w_failed * self.failed(c) as f64).clamp(0.0, c.cap)) } + /// The stance's `drives.curiosity` (and the line's receipt): the level and its two inputs + pub fn doc(&self, c: &Curio) -> Json { + Json::Obj(vec![("failed_contexts".into(), Json::Num(self.failed(c) as f64)), ("level".into(), Json::Num(self.level(c))), ("surprise_recent".into(), Json::Num(self.s))]) + } + /// One event folded in: `key` its combo, `z` its z (None while cold), `kv` its logged inputs. Before the expectation's normal + /// reads the event (z is the deviation's z). Every number at 6 decimals. + pub fn step(&mut self, c: &Curio, key: &str, z: Option, kv: &[(String, Json)]) { + // one-sided, as the deviation's `registers`: a prediction error below the learned normal (z < 0) is no surprise + if let Some(z) = z { let zp = r6(z.max(0.0)); let lam = 0.5f64.powf(1.0 / c.hl); self.s = r6(lam * self.s + (1.0 - lam) * zp); + let miss = zp > c.miss_z; + match self.ctx.binary_search_by(|x| x.0.as_str().cmp(key)) { + Ok(i) => { let v = &mut self.ctx[i].1; v.push(miss); if v.len() > c.last_k { v.remove(0); } } + Err(i) => self.ctx.insert(i, (key.to_string(), vec![miss])) } } + let h = get(kv, "elapsed_hours").and_then(Json::as_f64).unwrap_or(0.0); + self.since = if matches!(get(kv, STIR_KEY), Some(Json::Str(_))) && c.stir.is_some() { Some(0.0) } else { self.since.map(|s| r6(s + h)) }; + self.asleep = c.stir.as_ref().and_then(|st| st.asleep.as_ref()).is_some_and(|(fl, _, _)| matches!(get(kv, fl), Some(Json::Bool(true)))); + } + /// The memory's contexts after the expectation's own prune: a context the expectation dropped is dropped here + pub fn prune(&mut self, keep: &[(String, f64, Json)]) { self.ctx.retain(|(k, _)| keep.iter().any(|c| &c.0 == k)); } + /// As a checkpoint line's `expect.curiosity`: {asleep, ctx: [[key, [miss, ...]], ...] in key order, s, since} + pub fn to_json(&self) -> Json { + Json::Obj(vec![("asleep".into(), Json::Bool(self.asleep)), + ("ctx".into(), Json::Arr(self.ctx.iter().map(|(k, v)| Json::Arr(vec![Json::Str(k.clone()), Json::Arr(v.iter().map(|m| Json::Bool(*m)).collect())])).collect())), + ("s".into(), Json::Num(self.s)), ("since".into(), self.since.map_or(Json::Null, Json::Num))]) + } + /// A checkpoint line's `expect.curiosity` -> the fold; Err(what is wrong) + pub fn read(c: &Curio, j: &Json) -> Result { + let kv = j.as_obj().filter(|kv| kv.len() == 4).ok_or("curiosity: an object {asleep, ctx, s, since}")?; + let f = |k: &str| get(kv, k).ok_or(format!("curiosity: no {k}")); + let s = f("s")?.as_f64().filter(|x| x.is_finite() && *x >= 0.0 && r6(*x) == *x).ok_or("curiosity.s: a number >= 0 at 6 decimals")?; + let since = match f("since")? { Json::Null => None, Json::Num(x) if x.is_finite() && *x >= 0.0 && r6(*x) == *x => Some(*x), _ => return Err("curiosity.since: null or hours >= 0 at 6 decimals".into()) }; + let Json::Bool(asleep) = f("asleep")? else { return Err("curiosity.asleep: true or false".into()) }; + let mut ctx: Vec<(String, Vec)> = vec![]; + for x in f("ctx")?.as_arr().ok_or("curiosity.ctx: a list")? { + let Some([Json::Str(k), Json::Arr(m)]) = x.as_arr() else { return Err("curiosity.ctx: each [key, [miss, ...]]".into()) }; + if ctx.last().is_some_and(|l| l.0 >= *k) { return Err("curiosity.ctx: in key order, each key once".into()); } + let m: Option> = m.iter().map(|b| if let Json::Bool(b) = b { Some(*b) } else { None }).collect(); + let m = m.filter(|m| !m.is_empty() && m.len() <= c.last_k).ok_or("curiosity.ctx: 1 to last_k misses (true / false)")?; + ctx.push((k.clone(), m)); } + Ok(CurMem { s, since, asleep: *asleep, ctx }) + } +} + +/// A self-started idle event that is due: its class, whether it may speak, whether the individual was asleep, the level that +/// gated it, and the minutes since the last stir (None: none yet) +#[derive(Clone, Debug, PartialEq)] +pub struct Stir { pub class: String, pub speaks: bool, pub asleep: bool, pub level: f64, pub since_minutes: Option } +impl Stir { + /// The control query's answer and the event a host feeds when it acts on it (`{"stir": class}` plus its own readings) + pub fn to_json(&self) -> Json { + Json::Obj(vec![("asleep".into(), Json::Bool(self.asleep)), ("class".into(), Json::Str(self.class.clone())), ("level".into(), Json::Num(self.level)), + ("since_minutes".into(), self.since_minutes.map_or(Json::Null, |m| Json::Num(r6(m)))), ("speaks".into(), Json::Bool(self.speaks))]) + } +} +/// Is a stir due `idle_hours` after the last event? Exactly from the fold: the level at least the threshold; while asleep only +/// with `when_asleep` (its class, its rate); at most one per the declared minutes since the last stir (none yet: due) +pub fn stir_due(c: &Curio, m: &CurMem, idle_hours: f64) -> Option { + let sp = c.stir.as_ref()?; let level = m.level(c); + if level < sp.threshold { return None; } + let (every, class, speaks) = if m.asleep { let (_, e, cl) = sp.asleep.as_ref()?; (*e, cl.clone(), false) } else { (sp.every_min, sp.class.clone(), sp.speaks) }; + let since = m.since.map(|s| r6((s + idle_hours.max(0.0)) * 60.0)); + if since.is_some_and(|s| s < every) { return None; } + Some(Stir { class, speaks, asleep: m.asleep, level, since_minutes: since }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::strand::{self, Clock, Live}; + fn run(prog: &Json, f: &Flags) -> Json { run_program(prog, f, 1, 100, 0) } + fn g<'a>(j: &'a Json, ks: &[&str]) -> &'a Json { static NULL: Json = Json::Null; ks.iter().try_fold(j, |x, k| x.get(k)).unwrap_or(&NULL) } + const DOC: &str = r#"{"probbit_persona":1,"identity":{"name":"C","version":"1","seed":3}, + "traits":[{"id":"verbosity","levels":["terse","short","full"],"logw":[0.2,0.5,0.1]},{"id":"initiative","levels":["wait","offer","lead"],"logw":[0.3,0.2,0]}], + "moods":[{"id":"valence","levels":["down","even","up"],"inertia":0.5,"half_life_hours":6}], + "inputs":[{"id":"night","kind":"flag","effects":{"verbosity":-0.5}},{"id":"ping","kind":"flag","effects":{"valence":0.6}},{"id":"odd","kind":"number","max":3,"effects":{"valence":-0.8,"verbosity":0.6}}], + "habits":[{"id":"quiet_at_night","when":{"night":true},"then":{"verbosity":["terse"],"initiative":["wait","offer"]}}], + "expectation":{"half_life_events":8,"max_combos":6}CUR}"#; + const CUR: &str = r#","curiosity":{"half_life_events":4,"quiet_z":0.5,"miss_z":1.5,"last_k":2,"weights":{"quiet":1.5,"failed":0.5},"cap":2,"effects":{"initiative":0.8}, + "stir":{"threshold":0.6,"every_minutes":30,"class":"check","when_asleep":{"flag":"night","every_minutes":120,"class":"glance"}}}"#; + fn doc(c: &str) -> Json { json::parse(&DOC.replace("CUR", c)).unwrap() } + fn refused(c: &str) -> String { match build(&doc(c)) { Err(e) => format!("{}: {}", e.path, e.msg), Ok(_) => "ok".into() } } + fn ev(s: &str) -> Json { json::parse(s).unwrap() } + /// A day of varied events, then a dark night of identical sensed events every 10 minutes; the host polls `stir_due` after each + /// event and feeds the stir it is offered -> (the strand's text, the individual, per night tick the level and the stir fed) + fn life(with_checkpoints: bool) -> (String, Box, Vec<(f64, Option)>) { + let d = doc(CUR); let p = build(&d).unwrap(); + let (mut lv, header) = Live::start(p.clone(), &d, init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.14.0"); + let mut text = format!("{header}\n"); let mut night = vec![]; + let mut feed = |lv: &mut Live, e: &Json, text: &mut String| { let (s, l) = lv.event(e, &run).unwrap(); text.push_str(&l); text.push('\n'); + if with_checkpoints && lv.n % 10 == 0 { text.push_str(&lv.checkpoint(&s)); text.push('\n'); } s }; + for k in 0..24 { let e = match k % 4 { 0 => r#"{"ping":true,"elapsed_hours":0.25}"#.to_string(), 1 => format!(r#"{{"odd":{},"elapsed_hours":0.25}}"#, k % 3 + 1), + 2 => r#"{"elapsed_hours":0.25}"#.to_string(), _ => r#"{"ping":true,"odd":2,"elapsed_hours":0.25}"#.to_string() }; + feed(&mut lv, &ev(&e), &mut text); } + for _ in 0..48 { feed(&mut lv, &ev(r#"{"night":true,"elapsed_hours":0.166667}"#), &mut text); + let lvl = crate::persona::cur_level(&lv.p, lv.expect.as_ref().unwrap()).unwrap(); + let due = lv.stir_due(0.0); + if let Some(s) = &due { let st = feed(&mut lv, &ev(&format!(r#"{{"night":true,"stir":"{}","elapsed_hours":0}}"#, s.class)), &mut text); + assert!(g(&st, &["habits", "active"]).as_arr().unwrap().iter().any(|h| h.as_str() == Some("quiet_at_night")), "{}", canon(&st)); + assert_eq!(g(&st, &["stance", "verbosity", "level"]).as_str(), Some("terse"), "the habit binds on the stir"); + assert_eq!(g(&st, &["drives", "curiosity", "stir"]).as_str(), Some(s.class.as_str())); } + night.push((lvl, due)); } + (text, Box::new(lv), night) + } + #[test] + fn the_block_is_read_strictly() { + assert_eq!(refused(""), "ok"); + assert_eq!(refused(CUR), "ok"); + assert_eq!(match build(&json::parse(&DOC.replace("CUR", CUR).replace(r#""expectation":{"half_life_events":8,"max_combos":6}"#, r#""comment":"no expectation""#)).unwrap()) { Err(e) => format!("{}: {}", e.path, e.msg), Ok(_) => "ok".into() }, + "curiosity: a curiosity block needs an expectation block (the drive reads its measured surprise)"); + assert_eq!(refused(r#","curiosity":{"effects":{"pursue":1}}"#), "curiosity.effects.pursue: the curiosity drive never moves pursue"); + assert_eq!(refused(r#","curiosity":{"effects":{"nope":1}}"#), r#"curiosity.effects.nope: unknown trait or mood "nope""#); + assert_eq!(refused(r#","curiosity":{"effects":{}}"#), "curiosity.effects: a mapping trait or mood -> ordinal shift per unit of the level"); + assert_eq!(refused(r#","curiosity":{"effects":{"initiative":1},"quiet_z":0}"#), "curiosity.quiet_z: must be > 0 (the recent surprise below which nothing is new)"); + assert_eq!(refused(r#","curiosity":{"effects":{"initiative":1},"stir":{"threshold":0.5,"when_asleep":{"flag":"ping2"}}}"#), + "curiosity.stir.when_asleep.flag: a declared flag input (true on the events of the asleep hours)"); + assert_eq!(refused(r#","curiosity":{"effects":{"initiative":1},"stir":{"threshold":0.5,"every_minutes":60,"when_asleep":{"flag":"night","every_minutes":30}}}"#), + "curiosity.stir.when_asleep.every_minutes: at least the awake every_minutes (asleep stirs are rarer)"); + assert_eq!(refused(r#","curiosity":{"effects":{"initiative":1},"stir":{"threshold":3}}"#).split(':').next(), Some("curiosity.stir.threshold")); + let d = DOC.replace("CUR", CUR).replace(r#"{"id":"ping","kind":"flag""#, r#"{"id":"cur_level","kind":"flag""#).replace(r#""ping":true"#, r#""cur_level":true"#); + assert_eq!(build(&json::parse(&d).unwrap()).err().map(|e| e.path), Some("inputs[1].id".to_string())); + // off by default: without the block no cur_level, no drives key, and the digest is another + let (a, b) = (build(&doc("")).unwrap(), build(&doc(CUR)).unwrap()); assert_ne!(a.digest, b.digest); + let (sa, _) = turn(&a, &init(&a, Some(1), true, &run), &ev(r#"{"ping":true}"#), false, &run, false).unwrap(); + assert!(sa.get("drives").is_none() && g(&sa, &["inputs", "cur_level"]).is_null(), "{}", canon(&sa)); + let (sb, _) = turn(&b, &init(&b, Some(1), true, &run), &ev(r#"{"ping":true}"#), false, &run, false).unwrap(); + assert!(g(&sb, &["inputs", "cur_level"]).is_null(), "cur_level leaves inputs: {}", canon(&sb)); + assert_eq!(turn(&b, &init(&b, Some(1), true, &run), &ev(r#"{"cur_level":1}"#), false, &run, false).err().map(|e| e.path), Some("inputs.cur_level".to_string())); + // the expectation block outside the digest still: with and without it the curiosity persona's digest... is another persona + let d = doc(CUR); let p = build(&d).unwrap(); let (mut lv, _) = Live::start(p.clone(), &d, init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.14.0"); + assert_eq!(lv.event(&ev(r#"{"stir":"hum"}"#), &run).err().map(|e| format!("{}: {}", e.path, e.msg)), Some("event.stir: a declared stir class: check | glance".to_string())); + } + #[test] + fn a_quiet_night_stirs_at_the_declared_rate_and_the_habit_binds() { + let (_, lv, night) = life(false); + let stirs: Vec<(usize, &Stir)> = night.iter().enumerate().filter_map(|(i, (_, s))| s.as_ref().map(|s| (i, s))).collect(); + eprintln!("R16 night: levels {:?}", night.iter().map(|x| x.0).collect::>()); + eprintln!("R16 night: stirs at ticks {:?}", stirs.iter().map(|x| x.0).collect::>()); + assert!(night[0].0 < 0.6, "the night starts below the threshold: {}", night[0].0); + assert!(stirs.len() >= 2, "the quiet night raises the drive past the threshold"); + for w in stirs.windows(2) { assert!((w[1].0 - w[0].0) * 10 >= 120, "asleep: at most one stir per 120 minutes: {:?}", stirs); } + assert!(stirs.iter().all(|(_, s)| s.class == "glance" && !s.speaks && s.asleep)); + // awake, the same level stirs at the awake class and rate + assert!(lv.stir_due(10.0).is_some_and(|s| s.asleep), "the last event was a night event"); + } + #[test] + fn the_drive_replays_resumes_and_restarts_from_a_checkpoint() { + for cps in [false, true] { + let (text, lv, _) = life(cps); + let v = strand::verify(&text, &run).unwrap_or_else(|(i, w)| panic!("line {i}: {w}")); + assert_eq!(v.get("events").and_then(Json::as_f64), Some(lv.n as f64), "{}", canon(&v)); + let ls = strand::lines(&text); let (mut r, s0) = strand::Replay::open_lines(&ls).unwrap(); + for i in s0..ls.len() { r.step_at(&ls, i, &run).unwrap(); } + for h in [0.0, 0.5, 3.0] { assert_eq!(r.live.stir_due(h), lv.stir_due(h)); } + assert_eq!(r.live.expect.as_ref().unwrap().cur, lv.expect.as_ref().unwrap().cur); + let rs = strand::resume(&text, &lv.p, lv.st.clone(), Clock::Fixed).unwrap(); + assert_eq!((rs.n, rs.head(), rs.expect.as_ref().unwrap().cur.clone()), (lv.n, lv.head(), lv.expect.as_ref().unwrap().cur.clone())); + if cps { strand::verify_from_checkpoint(&text, &run).unwrap_or_else(|(i, w)| panic!("from checkpoint: line {i}: {w}")); + assert!(text.contains(r#""curiosity":{"asleep":"#), "the checkpoint carries the fold"); } + // an edited receipt diverges at its line + let i = text.find(r#""surprise_recent":"#).unwrap(); let mut t2 = text.clone(); t2.replace_range(i..i + 18, r#""surprise_recent":9"#); + assert!(strand::verify(&t2, &run).is_err()); + } + } +} diff --git a/probbit-cli/src/evaluate.rs b/probbit-persona/src/evaluate.rs similarity index 98% rename from probbit-cli/src/evaluate.rs rename to probbit-persona/src/evaluate.rs index 908374f..04b294d 100644 --- a/probbit-cli/src/evaluate.rs +++ b/probbit-persona/src/evaluate.rs @@ -13,6 +13,8 @@ //! copied as they are. The program runs exactly as `probbit run` runs it (`--program` prints it). //! Response = `model`, `answers` (keyed by question id, each in the judge's answer shape plus a `probbit` object), `usage`, then every //! field of the `probbit run` document. Docs: docs/probbit-ir-json.md "Decision API". +#[allow(unused_imports)] +use std::prelude::rust_2021::*; use crate::json::{arr, at, fields, keyed, num, number, obj, opt, req, schema, str as jstr, text, value, weight, InErr, Json}; use std::collections::{HashMap, HashSet}; @@ -105,6 +107,7 @@ pub fn compile(j: &Json) -> Result { let mut w = vec![0.0; q.options.len()]; for (o, v) in keyed(x, &p)? { let k = q.options.iter().position(|y| y == o).ok_or_else(|| value(&at(&p, o), format!("question {id}: unknown option {o}")))?; w[k] = weight(v, &at(&p, o))?; } set(i, w, "probbit.logw", &p)?; } } + drop(qidx); // no_std: hashbrown's map drop is not `may_dangle`, so the borrow of `qs` ends here, not at the scope end (as run.rs) for (q, w) in qs.iter_mut().zip(&h) { q.w = w.clone(); } // the alphabet: score levels first in order (precedes / linear read values as ordered slots), then "false", "true", then // every choice option in order of first appearance; each variable allows its own options only diff --git a/probbit-persona/src/hooks.rs b/probbit-persona/src/hooks.rs new file mode 100644 index 0000000..5312e21 --- /dev/null +++ b/probbit-persona/src/hooks.rs @@ -0,0 +1,34 @@ +//! The host's hooks into the decision pipeline: the CLI's `--top` monitor (the tier that is running, what the gate saw) and its +//! configuration echo (`telemetry.config`, the probbit.json in effect). Unset, the monitor calls do nothing and the echo is empty, +//! which is what probbit-wasm and a firmware want. The CLI installs its own at start (probbit-cli/src/main.rs). +#[allow(unused_imports)] +use std::prelude::rust_2021::*; +use crate::json::Json; +use std::sync::atomic::{AtomicUsize, Ordering}; + +// fn pointers as usize: an atomic store works on every target the engine builds for, with or without std +static TIER: AtomicUsize = AtomicUsize::new(0); +static GATE_SEEN: AtomicUsize = AtomicUsize::new(0); +static CONFIG_ECHO: AtomicUsize = AtomicUsize::new(0); + +/// Install the monitor: `tier` gets the name of each tier as it starts, `gate_seen` the gate's R-hat, its TV bound, the released +/// variables and all of them. +pub fn set_monitor(tier: fn(&'static str), gate_seen: fn(f64, f64, usize, usize)) { + TIER.store(tier as usize, Ordering::Relaxed); GATE_SEEN.store(gate_seen as usize, Ordering::Relaxed); +} +/// Install the configuration echo: the telemetry fields that name the configuration in effect. +pub fn set_config_echo(f: fn() -> Vec<(&'static str, Json)>) { CONFIG_ECHO.store(f as usize, Ordering::Relaxed) } + +pub fn tier(t: &'static str) { + let p = TIER.load(Ordering::Relaxed); + if p != 0 { let f: fn(&'static str) = unsafe { core::mem::transmute(p) }; f(t) } +} +pub fn gate_seen(rhat: f64, tv: f64, released: usize, n: usize) { + let p = GATE_SEEN.load(Ordering::Relaxed); + if p != 0 { let f: fn(f64, f64, usize, usize) = unsafe { core::mem::transmute(p) }; f(rhat, tv, released, n) } +} +pub fn config_echo() -> Vec<(&'static str, Json)> { + let p = CONFIG_ECHO.load(Ordering::Relaxed); + if p == 0 { return vec![]; } + let f: fn() -> Vec<(&'static str, Json)> = unsafe { core::mem::transmute(p) }; f() +} diff --git a/probbit-cli/src/hostlint.rs b/probbit-persona/src/hostlint.rs similarity index 100% rename from probbit-cli/src/hostlint.rs rename to probbit-persona/src/hostlint.rs diff --git a/probbit-persona/src/integrity.rs b/probbit-persona/src/integrity.rs new file mode 100644 index 0000000..86f363d --- /dev/null +++ b/probbit-persona/src/integrity.rs @@ -0,0 +1,249 @@ +//! RFC 0014 integrity (docs/rfcs/0014-integrity.md): per turn, how far the individual has drifted from the self its persona +//! declares, exact from the state and the persona document (mechanism: arithmetic over declared numbers; nothing here writes a +//! state). Off by default: only a persona whose `identity` block declares one of the self keys (`keep`, `warn`, `cue`, `traits`) +//! gets it, and those keys stay out of the digest (as RFC 0004's `bound`, amendment A1), so every state is as without them. +//! +//! - `drift` per trait: the total variation (half the L1 over the trait's levels) between the odds at the rest probe (one event of +//! the rest hours idle, no inputs: RFC 0004's probe) of the current individual rested, and of the same seed's fresh individual +//! (the declared persona); summed over the traits. Rested (`rested`): the channels of the hour (moods with `half_life_hours`, +//! the pace average, the load, the credit awaiting feedback, the last levels and stance, the agenda's step) at the fresh +//! individual's values, everything that persists as it is (learned deltas, history, counted moods, drives, the senses' normals, +//! the gates): the drift +//! measures what stays, not the mood of the hour, and the rest probe's cache hits whenever only the hour moved. +//! - `allowance` per trait: RFC 0004's epsilon at the individual's own seed when the persona declares a `bound` block and the +//! trait is learned (null otherwise); `bound_used` = drift / allowance (0 without an allowance). +//! - `used`: the summed drift over the traits with an allowance as a share of the summed allowance (0 without one): the dial. +//! - `habits_held`: of the habits `identity.keep` names, those that bound this turn (`held`) and those that did not (`not_fired`). +//! - `cue`: the persona's own `identity.cue`, when a trait's `bound_used` reaches `identity.warn` (null otherwise). +use super::*; + +/// The self keys of the `identity` block +pub const KEYS: [&str; 4] = ["keep", "warn", "cue", "traits"]; +/// The declared self: identity-bearing habits, the warn share, the cue, the traits measured, the rest probe's hours and the +/// `bound` block (when declared) that gives each learned trait its allowance +#[derive(Clone)] +pub struct Ident { pub keep: Vec, pub warn: f64, pub cue: Option, pub traits: Vec, pub rest: f64, pub bound: Option } +/// Whether an `identity` block declares a self key +pub fn declared(idn: &Json) -> bool { idn.as_obj().is_some_and(|kv| kv.iter().any(|(k, _)| KEYS.contains(&k.as_str()))) } +/// The strict reader of the self keys (`idn`: the whole `identity` block), against the persona built without them; `doc` is the +/// whole document (its `bound` block, when present, gives the allowances) +pub fn spec(idn: &Json, doc: &Json, p: &Persona) -> R { + let kv = idn.as_obj().ok_or_else(|| perr("identity", "must be a mapping"))?; + let mut keep: Vec = vec![]; + for (i, h) in list(get(kv, "keep"), "identity.keep")?.iter().enumerate() { let path = ix("identity.keep", i); + let h = h.as_str().filter(|h| p.habits.iter().any(|x| x.id == *h)).ok_or_else(|| perr(&path, "a habit id of this persona"))?; + need(!keep.iter().any(|x| x == h), &path, "listed twice")?; keep.push(h.to_string()); } + let warn = num_or(kv, "warn", 0.8, "identity", Some(0.0), Some(1.0), false)?; + need(warn > 0.0, "identity.warn", "a share of the allowance in (0, 1]")?; + let cue = match get(kv, "cue") { None | Some(Json::Null) => None, + Some(c) => Some(c.as_str().filter(|c| !c.trim().is_empty() && c.chars().count() <= 120 && !c.contains(['\n', '\r'])).ok_or_else(|| perr("identity.cue", "a short line of text (1-120 characters, one line)"))?.to_string()) }; + let mut traits: Vec = vec![]; + for (i, t) in list(get(kv, "traits"), "identity.traits")?.iter().enumerate() { let path = ix("identity.traits", i); + let t = t.as_str().filter(|t| p.var(t).is_some_and(|v| !v.mood)).ok_or_else(|| perr(&path, "a trait id of this persona"))?; + need(!traits.iter().any(|x| x == t), &path, "listed twice")?; traits.push(t.to_string()); } + if traits.is_empty() { traits = match &p.learning { Some(l) => l.traits.clone(), None => p.vars.iter().filter(|v| !v.mood).map(|v| v.id.clone()).collect() }; } + let bound = bound::of_doc(doc, p)?; + let rest = bound.as_ref().and_then(|b| b.rest_hours).unwrap_or_else(|| bound::default_rest(p)); + Ok(Ident { keep, warn, cue, traits, rest, bound }) +} + +/// One turn's integrity: the canonical `integrity` document, its summed drift and the dial's share +#[derive(Clone, Debug, PartialEq)] +pub struct Integrity { pub doc: Json, pub drift: f64, pub used: f64 } + +/// The fresh individual's odds at the rest probe and the allowances (both fixed per seed), and the last probe twice keyed: by +/// state (the rested state's body without its `turn` and digest: a turn that changes nothing else leaves the resting odds as they +/// were) and by program (the probe's first program without its comment, `prog_key` as the members' cache keys them: a rested state +/// whose senses or drives moved but whose resting program did not gives the same odds). A program row is kept only when that +/// program's answer was exact and the stance is that answer's (no yield, hold or conflict): exact marginals read the program alone. +pub struct Cache { base: Option>, key: [u8; 32], odds: Vec>, prog: Option<[u8; 32]>, + /// probes answered from the cache (by state or by program), probes solved, and of the hits those by program + pub hits: u64, pub misses: u64, pub by_program: u64 } +impl Cache { pub const fn new() -> Cache { Cache { base: None, key: [0; 32], odds: Vec::new(), prog: None, hits: 0, misses: 0, by_program: 0 } } } +impl Default for Cache { fn default() -> Cache { Cache::new() } } + +/// Per seed: the seed, the fresh individual's odds at the rest probe, the allowances, the fresh individual +type Base = (u64, Vec>, Vec>, State); +/// The rest probe's inputs: one event of `id.rest` idle hours +fn rest_in(id: &Ident) -> Json { Json::Obj(vec![("elapsed_hours".into(), Json::Num(id.rest))]) } +/// Per measured trait its odds in level order, read from a stance document +fn odds_of(p: &Persona, doc: &Json, id: &Ident) -> Vec> { + id.traits.iter().map(|t| { let o = doc.get("stance").and_then(|s| s.get(t)).and_then(|x| x.get("odds")); + p.var(t).map_or(vec![], |v| v.levels.iter().map(|lv| o.and_then(|o| o.get(lv)).and_then(Json::as_f64).unwrap_or(0.0)).collect()) }).collect() +} +/// The rest probe: one event of `id.rest` idle hours, no inputs -> per measured trait its odds in level order +fn probe(p: &Persona, st: &State, id: &Ident, eng: Engine) -> R>> { + let (doc, _) = turn_boxed(p, st, &rest_in(id), false, eng, false)?; Ok(odds_of(p, &doc, id)) +} +/// The rest probe through the program key: the turn's steps as `turn_boxed` runs them, its first program compiled here and keyed; +/// when the key is `known` -> (key, None), else the turn finishes on that program -> (key, Some((odds, whether a row may keep them))) +#[inline(never)] +fn probe_keyed(p: &Persona, st: &State, id: &Ident, eng: Engine, known: Option<&[u8; 32]>) -> R<([u8; 32], Option<(Vec>, bool)>)> { + let pr = turn_prep(p, st, &rest_in(id), true)?; + let first = Box::new(compile(p, &pr.st, &pr.raw, &Opts::default())?); + let key = prog_key(&first.0); + if known == Some(&key) { return Ok((key, None)); } + let sv = turn_solve(p, &pr, false, eng, false, false, Some(first))?; + let o = &sv.out; let row = o.verdict.as_str() == Some("exact") && o.held.is_empty() && o.yielded.is_empty() && o.conflict.is_empty(); + let (doc, _) = turn_doc(p, pr, sv); Ok((key, Some((odds_of(p, &doc, id), row)))) +} +/// The cache key of a rested state: its body without `turn` and `digest` (a rested state has no open decision, whose credit +/// would age in turns), hashed by structure (`prog_key`) +#[inline(never)] +fn key_of(p: &Persona, st: &State) -> [u8; 32] { + let mut b = st.body(p); if let Json::Obj(v) = &mut b { v.retain(|(k, _)| k != "turn" && k != "digest"); } prog_key(&b) +} +/// The individual rested: the channels of the hour at the fresh individual's values `st0` (moods with a half-life, the pace +/// average, the load, the rest pressure, the eligibility trace and the credit awaiting feedback with its open decisions, the +/// levels and stance taken last, the agenda's step); what persists as it is (learned deltas, history, moods without a +/// half-life, drives, senses, the surprise gates, genes) +pub fn rested(p: &Persona, st: &State, st0: &State) -> State { *rested_boxed(p, st, st0) } +/// `rested` on the heap (the probe's path: no frame holds a `State` by value) +#[inline(never)] +pub fn rested_boxed(p: &Persona, st: &State, st0: &State) -> Box { + let mut s = Box::new(st.clone()); + for (id, acc) in s.mood.iter_mut() { if p.var(id).is_some_and(|v| v.half_life.is_some()) { if let Some((_, a0)) = st0.mood.iter().find(|(k, _)| k == id) { acc.clone_from(a0); } } } + s.rbar = st0.rbar; s.load_l = st0.load_l; s.pressure = st0.pressure; s.elig = st0.elig; + s.prev.clone_from(&st0.prev); s.prev_stance.clone_from(&st0.prev_stance); s.agenda.clone_from(&st0.agenda); + s.credit.clone_from(&st0.credit); s.open.clone_from(&st0.open); + // RFC 0016: the curiosity drive's level moves with every event (a channel of the hour): at the fresh individual's (0) + s.cur = st0.cur; s.seal(p); s +} + +/// `integrity` of the state `st` (after a turn) whose stance document is `stance` (its `habits.bound`), from a fresh cache +pub fn compute(p: &Persona, st: &State, stance: &Json, eng: Engine) -> R> { compute_cached(p, st, stance, &mut Cache::new(), eng) } +/// `compute` through a cache: one rest-probe solve when the probe's program (without its comment) and the state's body (without +/// `turn`) both changed since the last solve, else none; the fresh individual's probe and the allowances once per seed. None +/// without the self keys. +pub fn compute_cached(p: &Persona, st: &State, stance: &Json, cache: &mut Cache, eng: Engine) -> R> { + let Some(id) = p.ident.as_ref() else { return Ok(None) }; + if cache.base.as_ref().map_or(true, |b| b.0 != st.seed) { cache.base = Some(base(p, st.seed, id, eng)?); cache.key = [0; 32]; cache.prog = None; } + resting(p, st, id, cache, eng)?; + document(stance, id, cache).map(Some) +} +/// The fresh individual of `seed`: its rest probe and the allowances (on the heap, once per seed) +#[inline(never)] +fn base(p: &Persona, seed: u64, id: &Ident, eng: Engine) -> R> { + let st0 = init_boxed(p, Some(seed), true, eng); let p0 = probe(p, &st0, id, eng)?; + let mut eps = vec![]; + for (i, t) in id.traits.iter().enumerate() { + let k = p.learning.as_ref().and_then(|l| l.traits.iter().position(|x| x == t)); + eps.push(match (&id.bound, &p.learning, k) { + (Some(b), Some(l), Some(k)) => { let sc = bound::scope(p, b, t, id.rest, Some(&*st0)); + match bound::bases(p, &sc, &st0) { Some(bs) => Some(bound::search(p, k, &sc.coords, l.total_cap, &bs, &p0[i], id.rest, eng)?.0), None => None } } + _ => None }); } + Ok(Box::new((seed, p0, eps, *st0))) +} +/// The resting odds of `st` into `cache.odds`: from the program key while the last solved probe left a row (an equal rested state +/// would compile the same program, so the state key would answer the same: it is not built), else from the state key, else one +/// rest probe; after a solve without a row the state key is that state's, as before +#[inline(never)] +fn resting(p: &Persona, st: &State, id: &Ident, cache: &mut Cache, eng: Engine) -> R<()> { + let rs = rested_boxed(p, st, &cache.base.as_ref().unwrap().3); + if cache.prog.is_none() { let key = key_of(p, &rs); if key == cache.key { cache.hits += 1; return Ok(()); } cache.key = key; } + match probe_keyed(p, &rs, id, eng, cache.prog.as_ref())? { + (_, None) => { cache.hits += 1; cache.by_program += 1; } + (k, Some((odds, row))) => { cache.odds = odds; cache.misses += 1; + if row { cache.prog = Some(k); } else if cache.prog.take().is_some() { cache.key = key_of(p, &rs); } } } + Ok(()) +} +/// The integrity document from the resting odds in `cache` and the stance's bound habits +#[inline(never)] +fn document(stance: &Json, id: &Ident, cache: &Cache) -> R { + let b = cache.base.as_ref().unwrap(); let (p0, eps) = (&b.1, &b.2); + let (mut sum, mut sum_e, mut sum_d) = (0.0f64, 0.0f64, 0.0f64); let mut rows = vec![]; let mut crossed = vec![]; + for (i, t) in id.traits.iter().enumerate() { + let d = r6(bound::tv(&cache.odds[i], &p0[i])); sum += d; + let u = match eps[i] { Some(e) if e > 0.0 => { sum_e += e; sum_d += d; r6(d / e) } _ => 0.0 }; + if u >= id.warn { crossed.push(t.clone()); } + rows.push((t.clone(), Json::Obj(vec![("drift".into(), Json::Num(d)), ("allowance".into(), eps[i].map_or(Json::Null, Json::Num)), ("bound_used".into(), Json::Num(u))]))); } + let used = if sum_e > 0.0 { r6(sum_d / sum_e) } else { 0.0 }; + let bound_now: Vec = stance.get("habits").and_then(|h| h.get("bound")).and_then(strs).unwrap_or_default(); + let (held, not_fired): (Vec, Vec) = id.keep.iter().cloned().partition(|h| bound_now.contains(h)); + let cue = if crossed.is_empty() { Json::Null } else { id.cue.as_deref().map_or(Json::Null, |c| Json::Str(c.into())) }; + let drift = r6(sum); + let doc = Json::Obj(vec![("drift".into(), Json::Num(drift)), ("allowance".into(), Json::Num(r6(sum_e))), ("used".into(), Json::Num(used)), + ("traits".into(), Json::Obj(rows)), ("warn".into(), Json::Num(id.warn)), ("crossed".into(), jstrs(&crossed)), + ("habits_held".into(), Json::Obj(vec![("held".into(), jstrs(&held)), ("not_fired".into(), jstrs(¬_fired))])), ("cue".into(), cue)]); + // canonical key order (the order the strand line is written in), so the stance's copy equals the line's + let doc = json::parse(&canon(&doc)).map_err(|e| perr("integrity", e.msg))?; + Ok(Integrity { doc, drift, used }) +} + +#[cfg(test)] +mod tests { + use super::*; + fn run(prog: &Json, f: &Flags) -> Json { run_program(prog, f, 1, 100, 0) } + const REF: &str = r#"{"probbit_persona":1,"identity":{"name":"I","version":"1","seed":0SELF}, + "traits":[{"id":"verbosity","levels":["terse","short","full"],"spread":0.5},{"id":"humour","levels":["none","light","playful"],"spread":0.5}], + "moods":[{"id":"valence","levels":["down","even","up"],"inertia":0.6,"half_life_hours":6}],"couplings":[{"vars":["valence","humour"],"align":0.8}], + "inputs":[{"id":"praise","kind":"flag"},{"id":"criticism","kind":"flag"},{"id":"loss","kind":"flag","effects":{"valence":-1.0}}], + "habits":[{"id":"quiet_after_loss","when":{"loss":true},"then":{"verbosity":["terse","short"]}}], + "learning":{"from":["praise","criticism"],"traits":["verbosity","humour"],"rate":0.5,"step_cap":0.2,"total_cap":1},"bound":{"seeds":"0-1","rest_hours":200}}"#; + fn persona(s: &str) -> Persona { build(&json::parse(&REF.replace("SELF", s)).unwrap()).unwrap() } + fn refused(s: &str) -> String { match build(&json::parse(&REF.replace("SELF", s)).unwrap()) { Err(e) => format!("{}: {}", e.path, e.msg), Ok(_) => "ok".into() } } + #[test] + fn self_keys_stay_out_of_the_digest_and_are_read_strictly() { + let (a, b) = (persona(""), persona(r#","keep":["quiet_after_loss"],"warn":0.5,"cue":"hold the line""#)); + assert_eq!(a.digest, b.digest); assert!(a.ident.is_none() && b.ident.is_some()); + assert_eq!(canon(&init(&a, Some(1), true, &run).to_json(&a)), canon(&init(&b, Some(1), true, &run).to_json(&b))); + assert_eq!(refused(r#","keep":["nope"]"#), "identity.keep[0]: a habit id of this persona"); + assert_eq!(refused(r#","warn":0"#), "identity.warn: a share of the allowance in (0, 1]"); + assert_eq!(refused(r#","cue":"""#), "identity.cue: a short line of text (1-120 characters, one line)"); + assert_eq!(refused(r#","traits":["valence"]"#), "identity.traits[0]: a trait id of this persona"); + assert_eq!(refused(r#","traits":["humour","humour"]"#), "identity.traits[1]: listed twice"); + } + #[test] + fn fresh_is_zero_drift_and_a_campaign_stays_under_the_allowance() { + let p = persona(r#","keep":["quiet_after_loss"],"warn":0.5,"cue":"hold the line""#); + let mut c = Cache::new(); let st = init(&p, Some(1), true, &run); + let (doc, _) = turn(&p, &st, &Json::Obj(vec![]), false, &run, false).unwrap(); + let i0 = compute_cached(&p, &st, &doc, &mut c, &run).unwrap().unwrap(); assert_eq!(i0.drift, 0.0, "{}", canon(&i0.doc)); + let mut s = st; let mut cued = false; + for n in 0..60 { let f = if n % 5 == 0 { "criticism" } else { "praise" }; + let (d, ns) = turn(&p, &s, &Json::Obj(vec![(f.into(), Json::Bool(true)), ("elapsed_hours".into(), Json::Num(0.1))]), false, &run, false).unwrap(); s = ns; + let i = compute_cached(&p, &s, &d, &mut c, &run).unwrap().unwrap(); + assert_eq!(canon(&i.doc), canon(&compute(&p, &s, &d, &run).unwrap().unwrap().doc), "the cache gives the uncached number"); + for t in ["verbosity", "humour"] { let u = i.doc.get("traits").and_then(|x| x.get(t)).and_then(|x| x.get("bound_used")).and_then(Json::as_f64).unwrap(); assert!(u <= 1.0 + 1e-5, "{t} {u}"); } + if i.doc.get("cue") == Some(&Json::Str("hold the line".into())) { cued = true; } } + assert!(cued, "the campaign crosses warn 0.5 somewhere"); + assert!(c.misses >= 2); + // moods move every loss event; the rested state does not see them: a run of losses after the campaign hits the cache + let before = (c.hits, c.misses); + for _ in 0..5 { let (d, ns) = turn(&p, &s, &Json::Obj(vec![("loss".into(), Json::Bool(true)), ("elapsed_hours".into(), Json::Num(0.1))]), false, &run, false).unwrap(); s = ns; + compute_cached(&p, &s, &d, &mut c, &run).unwrap().unwrap(); } + assert_eq!((c.hits - before.0, c.misses - before.1), (5, 0), "five mood-only events: five hits"); + } + /// RFC 0014-b: senses that move every event (the rested state's key never repeats): through the program key every number + /// equals the uncached computation's + #[cfg(feature = "std")] + #[test] + fn the_program_key_moves_no_number_when_the_senses_move() { + let f = concat!(env!("CARGO_MANIFEST_DIR"), "/../bench/bound/fixtures/sensed.yaml"); + let mut d = crate::yaml::load(&std::fs::read_to_string(f).unwrap()).unwrap_or_else(|e| panic!("{}", e.msg)); + if let Json::Obj(v) = &mut d { if let Some((_, Json::Obj(i))) = v.iter_mut().find(|(k, _)| k == "identity") { i.push(("warn".into(), Json::Num(0.8))); } } + let p = build(&d).unwrap(); let mut c = Cache::new(); let mut s = init(&p, Some(3), true, &run); + for n in 0..40u32 { let e = format!(r#"{{"senses":{{"tool_latency":{},"tempo":{}}},"src":"env","elapsed_hours":0.1{}}}"#, 0.5 + f64::from(n % 7) * 0.37, n % 5, if n % 9 == 4 { r#","praise":true"# } else { "" }); + let (doc, ns) = turn(&p, &s, &json::parse(&e).unwrap(), false, &run, false).unwrap(); s = ns; + let i = compute_cached(&p, &s, &doc, &mut c, &run).unwrap().unwrap(); + assert_eq!(canon(&i.doc), canon(&compute(&p, &s, &doc, &run).unwrap().unwrap().doc), "event {n}: the cache gives the uncached number"); } + eprintln!("R14B sensed: hits {} (by program {}) misses {}", c.hits, c.by_program, c.misses); + } + /// B3: every repo persona (examples/persona, bench/bound/fixtures) with the self keys declared: a fresh individual drifts 0 + #[cfg(feature = "std")] + #[test] + fn fresh_individual_has_zero_drift_on_every_repo_persona() { + let root = concat!(env!("CARGO_MANIFEST_DIR"), "/.."); + let mut n = 0; + for dir in ["examples/persona", "bench/bound/fixtures"] { + let mut fs: Vec<_> = std::fs::read_dir(format!("{root}/{dir}")).unwrap().filter_map(|e| e.ok().map(|e| e.path())).filter(|p| p.extension().is_some_and(|x| x == "yaml")).collect(); fs.sort(); + for f in fs { let mut d = crate::yaml::load(&std::fs::read_to_string(&f).unwrap()).unwrap_or_else(|e| panic!("{}: {}", f.display(), e.msg)); if d.get("probbit_persona").is_none() { continue; } + if let Json::Obj(v) = &mut d { if let Some((_, Json::Obj(i))) = v.iter_mut().find(|(k, _)| k == "identity") { i.push(("warn".into(), Json::Num(0.8))); } } + let p = build(&d).unwrap_or_else(|e| panic!("{}: {}: {}", f.display(), e.path, e.msg)); + for seed in [0u64, 7] { let st = init(&p, Some(seed), true, &run); + let i = compute(&p, &st, &Json::Null, &run).unwrap().unwrap(); + eprintln!("B3 fresh\t{dir}/{}\tseed {seed}\tdrift {}\tused {}\tallowance {}", f.file_name().unwrap().to_string_lossy(), i.drift, i.used, canon(i.doc.get("allowance").unwrap())); + assert_eq!(i.drift, 0.0, "{}", f.display()); n += 1; } } } + assert!(n >= 20, "{n}"); + } +} diff --git a/probbit-cli/src/json.rs b/probbit-persona/src/json.rs similarity index 99% rename from probbit-cli/src/json.rs rename to probbit-persona/src/json.rs index 6cebfaf..8f11cc1 100644 --- a/probbit-cli/src/json.rs +++ b/probbit-persona/src/json.rs @@ -1,4 +1,6 @@ //! Minimal JSON reader/writer (no external crates). Enough for the probbit problem/decision documents. +#[allow(unused_imports)] +use std::prelude::rust_2021::*; #[derive(Clone, Debug, PartialEq)] pub enum Json { Null, Bool(bool), Num(f64), Str(String), Arr(Vec), Obj(Vec<(String, Json)>) } diff --git a/probbit-persona/src/lib.rs b/probbit-persona/src/lib.rs new file mode 100644 index 0000000..313703c --- /dev/null +++ b/probbit-persona/src/lib.rs @@ -0,0 +1,43 @@ +//! probbit-persona: probbit's document layer and strand writer as a library, for the CLI, the browser module and a firmware. +//! +//! - `json`, `yaml`: the reader/writer of the JSON documents and the YAML subset of persona files (no external crates). +//! - `persona`: the individuality layer (docs/persona.md): parse, build, lint, prove, bound, hostlint, the turn, `display.glyphs`. +//! - `run`: `probbit run` on a probbit-ir JSON program; `router` (feature `decide`): `probbit decide` on a router document; +//! `evaluate` (feature `decide`): the decision-API adapter. +//! - `strand`: the strand writer (`Live`: start, event, control, checkpoint and rest lines, the `prev` chain) and its checks over +//! a `&str` (`Replay`, `verify*`, `read_actions`, `first_difference`), with no file I/O. +//! - `integrity`: RFC 0014, the drift from the declared self per turn (off without the `identity` self keys). +//! - `reflect`: RFC 0013, self-report: the canonical account of a window of a strand's events, from its lines alone (no solve). +//! - `trace`: with feature `turn_trace`, one event's time and a host probe's bytes by phase (`Live::event`). +//! - `sys`: the process's resource usage (getrusage on 64-bit Unix; `None` elsewhere). `hooks`: the host's monitor and config echo. +//! +//! Features: `std` (default) builds on std; `nostd` builds `no_std` + `alloc` for a target without an operating system, with +//! the std subset from probbit-nostd (docs/embedding.md). `decide` (default) adds `router` and `evaluate` (probbit-decide). `turn_trace` (off) records the +//! last event's phases. +#![cfg_attr(not(feature = "std"), no_std)] +#[cfg(not(feature = "std"))] +#[allow(unused_imports)] +#[macro_use] +extern crate probbit_nostd as std; +#[cfg(not(feature = "std"))] +#[allow(unused_imports)] +#[macro_use] +extern crate alloc; +#[cfg(not(any(feature = "std", feature = "nostd")))] +compile_error!("probbit-persona needs feature `std` (the default) or `nostd`"); + +#[cfg(feature = "decide")] +pub mod evaluate; +pub mod hooks; +pub mod json; +pub mod persona; +/// RFC 0014: `integrity::{Integrity, compute}` (the module lives under `persona`, beside RFC 0004's `bound`) +pub use persona::integrity; +pub mod reflect; +#[cfg(feature = "decide")] +pub mod router; +pub mod run; +pub mod strand; +pub mod sys; +pub mod trace; +pub mod yaml; diff --git a/probbit-cli/src/persona.rs b/probbit-persona/src/persona.rs similarity index 85% rename from probbit-cli/src/persona.rs rename to probbit-persona/src/persona.rs index 78431dc..20f045b 100644 --- a/probbit-cli/src/persona.rs +++ b/probbit-persona/src/persona.rs @@ -9,6 +9,8 @@ //! canonical number rule `canon_num`) and the reference implementation (the R30 prototype, Python) gives the same documents //! byte for byte on the examples (`tests/persona.rs`). The engine is a parameter (`Engine`): the CLI and `probbit mcp` run //! `run::run` on this process's threads, the browser module without threads; the answer is the same at fixed work. +#[allow(unused_imports)] +use std::prelude::rust_2021::*; use crate::json::{self, at, ix, InErr, Json}; use probbit_core::rt::Instant; use std::collections::HashMap; @@ -18,6 +20,16 @@ pub mod bound; /// The dead-rule lint: habits a host can never bring into force (`persona lint --host`) #[path = "hostlint.rs"] pub mod hostlint; +/// RFC 0014 integrity: the drift from the self the persona declares (off without the `identity` self keys) +#[path = "integrity.rs"] +pub mod integrity; +/// RFC 0016 curiosity: a declared drive fed by the measured surprise, and the self-start it gates (off without the block) +#[path = "curiosity.rs"] +pub mod curiosity; +pub use curiosity::{Curio, CUR_ID}; +/// RFC 0015 the rest pass: the expectation's memory consolidated at rest, one `consolidation` line (off without `expectation.rest`) +#[path = "rest.rs"] +pub mod rest; const RULE_KEYS: [&str; 6] = ["caps", "implies", "tables", "precedes", "linear", "all_different"]; const RESERVED_LEVELS: [&str; 9] = ["true", "false", "null", "yes", "no", "on", "off", "y", "n"]; @@ -47,48 +59,72 @@ pub fn py_repr(x: f64) -> String { } /// The canonical number (docs/persona.md §5.2): a whole number of magnitude <= 2^53 prints as an integer (`1`, `-2`, `0`), /// any other number as Python's repr (`0.43543`, `1e-05`). So `1`, `1.0` and `1e0` are one number in every digest. -fn canon_num(x: f64, s: &mut String) { +/// Where the canonical writer puts its text: a `String`, or a header streamed in pieces (`strand::write_header`) +pub trait Sink { fn push(&mut self, c: char); fn push_str(&mut self, t: &str); } +impl Sink for String { fn push(&mut self, c: char) { String::push(self, c) } fn push_str(&mut self, t: &str) { String::push_str(self, t) } } +fn canon_num(x: f64, s: &mut impl Sink) { if !x.is_finite() { s.push_str("null"); } else if x.fract() == 0.0 && x.abs() <= 9_007_199_254_740_992.0 { s.push_str(&format!("{}", x as i64)); } else { s.push_str(&py_repr(x)); } } /// Python's `json.dumps(..., ensure_ascii=False)` string escapes -fn esc(t: &str, s: &mut String) { +fn esc(t: &str, s: &mut impl Sink) { s.push('"'); for c in t.chars() { match c { '"' => s.push_str("\\\""), '\\' => s.push_str("\\\\"), '\n' => s.push_str("\\n"), '\r' => s.push_str("\\r"), '\t' => s.push_str("\\t"), '\u{8}' => s.push_str("\\b"), '\u{c}' => s.push_str("\\f"), c if (c as u32) < 0x20 => s.push_str(&format!("\\u{:04x}", c as u32)), c => s.push(c) } } s.push('"'); } -fn write(j: &Json, sort: bool, s: &mut String) { +/// Compact JSON of `j` into `s` (`sort`: keys sorted at every level, as `canon`; else insertion order, as `text`) +pub fn write_to(j: &Json, sort: bool, s: &mut impl Sink) { match j { Json::Null => s.push_str("null"), Json::Bool(b) => s.push_str(if *b { "true" } else { "false" }), Json::Num(x) => canon_num(*x, s), Json::Str(t) => esc(t, s), - Json::Arr(v) => { s.push('['); for (k, x) in v.iter().enumerate() { if k > 0 { s.push(','); } write(x, sort, s); } s.push(']'); } - Json::Obj(v) => { let mut o: Vec<&(String, Json)> = v.iter().collect(); if sort { o.sort_by(|a, b| a.0.cmp(&b.0)); } - s.push('{'); for (k, (key, x)) in o.into_iter().enumerate() { if k > 0 { s.push(','); } esc(key, s); s.push(':'); write(x, sort, s); } s.push('}'); } + Json::Arr(v) => { s.push('['); for (k, x) in v.iter().enumerate() { if k > 0 { s.push(','); } write_to(x, sort, s); } s.push(']'); } + Json::Obj(v) => { let mut o: Vec<&(String, Json)> = v.iter().collect(); if sort { sort_keys(&mut o); } + s.push('{'); for (k, (key, x)) in o.into_iter().enumerate() { if k > 0 { s.push(','); } esc(key, s); s.push(':'); write_to(x, sort, s); } s.push('}'); } } } +/// An object's entries sorted by key (by code point), stably: a binary insertion sort in place up to 512 entries (the library's +/// stable sort keeps a 4 KB scratch buffer in its frame, the largest frame of a canonical write on a small stack), the library's +/// above. Both are stable, so both give the one stable order: the bytes are the same. +fn sort_keys(o: &mut [&(String, Json)]) { + if o.len() > 512 { o.sort_by(|a, b| a.0.cmp(&b.0)); return; } + for i in 1..o.len() { let at = o[..i].partition_point(|x| x.0 <= o[i].0); o[at..=i].rotate_right(1); } +} /// Canonical JSON: keys sorted (by code point), no whitespace, UTF-8, canonical numbers. Every persona document is printed so. -pub fn canon(j: &Json) -> String { let mut s = String::new(); write(j, true, &mut s); s } +pub fn canon(j: &Json) -> String { let mut s = String::new(); write_to(j, true, &mut s); s } /// A program's text: insertion order (the compiler's fixed order), compact, canonical numbers. `engine.program` is its sha256. -pub fn text(j: &Json) -> String { let mut s = String::new(); write(j, false, &mut s); s } -pub fn sha256(data: &[u8]) -> [u8; 32] { - const K: [u32; 64] = [0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5, 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, - 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174, 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da, 0x983e5152, 0xa831c66d, - 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967, 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85, - 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070, 0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, - 0x5b9cca4f, 0x682e6ff3, 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2]; - let mut h: [u32; 8] = [0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19]; - let mut m = data.to_vec(); let bits = (data.len() as u64).wrapping_mul(8); m.push(0x80); while m.len() % 64 != 56 { m.push(0); } m.extend_from_slice(&bits.to_be_bytes()); - for c in m.chunks(64) { +pub fn text(j: &Json) -> String { let mut s = String::new(); write_to(j, false, &mut s); s } +/// sha256 of `data` (no copy of it: the streamed `Sha256`) +pub fn sha256(data: &[u8]) -> [u8; 32] { let mut h = Sha256::new(); h.update(data); h.finish() } +const K: [u32; 64] = [0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5, 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, + 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174, 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da, 0x983e5152, 0xa831c66d, + 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967, 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85, + 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070, 0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, + 0x5b9cca4f, 0x682e6ff3, 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2]; +/// sha256 fed in pieces: `update` any number of times, then `finish` (= `sha256` of the pieces concatenated) +pub struct Sha256 { h: [u32; 8], buf: [u8; 64], fill: usize, len: u64 } +impl Default for Sha256 { fn default() -> Self { Self::new() } } +impl Sha256 { + pub fn new() -> Sha256 { Sha256 { h: [0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19], buf: [0; 64], fill: 0, len: 0 } } + pub fn update(&mut self, mut d: &[u8]) { + self.len = self.len.wrapping_add(d.len() as u64); + while !d.is_empty() { let k = (64 - self.fill).min(d.len()); self.buf[self.fill..self.fill + k].copy_from_slice(&d[..k]); self.fill += k; d = &d[k..]; + if self.fill == 64 { let b = self.buf; self.block(&b); self.fill = 0; } } + } + pub fn finish(mut self) -> [u8; 32] { + let bits = self.len.wrapping_mul(8); let mut pad = [0u8; 72]; pad[0] = 0x80; let z = (119 - self.fill) % 64; // 0x80, zeros, then 8 length bytes: to a 64-byte boundary + pad[1 + z..9 + z].copy_from_slice(&bits.to_be_bytes()); let len = self.len; self.update(&pad[..9 + z]); self.len = len; + let mut out = [0u8; 32]; for (i, x) in self.h.iter().enumerate() { out[4 * i..4 * i + 4].copy_from_slice(&x.to_be_bytes()); } out + } + fn block(&mut self, c: &[u8; 64]) { let mut w = [0u32; 64]; for t in 0..16 { w[t] = u32::from_be_bytes([c[4 * t], c[4 * t + 1], c[4 * t + 2], c[4 * t + 3]]); } for t in 16..64 { let s0 = w[t - 15].rotate_right(7) ^ w[t - 15].rotate_right(18) ^ (w[t - 15] >> 3); let s1 = w[t - 2].rotate_right(17) ^ w[t - 2].rotate_right(19) ^ (w[t - 2] >> 10); w[t] = w[t - 16].wrapping_add(s0).wrapping_add(w[t - 7]).wrapping_add(s1); } - let mut v = h; + let mut v = self.h; for t in 0..64 { let (a, b, cc, d, e, f, g, hh) = (v[0], v[1], v[2], v[3], v[4], v[5], v[6], v[7]); let t1 = hh.wrapping_add(e.rotate_right(6) ^ e.rotate_right(11) ^ e.rotate_right(25)).wrapping_add((e & f) ^ (!e & g)).wrapping_add(K[t]).wrapping_add(w[t]); let t2 = (a.rotate_right(2) ^ a.rotate_right(13) ^ a.rotate_right(22)).wrapping_add((a & b) ^ (a & cc) ^ (b & cc)); v = [t1.wrapping_add(t2), a, b, cc, d.wrapping_add(t1), e, f, g]; } - for (x, y) in h.iter_mut().zip(v) { *x = x.wrapping_add(y); } + for (x, y) in self.h.iter_mut().zip(v) { *x = x.wrapping_add(y); } } - let mut out = [0u8; 32]; for (i, x) in h.iter().enumerate() { out[4 * i..4 * i + 4].copy_from_slice(&x.to_be_bytes()); } out } pub fn hex(b: &[u8]) -> String { b.iter().map(|x| format!("{x:02x}")).collect() } /// "sha256:" of a text @@ -100,7 +136,7 @@ pub fn sha(j: &Json) -> String { digest_of(&canon(j)) } /// the same way; equal on 300,015 test doubles) fn round_to(x: f64, n: usize) -> f64 { format!("{x:.n$}").parse().unwrap_or(x) } /// Weights and odds: rounded to 6 decimals, -0 -> 0 -pub(crate) fn r6(x: f64) -> f64 { let v = round_to(x, 6); if v == 0.0 { 0.0 } else { v } } +pub fn r6(x: f64) -> f64 { let v = round_to(x, 6); if v == 0.0 { 0.0 } else { v } } /// A number as Python's `%s` prints it in a message (bounds are whole numbers or Python floats) fn pyn(x: f64) -> String { let mut s = String::new(); canon_num(x, &mut s); s } /// A standard normal number that is a pure function of its parts: Box-Muller on the first 16 bytes of @@ -137,7 +173,11 @@ struct Hist { id: String, of: String, streak: bool, window: usize, cap: usize, s struct Habit { id: String, when: Vec<(String, Json)>, then: Vec<(String, Json)>, rules: Vec<(String, Vec)>, say: String, priority: f64 } /// engine settings (docs/persona.md §2.7) #[derive(Clone)] -struct Eng { op: String, sweeps: usize, polish_sweeps: usize, exact_limit: u64, chains: usize, twin: bool, positional: bool, yields: bool } +struct Eng { op: String, sweeps: usize, polish_sweeps: usize, exact_limit: u64, chains: usize, twin: bool, positional: bool, yields: bool, + /// RFC 0011 (research): `tier: amplitude` (off when absent) and its `q_max` (default 12); round 4: `amplify` (default false) + amp: bool, q_max: usize, amplify: bool, + /// round 5: `schedule` (a name in `amp::SCHEDULE_NAMES`; absent = the default sweep, the receipt unchanged) + schedule: Option } /// bounded learning (docs/persona.md §2.8): `from` = the reward flag, then (optional) the correction flag; the traits whose learned /// deltas move; the step and total caps (natural-log weights); `credit` = action-addressed feedback, when declared #[derive(Clone)] @@ -181,7 +221,11 @@ pub struct Persona { pub name: String, pub version: String, pub seed: u64, pub d /// RFC 0009: the `expectation` block and `engine.trajectory` (both outside the digest; effects {}) pub expect: Option, pub traj: Option, /// §2.18: the `display.glyphs` block (outside the digest; effects {}) - pub glyphs: Option } + pub glyphs: Option, + /// RFC 0014: the `identity` block's self keys (outside the digest; effects {}) + pub ident: Option, + /// RFC 0016: the `curiosity` drive (in the digest: its level moves the stance through `cur_level`) + pub cur: Option } impl Persona { /// `line.why_past` (§2.11): `why` names what the learned weights came from; default on with a senses block, off without fn why_past(&self) -> bool { self.why_past.unwrap_or(self.senses.is_some()) } @@ -203,7 +247,7 @@ impl Persona { /// `load_level` with a load block (RFC 0002 part 2): the engine sets it from L fn is_load(&self, id: &str) -> bool { self.load.is_some() && id == LOAD_ID } /// An input the engine sets from its own state (pace's two, load's one): never the host's - fn is_owned(&self, id: &str) -> bool { self.is_pace(id) || self.is_load(id) } + fn is_owned(&self, id: &str) -> bool { self.is_pace(id) || self.is_load(id) || (self.cur.is_some() && id == CUR_ID) } fn var(&self, id: &str) -> Option<&Var> { self.vars.iter().find(|v| v.id == id) } fn input(&self, id: &str) -> Option<&Input> { self.inputs.iter().find(|x| x.id == id) } fn habit(&self, id: &str) -> &Habit { self.habits.iter().find(|h| h.id == id).expect("habit id") } @@ -292,7 +336,12 @@ fn effect(e: &Json, path: &str, levels: &[String]) -> R> { } } +/// A persona file: `.json` = JSON, anything else the YAML subset. -> (persona, the document). Without `std` there are no files: +/// an error (the document comes in as a string: `parse_doc` + `build`). +#[cfg(not(feature = "std"))] +pub fn load(path: &str) -> R<(Persona, Json)> { Err(perr(path, "cannot read the persona file: no file system on this target (pass the document as a string: parse_doc + build)")) } /// A persona file: `.json` = JSON, anything else the YAML subset. -> (persona, the document) +#[cfg(feature = "std")] pub fn load(path: &str) -> R<(Persona, Json)> { let b = std::fs::read(path).map_err(|e| perr(path, format!("cannot read the persona file: {e}")))?; let t = String::from_utf8(b).map_err(|e| perr(path, format!("not UTF-8 (byte {})", e.utf8_error().valid_up_to())))?; @@ -306,15 +355,29 @@ pub fn parse_doc(t: &str, is_json: bool) -> Result { } /// The strict validator: document -> persona (every field type-checked, unknown fields refused at their path). -pub fn build(doc: &Json) -> R { +pub fn build(doc: &Json) -> R { build_boxed(doc).map(|p| *p) } +/// `build` on the heap: the persona is returned boxed and every layer below (`display`, `bound`, `expectation`, the lowered +/// blocks) passes it on boxed, so no frame on the way holds a `Persona` by value; the heavy readers are out of line, so the +/// recursion through the layers holds only this dispatcher's frame (docs/embedding.md "Stack budget"). +#[inline(never)] +pub fn build_boxed(doc: &Json) -> R> { + // RFC 0014: the `identity` block's self keys move nothing (effects {}): they are read against the persona built without them + // and stay out of the digest, so declaring them on a persona with a recorded life leaves every state as it was + if let Some(idn) = doc.get("identity").filter(|i| integrity::declared(i)) { + let mut d = doc.clone(); + if let Json::Obj(v) = &mut d { if let Some((_, Json::Obj(i))) = v.iter_mut().find(|(k, _)| k == "identity") { i.retain(|(k, _)| !integrity::KEYS.contains(&k.as_str())); } } + let mut p = build_boxed(&d)?; let id = integrity::spec(idn, doc, &p)?; p.ident = Some(id); return Ok(p); } // §2.18: `display` moves nothing (effects {}): it is read against the persona built without it and stays out of the digest, // so adding it to a persona with a recorded life leaves every state and stance digest as it was if let Some(dj) = doc.get("display") { let mut d = doc.clone(); if let Json::Obj(v) = &mut d { v.retain(|(k, _)| k != "display"); } - let mut p = build(&d)?; p.glyphs = display_spec(dj, &p)?; return Ok(p); } + let mut p = build_boxed(&d)?; p.glyphs = display_spec(dj, &p)?; return Ok(p); } // RFC 0004: a `bound` block is read against the persona built without it, and it stays out of the digest, so adding it to a // persona with a recorded life leaves every state, strand and turn byte for byte the same (amendment A1) if let Some(b) = doc.get("bound") { let mut d = doc.clone(); if let Json::Obj(v) = &mut d { v.retain(|(k, _)| k != "bound"); } - let p = build(&d)?; bound::spec(b, &p)?; return Ok(p); } + let p = build_boxed(&d)?; bound::spec(b, &p)?; return Ok(p); } + // RFC 0016: `curiosity` lowers to the engine-owned number input `cur_level` (box [0, cap], carrying the block's effects, as + // pace lowers to its two) and enters the digest; it needs an expectation block (the drive reads its measured surprise) + if let Some(cj) = doc.get("curiosity").filter(|c| !c.is_null()) { return build_curiosity(doc, cj); } // RFC 0009: `expectation` and `engine.trajectory` move nothing (effects {}): they are read against the persona built without // them and stay out of the digest, so adding either to a persona with a recorded life leaves every state and stance digest as // it was (an `engine` mapping left empty is dropped with them) @@ -325,10 +388,43 @@ pub fn build(doc: &Json) -> R { if let Some(i) = v.iter().position(|(k, _)| k == "engine") { if let Json::Obj(e) = &mut v[i].1 { if let Some(j) = e.iter().position(|(k, _)| k == "trajectory") { tj = Some(e.remove(j).1); } } if v[i].1.as_obj().is_some_and(|e| e.is_empty()) { v.remove(i); } } } - let mut p = build(&d)?; + let mut p = build_boxed(&d)?; p.expect = match ex { None | Some(Json::Null) => None, Some(x) => Some(expect_spec(&x)?) }; p.traj = match tj { None | Some(Json::Null) => None, Some(x) => Some(traj_spec(&x)?) }; return Ok(p); } + build_base(doc) +} +/// RFC 0016: the persona with a `curiosity` block (C-R1 .. C-R3 refuse at the field's path): built without the block and with its +/// input `cur_level` after the declared inputs; the digest is the document's as given (without the keys outside the digest) +#[inline(never)] +fn build_curiosity(doc: &Json, cj: &Json) -> R> { + need(doc.get("expectation").is_some_and(|e| !e.is_null()), "curiosity", "a curiosity block needs an expectation block (the drive reads its measured surprise)")?; + let ids_of = |k: &str| -> Vec { match doc.get(k) { Some(Json::Arr(a)) => a.iter().filter_map(|t| t.get("id").and_then(Json::as_str).map(str::to_string)).collect(), _ => vec![] } }; + let mut ids = ids_of("traits"); ids.extend(ids_of("moods")); + let ins: Vec = match doc.get("inputs") { Some(Json::Arr(a)) => a.clone(), _ => vec![] }; + let flags: Vec = ins.iter().filter(|x| x.get("kind").and_then(Json::as_str) == Some("flag")).filter_map(|x| x.get("id").and_then(Json::as_str).map(str::to_string)).collect(); + let (cu, eff) = curiosity::spec(cj, &ids, &flags)?; + for (j, x) in ins.iter().enumerate() { if let Some(id) = x.get("id").and_then(Json::as_str) { + need(id != CUR_ID && id != curiosity::STIR_KEY, &format!("inputs[{j}].id"), "reserved for the curiosity block (cur_level, stir)")?; } } + if let Some(Json::Arr(hs)) = doc.get("habits") { for (i, h) in hs.iter().enumerate() { if let Some(w) = h.get("when").and_then(Json::as_obj) { for (k, _) in w { + need(k != CUR_ID && k != curiosity::STIR_KEY, &format!("habits[{i}].when.{k}"), "reserved for the curiosity block")?; } } } } + let mut d = doc.clone(); let mut dg = doc.clone(); + if let Json::Obj(v) = &mut d { v.retain(|(k, _)| k != "curiosity"); + let x = Json::Obj(vec![("id".into(), Json::Str(CUR_ID.into())), ("kind".into(), Json::Str("number".into())), ("max".into(), Json::Num(cu.cap)), + ("say".into(), Json::Str("curiosity".into())), ("effects".into(), eff)]); + let mut ins = ins; ins.push(x); + match v.iter_mut().find(|(k, _)| k == "inputs") { Some(e) => e.1 = Json::Arr(ins), None => v.push(("inputs".into(), Json::Arr(ins))) } } + // the digest's document: as given, without `expectation` and `engine.trajectory` (outside every digest) + if let Json::Obj(v) = &mut dg { v.retain(|(k, _)| k != "expectation"); + if let Some(i) = v.iter().position(|(k, _)| k == "engine") { + if let Json::Obj(e) = &mut v[i].1 { e.retain(|(k, _)| k != "trajectory"); } + if v[i].1.as_obj().is_some_and(|e| e.is_empty()) { v.remove(i); } } } + let mut p = build_boxed(&d)?; p.digest = sha(&dg); p.cur = Some(cu); + Ok(p) +} +/// The persona without the layers `build_boxed` peels off: the top-level keys, then the lowered blocks or the base reader +#[inline(never)] +fn build_base(doc: &Json) -> R> { const TOP: [&str; 22] = ["probbit_persona", "identity", "traits", "moods", "couplings", "inputs", "history", "habits", "agenda", "engine", "line", "learning", "drives", "reward_from", "senses", "needs", "surprise", "pace", "dispositions", "rest", "load", "comment"]; let kv = keys(doc, "", &TOP, &["probbit_persona", "identity", "traits"])?; @@ -436,8 +532,8 @@ pub fn build(doc: &Json) -> R { for (i, hb) in list(get(kv, "habits"), "habits")?.iter().enumerate() { habits.push(habit_spec(hb, &ix("habits", i), &inputs, &history, &vars, &steps)?); } need((1..habits.len()).all(|i| habits[..i].iter().all(|h| h.id != habits[i].id)), "habits", "habit ids must be distinct")?; // engine - let mut eng = Eng { op: "decide".into(), sweeps: 2000, polish_sweeps: 200, exact_limit: 2_000_000, chains: 4, twin: true, positional: true, yields: false }; - if let Some(e) = get_t(kv, "engine") { let ekv = keys(e, "engine", &["op", "sweeps", "polish_sweeps", "exact_limit", "chains", "twin", "values", "on_conflict", "comment"], &[])?; + let mut eng = Eng { op: "decide".into(), sweeps: 2000, polish_sweeps: 200, exact_limit: 2_000_000, chains: 4, twin: true, positional: true, yields: false, amp: false, q_max: 12, amplify: false, schedule: None }; + if let Some(e) = get_t(kv, "engine") { let ekv = keys(e, "engine", &["op", "sweeps", "polish_sweeps", "exact_limit", "chains", "twin", "values", "on_conflict", "tier", "q_max", "amplify", "schedule", "comment"], &[])?; for (k, x) in ekv { let p = at("engine", k); match k.as_str() { "op" => eng.op = x.as_str().filter(|o| ["decide", "sample"].contains(o)).ok_or_else(|| perr(&p, "decide | sample"))?.to_string(), @@ -445,11 +541,17 @@ pub fn build(doc: &Json) -> R { "values" => eng.positional = match x.as_str() { Some("positional") => true, Some("semantic") => false, _ => return Err(perr(&p, "positional | semantic")) }, "on_conflict" => eng.yields = match x.as_str() { Some("fallback") => false, Some("yield") => true, _ => return Err(perr(&p, "fallback | yield")) }, "comment" => {} + "tier" => { need(x.as_str() == Some("amplitude"), &p, "amplitude (the one optional tier; absent = the default tiers)")?; eng.amp = true; } + "q_max" => eng.q_max = num(Some(x), &p, Some(1.0), Some(16.0), true)? as usize, + "amplify" => eng.amplify = if let Json::Bool(b) = x { *b } else { return Err(perr(&p, "true | false")) }, + "schedule" => eng.schedule = Some(x.as_str().filter(|n| SCHEDULE_NAMES.contains(n)).ok_or_else(|| perr(&p, "tf-linear-16x4 | tf-linear-64x16 | qaoa-p2"))?.to_string()), "polish_sweeps" => eng.polish_sweeps = num(Some(x), &p, Some(0.0), Some(1e9), true)? as usize, "sweeps" => eng.sweeps = num(Some(x), &p, Some(1.0), Some(1e9), true)? as usize, "exact_limit" => eng.exact_limit = num(Some(x), &p, Some(1.0), Some(1e9), true)? as u64, _ => eng.chains = num(Some(x), &p, Some(1.0), Some(MAX_CHAINS), true)? as usize, } } } + if eng.amplify && !eng.amp { return Err(perr(&at("engine", "amplify"), "true needs tier: amplitude (amplification acts on the amplitude register)")); } + if eng.schedule.is_some() && !eng.amp { return Err(perr(&at("engine", "schedule"), "needs tier: amplitude (the schedule is the amplitude register's evolution)")); } // line let lkv: &[(String, Json)] = match get_t(kv, "line") { Some(l) => keys(l, "line", &["max_tokens", "order", "prefix", "why_past", "block", "comment"], &[])?, None => &[] }; // RFC 0008 E: `line.block` puts the engine-rendered block in every stance document (a persona with a bound or a coupling has it) @@ -461,7 +563,7 @@ pub fn build(doc: &Json) -> R { let prefix = match get(lkv, "prefix") { None => "Stance: ".to_string(), Some(Json::Str(s)) => s.clone(), Some(_) => return Err(perr("line.prefix", "a string")) }; let learning = match get(kv, "learning").filter(|l| !l.is_null()) { None => None, Some(l) => Some(learning_spec(l, &vars, &inputs)?) }; let reward = match get(kv, "reward_from").filter(|x| !x.is_null()) { None => None, Some(rf) => Some(reward_spec(rf, &learning.as_ref().map_or(vec![], |l| l.from.clone()), &inputs)?) }; - let p = Persona { name, version, seed, digest, vars, steps, slots, say_order, prefer, couplings, inputs, history, habits, eng, max_tokens, order, prefix, learning, drives: None, reward, senses: None, why_past, surprise: None, pace: None, disp: None, rest_b: None, load: None, block, expect: None, traj: None, glyphs: None }; + let p = Box::new(Persona { name, version, seed, digest, vars, steps, slots, say_order, prefer, couplings, inputs, history, habits, eng, max_tokens, order, prefix, learning, drives: None, reward, senses: None, why_past, surprise: None, pace: None, disp: None, rest_b: None, load: None, block, expect: None, traj: None, glyphs: None, ident: None, cur: None }); // the fallback stance must obey every unconditional habit (checked once, here) let mut fb: HashMap = p.vars.iter().map(|v| (v.id.clone(), v.fallback.clone())).collect(); for (i, s) in p.steps.iter().enumerate() { fb.insert(format!("step.{s}"), p.slots[i].clone()); } @@ -479,7 +581,8 @@ const BODY: [&str; 5] = ["surprise", "pace", "dispositions", "rest", "load"]; /// field's path): `pace` lowers to the number inputs `pace_up` (box [0, cap − r0]) and `pace_down` (box [0, r0]) carrying its /// effects, after the declared inputs; `surprise` and `dispositions` lower to nothing (they act in the credit and drive steps). The /// rest of the document is read as it is without them. -fn build_body(doc: &Json) -> R { +#[inline(never)] +fn build_body(doc: &Json) -> R> { let kv = doc.as_obj().unwrap(); let block = |k: &str| get(kv, k).filter(|x| !x.is_null()); let has = |k: &str| block(k).is_some(); // Design B: the surprise gate (V-R1, V-R2); the defaults are the values the stand-in passed with let surprise = match block("surprise") { None => None, Some(sj) => { @@ -591,7 +694,7 @@ fn build_body(doc: &Json) -> R { if !ld.effects.is_empty() { x.push(("effects".into(), Json::Obj(ld.effects.iter().map(|(k, w)| (k.clone(), Json::Num(*w))).collect()))); } ins.push(Json::Obj(x)); match d2.iter_mut().find(|(k, _)| k == "inputs") { Some(x) => x.1 = Json::Arr(ins), None => d2.push(("inputs".into(), Json::Arr(ins))) } } - let mut p = build(&Json::Obj(d2))?; + let mut p = build_boxed(&Json::Obj(d2))?; p.digest = sha(doc); p.surprise = surprise; p.pace = pace; p.disp = disp; p.rest_b = rest_b; p.load = load; Ok(p) } @@ -1065,7 +1168,8 @@ fn needs_spec(nj: &Json, goals: &[Goal], senses: &[Sense], win_max: f64, moods: } /// A persona with a `drives` block: the block read strictly, then lowered (see `Drives`) and read as a 0.7.0 persona; the digest is /// the whole document's, the block included. A goal floor needs `pursue` free of multi-variable rules (the lift's soundness, §2.9). -fn build_drives(doc: &Json, senses: &[Sense]) -> R { +#[inline(never)] +fn build_drives(doc: &Json, senses: &[Sense]) -> R> { let kv = doc.as_obj().unwrap(); let drv = get(kv, "drives").unwrap(); let dkv = keys(drv, "drives", &["goals", "wanting", "afterglow", "expectation", "pursue", "effects", "learn_from_surprise", "valence", "comment"], &[])?; let gl = match get(dkv, "goals") { Some(Json::Arr(a)) if (2..=7).contains(&a.len()) => a, _ => return Err(perr("drives.goals", "2 to 7 goals (pursue is one variable, 2-7 levels)")) }; @@ -1178,7 +1282,7 @@ fn build_drives(doc: &Json, senses: &[Sense]) -> R { let synthetic: Vec = ins[n_decl..].iter().map(|x| x.get("id").and_then(Json::as_str).unwrap_or("").to_string()).collect(); let ii = slot(&mut d2, "inputs"); d2[ii].1 = Json::Arr(ins); // an error in a synthetic input is reported at the drives path it comes from - let mut p = build(&Json::Obj(d2)).map_err(|e| { for (j, o) in origins.iter().enumerate() { let pre = format!("inputs[{}]", n_decl + j); + let mut p = build_boxed(&Json::Obj(d2)).map_err(|e| { for (j, o) in origins.iter().enumerate() { let pre = format!("inputs[{}]", n_decl + j); if let Some(rest) = e.path.strip_prefix(&pre).filter(|r| r.is_empty() || r.starts_with('.') || r.starts_with('[')) { return perr(&format!("{o}{}", rest.strip_prefix(".effects").unwrap_or("")), e.msg); } } e })?; p.digest = sha(doc); if goals.iter().any(|g| g.floor > 0.0) { for h in &p.habits { for (k, lst) in &h.rules { for r in lst { let vs = rule_vars(k, r); @@ -1390,12 +1494,20 @@ fn host_inputs(d: &Drives, ds: &DState, c: &DriveCtx, raw: &[(String, Json)]) -> /// against the normals, `src` the event's source, `check` the provenance check), then with a drives block the stepped drives and /// the host inputs plus the drive aggregates; then with a pace block (RFC 0002 Design C) the pace step and its two inputs; else the /// state and inputs as given -fn prep(p: &Persona, st: &State, raw: &[(String, Json)], src: Option<&Json>, check: bool) -> R<(State, Vec<(String, Json)>)> { +fn prep(p: &Persona, st: &State, raw: &[(String, Json)], src: Option<&Json>, check: bool) -> R<(State, Vec<(String, Json)>)> { prep_n(p, st, raw, src, check, false) } +/// `prep`; `narrow`: an expectation member's (`State::prep_clone`) +#[inline(always)] +fn prep_n(p: &Persona, st: &State, raw: &[(String, Json)], src: Option<&Json>, check: bool, narrow: bool) -> R<(State, Vec<(String, Json)>)> { // RFC 0002 part 2: the engine sets load_level from L (the host never sends it) if p.load.is_some() { need(get(raw, LOAD_ID).is_none(), "inputs.load_level", "reserved for the load block (the engine sets it from the load)")?; } - let (mut s2, mut host) = prep_body(p, st, raw, src, check)?; + // RFC 0016: the engine sets cur_level from the drive's fold (the host never sends it) + if p.cur.is_some() { need(get(raw, CUR_ID).is_none(), "inputs.cur_level", "reserved for the curiosity block (the engine sets it from the drive)")?; } + let (mut s2, mut host) = prep_body(p, st, raw, src, check, narrow)?; pressure_step(p, &mut s2, raw)?; load_step(p, st, &mut s2, raw, &mut host)?; + // RFC 0016: the level the live writer set from the fold before this event (0 without a memory: `persona turn`); a stir's + // marker is the line's, not the compiler's + if p.cur.is_some() { host.retain(|(k, _)| k != curiosity::STIR_KEY); host.push((CUR_ID.into(), Json::Num(st.cur.unwrap_or(0.0)))); } Ok((s2, host)) } /// Does this event carry feedback (RFC 0002 part 2: a rest line never shares an event with a learning step)? A learning flag on, @@ -1425,6 +1537,9 @@ fn rest_apply(p: &Persona, st: &State) -> State { /// The rest line due before an event (RFC 0002 part 2), for the strand: its `rest` object (downscale, prune, the pressure, the /// learned weights after it, and with a surprise block each learned trait's gate as kept) and the state after it; None when no /// line is due. `ev` = the stamped event as the turn reads it. +/// `rest_line` as `Live::event` reads it: the `rest` object and the digest of the state after it (the turn applies the rest itself) +#[inline(never)] +pub(crate) fn rest_line_digest(p: &Persona, st: &State, ev: &Json) -> Option<(Json, String)> { rest_line(p, st, ev).map(|(r, s)| (r, s.digest)) } pub fn rest_line(p: &Persona, st: &State, ev: &Json) -> Option<(Json, State)> { let Json::Obj(raw) = ev else { return None }; if !rest_due(p, st, raw) { return None; } @@ -1465,8 +1580,8 @@ pub fn load_level(p: &Persona, st: &State) -> Option<(f64, f64)> { p.load.as_ref /// The rest pressure of a state with a rest block (RFC 0002 part 2): (S, pending); none without one pub fn rest_pressure(st: &State) -> Option<(f64, bool)> { st.pressure } /// prep without part 2: the sense, needs, drive and pace steps -fn prep_body(p: &Persona, st: &State, raw: &[(String, Json)], src: Option<&Json>, check: bool) -> R<(State, Vec<(String, Json)>)> { - let mut s2 = st.clone(); let mut x = raw.to_vec(); +fn prep_body(p: &Persona, st: &State, raw: &[(String, Json)], src: Option<&Json>, check: bool, narrow: bool) -> R<(State, Vec<(String, Json)>)> { + let mut s2 = st.prep_clone((p.senses.is_some() && st.senses.is_some(), p.drives.is_some() && st.drives.is_some()), narrow); let mut x = raw.to_vec(); // RFC 0002 Design C: the engine sets pace_up and pace_down from R̄ (the host never sends them) if p.pace.is_some() { for k in PACE_IDS { need(get(&x, k).is_none(), &format!("inputs.{k}"), "reserved for the pace block (the engine sets it from the average reward)")?; } } if let (Some(sn), Some(ss)) = (&p.senses, &st.senses) { let (ns, host) = sense_step(sn, ss, &x, src, check)?; s2.senses = Some(ns); x = host; } @@ -1498,10 +1613,11 @@ struct Pay { wait: bool, pays: Option<(u64, Option)>, cancel: Option bool { s.len() == 71 && s.starts_with("sha256:") && s[7..].bytes().all(|c| c.is_ascii_digit() || (b'a'..=b'f').contains(&c)) } -/// A stance's digest (§2.8): sha256 of the stance document's canonical JSON without `credit`, `state_digest` and `timing`. The +/// A stance's digest (§2.8): sha256 of the stance document's canonical JSON without `credit`, `state_digest`, `timing` and +/// `integrity` (RFC 0014, computed after the turn). The /// state remembers it (an open decision's, the previous stance's) without hashing itself, and `for: {turn, stance}` binds it. pub fn stance_digest(doc: &Json) -> String { - match doc { Json::Obj(v) => sha(&Json::Obj(v.iter().filter(|(k, _)| !["credit", "state_digest", "timing"].contains(&k.as_str())).cloned().collect())), _ => sha(doc) } + match doc { Json::Obj(v) => sha(&Json::Obj(v.iter().filter(|(k, _)| !["credit", "state_digest", "timing", "integrity"].contains(&k.as_str())).cloned().collect())), _ => sha(doc) } } /// With a credit block: `await`, `for` and `cancel` taken out of the inputs and read (B-R5: a malformed value refuses the event); /// without one they stay undeclared inputs (listed in `ignored`) @@ -1671,6 +1787,8 @@ fn stance_body(j: &mut Json, p: &Persona, st: &State) { // RFC 0002 part 2: load_level leaves `inputs`; `load: {l}` = L after this event if let Some(l) = st.load_l { if let Some((_, Json::Obj(ins))) = v.iter_mut().find(|(k, _)| k == "inputs") { ins.retain(|(k, _)| k != LOAD_ID); } v.push(("load".into(), Json::Obj(vec![("l".into(), Json::Num(l))]))); } + // RFC 0016: cur_level leaves `inputs` (the live writer adds `drives.curiosity` with the level and its two inputs) + if p.cur.is_some() { if let Some((_, Json::Obj(ins))) = v.iter_mut().find(|(k, _)| k == "inputs") { ins.retain(|(k, _)| k != CUR_ID); } } if let Some(gs) = &st.gate { v.push(("surprise".into(), Json::Obj(gs.iter().filter(|(_, g)| g.g > 1.0).map(|(t, g)| (t.clone(), Json::Num(g.g))).collect()))); } } /// The pace level of an average (RFC 0002 Design C): slow below `levels[0]`, brisk above `levels[1]`, steady between @@ -1879,7 +1997,8 @@ fn sense_cond_text(sn: &Senses, c: &SenseCond) -> String { } /// A persona with a `senses` block: the block read strictly, then lowered (see `Senses`) and read as a persona without it (a drives /// block, if any, lowers next); the digest is the whole document's, the block included -fn build_senses(doc: &Json) -> R { +#[inline(never)] +fn build_senses(doc: &Json) -> R> { let kv = doc.as_obj().unwrap(); let sl = match get(kv, "senses") { Some(Json::Arr(a)) if (1..=24).contains(&a.len()) => a, _ => return Err(perr("senses", "a list of 1 to 24 senses")) }; let mut senses: Vec = vec![]; @@ -1925,7 +2044,7 @@ fn build_senses(doc: &Json) -> R { let ii = slot(&mut d2, "inputs"); d2[ii].1 = Json::Arr(ins); // an error in a synthetic input is reported at the senses path it comes from // a drives block lowers next, with the senses at hand (a need may grow per reading of a number sense, RFC 0002 A.1) - let lowered = if get(&d2, "drives").is_some() { build_drives(&Json::Obj(d2), &senses) } else { build(&Json::Obj(d2)) }; + let lowered = if get(&d2, "drives").is_some() { build_drives(&Json::Obj(d2), &senses) } else { build_boxed(&Json::Obj(d2)) }; let mut p = lowered.map_err(|e| { for (j, o) in origins.iter().enumerate() { let pre = format!("inputs[{}]", n_decl + j); if let Some(rest) = e.path.strip_prefix(&pre).filter(|r| r.is_empty() || r.starts_with('.') || r.starts_with('[')) { return perr(&format!("{o}{}", rest.strip_prefix(".effects").unwrap_or("")), e.msg); } } e })?; p.digest = sha(doc); @@ -2162,7 +2281,10 @@ fn values_of(p: &Persona) -> Vec { pub struct State { pub seed: u64, pub turn: u64, persona: [String; 3], genes: Json, mood: Vec<(String, Vec)>, history: Vec<(String, Vec)>, prev: HashMap, agenda: Option>, rest: Option>, learned: Table, credit: Table, drives: Option, senses: Option, learned_from: Option>, open: Vec, prev_stance: Option, control_closed: Vec, elig: f64, pub digest: String, - gate: Option>, rbar: Option, disp: Vec<(String, f64)>, pressure: Option<(f64, bool)>, load_l: Option } + gate: Option>, rbar: Option, disp: Vec<(String, f64)>, pressure: Option<(f64, bool)>, load_l: Option, + /// RFC 0016: the curiosity drive's level the next turn reads (set from the expectation's memory before each live event; not + /// part of the state's JSON or digest: the memory is the fold of the logged lines, so a replay sets it the same way) + pub cur: Option } /// A decision awaiting its outcome (§2.8, `open_credit`): its turn, its credit (onehot(level) − odds per learned trait it released) /// and its stance's digest #[derive(Clone)] @@ -2253,6 +2375,18 @@ impl State { fn disp_of(&self, k: &str) -> Option { self.disp.iter().find(|(x, _)| x == k).map(|(_, v)| *v) } /// The surprise gain per learned trait in `learning.traits` order (RFC 0002 Design B); empty without a surprise block (gain 1) fn gains(&self) -> Vec { self.gate.as_ref().map_or(vec![], |gs| gs.iter().map(|(_, g)| g.g).collect()) } + /// The prep step's copy of the state (`prep_body`): the senses and drive states it steps are not copied (`stepped`: the + /// senses' then the drives' are replaced before anything reads them); `narrow` (an expectation member: its state is read + /// by the steps and the compiler only) also leaves out what neither reads: the persona's name, the agenda, the resting + /// stance, the learned-from counts and the digest + fn prep_clone(&self, stepped: (bool, bool), narrow: bool) -> State { + let w = |b: bool| !(narrow && b); + State { seed: self.seed, turn: self.turn, persona: if w(true) { self.persona.clone() } else { Default::default() }, genes: self.genes.clone(), mood: self.mood.clone(), + history: self.history.clone(), prev: self.prev.clone(), agenda: self.agenda.as_ref().filter(|_| w(true)).cloned(), rest: self.rest.as_ref().filter(|_| w(true)).cloned(), + learned: self.learned.clone(), credit: self.credit.clone(), drives: if stepped.1 { None } else { self.drives.clone() }, senses: if stepped.0 { None } else { self.senses.clone() }, + learned_from: self.learned_from.as_ref().filter(|_| w(true)).cloned(), open: self.open.clone(), prev_stance: self.prev_stance.clone(), control_closed: self.control_closed.clone(), + elig: self.elig, digest: if w(true) { self.digest.clone() } else { String::new() }, gate: self.gate.clone(), rbar: self.rbar, disp: self.disp.clone(), pressure: self.pressure, load_l: self.load_l, cur: self.cur } + } fn history_of(&self, of: &str) -> &[bool] { self.history.iter().find(|(k, _)| k == of).map_or(&[], |(_, v)| v) } /// The state document without its digest fn body(&self, p: &Persona) -> Json { @@ -2408,7 +2542,7 @@ impl State { _ => return Err(perr("state.prev_stance", "the previous stance's digest (sha256:<64 hex>); null before any turn")) }; let ca = get(kv, "control_closed").and_then(Json::as_arr).filter(|a| a.len() <= c.max_open).ok_or_else(|| perr("state.control_closed", format!("a list of at most {} turns", c.max_open)))?; for (i, x) in ca.iter().enumerate() { let n = turn_at(x, &ix("state.control_closed", i), control_closed.last().copied())?; control_closed.push(n); } } - Ok(State { seed, turn, persona, genes: g.clone(), mood, history, prev, agenda, rest, learned, credit, drives, senses, learned_from, open, prev_stance, control_closed, elig: 1.0, gate, rbar, disp, pressure, load_l, + Ok(State { seed, turn, persona, genes: g.clone(), mood, history, prev, agenda, rest, learned, credit, drives, senses, learned_from, open, prev_stance, control_closed, elig: 1.0, gate, rbar, disp, pressure, load_l, cur: None, digest: get(kv, "digest").and_then(Json::as_str).unwrap_or("").to_string() }) } } @@ -2440,10 +2574,17 @@ fn flags(p: &Persona, st: &State) -> Flags { #[derive(Clone)] struct Meta { vals: Vec<(String, Json)>, hist: Vec<(String, f64)>, ignored: Vec, active: Vec, conditional: Vec, active_inputs: Vec, rules: Vec>, owners: Vec, new_mood: Vec<(String, Vec)>, new_learned: Table, contrib: Vec)>>, field: Vec>, values: Vec, twin: bool, - lift: Vec<(String, f64)>, learn_sign: f64 } + lift: Vec<(String, f64)>, learn_sign: f64, seed: u64, turn: u64 } #[derive(Default, Clone)] -struct Opts { no_inertia: bool, clamps: Vec<(String, String)>, twin: Option, exclude: Vec, hist: Option> } +struct Opts { no_inertia: bool, clamps: Vec<(String, String)>, twin: Option, exclude: Vec, hist: Option>, + /// RFC 0009 (the expectation's members): the compiler's arithmetic and the rules in force, no program text (Json::Null) + meta_only: bool } +/// A compile meta with only what the expectation's members read of it (the field, the rules and habits in force, the values, +/// the twin flag, seed and turn: `member_text` and the rules' comparison), on the heap +fn slim(m: Meta) -> Box { + Box::new(Meta { vals: vec![], hist: vec![], ignored: vec![], conditional: vec![], active_inputs: vec![], owners: vec![], new_mood: vec![], new_learned: Default::default(), contrib: vec![], lift: vec![], ..m }) +} /// Inputs -> (values per declared input, history features, elapsed hours, ignored ids); a bad value is a persona error #[allow(clippy::type_complexity)] fn resolve_inputs(p: &Persona, st: &State, raw: &[(String, Json)]) -> R<(Vec<(String, Json)>, Vec<(String, f64)>, f64, Vec)> { @@ -2473,12 +2614,15 @@ fn cond_true(p: &Persona, k: &str, c: &Json, vals: &[(String, Json)], hist: &[(S fn jstrs(v: &[String]) -> Json { Json::Arr(v.iter().map(|s| Json::Str(s.clone())).collect()) } /// persona + state + inputs -> (the turn's probbit-ir program, meta). A pure function; every weight rounded to 6 decimals. +#[inline(never)] fn compile(p: &Persona, st: &State, raw: &[(String, Json)], o: &Opts) -> R<(Json, Meta)> { let (vals, mut hist, elapsed, ignored) = resolve_inputs(p, st, raw)?; if let Some(h) = &o.hist { hist = h.clone(); } // `prove`: history feature values given, not derived from the state let (nv, ns) = (p.vars.len(), p.steps.len()); let tix = |t: &str| -> usize { match t.strip_prefix("step.") { Some(s) => nv + p.steps.iter().position(|x| x == s).unwrap(), None => p.vars.iter().position(|v| v.id == t).unwrap() } }; let mut contrib: Vec)>> = vec![vec![]; nv + ns]; + // a contribution's source names the turn's `why` (and `explain`); a member's meta (`meta_only`) is never read for it: unnamed + let nm = |s: &str| if o.meta_only { String::new() } else { s.to_string() }; let mut mood_ev: Vec>> = p.vars.iter().map(|v| v.mood.then(|| vec![0.0; v.levels.len()])).collect(); // the learned deltas after this turn's feedback (the reward flag +1, the correction flag -1) on the previous stance's credit // with a drives block (§2.9): the turn's drive context, read from the (stepped) drive state; its surprise joins the learning sign @@ -2488,11 +2632,11 @@ fn compile(p: &Persona, st: &State, raw: &[(String, Json)], o: &Opts) -> R<(Json let sign = l.from.iter().enumerate().fold(0.0, |s, (i, id)| if matches!(get(&vals, id), Some(Json::Bool(true))) { s + if i == 0 { 1.0 } else { -1.0 } } else { s }); let sign = match &dctx { Some(c) => (sign + c.learn_sign).clamp(-1.0, 1.0), None => sign }; learn_sign = sign; learn(l, &st.credit, sign, st.elig, &st.learned, &st.gains(), st.disp_of("optimism")) } }; - for (i, v) in p.vars.iter().enumerate() { contrib[i].push(("prior".into(), v.base.clone())); - if let Some(g) = st.shift(&v.id) { contrib[i].push(("genes".into(), centered(v.levels.len()).iter().map(|c| g * c).collect())); } - if let Some((_, d)) = new_learned.iter().find(|(t, d)| *t == v.id && d.iter().any(|x| *x != 0.0)) { contrib[i].push(("learned".into(), d.clone())); } } - for (j, s) in p.steps.iter().enumerate() { contrib[nv + j].push(("prior".into(), vec![0.0; ns])); - if let Some((_, pf)) = p.prefer.iter().find(|(t, _)| *t == format!("step.{s}")) { contrib[nv + j].push(("prefer".into(), pf.clone())); } } + for (i, v) in p.vars.iter().enumerate() { contrib[i].push((nm("prior"), v.base.clone())); + if let Some(g) = st.shift(&v.id) { contrib[i].push((nm("genes"), centered(v.levels.len()).iter().map(|c| g * c).collect())); } + if let Some((_, d)) = new_learned.iter().find(|(t, d)| *t == v.id && d.iter().any(|x| *x != 0.0)) { contrib[i].push((nm("learned"), d.clone())); } } + for (j, s) in p.steps.iter().enumerate() { contrib[nv + j].push((nm("prior"), vec![0.0; ns])); + if let Some((_, pf)) = p.prefer.iter().find(|(t, _)| *t == format!("step.{s}")) { contrib[nv + j].push((nm("prefer"), pf.clone())); } } let mut active_inputs = vec![]; let mut push = |t: &str, src: String, w: Vec, contrib: &mut Vec)>>| { let i = tix(t); match mood_ev.get_mut(i).and_then(Option::as_mut) { Some(ev) => for (a, b) in ev.iter_mut().zip(&w) { *a += b; }, None => contrib[i].push((src, w)) } }; @@ -2502,17 +2646,17 @@ fn compile(p: &Persona, st: &State, raw: &[(String, Json)], o: &Opts) -> R<(Json Kind::Number => (&x.effects, v.as_f64().unwrap()), Kind::Level => (x.by_level.iter().find(|(l, _)| Some(l.as_str()) == v.as_str()).map_or(&[][..], |(_, e)| &e[..]), 1.0) }; if !eff.is_empty() && scale != 0.0 { active_inputs.push(x.id.clone()); } - for (t, vec) in eff { push(t, format!("input:{}", x.id), vec.iter().map(|y| g * scale * y).collect(), &mut contrib); } } + for (t, vec) in eff { push(t, if o.meta_only { String::new() } else { format!("input:{}", x.id) }, vec.iter().map(|y| g * scale * y).collect(), &mut contrib); } } for h in &p.history { let n = hist.iter().find(|(k, _)| *k == h.id).unwrap().1.min(h.cap as f64); if n != 0.0 && !h.effects.is_empty() { active_inputs.push(h.id.clone()); } - for (t, vec) in &h.effects { push(t, format!("history:{}", h.id), vec.iter().map(|y| n * y).collect(), &mut contrib); } } + for (t, vec) in &h.effects { push(t, if o.meta_only { String::new() } else { format!("history:{}", h.id) }, vec.iter().map(|y| n * y).collect(), &mut contrib); } } let mut new_mood = vec![]; for (i, v) in p.vars.iter().enumerate().filter(|(_, v)| v.mood) { let k = if o.no_inertia { 0.0 } else { v.inertia }; let decay = match v.half_life { Some(hl) if elapsed > 0.0 => 0.5f64.powf(elapsed / hl), _ => 1.0 }; let prev = &st.mood.iter().find(|(id, _)| *id == v.id).unwrap().1; let m: Vec = prev.iter().zip(mood_ev[i].as_ref().unwrap()).map(|(a, e)| r6(k * decay * a + (1.0 - k) * e)).collect(); - if m.iter().any(|&x| x != 0.0) { contrib[i].push(("mood_accumulator".into(), m.clone())); } + if m.iter().any(|&x| x != 0.0) { contrib[i].push((nm("mood_accumulator"), m.clone())); } new_mood.push((v.id.clone(), m)); } // habits in force and their rules let active: Vec<&Habit> = p.habits.iter().filter(|h| !o.exclude.contains(&h.id) && h.when.iter().all(|(k, c)| cond_true(p, k, c, &vals, &hist))).collect(); @@ -2520,15 +2664,10 @@ fn compile(p: &Persona, st: &State, raw: &[(String, Json)], o: &Opts) -> R<(Json let mut prev = st.prev.clone(); for v in &p.vars { prev.entry(v.id.clone()).or_insert_with(|| v.fallback.clone()); } for h in &active { habit_rules(p, h, &prev, &mut rules, &mut owners); } let owners = ordered_owners(owners); - // the program: values, vars (traits and moods, then the agenda steps), pairs, rules; positional = level i -> l - let pos_vals = p.eng.positional; let sem = values_of(p); - let lmax = p.vars.iter().map(|v| v.levels.len()).max().unwrap_or(0); - let values: Vec = if pos_vals { (0..lmax).map(|i| format!("l{i}")).chain(p.slots.iter().cloned()).collect() } else { sem.clone() }; - let name_of = |var: &str, val: &str| -> String { let id = var.strip_prefix("free.").unwrap_or(var); - match p.var(id) { Some(v) if pos_vals => format!("l{}", v.levels.iter().position(|l| l == val).unwrap_or(0)), _ => val.to_string() } }; + let sem = values_of(p); // `pursue`'s drive terms after its prior, learned, input and history contributions: genes, wanting, afterglow, deadline (§3 step 2b) let pix = dctx.as_ref().and_then(|_| p.vars.iter().position(|v| v.id == "pursue")); - if let (Some(c), Some(i)) = (&dctx, pix) { for (src, vec) in &c.terms { if vec.iter().any(|x| *x != 0.0) { contrib[i].push((src.to_string(), vec.clone())); } } } + if let (Some(c), Some(i)) = (&dctx, pix) { for (src, vec) in &c.terms { if vec.iter().any(|x| *x != 0.0) { contrib[i].push((nm(*src), vec.clone())); } } } let total = |c: &[(String, Vec)], n: usize| -> Vec { let mut tot = vec![0.0; n]; for (_, vec) in c { for (a, b) in tot.iter_mut().zip(vec) { *a += b; } } tot.into_iter().map(r6).collect() }; let mut field: Vec> = (0..nv + ns).map(|i| total(&contrib[i], if i < nv { p.vars[i].levels.len() } else { ns })).collect(); // the floor lift (§2.9): on the rounded field, over the levels the habits in force allow; then the field is summed again @@ -2536,28 +2675,50 @@ fn compile(p: &Persona, st: &State, raw: &[(String, Json)], o: &Opts) -> R<(Json if let (Some(d), Some(i)) = (&p.drives, pix) { if d.goals.iter().any(|g| g.floor > 0.0) { let ti = RULE_KEYS.iter().position(|k| *k == "tables").unwrap(); let l = floor_lifts(p, d, &field[i], &pursue_allowed(&p.vars[i].levels, &rules[ti])); - if l.iter().any(|x| *x != 0.0) { contrib[i].push(("floor".into(), l.clone())); field[i] = total(&contrib[i], l.len()); } + if l.iter().any(|x| *x != 0.0) { contrib[i].push((nm("floor"), l.clone())); field[i] = total(&contrib[i], l.len()); } lift = d.goals.iter().zip(&l).filter(|(_, x)| **x != 0.0).map(|(g, x)| (g.id.clone(), *x)).collect(); } } + let use_twin = o.twin.unwrap_or(p.eng.twin); + let ids: Vec = active.iter().map(|h| h.id.clone()).collect(); + // the text before the meta, as the compiler always wrote it (the meta's fields move in after) + let prog = if o.meta_only { Json::Null } else { emit(p, &field, &rules, &ids, (st.seed, st.turn), use_twin, &sem, &o.clamps, true) }; + let conditional = active.iter().filter(|h| !h.when.is_empty()).map(|h| h.id.clone()).collect(); + let meta = Meta { vals, hist, ignored, active: ids, conditional, active_inputs, rules, owners, new_mood, new_learned, contrib, field, values: sem, twin: use_twin, lift, learn_sign, + seed: st.seed, turn: st.turn }; + Ok((prog, meta)) +} +/// The program's text from the compiler's meta: values, vars (traits and moods, then the agenda steps) with their `h` from the +/// field, pairs, the rules in force (renamed), the comment (persona, seed, turn, active habits); positional = level i -> l. +/// Inlined in `compile` (its frame as before); a member's text is written in a frame of its own (`member_text`) +#[inline(always)] +#[allow(clippy::too_many_arguments)] +/// `with_pairs: false`: no `pairs` (T3-c: a member's rules read by `run::from_json_parts`, the pairs are the skeleton's) +#[allow(clippy::too_many_arguments)] +fn emit(p: &Persona, field: &[Vec], rules: &[Vec], active: &[String], (seed, turn): (u64, u64), use_twin: bool, sem: &[String], clamps: &[(String, String)], with_pairs: bool) -> Json { + let (nv, ns) = (p.vars.len(), p.steps.len()); + let pos_vals = p.eng.positional; + let lmax = p.vars.iter().map(|v| v.levels.len()).max().unwrap_or(0); + let values: Vec = if pos_vals { (0..lmax).map(|i| format!("l{i}")).chain(p.slots.iter().cloned()).collect() } else { sem.to_vec() }; + let name_of = |var: &str, val: &str| -> String { let id = var.strip_prefix("free.").unwrap_or(var); + match p.var(id) { Some(v) if pos_vals => format!("l{}", v.levels.iter().position(|l| l == val).unwrap_or(0)), _ => val.to_string() } }; let mut pv = vec![]; for (i, v) in p.vars.iter().enumerate() { let mut var = vec![("id".to_string(), Json::Str(v.id.clone())), ("allowed".into(), Json::Arr(v.levels.iter().map(|l| Json::Str(name_of(&v.id, l))).collect())), ("h".into(), Json::Obj(v.levels.iter().zip(&field[i]).filter(|(_, w)| **w != 0.0).map(|(l, w)| (name_of(&v.id, l), Json::Num(*w))).collect()))]; - if let Some((_, c)) = o.clamps.iter().find(|(t, _)| *t == v.id) { var.push(("clamp".into(), Json::Str(name_of(&v.id, c)))); } + if let Some((_, c)) = clamps.iter().find(|(t, _)| *t == v.id) { var.push(("clamp".into(), Json::Str(name_of(&v.id, c)))); } pv.push(Json::Obj(var)); } for (j, s) in p.steps.iter().enumerate() { pv.push(Json::Obj(vec![("id".into(), Json::Str(format!("step.{s}"))), ("allowed".into(), jstrs(&p.slots)), ("h".into(), Json::Obj(p.slots.iter().zip(&field[nv + j]).filter(|(_, w)| **w != 0.0).map(|(l, w)| (l.clone(), Json::Num(*w))).collect()))])); } let k = values.len(); let vpos = |v: &str| values.iter().position(|x| x == v).unwrap(); let mut pairs = vec![]; - for (a, b, tab) in &p.couplings { + for (a, b, tab) in p.couplings.iter().filter(|_| with_pairs) { let (la, lb) = (&p.var(a).unwrap().levels, &p.var(b).unwrap().levels); let mut t = vec![vec![0.0; k]; k]; for (x, l1) in la.iter().enumerate() { for (y, l2) in lb.iter().enumerate() { let w = r6(tab[x][y]); if w != 0.0 || !pos_vals { t[vpos(&name_of(a, l1))][vpos(&name_of(b, l2))] = w; } } } pairs.push((a.clone(), b.clone(), t)); } let pair_json = |i: &str, j: &str, t: &[Vec]| Json::Obj(vec![("i".into(), Json::Str(i.into())), ("j".into(), Json::Str(j.into())), ("table".into(), Json::Arr(t.iter().map(|r| Json::Arr(r.iter().map(|x| Json::Num(*x)).collect())).collect()))]); - let comment = format!("probbit persona {} {} seed {} turn {}; active habits: {}", p.name, p.version, st.seed, st.turn, - if active.is_empty() { "none".to_string() } else { active.iter().map(|h| h.id.as_str()).collect::>().join(", ") }); - let use_twin = o.twin.unwrap_or(p.eng.twin); + let comment = format!("probbit persona {} {} seed {} turn {}; active habits: {}", p.name, p.version, seed, turn, + if active.is_empty() { "none".to_string() } else { active.join(", ") }); let mut twin_vars = vec![]; if use_twin { for v in &pv { let Json::Obj(f) = v else { continue }; twin_vars.push(Json::Obj(f.iter().filter(|(k, _)| k != "clamp") .map(|(k, x)| if k == "id" { (k.clone(), Json::Str(format!("free.{}", x.as_str().unwrap()))) } else { (k.clone(), x.clone()) }).collect())); } } @@ -2572,9 +2733,7 @@ fn compile(p: &Persona, st: &State, raw: &[(String, Json)], o: &Opts) -> R<(Json lst.extend(rules[ki].iter().map(|r| rename(r, key, &name_of))); if *key == "all_different" && ns > 0 && use_twin { lst.push(Json::Obj(vec![("vars".into(), Json::Arr(p.steps.iter().map(|s| Json::Str(format!("free.step.{s}"))).collect()))])); } if !lst.is_empty() { prog.push((key.to_string(), Json::Arr(lst))); } } - let conditional = active.iter().filter(|h| !h.when.is_empty()).map(|h| h.id.clone()).collect(); - let meta = Meta { vals, hist, ignored, active: active.iter().map(|h| h.id.clone()).collect(), conditional, active_inputs, rules, owners, new_mood, new_learned, contrib, field, values: sem, twin: use_twin, lift, learn_sign }; - Ok((Json::Obj(prog), meta)) + Json::Obj(prog) } /// A habit rule with its level names renamed to the program's values (positional: l) fn rename(r: &Json, key: &str, name_of: &dyn Fn(&str, &str) -> String) -> Json { @@ -2596,7 +2755,9 @@ struct Entry { level: String, p: f64, odds: Vec<(String, f64)>, released: bool, /// A stance document under construction (docs/persona.md §4) struct Out { status: String, verdict: Json, tier: Json, program: String, stance: Vec<(String, Entry)>, mood: Vec<(String, Entry)>, active: Vec, bound: Vec, violations: usize, conflict: Vec, yielded: Vec, unsure: Vec, escalate: Option, inputs: Vec<(String, Json)>, ignored: Vec, - line: String, why: String, agenda: Option>, state_digest: String, held: Vec, timing: Option, lift: Vec<(String, f64)> } + line: String, why: String, agenda: Option>, state_digest: String, held: Vec, timing: Option, lift: Vec<(String, f64)>, + /// RFC 0011: the amplitude tier's receipt (an object) or why it declined (a string); None without the flag + amp: Option } impl Out { fn to_json(&self, p: &Persona, st: &State) -> Json { let e = |m: &[(String, Entry)]| Json::Obj(m.iter().map(|(k, e)| { let odds = |o: &[(String, f64)]| Json::Obj(o.iter().map(|(l, x)| (l.clone(), Json::Num(*x))).collect()); @@ -2605,7 +2766,8 @@ impl Out { let s = |x: &str| Json::Str(x.to_string()); let mut v = vec![("probbit_persona_turn".to_string(), Json::Num(1.0)), ("persona".into(), Json::Obj(vec![("name".into(), s(&p.name)), ("version".into(), s(&p.version)), ("digest".into(), s(&p.digest))])), ("seed".into(), Json::Num(st.seed as f64)), ("turn".into(), Json::Num(st.turn as f64)), ("status".into(), s(&self.status)), - ("engine".into(), Json::Obj(vec![("verdict".into(), self.verdict.clone()), ("tier".into(), self.tier.clone()), ("program".into(), s(&self.program))])), + ("engine".into(), { let mut e = vec![("verdict".into(), self.verdict.clone()), ("tier".into(), self.tier.clone()), ("program".into(), s(&self.program))]; + match &self.amp { Some(Json::Obj(r)) => e.extend(r.iter().cloned()), Some(w) => e.push(("amplitude".into(), w.clone())), None => {} } Json::Obj(e) }), ("stance".into(), e(&self.stance)), ("mood".into(), e(&self.mood)), ("habits".into(), Json::Obj(vec![("active".into(), jstrs(&self.active)), ("bound".into(), jstrs(&self.bound)), ("violations".into(), Json::Num(self.violations as f64)), ("conflict".into(), jstrs(&self.conflict)), ("yielded".into(), jstrs(&self.yielded))])), @@ -2716,6 +2878,7 @@ fn back(p: &Persona, var: &str, val: &str) -> String { fn printed(x: f64) -> f64 { r6(if x.fract() == 0.0 { x } else { (x * 1e6).round() / 1e6 }) } /// The engine's answer -> (stance document, new state) +#[inline(never)] fn decode(p: &Persona, st: &State, program: &str, meta: &Meta, doc: &Json, conflict: Option<&[String]>, yielded: Option<&[String]>) -> (Out, State) { let verdict = doc.get("verdict").cloned().unwrap_or(Json::Null); let vs = verdict.as_str().unwrap_or(""); let plan: HashMap = doc.get("plan").and_then(Json::as_obj).map_or_else(HashMap::new, |o| o.iter().map(|(k, v)| (k.clone(), back(p, k, v.as_str().unwrap_or("")))).collect()); @@ -2785,15 +2948,32 @@ fn decode(p: &Persona, st: &State, program: &str, meta: &Meta, doc: &Json, confl ns.seal(p); let out = Out { status, verdict: verdict.clone(), tier: doc.get("tier").cloned().unwrap_or(Json::Null), program: program.to_string(), stance, mood, active: meta.active.clone(), bound, violations: viol, conflict: conflict.to_vec(), yielded: yielded.to_vec(), unsure, escalate, inputs: meta.vals.iter().cloned().chain(meta.hist.iter().map(|(k, x)| (k.clone(), Json::Num(*x)))).collect(), - ignored: meta.ignored.clone(), line: ln, why: w, agenda: (!p.steps.is_empty()).then_some(order_steps), state_digest: ns.digest.clone(), held: vec![], timing: None, lift: meta.lift.clone() }; + ignored: meta.ignored.clone(), line: ln, why: w, agenda: (!p.steps.is_empty()).then_some(order_steps), state_digest: ns.digest.clone(), held: vec![], timing: None, lift: meta.lift.clone(), amp: doc.get("amplitude").cloned() }; (out, ns) } // ------------------------------------------------------------------------------------------------------------ turns -fn solve(p: &Persona, st: &State, prog: &Json, eng: Engine, calls: &mut usize) -> Json { *calls += 1; eng(prog, &flags(p, st)) } +fn solve(p: &Persona, st: &State, prog: &Json, eng: Engine, calls: &mut usize) -> Json { + *calls += 1; let was = crate::trace::enter(crate::trace::Phase::Solve); + let r = if p.eng.amp { amplitude_tier(prog, p.eng.q_max, p.eng.amplify, p.eng.schedule.as_deref()).unwrap_or_else(|why| declined(eng(prog, &flags(p, st)), &why)) } else { eng(prog, &flags(p, st)) }; + crate::trace::enter(was); r +} +/// RFC 0011 (research): the amplitude tier on a program (`run::amplitude`); Err(why) when it does not take it +#[cfg(feature = "decide")] +fn amplitude_tier(prog: &Json, q_max: usize, amplify: bool, schedule: Option<&str>) -> Result { + match schedule.and_then(probbit_decide::amp::Evolution::named) { Some(ev) => crate::run::amplitude_with(prog, q_max, amplify, &ev), None => crate::run::amplitude(prog, q_max, amplify) } +} +#[cfg(not(feature = "decide"))] +fn amplitude_tier(_: &Json, _: usize, _: bool, _: Option<&str>) -> Result { Err("this build has no amplitude tier (feature decide)".into()) } +/// RFC 0011 round 5 (research): the names `engine.schedule` accepts (`probbit_decide::amp::SCHEDULE_NAMES`; checked by a test) +const SCHEDULE_NAMES: [&str; 3] = ["tf-linear-16x4", "tf-linear-64x16", "qaoa-p2"]; +/// The existing tiers' answer, with the receipt of why the amplitude tier did not run +fn declined(mut doc: Json, why: &str) -> Json { if let Json::Obj(v) = &mut doc { v.push(("amplitude".into(), Json::Str(format!("declined: {why}")))); } doc +} fn ms(t: Instant) -> f64 { t.elapsed().as_secs_f64() * 1e3 } /// A minimal set of active habits that cannot hold together (deletion filter: drop each habit in turn, keep it only if the rest /// becomes feasible without it); [] when the rules without any habit are already infeasible +#[inline(never)] fn habit_conflict(p: &Persona, st: &State, raw: &[(String, Json)], active: &[String], o: &Opts, eng: Engine, calls: &mut usize) -> R> { let mut feasible = |ex: &[String]| -> R { let mut oo = Opts { no_inertia: o.no_inertia, twin: Some(false), hist: o.hist.clone(), ..Default::default() }; oo.exclude = o.exclude.iter().chain(ex).cloned().collect(); let (prog, _) = compile(p, st, raw, &oo)?; Ok(solve(p, st, &prog, eng, calls).get("verdict").and_then(Json::as_str) != Some("infeasible")) }; @@ -2805,11 +2985,46 @@ fn habit_conflict(p: &Persona, st: &State, raw: &[(String, Json)], active: &[Str } /// One persona turn: (stance document, new state). A pure function of (persona, state, inputs, engine version). `timing` adds a /// non-canonical `timing` object (compile / engine / decode ms, engine calls, the 1-minute load average). -pub fn turn(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, timing: bool) -> R<(Json, State)> { turn_src(p, st, raw, no_inertia, eng, timing, true) } +pub fn turn(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, timing: bool) -> R<(Json, State)> { turn_src(p, st, raw, no_inertia, eng, timing, true).map(|(j, s)| (j, *s)) } +/// `turn` with the next state on the heap (`Live::event` runs it): the turn's steps are out of line and pass the state boxed, so +/// no frame of the turn holds a `State` by value and the deepest frame is one step's (docs/embedding.md "Stack budget") +pub fn turn_boxed(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, timing: bool) -> R<(Json, Box)> { turn_src(p, st, raw, no_inertia, eng, timing, true) } /// `turn` without the provenance check (§2.10), for `fuzz` and `prove`: they search over stances, not over sources, so their events /// carry no `src` and stand for events of an accepted source -pub fn turn_any_source(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, timing: bool) -> R<(Json, State)> { turn_src(p, st, raw, no_inertia, eng, timing, false) } -fn turn_src(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, timing: bool, check: bool) -> R<(Json, State)> { +pub fn turn_any_source(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, timing: bool) -> R<(Json, State)> { turn_src(p, st, raw, no_inertia, eng, timing, false).map(|(j, s)| (j, *s)) } +/// A turn's inputs and state after the steps before the compiler (`turn_prep`): the stepped state, the compiler's inputs, the +/// inputs as given, the event's `src`, its payments and what they settled +struct Prepped { st: Box, raw: Vec<(String, Json)>, given: Vec<(String, Json)>, src: Option, pay: Option, settled: Option, fresh: bool } +/// The engine's answer read back (`turn_solve`): the stance under construction, the next state and, when asked for, what the +/// expectation reads of the turn's first program (`At`; None inside: that answer was not exact) +struct Solved { out: Box, ns: Box, at: Option>> } +/// RFC 0009: the turn's first program as the deviation's A_t (the same program: `turn_at`): its exact marginals read as +/// `exact_marg` reads them and its compile meta (the fields the members read), so A_t is neither built nor solved again +pub struct At { marg: Marg, meta: Box } +#[inline(never)] +fn turn_src(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, timing: bool, check: bool) -> R<(Json, Box)> { + turn_src_at(p, st, raw, no_inertia, eng, timing, check, false).map(|(j, s, _)| (j, s)) +} +/// `turn_boxed` (no inertia off, no timing, the provenance check on) that also hands the expectation its A_t when `want` +/// (`Live::event` with the cache on): Some when the turn's first program is the program the deviation would build for the +/// event (no rest due before it, `src` taken out alike, the persona's op `decide`), Some(None) inside when its answer is not +/// exact (then there is no deviation); None: the deviation builds and solves A_t itself +#[inline(never)] +pub fn turn_at(p: &Persona, st: &State, raw: &Json, eng: Engine, want: bool) -> R<(Json, Box, Option>>)> { turn_src_at(p, st, raw, false, eng, false, true, want) } +#[inline(always)] +#[allow(clippy::too_many_arguments)] +fn turn_src_at(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, timing: bool, check: bool, want: bool) -> R<(Json, Box, Option>>)> { + let pr = turn_prep(p, st, raw, check)?; + // the deviation's A_t is `program_meta` of the same state and inputs: no rest, `src` taken out only with these blocks + let want = want && check && pr.fresh && !no_inertia && p.eng.op == "decide" && (p.reward.is_none() || p.senses.is_some() || p.credit().is_some() || p.has_needs()); + let mut sv = turn_solve(p, &pr, no_inertia, eng, timing, want, None)?; let at = sv.at.take(); + let (j, s) = turn_doc(p, pr, sv); Ok((j, s, at)) +} +/// The steps before the compiler: the inputs read, `src` and the payments taken out, a rest due applied, then the sense / needs / +/// drive / pace steps, the credit step and the surprise gate, each on the heap +#[inline(never)] +fn turn_prep(p: &Persona, st: &State, raw: &Json, check: bool) -> R { + crate::trace::enter(crate::trace::Phase::Prep); let raw: Vec<(String, Json)> = match raw { Json::Null => vec![], Json::Obj(v) => v.clone(), _ => return Err(perr("inputs", "must be a JSON object")) }; for (n, (k, _)) in raw.iter().enumerate() { if raw[..n].iter().any(|(k2, _)| k2 == k) { return Err(perr(&format!("inputs.{k}"), "duplicate input")); } } // reward provenance (§2.10), first: with `reward_from` (or a senses block, §2.11), `src` is read here (not compiled, so not @@ -2821,38 +3036,71 @@ fn turn_src(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, // a senses block steps the normals and a drives block the drives before the turn, which runs on the stepped state with the // lowering's inputs; then the credit step settles what this turn's feedback pays // RFC 0002 part 2: a rest line due before this event scales the learned weights first (the strand logs it as its own line) - let rested; let st = if rest_due(p, st, &raw) { rested = rest_apply(p, st); &rested } else { st }; - let given = raw; let (st2, raw) = prep(p, st, &given, src.as_ref(), check)?; - let (st2, settled) = match &pay { Some(k) => { let (s, d) = credit_step(p, &st2, &raw, k, src.as_ref(), check)?; (s, Some(d)) } None => (st2, None) }; + let rested = if rest_due(p, st, &raw) { Some(rest_boxed(p, st)) } else { None }; + let st = rested.as_deref().unwrap_or(st); + let given = raw; let (st2, raw) = prep_boxed(p, st, &given, src.as_ref(), check)?; + let (st2, settled) = match &pay { Some(k) => { let (s, d) = credit_boxed(p, &st2, &raw, k, src.as_ref(), check)?; (s, Some(d)) } None => (st2, None) }; // RFC 0002 Design B: the surprise gate steps v, r̂, g and the counts of each paid learned trait before the learner reads g - let st2 = gate_step(p, st2, &raw)?; - let st = &st2; + let st2 = gate_boxed(p, st2, &raw)?; + Ok(Prepped { st: st2, raw, given, src, pay, settled, fresh: rested.is_none() }) +} +#[inline(never)] +fn rest_boxed(p: &Persona, st: &State) -> Box { Box::new(rest_apply(p, st)) } +#[inline(never)] +fn prep_boxed(p: &Persona, st: &State, raw: &[(String, Json)], src: Option<&Json>, check: bool) -> R<(Box, Vec<(String, Json)>)> { let (s, x) = prep(p, st, raw, src, check)?; Ok((Box::new(s), x)) } +/// `prep_boxed` for an expectation member (`narrow`) +#[inline(never)] +fn prep_boxed_n(p: &Persona, st: &State, raw: &[(String, Json)], src: Option<&Json>, check: bool, narrow: bool) -> R<(Box, Vec<(String, Json)>)> { let (s, x) = prep_n(p, st, raw, src, check, narrow)?; Ok((Box::new(s), x)) } +#[inline(never)] +fn credit_boxed(p: &Persona, st: &State, host: &[(String, Json)], k: &Pay, src: Option<&Json>, check: bool) -> R<(Box, Settled)> { let (s, d) = credit_step(p, st, host, k, src, check)?; Ok((Box::new(s), d)) } +#[inline(never)] +fn gate_boxed(p: &Persona, st: Box, raw: &[(String, Json)]) -> R> { Ok(Box::new(gate_step(p, *st, raw)?)) } +#[inline(never)] +fn decode_boxed(p: &Persona, st: &State, program: &str, meta: &Meta, doc: &Json, conflict: Option<&[String]>, yielded: Option<&[String]>) -> (Box, Box) { + let (o, s) = decode(p, st, program, meta, doc, conflict, yielded); (Box::new(o), Box::new(s)) +} +#[inline(never)] +fn settle_boxed(p: &Persona, j: &mut Json, ns: Box, k: &Pay, s: Settled) -> Box { Box::new(settle(p, j, *ns, k, s)) } +/// The compiler, the engine and the decoder: the program for the stepped state, its solve (a yield on a habit conflict, a second, +/// clamped solve for the persona's vouch rule) and the answer read back; `first`: the first program already compiled from the same +/// stepped state and inputs with these options (RFC 0014's rest probe keys its cache on it), so it is not compiled twice +#[inline(never)] +#[allow(clippy::too_many_arguments)] +fn turn_solve(p: &Persona, pr: &Prepped, no_inertia: bool, eng: Engine, timing: bool, want: bool, first: Option>) -> R { + use crate::trace::{enter, Phase}; + let (st, raw): (&State, &[(String, Json)]) = (&pr.st, &pr.raw); + enter(Phase::Compile); let t0 = Instant::now(); let mut calls = 0; let o = Opts { no_inertia, ..Default::default() }; - let (mut prog, mut meta) = compile(p, st, &raw, &o)?; let mut ptext = digest_of(&text(&prog)); + let (mut prog, mut meta) = match first { Some(b) => *b, None => compile(p, st, &raw, &o)? }; let mut ptext = digest_of(&text(&prog)); let compile_ms = ms(t0); let t1 = Instant::now(); let mut doc = solve(p, st, &prog, eng, &mut calls); + // the expectation's A_t: this answer's exact marginals (an exact answer never yields, so `meta` stays this program's) + let first = want.then(|| doc_marg(&doc)); let (mut conflict, mut yielded): (Option>, Option>) = (None, None); if doc.get("verdict").and_then(Json::as_str) == Some("infeasible") && !meta.active.is_empty() { let c = habit_conflict(p, st, &raw, &meta.active, &o, eng, &mut calls)?; conflict = Some(c.clone()); if p.eng.yields && !c.is_empty() { let mut y: Vec = vec![]; let mut c = c; loop { let mut rank = c.clone(); rank.sort_by(|a, b| { let (ha, hb) = (p.habit(a), p.habit(b)); ha.priority.partial_cmp(&hb.priority).unwrap().then(p.habit_index(b).cmp(&p.habit_index(a))) }); y.push(rank[0].clone()); - let oo = Opts { no_inertia, exclude: y.clone(), ..Default::default() }; let (pr, me) = compile(p, st, &raw, &oo)?; prog = pr; meta = me; ptext = digest_of(&text(&prog)); + enter(Phase::Compile); let oo = Opts { no_inertia, exclude: y.clone(), ..Default::default() }; let (pr, me) = compile(p, st, &raw, &oo)?; prog = pr; meta = me; ptext = digest_of(&text(&prog)); doc = solve(p, st, &prog, eng, &mut calls); if doc.get("verdict").and_then(Json::as_str) != Some("infeasible") { conflict = None; break; } c = habit_conflict(p, st, &raw, &meta.active, &oo, eng, &mut calls)?; conflict = Some(c.clone()); if c.is_empty() { break; } } yielded = Some(y); } } let mut engine_ms = ms(t1); - let (mut out, mut ns) = decode(p, st, &ptext, &meta, &doc, conflict.as_deref(), yielded.as_deref()); let lift = out.lift.clone(); + // the program is read no more (its digest is kept): not held through the decode and a hold's solve + drop(prog); + enter(Phase::Decode); + let (mut out, mut ns) = decode_boxed(p, st, &ptext, &meta, &doc, conflict.as_deref(), yielded.as_deref()); let lift = out.lift.clone(); // the persona's vouch rule: unsure traits with a hold level are held there (a second, clamped solve), consistent with every habit let holds: Vec<(String, String)> = out.unsure.iter().filter_map(|t| { let v = p.var(t).unwrap(); let e = &out.stance.iter().find(|(k, _)| k == t).unwrap().1; v.hold.as_ref().filter(|h| **h != e.level).map(|h| (t.clone(), h.clone())) }).collect(); let mut held = vec![]; if !holds.is_empty() { - let o2 = Opts { no_inertia, clamps: holds.clone(), ..Default::default() }; let (prog2, meta2) = compile(p, st, &raw, &o2)?; + enter(Phase::Compile); let o2 = Opts { no_inertia, clamps: holds.clone(), ..Default::default() }; let (prog2, meta2) = compile(p, st, &raw, &o2)?; enter(Phase::Decode); let te = Instant::now(); let doc2 = solve(p, st, &prog2, eng, &mut calls); engine_ms += ms(te); if matches!(doc2.get("verdict").and_then(Json::as_str), Some("exact" | "diagnostics_passed")) { - let (mut out2, ns2) = decode(p, st, &digest_of(&text(&prog2)), &meta2, &doc2, None, None); + let (mut out2, ns2) = decode_boxed(p, st, &digest_of(&text(&prog2)), &meta2, &doc2, None, None); out2.unsure = out.unsure.clone(); for (t, _) in &holds { let u = out.stance.iter().find(|(k, _)| k == t).map(|(_, e)| e.odds.clone()); if let Some(e) = out2.stance.iter_mut().find(|(k, _)| k == t) { e.1.unheld = u; } } out2.bound = out.bound.clone(); held = holds.iter().map(|(t, _)| t.clone()).collect(); held.sort(); @@ -2861,19 +3109,28 @@ fn turn_src(p: &Persona, st: &State, raw: &Json, no_inertia: bool, eng: Engine, out2.line = line(p, &levels, &meta2.active, &out.bound, &out.unsure, out2.agenda.as_deref().unwrap_or(&[]), &vouched, &meta2.conditional, &baseline(p, st)); out = out2; ns = ns2; } } out.held = held; out.lift = lift; + let at = first.map(|m| m.map(|marg| Box::new(At { marg, meta: slim(meta) }))); if timing { let r3 = |x: f64| Json::Num((x * 1e3).round() / 1e3); out.timing = Some(Json::Obj(vec![("compile_ms".into(), r3(compile_ms)), ("engine_ms".into(), r3(engine_ms)), ("decode_ms".into(), r3((ms(t0) - compile_ms - engine_ms).max(0.0))), ("turn_ms".into(), r3(ms(t0))), ("engine_calls".into(), Json::Num(calls as f64)), ("holds".into(), Json::Num(out.held.len() as f64)), ("load_avg_1m".into(), crate::sys::loadavg().map_or(Json::Null, |l| Json::Num((l * 100.0).round() / 100.0)))])); } + Ok(Solved { out, ns, at }) +} +/// The stance document from the decoded stance: its senses, drives and body blocks, `src` back in its inputs, the credit settled, +/// the injected block and the glyph +#[inline(never)] +fn turn_doc(p: &Persona, pr: Prepped, sv: Solved) -> (Json, Box) { + crate::trace::enter(crate::trace::Phase::Document); + let Prepped { st, given, src, pay, settled, .. } = pr; let st: &State = &st; let Solved { out, mut ns, .. } = sv; let mut j = out.to_json(p, st); if let (Some(sn), Some(ss)) = (&p.senses, &st.senses) { stance_senses(&mut j, sn, ss, &given); } if let (Some(d), Some(ds)) = (&p.drives, &st.drives) { stance_drives(&mut j, d, ds, &out.lift, &given); } stance_body(&mut j, p, st); if let (Some(s), Json::Obj(v)) = (src, &mut j) { if let Some((_, Json::Obj(ins))) = v.iter_mut().find(|(k, _)| k == "inputs") { ins.push(("src".into(), s)); } } - if let (Some(k), Some(d)) = (&pay, settled) { ns = settle(p, &mut j, ns, k, d); } + if let (Some(k), Some(d)) = (&pay, settled) { ns = settle_boxed(p, &mut j, ns, k, d); } if p.has_block() { let b = block(p, &j); if let Json::Obj(v) = &mut j { v.push(("block".into(), Json::Str(b))); } } // §2.18: the glyph last, from the finished document (`probbit live` sets it again once `deviation` is in) set_glyph(p, &mut j); - Ok((j, ns)) + (j, ns) } /// RFC 0008 E: the block a host injects verbatim, rendered from a stance document: every declared trait, then every mood, once @@ -2908,6 +3165,9 @@ pub fn deficits(p: &Persona, st: &State) -> Vec<(String, f64, f64)> { } /// A new individual: genes from the seed, an empty mood and history, the fallback levels as `prev`, and (with `rest`) the /// resting stance: this individual's joint plan with no live evidence and a neutral mood (one engine call). +/// `init` on the heap (the setup path of a small-stack target: docs/embedding.md "Stack budget") +#[inline(never)] +pub fn init_boxed(p: &Persona, seed: Option, rest: bool, eng: Engine) -> Box { Box::new(init(p, seed, rest, eng)) } pub fn init(p: &Persona, seed: Option, rest: bool, eng: Engine) -> State { let seed = seed.unwrap_or(p.seed); let mut st = State { seed, turn: 0, persona: [p.name.clone(), p.version.clone(), p.digest.clone()], genes: genes(p, seed), @@ -2916,7 +3176,7 @@ pub fn init(p: &Persona, seed: Option, rest: bool, eng: Engine) -> State { prev: p.vars.iter().map(|v| (v.id.clone(), v.fallback.clone())).collect(), agenda: (!p.steps.is_empty()).then(|| p.steps.clone()), rest: None, learned: zeros(p), credit: zeros(p), drives: p.drives.as_ref().map(|d| DState::zero(p, d, seed)), senses: p.senses.as_ref().map(|sn| SState::zero(&p.name, sn, seed)), learned_from: learned_from0(p), open: vec![], prev_stance: None, control_closed: vec![], elig: 1.0, digest: String::new(), - gate: gate0(p), rbar: p.pace.as_ref().map(|_| 0.0), disp: disp_genes(p, seed), pressure: p.rest_b.as_ref().map(|_| (0.0, false)), load_l: p.load.as_ref().map(|_| 0.0) }; + gate: gate0(p), rbar: p.pace.as_ref().map(|_| 0.0), disp: disp_genes(p, seed), pressure: p.rest_b.as_ref().map(|_| (0.0, false)), load_l: p.load.as_ref().map(|_| 0.0), cur: None }; if rest { if let Ok((prog, _)) = compile(p, &st, &[], &Opts { twin: Some(false), ..Default::default() }) { let doc = eng(&prog, &flags(p, &st)); if let Some(plan) = doc.get("plan").and_then(Json::as_obj) { @@ -2994,15 +3254,20 @@ pub fn replay(p: &Persona, seed: Option, turns: &[Json], no_inertia: bool, } /// `compile`: the turn's program (insertion order; its sha256 is the stance's `engine.program`) pub fn program(p: &Persona, st: &State, raw: &Json) -> R { Ok(text(&program_json(p, st, raw)?)) } -/// `program` as a document (RFC 0009: the expectation's members and the trajectory compile through it) -pub fn program_json(p: &Persona, st: &State, raw: &Json) -> R { +/// `program` as a document (RFC 0009: the expectation's members and the trajectory compile through it); the steps before the +/// compiler pass the state boxed, as in the turn +#[inline(never)] +pub fn program_json(p: &Persona, st: &State, raw: &Json) -> R { Ok(program_meta(p, st, raw, false)?.0) } +/// `program_json` with the compiler's meta; `meta_only`: no program text (Json::Null), the field and the rules in force only +#[inline(never)] +fn program_meta(p: &Persona, st: &State, raw: &Json, meta_only: bool) -> R<(Json, Box)> { let raw: Vec<(String, Json)> = match raw { Json::Null => vec![], Json::Obj(v) => v.clone(), _ => return Err(perr("inputs", "must be a JSON object")) }; let (src, raw) = if p.senses.is_some() || p.credit().is_some() || p.has_needs() { take_src(p, raw, true)? } else { (None, raw) }; let (pay, raw) = take_pay(p, raw)?; - let (st, raw) = prep(p, st, &raw, src.as_ref(), true)?; - let st = match &pay { Some(k) => credit_step(p, &st, &raw, k, src.as_ref(), true)?.0, None => st }; - let st = gate_step(p, st, &raw)?; - Ok(compile(p, &st, &raw, &Opts::default())?.0) + let (st, raw) = prep_boxed_n(p, st, &raw, src.as_ref(), true, meta_only)?; + let st = match &pay { Some(k) => credit_boxed(p, &st, &raw, k, src.as_ref(), true)?.0, None => st }; + let st = gate_boxed(p, st, &raw)?; + let (j, m) = compile(p, &st, &raw, &Opts { meta_only, ..Default::default() })?; Ok((j, Box::new(m))) } /// Static check: for every conditional habit and every pair of them (with every unconditional one), and every previous level of @@ -3789,14 +4054,28 @@ pub fn prove_seed(p: &Persona, pr: &Prop, seed: u64, eng: Engine) -> Result<(usi // --------------------------------------------------------------------------------- RFC 0009: the expectation and the trajectory /// RFC 0009 `expectation` (effects {}): the decayed combo counts' half-life in events and the most mixture members #[derive(Clone, Debug)] -pub struct Expect { pub half_life: f64, pub max_combos: usize } +pub struct Expect { pub half_life: f64, pub max_combos: usize, + /// RFC 0015: the rest pass (`expectation.rest`; None without the block: the pass is refused and nothing is folded for it) + pub rest: Option } +/// RFC 0015 `expectation.rest`: the rest pass's half-life in events for the window it re-folds (default the expectation's own, +/// which leaves every count as it is), the bins of a declared number input in the merge key (1-3; 3 = the waking key's thirds, +/// so nothing merges), the gain of a context's mean surprise in its weight (0: none) and the factor on the learned deltas (1: none) +#[derive(Clone, Debug, PartialEq)] +pub struct RestPass { pub half_life: f64, pub bucket: u8, pub surprise_gain: f64, pub decay: f64 } /// RFC 0009 `engine.trajectory`: K steps of beta from beta0 to 1 (linear), each solved by the exact tier #[derive(Clone, Debug)] pub struct Traj { pub k: usize, pub beta0: f64 } fn expect_spec(x: &Json) -> R { - let kv = keys(x, "expectation", &["half_life_events", "max_combos", "comment"], &[])?; - Ok(Expect { half_life: num_or(kv, "half_life_events", 32.0, "expectation", Some(1.0), Some(1e6), false)?, - max_combos: num_or(kv, "max_combos", 16.0, "expectation", Some(1.0), Some(64.0), true)? as usize }) + let kv = keys(x, "expectation", &["half_life_events", "max_combos", "rest", "comment"], &[])?; + let half_life = num_or(kv, "half_life_events", 32.0, "expectation", Some(1.0), Some(1e6), false)?; + // RFC 0015: `rest` (every key optional, with the defaults that change nothing) + let rest = match get(kv, "rest") { None => None, Some(r) => { + let rk = keys(r, "expectation.rest", &["half_life", "bucket", "surprise_gain", "decay", "comment"], &[])?; + Some(RestPass { half_life: num_or(rk, "half_life", half_life, "expectation.rest", Some(1.0), Some(1e6), false)?, + bucket: num_or(rk, "bucket", 3.0, "expectation.rest", Some(1.0), Some(3.0), true)? as u8, + surprise_gain: num_or(rk, "surprise_gain", 0.0, "expectation.rest", Some(0.0), Some(4.0), false)?, + decay: num_or(rk, "decay", 1.0, "expectation.rest", Some(0.0), Some(1.0), false)? }) } }; + Ok(Expect { half_life, max_combos: num_or(kv, "max_combos", 16.0, "expectation", Some(1.0), Some(64.0), true)? as usize, rest }) } fn traj_spec(x: &Json) -> R { let kv = keys(x, "engine.trajectory", &["k", "beta0", "comment"], &[])?; @@ -3808,13 +4087,70 @@ fn traj_spec(x: &Json) -> R { /// latest event that had it (that combo's member), whether a non-time combo has been seen, and the learned normal of /// ln(1 + pred). A fold over the logged events (their inputs and deviation receipts), so it lives with a strand's replay and /// never in the state: the state digest is the same with the block on or off. +/// +/// The canonical fold (`canonical`, strands of engine 0.14.0 on): after every step each count, the mean and the variance are at +/// 6 decimals (a count at 0 is dropped) and at most `max_combos - 1` named combos are kept, the one last in the mixture's order +/// moving its count to `rest` (added to the empty member's weight, where the mixture puts every combo past the kept ones). The +/// memory is then a small exact object a checkpoint line carries (`to_json`, `read`), the same bytes on every libm, and a fold +/// restarted from it equals the fold from the strand's header. #[derive(Clone, Debug, Default)] -pub struct ExpectMem { combos: Vec<(String, f64, Json)>, warm: bool, mean: f64, var: f64, n: u64 } +pub struct ExpectMem { combos: Vec<(String, f64, Json)>, warm: bool, mean: f64, var: f64, n: u64, rest: f64, canon: bool, + /// RFC 0016: the curiosity drive's fold (canonical memory with a `curiosity` block only; None before the first event) + pub cur: Option, + /// RFC 0015: the rest pass's window (canonical memory with an `expectation.rest` block only; None before the first event) + pub win: Option } const DEV_PRIOR_VAR: f64 = 0.04; const DEV_VAR_FLOOR: f64 = 1e-4; /// z at or above this registers the deviation (the RFC 0001 sense threshold) pub const DEV_Z: f64 = 1.5; -impl ExpectMem { pub fn new() -> ExpectMem { ExpectMem { var: DEV_PRIOR_VAR, ..Default::default() } } } +impl ExpectMem { + pub fn new() -> ExpectMem { ExpectMem { var: DEV_PRIOR_VAR, ..Default::default() } } + /// The empty memory of the canonical fold (strands of engine 0.14.0 on) + pub fn canonical() -> ExpectMem { ExpectMem { canon: true, ..ExpectMem::new() } } + /// Whether this is the canonical fold (its memory goes on checkpoint lines) + pub fn is_canonical(&self) -> bool { self.canon } + /// The memory as a checkpoint line's `expect`: {combos: [[count, inputs], ...] in key order, mean, n, rest, var, warm}; the key of + /// a combo is a function of its inputs (`combo_key`), so it is not written + pub fn to_json(&self) -> Json { + let mut c: Vec<&(String, f64, Json)> = self.combos.iter().collect(); c.sort_by(|a, b| a.0.cmp(&b.0)); + let j = Json::Obj(vec![("combos".into(), Json::Arr(c.iter().map(|x| Json::Arr(vec![Json::Num(x.1), x.2.clone()])).collect())), ("mean".into(), Json::Num(self.mean)), + ("n".into(), Json::Num(self.n as f64)), ("rest".into(), Json::Num(self.rest)), ("var".into(), Json::Num(self.var)), ("warm".into(), Json::Bool(self.warm))]); + // RFC 0016: the drive's fold rides with the memory (only with a curiosity block) + let mut j = j; if let (Some(c), Json::Obj(v)) = (&self.cur, &mut j) { v.insert(1, ("curiosity".into(), c.to_json())); } + // RFC 0015: the rest pass's window rides with the memory (only with an `expectation.rest` block), its rows in key order + if let (Some(w), Json::Obj(v)) = (&self.win, &mut j) { let ks: Vec<&str> = c.iter().map(|x| x.0.as_str()).collect(); v.push(("window".into(), w.to_json(&ks))); } + j + } + /// A checkpoint line's `expect` for persona `p` -> the canonical memory; Err(what is wrong). It must be what the canonical fold + /// can hold: combos in key order, each count > 0 at 6 decimals, distinct keys, at most `max_combos - 1` named ones + pub fn read(p: &Persona, j: &Json) -> Result { + let ex = p.expect.as_ref().ok_or("the persona has no expectation block")?; + // RFC 0016: with a curiosity block the memory has a seventh key, `curiosity` (the drive's fold) + // RFC 0015: with an `expectation.rest` block one more key, `window` (the rest pass's window) + let rw = ex.rest.is_some(); + let want = 6 + usize::from(p.cur.is_some()) + usize::from(rw); + let shape = match (p.cur.is_some(), rw) { (true, false) => "an object {combos, curiosity, mean, n, rest, var, warm}", (false, false) => "an object {combos, mean, n, rest, var, warm}", + (true, true) => "an object {combos, curiosity, mean, n, rest, var, warm, window}", (false, true) => "an object {combos, mean, n, rest, var, warm, window}" }; + let kv = j.as_obj().filter(|kv| kv.len() == want).ok_or(shape)?; + let f = |k: &str| kv.iter().find(|(x, _)| x == k).map(|(_, v)| v).ok_or(format!("no {k}")); + let r6n = |k: &str| -> Result { f(k)?.as_f64().filter(|x| x.is_finite() && *x >= 0.0 && r6(*x) == *x).ok_or(format!("{k}: a number >= 0 at 6 decimals")) }; + let mean = f("mean")?.as_f64().filter(|x| x.is_finite() && r6(*x) == *x).ok_or("mean: a number at 6 decimals")?; + let (var, rest) = (r6n("var")?, r6n("rest")?); + let n = f("n")?.as_f64().filter(|x| *x >= 0.0 && x.fract() == 0.0 && *x <= 9_007_199_254_740_992.0).ok_or("n: a whole number >= 0")? as u64; + let Json::Bool(warm) = f("warm")? else { return Err("warm: true or false".into()) }; + let mut combos: Vec<(String, f64, Json)> = vec![]; + for c in f("combos")?.as_arr().ok_or("combos: a list of [count, inputs]")? { + let (cnt, rep) = match c.as_arr() { Some([Json::Num(x), r @ Json::Obj(_)]) => (*x, r), _ => return Err("combos: each [count, inputs]".into()) }; + if !(cnt.is_finite() && cnt > 0.0 && r6(cnt) == cnt) { return Err("combos: each count > 0 at 6 decimals".into()); } + let key = combo_key(p, rep.as_obj().unwrap_or(&[])); + if combos.last().is_some_and(|l| l.0 >= key) { return Err("combos: in key order, each key once".into()); } + combos.push((key, cnt, rep.clone())); } + if combos.iter().filter(|c| c.0 != "{}").count() > ex.max_combos - 1 { return Err(format!("combos: at most {} named", ex.max_combos - 1)); } + let cur = match &p.cur { Some(c) => Some(curiosity::CurMem::read(c, f("curiosity")?)?), None => None }; + let win = if rw { let ks: Vec = combos.iter().map(|c| c.0.clone()).collect(); Some(rest::RestWin::read(f("window")?, &ks)?) } else { None }; + Ok(ExpectMem { combos, warm: *warm, mean, var, n, rest, canon: true, cur, win }) + } +} /// One event's deviation from the engine's own expectation (nats): `s_self` = sum over the non-twin variables of KL(A_i || E_i), /// `pred` = the same against the mixture Pi_i, `input` = -ln of the mixture weight of the event's combo (its own member's, /// else the empty member's), `cold` until the first non-time combo; `member`: the combo has its own member; `z` against the @@ -3841,6 +4177,8 @@ pub fn expect_weights(ex: &Expect, mem: &ExpectMem) -> Vec<(String, f64)> { let keep = ex.max_combos - 1; let mut empty = 1.0 + mem.combos.iter().find(|x| x.0 == "{}").map_or(0.0, |x| x.1); for x in c.iter().skip(keep) { empty += x.1; } + // the canonical fold's evicted counts (0 in the fold before 0.14.0: the sum is unchanged) + if mem.canon { empty += mem.rest; } c.truncate(keep); let tot = c.iter().fold(empty, |a, x| a + x.1); let mut out = vec![("{}".to_string(), empty / tot)]; out.extend(c.into_iter().map(|(k, n)| (k, n / tot))); out @@ -3848,12 +4186,112 @@ pub fn expect_weights(ex: &Expect, mem: &ExpectMem) -> Vec<(String, f64)> { /// Per non-twin variable its exact marginal, levels sorted type Marg = Vec<(String, Vec<(String, f64)>)>; /// The non-twin exact marginals of a program -> None unless the exact tier solved it +#[inline(never)] fn exact_marg(p: &Persona, st: &State, prog: &Json, eng: Engine) -> Option { - let doc = eng(prog, &Flags { op: "decide".into(), ..flags(p, st) }); + doc_marg(&eng(prog, &Flags { op: "decide".into(), ..flags(p, st) })) +} +/// An answer's non-twin marginals -> None unless its verdict is exact +fn doc_marg(doc: &Json) -> Option { if doc.get("verdict").and_then(Json::as_str) != Some("exact") { return None; } Some(doc.get("marginals")?.as_obj()?.iter().filter(|(k, _)| !k.starts_with("free.")).map(|(k, v)| (k.clone(), v.as_obj().map_or(vec![], |o| { let mut r: Vec<(String, f64)> = o.iter().map(|(l, x)| (l.clone(), x.as_f64().unwrap_or(0.0))).collect(); r.sort_by(|a, b| a.0.cmp(&b.0)); r }))).collect()) } +/// The expectation's cache: per program the deviation solved, its non-twin exact marginals, keyed by the sha256 of the program +/// without its `comment` (the only key that names the turn and the seed). The exact tier's marginals are a function of the +/// program's model alone (the seed, the sweeps and the chains are the sampler's), so a hit is the value the solve would return, +/// bit for bit: every `pred`, `self`, `z`, stance and strand byte is the same with the cache on or off. The members' programs +/// are rebuilt every event (the state moves every event: its digest carries the turn), only the solves are skipped; decay +/// alone never invalidates a row (marginals do not read counts). After each event only the rows that event used are kept: at +/// most `max_combos + 1` (the event's own program, the empty program, the other members). The key is the sha256 of the program's +/// structure (each number by its bits, -0 as 0 as the text writes it, each string with its length): two programs share a row +/// only when their texts are equal. A life whose programs move every event (a state that moves every event) gets no hit: after an event without +/// one the cache is not consulted for the next 1, 2, 4, ... 32 events (a hit resets it; the rows are dropped meanwhile and the event +/// after the pause fills them for the one after it to compare), so such a life keys two events in 34. +#[derive(Clone, Debug)] +pub struct ExpectCache { on: bool, rows: Vec<([u8; 32], Marg)>, used: Vec, hits: u32, solves: u32, skip: u32, back: u32, live: bool, had: bool, modeled: u32, + /// T3-b: A_t from the turn (`turn_at`); `check`: also built and solved again, and compared + reuse: bool, check: bool, reused: u32 } +impl Default for ExpectCache { fn default() -> ExpectCache { ExpectCache::new(true) } } +impl ExpectCache { + /// An empty cache; `on: false` solves every program (the receipt is the same) + pub fn new(on: bool) -> ExpectCache { ExpectCache { on, rows: vec![], used: vec![], hits: 0, solves: 0, skip: 0, back: 0, live: false, had: false, modeled: 0, reuse: true, check: false, reused: 0 } } + /// The same cache with the turn's own program read as A_t (`reuse`, the default: `Live::event` runs the turn first and hands + /// the deviation its first program's meta and exact marginals) or built and solved again by the deviation; `check`: when + /// handed, A_t is built and solved again as well and the two compared (a mismatch panics: a test's instrument) + pub fn with_reuse(mut self, reuse: bool, check: bool) -> ExpectCache { self.reuse = reuse; self.check = check; self } + /// Whether `Live::event` hands the deviation the turn's own program (the cache on, reuse on) + pub fn reuses(&self) -> bool { self.on && self.reuse } + /// The last event's A_t read from the turn (1) or not (0) + pub fn reused(&self) -> u32 { self.reused } + /// Whether the cache is used + pub fn is_on(&self) -> bool { self.on } + /// RFC 0015: every row dropped and the pause reset (the rest pass changed the memory the rows were built for); the switches + /// and the counters stay. The receipts are the same with or without it: a row is keyed by its program's text. + pub fn invalidate(&mut self) { self.rows.clear(); self.used.clear(); self.skip = 0; self.back = 0; self.had = false; } + /// The last event's programs answered from the cache and solved + pub fn last(&self) -> (u32, u32) { (self.hits, self.solves) } + /// The last event's programs solved as the event's own model with their own `h` (the rest of the solves: built and run) + pub fn modeled(&self) -> u32 { self.modeled } + /// The rows held (at most the persona's `max_combos + 1`) + pub fn len(&self) -> usize { self.rows.len() } + /// Whether no row is held + pub fn is_empty(&self) -> bool { self.rows.is_empty() } + /// The heap bytes the rows hold on this target (each row: its key, per variable its id and per level its name and value) + pub fn bytes(&self) -> usize { + use core::mem::size_of; + self.rows.capacity() * size_of::<([u8; 32], Marg)>() + self.used.capacity() + self.rows.iter().map(|(_, m)| m.capacity() * size_of::<(String, Vec<(String, f64)>)>() + + m.iter().map(|(id, ls)| id.capacity() + ls.capacity() * size_of::<(String, f64)>() + ls.iter().map(|(l, _)| l.capacity()).sum::()).sum::()).sum::() + } + fn begin(&mut self) { self.hits = 0; self.solves = 0; self.modeled = 0; self.reused = 0; self.used.clear(); self.used.resize(self.rows.len(), false); + self.live = self.on && self.skip == 0; self.had = !self.rows.is_empty(); if self.skip > 0 { self.skip -= 1; } } + /// Only the rows this event has used so far are kept (`deviation_pre`, before the turn) + fn prune(&mut self) { let mut i = 0; self.rows.retain(|_| { i += 1; self.used[i - 1] }); self.used.clear(); self.used.resize(self.rows.len(), true); } + fn end(&mut self) { + // an event that had rows to compare and found none backs off; one that started empty only fills (the next one compares) + if self.live { if self.hits > 0 { self.back = 0; } else if self.had { self.back = (self.back * 2).clamp(1, 32); self.skip = self.back; } } + let mut i = 0; self.rows.retain(|_| { i += 1; self.used[i - 1] }); self.used.clear(); } + /// A row by key (None when the cache is paused or has none) + fn find(&mut self, key: &[u8; 32]) -> Option { + if !self.live { return None; } + let i = self.rows.iter().position(|r| r.0 == *key)?; self.used[i] = true; self.hits += 1; Some(self.rows[i].1.clone()) + } + /// A_t's marginals from the turn, kept under `key` while the cache is live (not a solve of the deviation's) + fn put(&mut self, key: [u8; 32], m: &Marg) { + if !self.live { return; } + match self.rows.iter().position(|r| r.0 == key) { Some(i) => self.used[i] = true, None => { self.rows.push((key, m.clone())); self.used.push(true); } } + } + /// A program solved here, not by `marg`: counted, and kept under `key` while the cache is live + fn keep(&mut self, key: Option<[u8; 32]>, m: &Option) { + self.solves += 1; + if let (true, Some(k), Some(m)) = (self.live, key, m) { self.rows.push((k, m.clone())); self.used.push(true); } + } + /// The program's exact marginals: from the cache when the program (without its comment) was solved before, else solved (the + /// key's hasher lives in its own frame, not in the one that calls the engine) + #[inline(never)] + fn marg(&mut self, p: &Persona, st: &State, prog: &Json, eng: Engine) -> Option { + if !self.live { self.solves += 1; return exact_marg(p, st, prog, eng); } + let key = prog_key(prog); + if let Some(i) = self.rows.iter().position(|r| r.0 == key) { self.used[i] = true; self.hits += 1; return Some(self.rows[i].1.clone()); } + self.solves += 1; let m = exact_marg(p, st, prog, eng)?; + self.rows.push((key, m.clone())); self.used.push(true); Some(m) + } +} +/// The cache's key of a program: the sha256 of its structure without `comment` +#[inline(never)] +fn prog_key(prog: &Json) -> [u8; 32] { + let mut h = Sha256::new(); + match prog { Json::Obj(v) => for (k, x) in v.iter().filter(|(k, _)| k != "comment") { hash_str(&mut h, k); hash_json(&mut h, x); }, x => hash_json(&mut h, x) } + h.finish() +} +fn hash_str(h: &mut Sha256, s: &str) { h.update(&(s.len() as u64).to_le_bytes()); h.update(s.as_bytes()); } +/// A value's structure into `h`: a tag per kind, numbers by their bits (-0 as 0, as the program's text writes both), strings and +/// lists with their lengths +fn hash_json(h: &mut Sha256, j: &Json) { + match j { Json::Null => h.update(b"n"), Json::Bool(b) => h.update(if *b { b"t" } else { b"f" }), Json::Num(x) => { h.update(b"#"); h.update(&(if *x == 0.0 { 0 } else { x.to_bits() }).to_le_bytes()) } + Json::Str(s) => { h.update(b"s"); hash_str(h, s) } + Json::Arr(v) => { h.update(b"["); h.update(&(v.len() as u64).to_le_bytes()); for x in v { hash_json(h, x); } } + Json::Obj(v) => { h.update(b"{"); h.update(&(v.len() as u64).to_le_bytes()); for (k, x) in v { hash_str(h, k); hash_json(h, x); } } } +} fn with_elapsed(rep: &Json, e: &Json) -> Json { let mut kv: Vec<(String, Json)> = rep.as_obj().map_or(vec![], |o| o.to_vec()); kv.push(("elapsed_hours".into(), e.clone())); Json::Obj(kv) } @@ -3870,20 +4308,41 @@ fn kl_marg(a: &Marg, b: &Marg) -> Option<(f64, Vec)> { /// programs (each combo's latest inputs with this event's hours; the event's own combo with its own inputs, so A_t is a member /// whenever its combo is). Mixture marginals are linear in the members' exact marginals, so every number is exact from the /// M + 1 solves. None without the block, or when a program is not solved exactly or a divergence is not finite. -pub fn deviation(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine) -> Option { +#[inline(never)] +pub fn deviation(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine) -> Option { deviation_cached(p, st, raw, mem, eng, &mut ExpectCache::new(false)) } +/// `deviation` with the members' marginals from `cache` (`ExpectCache`): the same value, fewer solves +#[inline(never)] +pub fn deviation_cached(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine, cache: &mut ExpectCache) -> Option { deviation_at(p, st, raw, mem, eng, cache, None) } +/// `deviation_cached` after the turn: `at` is what `turn_at` handed of the event's own program (None: A_t built and solved here) +#[inline(never)] +pub fn deviation_at(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine, cache: &mut ExpectCache, at: Option>>) -> Option { + cache.begin(); let d = deviation_in(p, st, raw, mem, eng, cache, at); cache.end(); crate::trace::expect_cache(cache.hits, cache.solves); d +} +#[inline(always)] +fn deviation_in(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine, cache: &mut ExpectCache, at: Option>>) -> Option { let ex = p.expect.as_ref()?; let kv = raw.as_obj()?; let e = get(kv, "elapsed_hours").cloned().unwrap_or(Json::Num(0.0)); let key = combo_key(p, kv); - let solve = |ins: &Json| program_json(p, st, ins).ok().and_then(|prog| exact_marg(p, st, &prog, eng)); - let a = solve(raw)?; let ed = solve(&with_elapsed(&Json::Obj(vec![]), &e))?; let ws = expect_weights(ex, mem); let member = ws.iter().any(|(k, _)| *k == key); + // the programs: the event's own (A_t), the one with the hours only (E_t), then each other member's (its combo's latest inputs + // with this event's hours), in the mixture's order + let mut ins = vec![with_elapsed(&Json::Obj(vec![]), &e)]; + for (k, _) in &ws { if *k == key || k == "{}" { continue; } ins.push(with_elapsed(&mem.combos.iter().find(|x| x.0 == *k).map(|x| x.2.clone()).unwrap_or(Json::Obj(vec![])), &e)); } + let ms = members_marg(p, st, raw, &ins, eng, cache, at)?; + mixture(mem, &ws, &key, member, ms) +} +/// The deviation from the programs' marginals (A_t, E_t, then the other members in the mixture's order), out of the frame the +/// members are built under +#[inline(never)] +fn mixture(mem: &ExpectMem, ws: &[(String, f64)], key: &str, member: bool, ms: Vec>) -> Option { + let mut ms = ms.into_iter(); + let a = ms.next()??; let ed = ms.next()??; let cold = !mem.warm && key == "{}"; // the members' marginals, weighted; a member that does not compile or solve exactly folds into the empty member let mut parts: Vec<(f64, Marg)> = vec![]; let mut empty_w = 0.0; - for (k, w) in &ws { + for (k, w) in ws { if *k == key { parts.push((*w, a.clone())); continue; } if k == "{}" { empty_w += w; continue; } - let rep = mem.combos.iter().find(|x| x.0 == *k).map(|x| x.2.clone()).unwrap_or(Json::Obj(vec![])); - match solve(&with_elapsed(&rep, &e)) { Some(m) => parts.push((*w, m)), None => empty_w += w } } + match ms.next().flatten() { Some(m) => parts.push((*w, m)), None => empty_w += w } } // the empty member: E_t, or A_t itself when the event is time-only (ws[0], pushed first) if key != "{}" { parts.push((empty_w, ed.clone())); } else { parts[0].0 += empty_w; } let pi: Marg = a.iter().map(|(id, la)| (id.clone(), la.iter().map(|(l, _)| (l.clone(), parts.iter().fold(0.0, |acc, (w, m)| @@ -3893,22 +4352,363 @@ pub fn deviation(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engi let z = (!cold).then(|| ((1.0 + sig5(pred)).ln() - mem.mean) / (mem.var + DEV_VAR_FLOOR).sqrt()); Some(Deviation { s_self, pred, input: -w_cur.ln(), cold, member, z, members: ws.len(), terms: a.iter().zip(tp.iter().zip(&ts)).map(|((id, _), (x, y))| (id.clone(), *x, *y)).collect() }) } +/// A member's program text from its compile meta (no clamps), in a frame of its own +#[inline(never)] +fn member_text(p: &Persona, mm: &Meta) -> Json { emit(p, &mm.field, &mm.rules, &mm.active, (mm.seed, mm.turn), mm.twin, &mm.values, &[], true) } +/// T3-c: a member's text without its pairs (`emit`, `with_pairs: false`), in a frame of its own +#[inline(never)] +fn member_stub(p: &Persona, mm: &Meta) -> Json { emit(p, &mm.field, &mm.rules, &mm.active, (mm.seed, mm.turn), mm.twin, &mm.values, &[], false) } +/// T3-c: a member's rules in force lowered to caps as `run::from_json` lowers its text's (its text without the pairs, read), +/// with the program's variables and values; None: not read +#[inline(never)] +fn member_caps(p: &Persona, mm: &Meta) -> Option<(Vec, Vec, Vec)> { + let t = member_stub(p, mm); + crate::run::from_json_parts(&t).ok().map(crate::run::Parts::into_caps) +} +/// A program's exact marginals the reference way: its text built (`program_json`) and run by the engine (`ExpectCache::marg`); +/// turn_trace: the build and the solve timed apart (`expect_build_us`, `expect_solve_us`) +#[inline(never)] +fn text_marg(p: &Persona, st: &State, ins: &Json, eng: Engine, cache: &mut ExpectCache) -> Option { + let t = crate::trace::Stopwatch::start(); let prog = program_json(p, st, ins).ok(); let b = t.us(); + let m = prog.and_then(|prog| cache.marg(p, st, &prog, eng)); crate::trace::expect_split(b, t.us() - b); m +} +/// The event's own program read once from its text (`run::from_json_parts`) into the members' one model (`run::Members`: its +/// parts moved in, no second copy), its variables, values and own `h`, with per variable of the persona (traits and moods, then +/// the agenda steps) the value index of each of its levels in the program +struct Skel { vars: Vec, values: Vec, h0: Vec, mm: crate::run::Members, idx: Vec>, + /// T3-c: the key of the skeleton's rules in force, of the rules whose caps the model holds now, and the skeleton's caps + base_rk: [u8; 32], cur_rk: [u8; 32], base_caps: Vec } +/// The expectation's programs -> per program its exact marginals (None: it does not compile or is not solved exactly); None when +/// A_t or E_t has none (no deviation). With the cache off (`persona::deviation`, the reference) every program's text is built +/// and run by the engine. With it on, A_t's text is built and read once; every other program is compiled without its text +/// (`Opts::meta_only`) and, when its rules in force are A_t's, solved as A_t's model with its own `h` by the enumeration +/// `run_program` would answer with (`run::enumerated`): the same model, the same marginals, bit for bit. Any other program +/// (other rules, another tier) is built and run as the reference does. The cache's key for these programs: A_t's text without +/// `comment` and the vars' `h` (its skeleton), then the program's field: equal keys, equal texts. Inlined in the deviation's frame +/// (each member's build runs right under it, as before) +#[inline(always)] +fn members_marg(p: &Persona, st: &State, raw: &Json, ins: &[Json], eng: Engine, cache: &mut ExpectCache, at: Option>>) -> Option>> { + let mut out = Vec::with_capacity(ins.len() + 1); + if !cache.on { out.push(text_marg(p, st, raw, eng, cache)); if out[0].is_none() { return None; } + for (i, x) in ins.iter().enumerate() { let m = text_marg(p, st, x, eng, cache); if i == 0 && m.is_none() { return None; } out.push(m); } + return Some(out); } + let t = crate::trace::Stopwatch::start(); + // A_t: handed by the turn (its first program: the text written again from its meta, its answer's marginals), or built here + let (prog, meta, own) = match at { + Some(None) => { cache.reused = 1; if cache.check { assert!(program_json(p, st, raw).ok().and_then(|x| exact_marg(p, st, &x, eng)).is_none(), "A_t: not exact either way"); } return None } + Some(Some(a)) => { let At { marg, meta } = *a; cache.reused = 1; (member_text(p, &meta), meta, Some(marg)) } + None => { let Ok((prog, meta)) = program_meta(p, st, raw, false) else { crate::trace::expect_split(t.us(), 0); return None }; (prog, meta, None) } }; + let mut prog = Some(prog); + let (b, t) = (t.us(), crate::trace::Stopwatch::start()); + let skel = cache.live.then(|| skel_key(prog.as_ref().unwrap())); + let mut sk: Option>> = None; // read on the first program the cache does not answer; Some(None): not usable + let a = match own { + Some(m) => { if cache.check { at_check(p, st, raw, prog.as_ref().unwrap(), &m, eng); } if let Some(s) = &skel { cache.put(field_key(s, &meta.field), &m); } Some(m) } + None => model_marg(p, st, &mut prog, &meta, skel.as_ref(), &mut sk, &meta, true, None, eng, cache).flatten() }; + crate::trace::expect_split(b, t.us()); + a.as_ref()?; out.push(a); + for (i, x) in ins.iter().enumerate() { + // A_t's text is read no more once its skeleton is (the members' model holds what they need) + if sk.is_some() { prog = None; } + let t = crate::trace::Stopwatch::start(); + let m = match program_meta(p, st, x, true) { + Err(_) => { crate::trace::expect_split(t.us(), 0); None } + Ok((_, mm)) if mm.rules == meta.rules => { let (b, t) = (t.us(), crate::trace::Stopwatch::start()); + let r = model_marg(p, st, &mut prog, &mm, skel.as_ref(), &mut sk, &meta, false, None, eng, cache); + // another tier (or no usable skeleton): its text from its meta, run by the engine + let r = match r { Some(r) => r, None => { let prog = member_text(p, &mm); cache.marg(p, st, &prog, eng) } }; crate::trace::expect_split(b, t.us()); r } + // other rules in force: its text from its meta (no second build), run by the engine + Ok((_, mm)) => { let prog = member_text(p, &mm); let (b, t) = (t.us(), crate::trace::Stopwatch::start()); let m = cache.marg(p, st, &prog, eng); crate::trace::expect_split(b, t.us()); m } }; + if i == 0 && m.is_none() { return None; } + out.push(m); } + Some(out) +} +/// T3-c: one member's exact marginals held through the turn: as a row of f64 in the order of the members' layout (`Held::lay`), +/// or by name when its layout is another +enum HeldM { Row(Vec), Named(Marg) } +/// T3-c: what the deviation holds through the turn when the members run before it (`deviation_pre`): per member but A_t (E_t, +/// then the other members in the mixture's order) its marginals or None (it does not compile or is not solved exactly); the +/// sha256 of the rows' layout (each variable's id and its levels' names, in order); the skeleton's key and the key of its rules +/// in force (A_t's row goes into the cache under the skeleton when its rules are the skeleton's) +pub struct Held { ms: Vec>, lay: [u8; 32], skel: Option<[u8; 32]>, rk: [u8; 32] } +/// The sha256 of a marginals' layout: per variable its id and its levels' names, in order +#[inline(never)] +fn lay_key(m: &Marg) -> [u8; 32] { + let mut h = Sha256::new(); h.update(&(m.len() as u64).to_le_bytes()); + for (id, ls) in m { hash_str(&mut h, id); h.update(&(ls.len() as u64).to_le_bytes()); for (l, _) in ls { hash_str(&mut h, l); } } + h.finish() +} +/// The sha256 of a program's rules in force +#[inline(never)] +fn rules_key(rules: &[Vec]) -> [u8; 32] { + let mut h = Sha256::new(); h.update(&(rules.len() as u64).to_le_bytes()); + for r in rules { hash_json(&mut h, &Json::Arr(r.clone())); } + h.finish() +} +/// T3-c, the members before the turn (`Live::event` with the cache reusing the turn's program): every member but A_t is solved +/// from the pre-event state as `deviation_cached` solves it, nothing of the turn held; their marginals are kept as rows (`Held`) +/// for `deviation_post`, which reads A_t from the turn. The members' one model is read from the first member's own text (its +/// skeleton); a member with other rules in force takes it with its own caps (its rules lowered, `member_caps`: the pairs and +/// folds are the skeleton's); a member the model does not answer is built and run by the engine. E_t comes first, the skeleton +/// is its. None: no deviation (no block, or E_t has no exact marginals: the other members are then not solved); +/// `deviation_post` runs either way (it closes the cache's event) +#[inline(never)] +pub fn deviation_pre(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine, cache: &mut ExpectCache) -> Option> { + cache.begin(); let h = pre_in(p, st, raw, mem, eng, cache); + // only this event's rows are held through the turn (A_t's goes in after it) + cache.prune(); h +} +#[inline(always)] +fn pre_in(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine, cache: &mut ExpectCache) -> Option> { + let ex = p.expect.as_ref()?; let kv = raw.as_obj()?; + let e = get(kv, "elapsed_hours").cloned().unwrap_or(Json::Num(0.0)); let key = combo_key(p, kv); + let ws = expect_weights(ex, mem); + let mut ins = vec![with_elapsed(&Json::Obj(vec![]), &e)]; + for (k, _) in &ws { if *k == key || k == "{}" { continue; } ins.push(with_elapsed(&mem.combos.iter().find(|x| x.0 == *k).map(|x| x.2.clone()).unwrap_or(Json::Obj(vec![])), &e)); } + drop(ws); + let mut out: Vec> = (0..ins.len()).map(|_| None).collect(); + let (mut sk, mut base, mut skel): (Option>>, Option>, Option<[u8; 32]>) = (None, None, None); + let mut prog: Option = None; + for i in 0..ins.len() { + if sk.is_some() { prog = None; } + let t = crate::trace::Stopwatch::start(); + let m = match program_meta(p, st, &ins[i], true) { + Err(_) => { crate::trace::expect_split(t.us(), 0); None } + Ok((_, mm)) => { + // the skeleton from the first member: its text without the pairs (read with the persona's couplings), no program text + let same = match &base { None => { prog = Some(member_stub(p, &mm)); + skel = cache.live.then(|| stub_key(p, prog.as_ref().unwrap())); true } Some(b) => mm.rules == b.rules }; + let (b, t) = (t.us(), crate::trace::Stopwatch::start()); + // other rules in force: the same model with the member's caps (its rules lowered) + let rk = (!same).then(|| rules_key(&mm.rules)); + let r = { let am = base.as_deref().unwrap_or(&mm); model_marg(p, st, &mut prog, &mm, skel.as_ref(), &mut sk, am, false, rk.as_ref(), eng, cache) }; + // another tier, rules not read or no usable skeleton: its text from its meta, run by the engine + let r = match r { Some(r) => r, None => { let prog = member_text(p, &mm); cache.marg(p, st, &prog, eng) } }; + if base.is_none() { base = Some(mm); } + crate::trace::expect_split(b, t.us()); r } }; + // E_t without exact marginals: no deviation (the other members are not solved) + if i == 0 && m.is_none() { return None; } + out[i] = m; } + drop((sk, prog)); + out[0].as_ref()?; + let lay = lay_key(out[0].as_ref().unwrap()); + let ms = out.into_iter().map(|m| m.map(|m| if lay_key(&m) == lay { HeldM::Row(m.iter().flat_map(|(_, ls)| ls.iter().map(|x| x.1)).collect()) } else { HeldM::Named(m) })).collect(); + Some(Box::new(Held { ms, lay, skel, rk: base.as_ref().map_or([0; 32], |b| rules_key(&b.rules)) })) +} +/// T3-c, after the turn: A_t from the turn (`at`; None: built and solved here), the held members named again from A_t's layout +/// (the same values, the same names, in the same order), then the mixture as `deviation_cached` computes it. A layout that is +/// not A_t's (never seen) solves the members again +#[inline(never)] +pub fn deviation_post(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine, cache: &mut ExpectCache, held: Option>, at: Option>>) -> Option { + let d = post_in(p, st, raw, mem, eng, cache, held, at); cache.end(); crate::trace::expect_cache(cache.hits, cache.solves); d +} +#[inline(always)] +#[allow(clippy::too_many_arguments)] +fn post_in(p: &Persona, st: &State, raw: &Json, mem: &ExpectMem, eng: Engine, cache: &mut ExpectCache, held: Option>, at: Option>>) -> Option { + let ex = p.expect.as_ref()?; let kv = raw.as_obj()?; let h = held?; + let a = match at { + Some(None) => { cache.reused = 1; if cache.check { assert!(program_json(p, st, raw).ok().and_then(|x| exact_marg(p, st, &x, eng)).is_none(), "A_t: not exact either way"); } return None } + Some(Some(a)) => { let At { marg, meta } = *a; cache.reused = 1; + if cache.check { at_check(p, st, raw, &member_text(p, &meta), &marg, eng); } + if let Some(s) = &h.skel { if rules_key(&meta.rules) == h.rk { cache.put(field_key(s, &meta.field), &marg); } } + marg } + None => text_marg(p, st, raw, eng, cache)? }; + let key = combo_key(p, kv); let ws = expect_weights(ex, mem); let member = ws.iter().any(|(k, _)| *k == key); + let mut ms: Vec> = Vec::with_capacity(h.ms.len() + 1); + if lay_key(&a) == h.lay { + let Held { ms: hm, .. } = *h; + for m in hm { ms.push(m.map(|m| match m { HeldM::Named(m) => m, HeldM::Row(r) => { let mut r = r.into_iter(); + a.iter().map(|(id, ls)| (id.clone(), ls.iter().map(|(l, _)| (l.clone(), r.next().unwrap_or(0.0))).collect())).collect() } })); } + } else { + drop(h); let e = get(kv, "elapsed_hours").cloned().unwrap_or(Json::Num(0.0)); + let mut ins = vec![with_elapsed(&Json::Obj(vec![]), &e)]; + for (k, _) in &ws { if *k == key || k == "{}" { continue; } ins.push(with_elapsed(&mem.combos.iter().find(|x| x.0 == *k).map(|x| x.2.clone()).unwrap_or(Json::Obj(vec![])), &e)); } + for x in &ins { ms.push(text_marg(p, st, x, eng, cache)); } + } + ms.insert(0, Some(a)); + mixture(mem, &ws, &key, member, ms) +} +/// The test instrument (`ExpectCache::with_reuse(_, true)`): the A_t the turn handed is the program the deviation builds for the +/// event, text for text, and its marginals are that program's run by the engine, bit for bit +#[inline(never)] +fn at_check(p: &Persona, st: &State, raw: &Json, prog: &Json, m: &Marg, eng: Engine) { + let own = program_json(p, st, raw).expect("A_t builds"); + assert_eq!(text(&own), text(prog), "A_t: the turn's first program is the deviation's"); + let r = exact_marg(p, st, &own, eng).expect("A_t exact"); + assert!(r.len() == m.len() && r.iter().zip(m).all(|(a, b)| a.0 == b.0 && a.1.len() == b.1.len() && a.1.iter().zip(&b.1).all(|(x, y)| x.0 == y.0 && x.1.to_bits() == y.1.to_bits())), "A_t: the turn's marginals are the engine's"); +} +/// One program of the event as A_t's model with its own `h` (from its compile meta `mm`; A_t's is `am`, its skeleton read with +/// it; `own`: A_t itself, its parsed `h`): +/// Some(marginals or None) as the engine would answer; None when the enumeration would not answer it (the caller runs it) +#[inline(never)] +#[allow(clippy::too_many_arguments)] +fn model_marg(p: &Persona, st: &State, prog: &mut Option, mm: &Meta, skel: Option<&[u8; 32]>, sk: &mut Option>>, am: &Meta, own: bool, rk: Option<&[u8; 32]>, eng: Engine, cache: &mut ExpectCache) -> Option> { + // T3-c: a member with other rules in force than the skeleton's (`rk`, its rules' key) is keyed with them + let key = skel.map(|s| match rk { None => field_key(s, &mm.field), Some(r) => field_key(&rules_skel(s, r), &mm.field) }); + if let Some(k) = &key { if let Some(m) = cache.find(k) { return Some(Some(m)); } } + // A_t's text is read into its parts and dropped before the members' model is built from them (A_t itself, unhanded, keeps + // its text: its own fallback runs it) + if sk.is_none() { *sk = Some(if own { prog.as_ref().and_then(|x| skel_parts(p, x, am)) } else { prog.take().and_then(|x| skel_parts_owned(p, x, am)) }); } + // T3-c: the model's caps are the member's rules lowered (the skeleton's again for a member with its rules) + if let Some(Some(s)) = sk.as_ref() { let want = rk.copied().unwrap_or(s.base_rk); if want != s.cur_rk && !swap_caps(p, sk, mm, want) { return None; } } + let own_run = |prog: &Option, cache: &mut ExpectCache| prog.as_ref().and_then(|x| cache.marg(p, st, x, eng)); + let Some(Some(s)) = sk.as_mut() else { return if own { Some(own_run(prog, cache)) } else { None } }; + let h = if own { s.h0.clone() } else { field_h(s, &mm.field) }; + if s.mm.set_h(h).is_err() { cache.keep(key, &None); return Some(None) } + let Some(r) = s.mm.enumerated(p.eng.exact_limit, p.eng.sweeps) else { return if own { Some(own_run(prog, cache)) } else { None } }; + let r = r.map(|mg| { let m = &s.mm.m; let k = m.k; s.vars.iter().enumerate().filter(|(_, id)| !id.starts_with("free.")).map(|(i, id)| (id.clone(), { + let mut l: Vec<(String, f64)> = (0..k).filter(|&q| m.allowed[i * k + q]).map(|q| (s.values[q].clone(), mg[i * k + q])).collect(); l.sort_by(|a, b| a.0.cmp(&b.0)); l })).collect() }); + cache.modeled += 1; cache.keep(key, &r); Some(r) +} +/// T3-c: the members' model with the caps of the rules `want` (the skeleton's, or member `mm`'s lowered) -> false when they are +/// not read (the member takes the engine's path); the skeleton is dropped when `Model::new` refuses them +#[inline(never)] +fn swap_caps(p: &Persona, sk: &mut Option>>, mm: &Meta, want: [u8; 32]) -> bool { + let Some(Some(s)) = sk.as_mut() else { return false }; + let r = if want == s.base_rk { s.mm.set_caps(s.base_caps.clone(), true) } else { + let Some((vs, vals, caps)) = member_caps(p, mm) else { return false }; if vs != s.vars || vals != s.values { return false; } s.mm.set_caps(caps, false) }; + if r.is_err() { *sk = Some(None); return false; } + s.cur_rk = want; true +} +/// A_t's text read into its parts, and the value index of every level -> None when the parts do not have the shape the compiler +/// writes (then every program is run the reference way); checked: A_t's own field mapped to `h` is the `h` its text carries +#[inline(never)] +fn skel_parts(p: &Persona, prog: &Json, mm: &Meta) -> Option> { skel_of(p, crate::run::from_json_parts(prog).ok()?, mm) } +/// `skel_parts`, the text dropped once its parts are read; T3-c: a text written without its pairs (`emit`, `with_pairs: false`) +/// gets the persona's couplings as `from_json` reads them from the full text (`persona_pairs`) +#[inline(never)] +fn skel_parts_owned(p: &Persona, prog: Json, mm: &Meta) -> Option> { + let bare = prog.get("pairs").is_none(); let mut parts = crate::run::from_json_parts(&prog).ok()?; drop(prog); + if bare { let ps = persona_pairs(p, &parts.vars, &parts.values, mm.twin)?; parts.set_pairs(ps); } + skel_of(p, parts, mm) +} +/// T3-c: the pairs `emit` writes for persona `p` (its couplings, then the twins' when the program has them), read as +/// `run::from_json_parts` reads them: per pair its variables' indices and its k x k table (r6 of the persona's, at the values' +/// indices; each weight checked as `json::weight` checks it) -> None when the text's would not read (then no skeleton) +#[inline(never)] +fn persona_pairs(p: &Persona, vars: &[String], values: &[String], twin: bool) -> Option> { + let k = values.len(); let pos_vals = p.eng.positional; + let name_of = |var: &str, val: &str| -> String { match p.var(var) { Some(v) if pos_vals => format!("l{}", v.levels.iter().position(|l| l == val).unwrap_or(0)), _ => val.to_string() } }; + let vi = |id: &str| vars.iter().position(|x| x == id); let vpos = |v: &str| values.iter().position(|x| x == v); + let mut out = vec![]; + let mut tabs = vec![]; + for (a, b, tab) in &p.couplings { + let (la, lb) = (&p.var(a)?.levels, &p.var(b)?.levels); let mut t = vec![0.0; k * k]; + for (x, l1) in la.iter().enumerate() { for (y, l2) in lb.iter().enumerate() { let w = r6(tab[x][y]); if w != 0.0 || !pos_vals { t[vpos(&name_of(a, l1))? * k + vpos(&name_of(b, l2))?] = w; } } } + if t.iter().any(|x| !x.is_finite() || x.abs() > crate::json::MAX_WEIGHT) { return None; } + let (i, j) = (vi(a)?, vi(b)?); if i == j { return None; } + out.push(probbit_ir::Pair { i, j, c: probbit_ir::Coupling::Table(t.clone()) }); tabs.push((a, b, t)); } + if twin { for (a, b, t) in tabs { let (i, j) = (vi(&format!("free.{a}"))?, vi(&format!("free.{b}"))?); out.push(probbit_ir::Pair { i, j, c: probbit_ir::Coupling::Table(t) }); } } + Some(out) +} +/// T3-c: the skeleton's key from a member's text without its pairs: the persona's digest (its couplings) and the text without +/// `comment` and the vars' `h`, as `skel_key` hashes a full text +#[inline(never)] +fn stub_key(p: &Persona, stub: &Json) -> [u8; 32] { let mut h = Sha256::new(); h.update(b"without pairs"); hash_str(&mut h, &p.digest); h.update(&skel_key(stub)); h.finish() } +/// The skeleton from A_t's parts (`skel_parts`) +fn skel_of(p: &Persona, parts: crate::run::Parts, mm: &Meta) -> Option> { + let (nv, ns) = (p.vars.len(), p.steps.len()); + if parts.vars.len() != (nv + ns) * if mm.twin { 2 } else { 1 } { return None; } + let q = |name: &str| parts.values.iter().position(|x| x == name); + let mut idx = vec![]; + for v in &p.vars { idx.push(v.levels.iter().enumerate().map(|(j, l)| q(&if p.eng.positional { format!("l{j}") } else { l.clone() })).collect::>>()?); } + for _ in &p.steps { idx.push(p.slots.iter().map(|l| q(l)).collect::>>()?); } + let (n, k) = (parts.vars.len(), parts.values.len()); + if !field_hn(n, k, &idx, &mm.field).iter().zip(parts.h()).all(|(a, b)| a.to_bits() == b.to_bits()) { return None; } + let (vars, values, h0, mm2) = parts.into_members()?; let rk = rules_key(&mm.rules); + let base_caps = mm2.m.caps.clone(); + Some(Box::new(Skel { vars, values, h0, mm: mm2, idx, base_rk: rk, cur_rk: rk, base_caps })) +} +/// A field (per persona variable, per level) as the program's `h` (n x k): the value index each level gets, 0 where the text +/// omits a weight, the twins' rows copied +fn field_h(s: &Skel, field: &[Vec]) -> Vec { field_hn(s.vars.len(), s.values.len(), &s.idx, field) } +fn field_hn(n: usize, k: usize, idx: &[Vec], field: &[Vec]) -> Vec { + let m = idx.len(); let mut h = vec![0.0; n * k]; + for r in 0..n { let i = r % m; for (&q, &w) in idx[i].iter().zip(&field[i]) { if w != 0.0 { h[r * k + q] = w; } } } + h +} +/// A_t's text without `comment` and without the vars' `h`, hashed as `prog_key` hashes a program +#[inline(never)] +fn skel_key(prog: &Json) -> [u8; 32] { + let mut h = Sha256::new(); h.update(b"skeleton"); + if let Json::Obj(v) = prog { for (k, x) in v.iter().filter(|(k, _)| k != "comment") { hash_str(&mut h, k); + match (k.as_str(), x) { ("vars", Json::Arr(vs)) => { h.update(b"["); h.update(&(vs.len() as u64).to_le_bytes()); + for var in vs { match var { Json::Obj(f) => { let f: Vec<&(String, Json)> = f.iter().filter(|(k, _)| k != "h").collect(); h.update(b"{"); h.update(&(f.len() as u64).to_le_bytes()); + for (k, x) in f { hash_str(&mut h, k); hash_json(&mut h, x); } } x => hash_json(&mut h, x) } } } + _ => hash_json(&mut h, x) } } } + h.finish() +} +/// T3-c: the skeleton's key for a member with other rules in force (`rk`, its rules' key): its text is the skeleton's with its +/// own rules and `h`, so (skeleton, rules, field) names it +#[inline(never)] +fn rules_skel(skel: &[u8; 32], rk: &[u8; 32]) -> [u8; 32] { let mut h = Sha256::new(); h.update(skel); h.update(b"rules"); h.update(rk); h.finish() } +/// A program's key from A_t's skeleton and its field (each number by its bits, -0 as 0) +fn field_key(skel: &[u8; 32], field: &[Vec]) -> [u8; 32] { + let mut h = Sha256::new(); h.update(skel); + for row in field { h.update(&(row.len() as u64).to_le_bytes()); for x in row { h.update(&(if *x == 0.0 { 0 } else { x.to_bits() }).to_le_bytes()); } } + h.finish() +} /// The memory after an event: every count decays by 2^(-1/half_life), the event's combo gains one and keeps its inputs as its /// member; the learned normal (after the cold start) reads ln(1 + the logged pred) with the same half-life pub fn expect_step(p: &Persona, mem: &mut ExpectMem, raw: &Json, logged_pred: Option) { let Some(ex) = p.expect.as_ref() else { return }; let Some(kv) = raw.as_obj() else { return }; let d = 0.5f64.powf(1.0 / ex.half_life); let key = combo_key(p, kv); let rep = Json::Obj(kv.iter().filter(|(k, _)| k != "elapsed_hours").cloned().collect()); + // the canonical fold keeps a member's inputs with their keys sorted at every level, as a checkpoint line carries them + let rep = if mem.canon { key_sorted(&rep) } else { rep }; for c in &mut mem.combos { c.1 *= d; } match mem.combos.iter_mut().find(|c| c.0 == key) { Some(c) => { c.1 += 1.0; c.2 = rep; } None => mem.combos.push((key.clone(), 1.0, rep)) } if key != "{}" { mem.warm = true; } + // RFC 0016 (canonical memory): the drive's fold reads the event's z as the deviation computed it (the logged prediction + // against the normal before this event), before the normal moves + if let (Some(c), true) = (p.cur.as_ref(), mem.canon) { + let z = logged_pred.map(|x| ((1.0 + x).ln() - mem.mean) / (mem.var + DEV_VAR_FLOOR).sqrt()); + mem.cur.get_or_insert_with(|| curiosity::CurMem::new(c)).step(c, &key, z, kv); } + // RFC 0015 (canonical memory): the rest pass's window reads the same z (one-sided: z+), before the normal moves + if let (Some(rp), true) = (ex.rest.as_ref(), mem.canon) { + let z = logged_pred.map(|x| ((1.0 + x).ln() - mem.mean) / (mem.var + DEV_VAR_FLOOR).sqrt()); + mem.win.get_or_insert_with(Default::default).step(&key, d, 0.5f64.powf(1.0 / rp.half_life), z.map_or(0.0, |z| r6(z.max(0.0)))); } if let Some(x) = logged_pred { let (x, a) = ((1.0 + x).ln(), 1.0 - d); let dx = x - mem.mean; mem.mean += a * dx; mem.var = (1.0 - a) * (mem.var + a * dx * dx); mem.n += 1; } + if mem.canon { + mem.rest = r6(mem.rest * d); + for c in &mut mem.combos { c.1 = r6(c.1); } + mem.combos.retain(|c| c.1 > 0.0); + // past max_combos - 1 named combos: the last in the mixture's order (count desc, key asc) moves its count to `rest` + while mem.combos.iter().filter(|c| c.0 != "{}").count() > ex.max_combos - 1 { + let Some(i) = (0..mem.combos.len()).filter(|&i| mem.combos[i].0 != "{}").min_by(|&a, &b| { let (x, y) = (&mem.combos[a], &mem.combos[b]); + x.1.partial_cmp(&y.1).unwrap_or(std::cmp::Ordering::Equal).then(y.0.cmp(&x.0)) }) else { break }; + let c = mem.combos.remove(i); mem.rest = r6(mem.rest + c.1); } + if let Some(cm) = mem.cur.as_mut() { cm.prune(&mem.combos); } + if let Some(w) = mem.win.as_mut() { w.settle(&mem.combos); } + mem.mean = r6(mem.mean); mem.var = r6(mem.var); + } +} +/// RFC 0016: the curiosity drive's level the next event's turn reads, from the expectation's memory (None without the block or +/// before the canonical fold, engine 0.14.0); a memory with no event yet reads the fold's start (level 0 from quiet) +pub fn cur_level(p: &Persona, mem: &ExpectMem) -> Option { + let c = p.cur.as_ref().filter(|_| mem.canon)?; Some(mem.cur.as_ref().map_or_else(|| curiosity::CurMem::new(c).level(c), |m| m.level(c))) +} +/// RFC 0016: the stance's `drives.curiosity` (the level and its two inputs, as the turn read them) from the memory before the event +pub fn cur_doc(p: &Persona, mem: &ExpectMem) -> Option { + let c = p.cur.as_ref().filter(|_| mem.canon)?; Some(mem.cur.as_ref().map_or_else(|| curiosity::CurMem::new(c).doc(c), |m| m.doc(c))) +} +/// RFC 0016: is a self-started idle event due `idle_hours` after the last event (`curiosity::stir_due` on the memory's fold) +pub fn stir_due(p: &Persona, mem: &ExpectMem, idle_hours: f64) -> Option { + let c = p.cur.as_ref().filter(|_| mem.canon)?; let start; let m = match mem.cur.as_ref() { Some(m) => m, None => { start = curiosity::CurMem::new(c); &start } }; + curiosity::stir_due(c, m, idle_hours) +} +/// `j` with every object's keys sorted (by code point) at every level: the value canonical JSON writes, as a value +fn key_sorted(j: &Json) -> Json { + match j { Json::Obj(v) => { let mut o: Vec<(String, Json)> = v.iter().map(|(k, x)| (k.clone(), key_sorted(x))).collect(); o.sort_by(|a, b| a.0.cmp(&b.0)); Json::Obj(o) } + Json::Arr(v) => Json::Arr(v.iter().map(key_sorted).collect()), x => x.clone() } } impl Deviation { - /// The stance's `deviation` (full precision) - pub fn to_json(&self) -> Json { - Json::Obj(vec![("self".into(), Json::Num(self.s_self)), ("pred".into(), Json::Num(self.pred)), ("input".into(), Json::Num(self.input)), ("cold".into(), Json::Bool(self.cold)), + /// The stance's `deviation`: every number at 6 decimals (the canonical number rule), so a checkpoint line carries the same + /// bytes on every platform's libm; `full`: self, pred and input at full precision, as engines before 0.14.0 wrote them + pub fn to_json(&self, full: bool) -> Json { + let n = |x: f64| Json::Num(if full { x } else { r6(x) }); + Json::Obj(vec![("self".into(), n(self.s_self)), ("pred".into(), n(self.pred)), ("input".into(), n(self.input)), ("cold".into(), Json::Bool(self.cold)), ("z".into(), self.z.map_or(Json::Null, |z| Json::Num(r6(z)))), ("registers".into(), Json::Bool(self.z.is_some_and(|z| z >= DEV_Z)))]) } /// The strand line's receipt: self, pred and input at 5 significant digits, `cold: true` while cold @@ -3938,14 +4738,19 @@ pub fn trajectory_program(prog: &Json, k: usize, beta0: f64, limit: u64) -> Opti Some(Json::Obj(vec![("k".into(), Json::Num(k as f64)), ("beta0".into(), Json::Num(beta0)), ("steps".into(), Json::Arr(steps))])) } /// The event's trajectory with the persona's `engine.trajectory` (the program `persona compile` gives for the event) +#[inline(never)] pub fn trajectory(p: &Persona, st: &State, raw: &Json) -> Option { let t = p.traj.as_ref()?; trajectory_program(&program_json(p, st, raw).ok()?, t.k, t.beta0, p.eng.exact_limit) } -/// A stance document's sha256 as strand lines log it: without `deviation` (RFC 0009) and without `glyph` and `glyph_why` (§2.18); -/// both have effects {}: the blocks on or off, one digest +/// A stance document's sha256 as strand lines log it: without `deviation` (RFC 0009), without `glyph` and `glyph_why` (§2.18) and +/// without `integrity` (RFC 0014); all have effects {}: the blocks on or off, one digest pub fn stance_sha(doc: &Json) -> String { - const OUT: [&str; 3] = ["deviation", "glyph", "glyph_why"]; - match doc { Json::Obj(v) if v.iter().any(|(k, _)| OUT.contains(&k.as_str())) => sha(&Json::Obj(v.iter().filter(|(k, _)| !OUT.contains(&k.as_str())).cloned().collect())), _ => sha(doc) } + const OUT: [&str; 4] = ["deviation", "glyph", "glyph_why", "integrity"]; + // RFC 0016: `drives.curiosity` is the live writer's (from the expectation's memory), outside the digest as `deviation` is + let cur = |x: &Json| x.get("curiosity").is_some(); + match doc { Json::Obj(v) if v.iter().any(|(k, x)| OUT.contains(&k.as_str()) || (k == "drives" && cur(x))) => sha(&Json::Obj(v.iter().filter(|(k, _)| !OUT.contains(&k.as_str())) + .filter_map(|(k, x)| match x { Json::Obj(d) if k == "drives" && cur(x) => { let r: Vec<(String, Json)> = d.iter().filter(|(j, _)| j != "curiosity").cloned().collect(); + (!r.is_empty()).then(|| (k.clone(), Json::Obj(r))) } _ => Some((k.clone(), x.clone())) }).collect())), _ => sha(doc) } } /// §2.18 `display.glyphs`: one glyph per stance, computed from the stance document. `moods`: the two mood ids (rows, then @@ -4050,6 +4855,181 @@ mod tests { fn inputs(pr: bool, cr: bool, loss: bool) -> Json { json::parse(&format!(r#"{{"praise":{pr},"criticism":{cr},"loss":{loss}}}"#)).unwrap() } fn err(r: R) -> String { r.err().map(|e| format!("{}: {}", e.path, e.msg)).unwrap_or_default() } + /// The streamed sha256 is the one-shot one at every length around the 64-byte block and padding edges, fed in any pieces + #[test] + fn sha256_fed_in_pieces_is_the_one_shot_digest() { + assert_eq!(hex(&sha256(b"")), "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"); + assert_eq!(hex(&sha256(b"abc")), "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"); + let data: Vec = (0..300u32).map(|i| (i * 7 + 3) as u8).collect(); + for n in [0, 1, 55, 56, 57, 63, 64, 65, 119, 120, 128, 300] { let one = sha256(&data[..n]); + for piece in [1, 3, 64, 100] { let mut h = Sha256::new(); for c in data[..n].chunks(piece) { h.update(c); } assert_eq!(h.finish(), one, "{n} bytes in pieces of {piece}"); } } + } + /// RFC 0011 (research): `engine.tier: amplitude` on a persona of two-valued traits answers from the simulated register (its + /// receipt in the stance document, habits held, deterministic, a strand that verifies); round 2: three-valued traits are + /// answered too (two qubits each, one register per component), a component over `q_max` declines, the existing tiers answer + /// and the receipt says why; absent, nothing changes + #[cfg(feature = "decide")] + #[test] + fn the_amplitude_tier_answers_multi_level_programs_and_declines_past_q_max() { + use crate::strand::{self, Clock, Live}; + let text = include_str!("../../examples/persona/dispatch-policy.yaml").replace("\r\n", "\n").replace("engine:\n", "engine:\n tier: amplitude\n"); + let doc = crate::yaml::load(&text).ok().unwrap(); let p = build(&doc).unwrap(); let st = init(&p, Some(2), true, &run); + for raw in [r#"{"blast":0}"#, r#"{"blast":0.9}"#, r#"{"blast":0.3,"error":true}"#] { + let raw = json::parse(raw).unwrap(); let (a, _) = turn(&p, &st, &raw, false, &run, false).unwrap(); let (b, _) = turn(&p, &st, &raw, false, &run, false).unwrap(); + assert_eq!(canon(&a), canon(&b)); assert_eq!(g(&a, &["engine", "tier"]).as_str(), Some("amplitude"), "{}", canon(&a)); + let q = g(&a, &["engine", "qubits"]).as_f64().unwrap(); assert!((1.0..=4.0).contains(&q)); + for k in ["circuit", "schedule", "state"] { assert!(g(&a, &["engine", k]).as_str().is_some_and(is_digest), "{k}"); } + assert_eq!(g(&a, &["habits", "violations"]).as_f64(), Some(0.0)); assert_eq!(g(&a, &["status"]).as_str(), Some("ok")); } + let (mut live, header) = Live::start(p.clone(), &doc, st.clone(), Clock::Fixed, "probbit test"); let mut lines = vec![header]; + for (i, raw) in [r#"{"blast":0}"#, r#"{"blast":0.9}"#, r#"{"error":true}"#, r#"{"error":true}"#, r#"{"praise":true,"src":"human"}"#].iter().enumerate() { + let (_, l) = live.event(&json::parse(raw).unwrap(), &run).unwrap_or_else(|e| panic!("event {i}: {}", e.msg)); lines.push(l); } + let strand_text = lines.join("\n") + "\n"; assert!(strand::verify(&strand_text, &run).is_ok()); + // three-valued traits: answered (encoded qubits, the components, the encoding and the mixer in the receipt) + let q = build(&json::parse(&DOC.replace("LEARN", r#","engine":{"tier":"amplitude"}"#)).unwrap()).unwrap(); let s0 = init(&q, Some(3), true, &run); + let (d, _) = turn(&q, &s0, &inputs(false, false, false), false, &run, false).unwrap(); let (d2, _) = turn(&q, &s0, &inputs(false, false, false), false, &run, false).unwrap(); + assert_eq!(canon(&d), canon(&d2)); assert_eq!(g(&d, &["engine", "tier"]).as_str(), Some("amplitude"), "{}", canon(&d)); + assert_eq!(g(&d, &["engine", "mixer"]).as_str(), Some("complete-graph")); assert!(g(&d, &["engine", "encoding"]).as_str().is_some_and(is_digest)); + assert!(g(&d, &["engine", "qubits"]).as_f64().unwrap() >= 2.0); assert_eq!(g(&d, &["habits", "violations"]).as_f64(), Some(0.0)); + // a component past q_max: declined, the existing tier answers, the receipt says so + let q1 = build(&json::parse(&DOC.replace("LEARN", r#","engine":{"tier":"amplitude","q_max":1}"#)).unwrap()).unwrap(); + let (d, _) = turn(&q1, &init(&q1, Some(3), true, &run), &inputs(false, false, false), false, &run, false).unwrap(); + assert!(g(&d, &["engine", "amplitude"]).as_str().is_some_and(|w| w.starts_with("declined: ") && w.contains("encoded qubits, q_max 1")), "{}", canon(&d)); + assert_ne!(g(&d, &["engine", "tier"]).as_str(), Some("amplitude")); + // the tutor example (three-level traits, 60 encoded qubits in components of at most 10): answered, and its strand verifies + let text = include_str!("../../examples/persona/tutor.yaml").replace("\r\n", "\n").replace("engine:\n", "engine:\n tier: amplitude\n"); + let doc = crate::yaml::load(&text).ok().unwrap(); let p = build(&doc).unwrap(); let st = init(&p, Some(2), true, &run); + let (mut live, header) = Live::start(p.clone(), &doc, st, Clock::Fixed, "probbit test"); let mut lines = vec![header]; + for raw in [r#"{"elapsed_hours":1,"praise":true}"#, r#"{"elapsed_hours":2}"#] { let (o, l) = live.event(&json::parse(raw).unwrap(), &run).unwrap(); lines.push(l); + assert_eq!(g(&o, &["engine", "tier"]).as_str(), Some("amplitude"), "{}", canon(&o)); } + assert!(strand::verify(&(lines.join("\n") + "\n"), &run).is_ok()); + let plain = build(&json::parse(&DOC.replace("LEARN", "")).unwrap()).unwrap(); let (e, _) = turn(&plain, &init(&plain, Some(3), true, &run), &inputs(false, false, false), false, &run, false).unwrap(); + assert!(g(&e, &["engine", "amplitude"]).is_null() && g(&e, &["engine", "qubits"]).is_null()); + assert_eq!(err(build(&json::parse(&DOC.replace("LEARN", r#","engine":{"tier":"pbit"}"#)).unwrap())), "engine.tier: amplitude (the one optional tier; absent = the default tiers)"); + } + /// RFC 0011 round 4 (research): `engine.amplify: true` rotates each component's register onto its feasible plans by exact + /// amplitude amplification instead of projecting: the stance is the projection's (plan, odds, line), the receipt carries + /// `amplify` and feasible mass 1, the strand verifies; `amplify: false` is the round-3 behaviour; without the tier it is refused + #[cfg(feature = "decide")] + #[test] + fn amplify_lands_on_the_projection_and_logs_it() { + use crate::strand::{self, Clock, Live}; + let base = include_str!("../../examples/persona/tutor.yaml").replace("\r\n", "\n"); + let mk = |extra: &str| { let doc = crate::yaml::load(&base.replace("engine:\n", &format!("engine:\n tier: amplitude\n{extra}"))).ok().unwrap(); (build(&doc).unwrap(), doc) }; + let ((pj, _), (pf, _), (pa, doc)) = (mk(""), mk(" amplify: false\n"), mk(" amplify: true\n")); + let keep = |d: &Json| { let mut d = d.clone(); if let Json::Obj(v) = &mut d { v.retain(|(k, _)| !["persona", "engine", "state_digest"].contains(&k.as_str())); } canon(&d) }; + let (sj, sf, sa) = (init(&pj, Some(2), true, &run), init(&pf, Some(2), true, &run), init(&pa, Some(2), true, &run)); + for raw in [r#"{"elapsed_hours":1,"praise":true}"#, r#"{"elapsed_hours":2}"#, r#"{"elapsed_hours":1,"correction":true}"#] { + let raw = json::parse(raw).unwrap(); let (a, _) = turn(&pj, &sj, &raw, false, &run, false).unwrap(); let (f, _) = turn(&pf, &sf, &raw, false, &run, false).unwrap(); + let (b, _) = turn(&pa, &sa, &raw, false, &run, false).unwrap(); + assert_eq!(canon(&g(&a, &["engine"])), canon(&g(&f, &["engine"]))); assert!(g(&a, &["engine", "amplify"]).is_null()); + assert_eq!(keep(&a), keep(&b), "the stance with amplify is the projection's"); + assert_eq!(g(&b, &["engine", "feasible_mass"]).as_f64(), Some(1.0)); let am = g(&b, &["engine", "amplify"]); + let it = am.get("iterations").and_then(Json::as_arr).unwrap(); assert_eq!(it.len(), g(&b, &["engine", "components"]).as_arr().unwrap().len()); + assert!(it.iter().any(|x| x.as_f64().unwrap() >= 1.0), "{}", canon(&am)); assert_eq!(am.get("mass_after").and_then(Json::as_f64), Some(1.0)); + assert_eq!(am.get("mass_before").and_then(Json::as_f64), g(&a, &["engine", "feasible_mass"]).as_f64()); } + let (mut live, header) = Live::start(pa.clone(), &doc, sa, Clock::Fixed, "probbit test"); let mut lines = vec![header]; + for raw in [r#"{"elapsed_hours":1,"praise":true}"#, r#"{"elapsed_hours":2}"#] { let (_, l) = live.event(&json::parse(raw).unwrap(), &run).unwrap(); lines.push(l); } + assert!(strand::verify(&(lines.join("\n") + "\n"), &run).is_ok()); + let e = err(build(&crate::yaml::load(&base.replace("engine:\n", "engine:\n amplify: true\n")).ok().unwrap())); + assert!(e.starts_with("engine.amplify: true needs tier: amplitude"), "{e}"); + } + /// RFC 0011 round 5 (research): `engine.schedule` names the register's evolution: the default name is the absent key (the + /// same documents, byte for byte), the other two name themselves in the receipt and their strands verify, an unknown name and + /// the key without the tier are refused; the name list is the decide crate's + #[cfg(feature = "decide")] + #[test] + fn the_schedule_key_names_the_evolution() { + use crate::strand::{self, Clock, Live}; + assert_eq!(SCHEDULE_NAMES, probbit_decide::amp::SCHEDULE_NAMES); + let base = include_str!("../../examples/persona/tutor.yaml").replace("\r\n", "\n"); + let mk = |extra: &str| { let doc = crate::yaml::load(&base.replace("engine:\n", &format!("engine:\n tier: amplitude\n amplify: true\n{extra}"))).ok().unwrap(); (build(&doc).unwrap(), doc) }; + let (p0, _) = mk(""); let (pd, _) = mk(" schedule: tf-linear-16x4\n"); let s0 = init(&p0, Some(2), true, &run); let sd = init(&pd, Some(2), true, &run); + let raw = json::parse(r#"{"elapsed_hours":1,"praise":true}"#).unwrap(); + let (a, _) = turn(&p0, &s0, &raw, false, &run, false).unwrap(); let (b, _) = turn(&pd, &sd, &raw, false, &run, false).unwrap(); + assert_eq!(canon(&g(&a, &["engine"])), canon(&g(&b, &["engine"]))); assert_eq!(g(&a, &["engine", "schedule_name"]).as_str(), Some("tf-linear-16x4.000000")); + for (name, shown) in [("tf-linear-64x16", "tf-linear-64x16.000000"), ("qaoa-p2", "qaoa-p2")] { + let (p, doc) = mk(&format!(" schedule: {name}\n")); let st = init(&p, Some(2), true, &run); + let (d, _) = turn(&p, &st, &raw, false, &run, false).unwrap(); assert_eq!(g(&d, &["engine", "schedule_name"]).as_str(), Some(shown)); + assert_eq!(g(&d, &["engine", "feasible_mass"]).as_f64(), Some(1.0)); + let (mut live, header) = Live::start(p.clone(), &doc, st, Clock::Fixed, "probbit test"); let mut lines = vec![header]; + for r in [r#"{"elapsed_hours":1,"praise":true}"#, r#"{"elapsed_hours":2}"#] { let (_, l) = live.event(&json::parse(r).unwrap(), &run).unwrap(); lines.push(l); } + assert!(strand::verify(&(lines.join("\n") + "\n"), &run).is_ok(), "{name}"); } + let e = err(build(&crate::yaml::load(&base.replace("engine:\n", "engine:\n tier: amplitude\n schedule: tf-linear-8x2\n")).ok().unwrap())); + assert_eq!(e, "engine.schedule: tf-linear-16x4 | tf-linear-64x16 | qaoa-p2"); + let e = err(build(&crate::yaml::load(&base.replace("engine:\n", "engine:\n schedule: qaoa-p2\n")).ok().unwrap())); + assert!(e.starts_with("engine.schedule: needs tier: amplitude"), "{e}"); + } + /// The boxed setup API is the by-value one: the same persona, state, header and individual + #[test] + fn the_boxed_setup_is_the_by_value_setup() { + use crate::strand::{self, Clock, Live}; + let doc = json::parse(&DOC.replace("LEARN", LEARN)).unwrap(); + let (p, b) = (build(&doc).unwrap(), build_boxed(&doc).unwrap()); assert_eq!(p.digest, b.digest); + let (st, sb) = (init(&p, Some(4), true, &run), init_boxed(&b, Some(4), true, &run)); assert_eq!(st.digest, sb.digest); + let (mut a, ha) = Live::start(p.clone(), &doc, st.clone(), Clock::Fixed, "probbit 0.14.0"); + let (mut c, hc) = Live::start_boxed(b, &doc, sb, Clock::Fixed, "probbit 0.14.0"); assert_eq!(ha, hc); assert_eq!(a.head(), c.head()); + let (_, la) = a.event(&inputs(true, false, false), &run).unwrap(); let (_, lc) = c.event(&inputs(true, false, false), &run).unwrap(); assert_eq!(la, lc); + let text = format!("{ha}\n{la}\n"); + let r1 = strand::resume(&text, &p, a.st.clone(), Clock::Fixed).unwrap(); let r2 = strand::resume_boxed(&text, &p, Box::new(c.st.clone()), Clock::Fixed).unwrap(); + assert_eq!((r1.n, r1.head(), &r1.st.digest), (r2.n, r2.head(), &r2.st.digest)); + } + /// The boxed turn is the turn: the same stance documents and states over a few events (a habit firing, feedback, none), and + /// `Live::event` (which runs it) chains the same lines as before + #[test] + fn the_boxed_turn_is_the_turn() { + let p = persona(LEARN).unwrap(); let mut st = init(&p, Some(5), true, &run); + for (pr, cr, loss) in [(false, false, false), (true, false, false), (false, true, true), (true, false, true), (false, false, false)] { + let ins = inputs(pr, cr, loss); + let (d1, s1) = turn(&p, &st, &ins, false, &run, false).unwrap(); let (d2, s2) = turn_boxed(&p, &st, &ins, false, &run, false).unwrap(); + assert_eq!(canon(&d1), canon(&d2)); assert_eq!(canon(&s1.to_json(&p)), canon(&s2.to_json(&p))); st = s1; } + assert_eq!(err(turn_boxed(&p, &st, &json::parse("[1]").unwrap(), false, &run, false)), "inputs: must be a JSON object"); + } + /// The chain check opens the header without a replay and compares it in pieces: a header that reads the same but is not as + /// written (one space more) is refused as before, and a strand with checkpoints still checks + #[test] + fn the_chain_check_compares_the_header_in_pieces() { + use crate::strand::{self, Clock, Live}; + let doc = json::parse(&DOC.replace("LEARN", LEARN)).unwrap(); let p = build(&doc).unwrap(); + let (mut lv, header) = Live::start(p.clone(), &doc, init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.14.0"); + let mut text = format!("{header}\n"); + for k in 0..4 { let (s, l) = lv.event(&inputs(k % 2 == 0, false, k == 3), &run).unwrap(); text += &l; text.push('\n'); if k == 1 { text += &lv.checkpoint(&s); text.push('\n'); } } + assert_eq!(strand::check_chain(&text).unwrap(), strand::Status::Active); + let r = strand::resume(&text, &p, lv.st.clone(), Clock::Fixed).unwrap(); assert_eq!((r.n, r.head()), (4, lv.head())); + let spaced = text.replacen(r#""seed":"#, r#""seed": "#, 1); + let e = strand::check_chain(&spaced).unwrap_err(); assert_eq!(e.msg, "line 1: the header is not as written"); + let e = strand::resume(&spaced, &p, lv.st.clone(), Clock::Fixed).err().unwrap(); assert_eq!(e.msg, "line 1: the header is not as written"); + } + /// With `turn_trace`, an event's breakdown: every phase entered once at least (a probe returning the phase's number + 1 is + /// kept per phase), an engine call counted, and the phases' microseconds within the event's + #[cfg(feature = "turn_trace")] + #[test] + fn turn_trace_reports_every_phase_of_an_event() { + use crate::{strand::{Clock, Live}, trace}; + let doc = json::parse(&DOC.replace("LEARN", LEARN)).unwrap(); let p = build(&doc).unwrap(); + let (mut lv, _) = Live::start(p.clone(), &doc, init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.14.0"); + fn probe(ph: trace::Phase) -> u64 { ph as u64 + 1 } + trace::set_probe(Some(probe)); lv.event(&inputs(true, false, false), &run).unwrap(); trace::set_probe(None); + let t = trace::last(); assert_eq!(t.bytes, [1, 2, 3, 4, 5, 6, 7, 8, 9]); assert!(t.solves >= 1); + assert!(t.us.iter().sum::() <= t.total_us + trace::PHASES as u64, "{t:?}"); + assert_eq!(g(&t.to_json(), &["us"]).as_obj().map(|o| o.len()), Some(trace::PHASES)); + } + /// The header written in pieces is `Live::start`'s header byte for byte, and its returned digest is the chain's first `prev`, + /// at every piece size (1 byte, a size that cuts inside UTF-8 characters, the default, one piece for the whole line) + #[test] + fn the_streamed_header_is_the_header_byte_for_byte() { + use crate::strand::{self, Clock, Live}; + // a document with non-ASCII text and escapes, and a display block (the header keeps the document as given) + let src = DOC.replace("LEARN", LEARN).replace(r#""name":"L""#, r#""name":"L","summary":"caf\u00e9 \u263a tab\t \"q\"""#); + let doc = json::parse(&src).unwrap(); let p = build(&doc).unwrap(); let st = init(&p, None, true, &run); + let (lv, header) = Live::start(p.clone(), &doc, st.clone(), Clock::Fixed, "probbit 0.14.0"); + for chunk in [0, 1, 7, 64, strand::HEADER_CHUNK, header.len(), header.len() + 9] { + let mut pieces: Vec> = vec![]; let d = strand::write_header(&p, &doc, &st, "probbit 0.14.0", chunk, &mut |c: &[u8]| pieces.push(c.to_vec())); + assert!(pieces.iter().all(|c| !c.is_empty() && c.len() <= chunk.max(1)), "piece size {chunk}"); + assert_eq!(pieces.concat(), header.as_bytes(), "piece size {chunk}"); assert_eq!(d, lv.head()); assert_eq!(d, digest_of(&header)); } + let mut out = vec![]; let lv2 = Live::start_to(Box::new(p), &doc, Box::new(st), Clock::Fixed, "probbit 0.14.0", 100, &mut |c: &[u8]| out.extend_from_slice(c)); + assert_eq!(out, header.as_bytes()); assert_eq!(lv2.head(), lv.head()); + } + /// Check both signs at the rounded cap and refusal beyond half a unit of the sixth decimal. #[test] fn state_cap_rounding_has_a_bounded_tolerance() { diff --git a/probbit-persona/src/reflect.rs b/probbit-persona/src/reflect.rs new file mode 100644 index 0000000..8bf5fff --- /dev/null +++ b/probbit-persona/src/reflect.rs @@ -0,0 +1,407 @@ +//! RFC 0013, self-report (docs/persona.md §5.8 "Self-report"): a canonical account of a window of a strand's events, computed +//! from the strand's lines alone: no turn is run, nothing is solved, nothing is drawn at random. Per event line the reflection +//! reads what the line logs (the inputs, the expectation's receipt, the glyph); the z of each receipt is recomputed from the +//! expectation's memory, a fold of the logged receipts (`persona::expect_step`'s mean and variance, nothing else); per +//! checkpoint line in the window it reads the stance and the state the line carries (the levels taken and their odds, the +//! moods, the drives, the habits that bound, the learned weights); control, rest and reflection lines are counted. A section +//! the window has no line for is `null` (absent, not invented): a window without a checkpoint line reports no levels. +//! +//! The document (`Reflection::doc`) is canonical JSON with every number at 6 decimals and `digest` = the sha256 of its +//! canonical text without `digest`; `summary` is 6 to 12 sentences (13 at most with a declared stir, RFC 0016) made by fixed templates from its numbers, part of the +//! document. Cost: one pass over the lines from the fold's start (the last checkpoint carrying the expectation's memory before +//! the window, or the header), each event line parsed once, each checkpoint line in the window parsed once. +use crate::json::{self, InErr, Json}; +use crate::persona::{self, r6}; +#[allow(unused_imports)] +use std::prelude::rust_2021::*; + +/// The document's format +pub const FORMAT: f64 = 1.0; +/// The surprises reported: the K events with the largest |z| +pub const TOP_K: usize = 5; +/// The turning points listed per mood (all are counted) +pub const TURNS_MAX: usize = 8; +/// The samples of a mood's path the document carries (the last ones) +pub const PATH_MAX: usize = 50; +/// The engine from which a strand may carry reflection lines +pub const REFLECT_SINCE: (u64, u64, u64) = (0, 14, 0); +/// z at or above this registers (the expectation's own threshold) +const DEV_Z: f64 = persona::DEV_Z; +/// The expectation's prior variance and variance floor (persona.rs) +const PRIOR_VAR: f64 = 0.04; +const VAR_FLOOR: f64 = 1e-4; + +/// Which events a reflection covers +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Window { All, Last(u64), SinceCheckpoint } +impl Window { + /// The window a reflection document names (its `window.last_n` and `window.since_checkpoint`) + pub fn of(doc: &Json) -> Option { + let w = doc.get("window")?; + if w.get("since_checkpoint") == Some(&Json::Bool(true)) { return Some(Window::SinceCheckpoint); } + match w.get("last_n") { Some(Json::Num(x)) if *x >= 1.0 && x.fract() == 0.0 => Some(Window::Last(*x as u64)), Some(Json::Null) => Some(Window::All), _ => None } + } +} +/// A reflection: the canonical document (with `digest`) and its digest +pub struct Reflection { pub doc: Json, pub digest: String } +impl Reflection { + /// The canonical text + pub fn text(&self) -> String { persona::canon(&self.doc) } + /// The summary's sentences + pub fn summary(&self) -> Vec { self.doc.get("summary").and_then(Json::as_arr).map_or(vec![], |v| v.iter().filter_map(|s| s.as_str().map(String::from)).collect()) } +} + +/// What kind a canonical strand line is, from its first bytes (`reflect` reads lines `verify` checks; it does not check them) +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum K { Header, Event, Control, Checkpoint, Rest, Reflection, Consolidation } +fn kind_of(l: &str) -> K { + if l.starts_with("{\"probbit_strand\":") { K::Header } else if l.starts_with("{\"checkpoint\":") { K::Checkpoint } else if l.starts_with("{\"at\":") { + // RFC 0015: a consolidation line also starts with `at` (its next key is `consolidation`, a control line's is `by`) + if json::parse(l).ok().is_some_and(|j| j.get("consolidation").is_some()) { K::Consolidation } else { K::Control } } + else if l.starts_with("{\"prev\":") { let t = l.get(80..).unwrap_or(""); if t.starts_with("\",\"reflection\":") { K::Reflection } else { K::Rest } } + else { K::Event } +} +/// True for a reflection line (canonical: `{"prev": ..., "reflection": ...}`) +pub fn is_reflection_line(l: &str) -> bool { kind_of(l) == K::Reflection } +fn bad(msg: impl Into) -> InErr { json::value("reflect", msg) } +fn n6(x: f64) -> Json { Json::Num(r6(x)) } +fn cnt(x: usize) -> Json { Json::Num(x as f64) } +fn f6(x: f64) -> String { format!("{:.6}", r6(x)) } +/// The declared ids and levels of a document's list (`traits`, `moods`) +fn declared(doc: &Json, key: &str) -> Vec<(String, Vec)> { + doc.get(key).and_then(Json::as_arr).map_or(vec![], |v| v.iter().filter_map(|t| Some((t.get("id")?.as_str()?.to_string(), + t.get("levels")?.as_arr()?.iter().filter_map(|l| l.as_str().map(String::from)).collect()))).collect()) +} +/// Where odds sit along the levels, 0 (the lowest) to 1 (the highest): the monitor's sparkline position +fn position(levels: &[String], odds: &Json) -> f64 { + let ps: Vec = levels.iter().map(|l| odds.get(l).and_then(Json::as_f64).unwrap_or(0.0)).collect(); + let (t, k) = (ps.iter().sum::(), ps.len()); + if k < 2 || t <= 0.0 { return 0.0; } + ps.iter().enumerate().map(|(i, p)| p * i as f64).sum::() / t / (k - 1) as f64 +} +/// An input counts as set when it is true or a non-zero number (the history and the sensed readings are not counted) +fn set_keys(inputs: &Json) -> Vec { + inputs.as_obj().map_or(vec![], |kv| kv.iter().filter(|(k, v)| !matches!(k.as_str(), "elapsed_hours" | "src" | "senses") + && match v { Json::Bool(b) => *b, Json::Num(x) => *x != 0.0, _ => false }).map(|(k, _)| k.clone()).collect()) +} +fn by_count(m: &[(String, usize)]) -> Vec<(String, usize)> { let mut v = m.to_vec(); v.sort_by(|a, b| b.1.cmp(&a.1).then(a.0.cmp(&b.0))); v } +fn bump(m: &mut Vec<(String, usize)>, k: &str) { match m.iter_mut().find(|x| x.0 == k) { Some(x) => x.1 += 1, None => m.push((k.to_string(), 1)) } } +fn obj_counts(m: &[(String, usize)]) -> Json { Json::Obj(m.iter().map(|(k, c)| (k.clone(), cnt(*c))).collect()) } +fn odds_obj(levels: &[String], o: &Json) -> Json { Json::Obj(levels.iter().map(|l| (l.clone(), n6(o.get(l).and_then(Json::as_f64).unwrap_or(0.0)))).collect()) } + +/// Per trait (and `pursue`): the levels taken, the odds at the first and last samples, the smallest and largest p of the level taken +struct Tr { id: String, levels: Vec, taken: Vec<(String, usize)>, first: Option, last: Option, pmin: f64, pmax: f64 } +/// Per mood: its position at each sample (event n, value) +struct Md { id: String, levels: Vec, path: Vec<(u64, f64)> } + +/// The reflection of the last `n` events of a strand (`None`: every event) +pub fn reflect(lines: &[&str], n: Option) -> Result { reflect_window(lines, n.map_or(Window::All, Window::Last)) } +/// The reflection of the events after the strand's last checkpoint line (the checkpoint itself is the first sample) +pub fn reflect_since_checkpoint(lines: &[&str]) -> Result { reflect_window(lines, Window::SinceCheckpoint) } +/// `reflect` over a strand's text +pub fn reflect_text(text: &str, w: Window) -> Result { reflect_window(&crate::strand::lines(text), w) } + +/// The reflection of window `w` of a strand's lines (its header first) +#[inline(never)] +pub fn reflect_window(lines: &[&str], w: Window) -> Result { + let head = json::parse(lines.first().copied().unwrap_or("")).map_err(|e| bad(format!("line 1: the header is not JSON: {}", e.msg)))?; + if kind_of(lines.first().copied().unwrap_or("")) != K::Header { return Err(bad("line 1 is not a strand header")); } + let engine = head.get("engine").and_then(Json::as_str).unwrap_or("").to_string(); + let canon = !crate::strand::engine_version(&engine).is_some_and(|v| v < crate::strand::DEV_R6_SINCE); + let doc = head.get("document").cloned().unwrap_or(Json::Null); + let kinds: Vec = lines.iter().map(|l| kind_of(l)).collect(); + let events: Vec = (1..lines.len()).filter(|&i| kinds[i] == K::Event).collect(); + // the window: [start, lines.len()) and its first event + let (start, first_ev) = match w { + Window::All => { let s = events.first().copied().ok_or_else(|| bad("the strand has no event line"))?; (s, 0) } + Window::Last(k) => { if k == 0 { return Err(bad("--last: a whole number >= 1")); } let f = events.len().saturating_sub(k as usize); + (events.get(f).copied().ok_or_else(|| bad("the strand has no event line"))?, f) } + Window::SinceCheckpoint => { let c = (1..lines.len()).rev().find(|&i| kinds[i] == K::Checkpoint).ok_or_else(|| bad("the strand has no checkpoint line"))?; + let f = events.iter().position(|&i| i > c).ok_or_else(|| bad("no event line after the last checkpoint"))?; (c, f) } + }; + let win: &[usize] = &events[first_ev..]; + // the expectation's memory before the window's first event: from the last checkpoint before it that carries the memory (a + // canonical strand: its mean and variance are the fold's, at 6 decimals), else folded from the header + let ex = doc.get("expectation"); + let d = ex.map(|e| 0.5f64.powf(1.0 / e.get("half_life_events").and_then(Json::as_f64).unwrap_or(32.0))); + let (mut mean, mut var) = (0.0f64, PRIOR_VAR); + let mut fold_from = 1; + if ex.is_some() && canon { + if let Some(c) = (1..win[0]).rev().find(|&i| kinds[i] == K::Checkpoint && lines[i].contains(",\"expect\":")) { + let j = json::parse(lines[c]).map_err(|e| bad(format!("line {}: {}", c + 1, e.msg)))?; + let m = j.get("expect").ok_or_else(|| bad(format!("line {}: the checkpoint has no expect", c + 1)))?; + mean = m.get("mean").and_then(Json::as_f64).unwrap_or(0.0); var = m.get("var").and_then(Json::as_f64).unwrap_or(PRIOR_VAR); fold_from = c + 1; + } + } + let (traits_decl, moods_decl) = (declared(&doc, "traits"), declared(&doc, "moods")); + let goals: Vec = doc.get("drives").and_then(|d| d.get("goals")).and_then(Json::as_arr).map_or(vec![], |v| v.iter().filter_map(|g| g.get("id").and_then(Json::as_str).map(String::from)).collect()); + let from: Vec = doc.get("learning").and_then(|l| l.get("from")).and_then(Json::as_arr).map_or(vec![], |v| v.iter().filter_map(|x| x.as_str().map(String::from)).collect()); + let mut trs: Vec = traits_decl.iter().map(|(id, lv)| Tr { id: id.clone(), levels: lv.clone(), taken: vec![], first: None, last: None, pmin: f64::INFINITY, pmax: f64::NEG_INFINITY }).collect(); + if !goals.is_empty() { trs.push(Tr { id: "pursue".into(), levels: goals.clone(), taken: vec![], first: None, last: None, pmin: f64::INFINITY, pmax: f64::NEG_INFINITY }); } + let mut mds: Vec = moods_decl.iter().map(|(id, lv)| Md { id: id.clone(), levels: lv.clone(), path: vec![] }).collect(); + let (mut inputs, mut habits, mut peaks): (Vec<(String, usize)>, Vec<(String, usize)>, Vec<(String, f64, u64)>) = (vec![], vec![], vec![]); + let (mut scored, mut registered, mut top): (usize, usize, Vec<(u64, f64, Json)>) = (0, 0, vec![]); + let (mut rewards, mut corrections, mut cancelled) = (0usize, 0usize, 0usize); + let (mut samples, mut drive_samples, mut learned): (usize, usize, Vec) = (0, 0, vec![]); + let mut ctl: Vec<(String, usize)> = ["erase", "pause", "resume", "retire"].iter().map(|k| (k.to_string(), 0)).collect(); + let (mut controls, mut rests, mut reflections) = (0usize, 0usize, 0usize); + // RFC 0015: the rest passes in the window (lines, contexts merged, weights moved) + let (mut cons, mut cons_merged, mut cons_moved) = (0usize, 0usize, 0.0f64); + let (mut first_n, mut last_n) = (0u64, 0u64); + // RFC 0016: with a curiosity block that declares a stir, the self-started events in the window (count, classes) + let stir_on = doc.get("curiosity").and_then(|c| c.get("stir")).is_some_and(|s| !s.is_null()); + let mut stirs: Vec<(String, usize)> = vec![]; + for i in fold_from.min(start)..lines.len() { + let in_win = i >= start; + match kinds[i] { + K::Event => { + let need = in_win || ex.is_some(); + if !need { continue; } + let j = json::parse(lines[i]).map_err(|e| bad(format!("line {}: {}", i + 1, e.msg)))?; + let n = j.get("n").and_then(Json::as_f64).unwrap_or(0.0) as u64; + let dv = j.get("deviation").filter(|x| !x.is_null()); + let cold = dv.is_some_and(|x| x.get("cold") == Some(&Json::Bool(true))); + let pred = dv.and_then(|x| x.get("pred")).and_then(Json::as_f64); + if in_win { + if first_n == 0 { first_n = n; } last_n = n; + let set = set_keys(j.get("inputs").unwrap_or(&Json::Null)); + for k in &set { bump(&mut inputs, k); } + if stir_on { if let Some(c) = j.get("inputs").and_then(|x| x.get("stir")).and_then(Json::as_str) { bump(&mut stirs, c); } } + let on = |k: Option<&String>| k.is_some_and(|k| set.contains(k)); + match (on(from.first()), on(from.get(1))) { (true, true) => cancelled += 1, (true, false) => rewards += 1, (false, true) => corrections += 1, _ => {} } + if let (Some(dv), Some(p), false, true) = (dv, pred, cold, ex.is_some()) { + let z = r6(((1.0 + p).ln() - mean) / (var + VAR_FLOOR).sqrt()); + scored += 1; if z >= DEV_Z { registered += 1; } + let f = |k: &str| n6(dv.get(k).and_then(Json::as_f64).unwrap_or(0.0)); + let mut set = set.clone(); set.sort(); + let e = Json::Obj(vec![("glyph".into(), j.get("glyph").cloned().unwrap_or(Json::Null)), ("input".into(), f("input")), ("n".into(), Json::Num(n as f64)), + ("pred".into(), f("pred")), ("self".into(), f("self")), ("set".into(), Json::Arr(set.into_iter().map(Json::Str).collect())), ("z".into(), Json::Num(z))]); + top.push((n, z, e)); + top.sort_by(|a, b| b.1.abs().partial_cmp(&a.1.abs()).unwrap_or(core::cmp::Ordering::Equal).then(a.0.cmp(&b.0))); top.truncate(TOP_K); + } + } + // the fold (persona::expect_step's mean and variance): a receipt that is not cold moves them + if let (Some(d), Some(p), false) = (d, pred, cold) { + let (x, a) = ((1.0 + p).ln(), 1.0 - d); let dx = x - mean; + mean += a * dx; var = (1.0 - a) * (var + a * dx * dx); + if canon { mean = r6(mean); var = r6(var); } + } + } + K::Checkpoint if in_win => { + let j = json::parse(lines[i]).map_err(|e| bad(format!("line {}: {}", i + 1, e.msg)))?; + let n = j.get("checkpoint").and_then(Json::as_f64).unwrap_or(0.0) as u64; + let Some(st) = j.get("stance") else { continue }; + samples += 1; + for t in trs.iter_mut() { + let x = if t.id == "pursue" { st.get("pursue") } else { st.get("stance").and_then(|s| s.get(&t.id)) }; + let Some(x) = x else { continue }; + let lv = if t.id == "pursue" { x.get("goal") } else { x.get("level") }; + if let Some(l) = lv.and_then(Json::as_str) { bump(&mut t.taken, l); } + if let Some(o) = x.get("odds") { let o = odds_obj(&t.levels, o); if t.first.is_none() { t.first = Some(o.clone()); } t.last = Some(o); } + if let Some(p) = x.get("p").and_then(Json::as_f64) { t.pmin = t.pmin.min(p); t.pmax = t.pmax.max(p); } + } + for m in mds.iter_mut() { if let Some(o) = st.get("mood").and_then(|s| s.get(&m.id)).and_then(|x| x.get("odds")) { m.path.push((n, r6(position(&m.levels, o)))); } } + if let Some(Json::Obj(want)) = st.get("drives").and_then(|d| d.get("want")) { drive_samples += 1; + for (g, v) in want { let v = v.as_f64().unwrap_or(0.0); + match peaks.iter_mut().find(|x| &x.0 == g) { Some(x) => if v > x.1 { x.1 = v; x.2 = n; }, None => peaks.push((g.clone(), v, n)) } } } + if let Some(b) = st.get("habits").and_then(|h| h.get("bound")).and_then(Json::as_arr) { for h in b.iter().filter_map(Json::as_str) { bump(&mut habits, h); } } + if let Some(l) = j.get("state").and_then(|s| s.get("learned")) { learned.push(l.clone()); } + } + K::Control if in_win => { + let j = json::parse(lines[i]).map_err(|e| bad(format!("line {}: {}", i + 1, e.msg)))?; + controls += 1; if let Some(c) = j.get("control").and_then(Json::as_str) { bump(&mut ctl, c); } + } + K::Rest if in_win => rests += 1, + K::Reflection if in_win => reflections += 1, + K::Consolidation if in_win => { let j = json::parse(lines[i]).map_err(|e| bad(format!("line {}: {}", i + 1, e.msg)))?; + let c = j.get("consolidation"); cons += 1; + cons_merged += c.and_then(|c| c.get("merged")).and_then(|m| m.get("count")).and_then(Json::as_f64).unwrap_or(0.0) as usize; + cons_moved = r6(cons_moved + c.and_then(|c| c.get("moved")).and_then(Json::as_f64).unwrap_or(0.0)); } + _ => {} + } + } + // the document + let none = |b: bool, j: Json| if b { j } else { Json::Null }; + let sampled = samples > 0; + let traits = Json::Obj(trs.iter().filter(|t| !t.taken.is_empty() || t.first.is_some()).map(|t| (t.id.clone(), Json::Obj(vec![ + ("levels".into(), obj_counts(&t.taken)), ("odds_first".into(), t.first.clone().unwrap_or(Json::Null)), ("odds_last".into(), t.last.clone().unwrap_or(Json::Null)), + ("p_max".into(), if t.pmax.is_finite() { n6(t.pmax) } else { Json::Null }), ("p_min".into(), if t.pmin.is_finite() { n6(t.pmin) } else { Json::Null })]))).collect()); + let mood = |m: &Md| -> Json { + let xs: Vec = m.path.iter().map(|p| p.1).collect(); + // the turning points: the samples where the first difference changes sign (a zero difference keeps the last sign) + let (mut turns, mut sign, mut at) = (0usize, 0i8, vec![]); + for k in 1..xs.len() { let dx = xs[k] - xs[k - 1]; if dx == 0.0 { continue; } let s = if dx > 0.0 { 1 } else { -1 }; + if sign != 0 && s != sign { turns += 1; if at.len() < TURNS_MAX { at.push(Json::Num(m.path[k - 1].0 as f64)); } } sign = s; } + let p0 = m.path.len().saturating_sub(PATH_MAX); + Json::Obj(vec![("end".into(), n6(*xs.last().unwrap_or(&0.0))), ("max".into(), n6(xs.iter().cloned().fold(f64::NEG_INFINITY, f64::max))), + ("min".into(), n6(xs.iter().cloned().fold(f64::INFINITY, f64::min))), ("path".into(), Json::Arr(m.path[p0..].iter().map(|(n, v)| Json::Arr(vec![Json::Num(*n as f64), n6(*v)])).collect())), + ("start".into(), n6(xs[0])), ("turning_points".into(), Json::Arr(at)), ("turns".into(), cnt(turns))]) + }; + let moods = Json::Obj(mds.iter().filter(|m| !m.path.is_empty()).map(|m| (m.id.clone(), mood(m))).collect()); + let drives = Json::Obj(peaks.iter().map(|(g, v, n)| (g.clone(), Json::Obj(vec![("at".into(), Json::Num(*n as f64)), ("peak".into(), n6(*v))]))).collect()); + let net = |a: &Json, b: &Json| -> Option { Some(Json::Obj(b.as_obj()?.iter().map(|(t, v)| (t.clone(), Json::Arr(v.as_arr().unwrap_or(&[]).iter().enumerate() + .map(|(k, x)| n6(x.as_f64().unwrap_or(0.0) - a.get(t).and_then(Json::as_arr).and_then(|w| w.get(k)).and_then(Json::as_f64).unwrap_or(0.0))).collect()))).collect())) }; + let netj = if learned.len() >= 2 { net(&learned[0], &learned[learned.len() - 1]) } else { None }; + let moved = netj.as_ref().and_then(Json::as_obj).map_or(0.0, |kv| kv.iter().flat_map(|(_, v)| v.as_arr().unwrap_or(&[]).iter().filter_map(Json::as_f64)).map(|x| x.abs()).sum::()); + let inputs_sorted = { let mut v = inputs.clone(); v.sort_by(|a, b| a.0.cmp(&b.0)); v }; + let events_n = win.len(); + let segment = head.get("segment").and_then(|s| s.get("index")).cloned().unwrap_or(Json::Null); + let has_learning = doc.get("learning").is_some(); + let tops: Vec = top.iter().map(|t| t.2.clone()).collect(); + let window = Json::Obj(vec![("checkpoints".into(), cnt(samples)), ("engine".into(), Json::Str(engine.clone())), ("events".into(), cnt(events_n)), + ("first".into(), Json::Num(first_n as f64)), ("last".into(), Json::Num(last_n as f64)), ("last_n".into(), match w { Window::Last(k) => Json::Num(k as f64), _ => Json::Null }), + ("lines".into(), cnt(lines.len() - start)), ("since_checkpoint".into(), Json::Bool(w == Window::SinceCheckpoint))]); + let mut kv: Vec<(String, Json)> = vec![ + ("controls".into(), Json::Obj(vec![("erase".into(), cnt(ctl[0].1)), ("pause".into(), cnt(ctl[1].1)), ("reflections".into(), cnt(reflections)), ("rests".into(), cnt(rests)), + ("resume".into(), cnt(ctl[2].1)), ("retire".into(), cnt(ctl[3].1)), ("segment".into(), segment), ("zero_credit".into(), cnt(controls))])), + ("credit".into(), none(has_learning, Json::Obj(vec![("cancelled".into(), cnt(cancelled)), ("corrections".into(), cnt(corrections)), ("rewards".into(), cnt(rewards)), ("zeroed".into(), cnt(controls))]))), + ("drives".into(), none(drive_samples > 0, drives)), + ("habits".into(), none(sampled, Json::Obj(vec![("bound".into(), obj_counts(&{ let mut v = habits.clone(); v.sort_by(|a, b| a.0.cmp(&b.0)); v })), ("samples".into(), cnt(samples))]))), + ("inputs".into(), obj_counts(&inputs_sorted)), + ("learned".into(), none(has_learning && !learned.is_empty(), Json::Obj(vec![("moved".into(), n6(moved)), ("net".into(), netj.clone().unwrap_or(Json::Null)), ("samples".into(), cnt(learned.len()))]))), + ("moods".into(), none(sampled && !mds.iter().all(|m| m.path.is_empty()), moods)), + ("probbit_reflection".into(), Json::Num(FORMAT)), + ("surprise".into(), none(ex.is_some(), Json::Obj(vec![("registered".into(), cnt(registered)), ("scored".into(), cnt(scored)), ("threshold".into(), n6(DEV_Z)), ("top".into(), Json::Arr(tops))]))), + ("traits".into(), none(sampled, traits)), + ("window".into(), window), + ]; + // RFC 0015: `consolidation` only when the window holds a consolidation line (every other reflection document keeps its bytes) + if cons > 0 { let at = kv.iter().position(|(k, _)| k.as_str() > "consolidation").unwrap_or(kv.len()); + kv.insert(at, ("consolidation".into(), Json::Obj(vec![("lines".into(), cnt(cons)), ("merged".into(), cnt(cons_merged)), ("moved".into(), n6(cons_moved))]))); } + // RFC 0016: `stirs` only with a declared stir (every other reflection document keeps its bytes) + if stir_on { let at = kv.iter().position(|(k, _)| k.as_str() > "stirs").unwrap_or(kv.len()); + kv.insert(at, ("stirs".into(), Json::Obj({ let mut v = stirs.clone(); v.sort_by(|a, b| a.0.cmp(&b.0)); v.into_iter().map(|(k, c)| (k, cnt(c))).collect() }))); } + let doc0 = Json::Obj(kv.clone()); + let summary = sentences(&doc0, &trs, &mds); + kv.push(("summary".into(), Json::Arr(summary.into_iter().map(Json::Str).collect()))); + let digest = persona::digest_of(&persona::canon(&Json::Obj(kv.clone()))); + kv.push(("digest".into(), Json::Str(digest.clone()))); + Ok(Reflection { doc: Json::Obj(kv), digest }) +} + +/// The summary: 6 to 12 sentences, each from a fixed template filled with the document's numbers (mechanism language only) +fn sentences(d: &Json, trs: &[Tr], mds: &[Md]) -> Vec { + let g = |p: &[&str]| -> Option<&Json> { let mut x = d; for k in p { x = x.get(k)?; } Some(x) }; + let u = |p: &[&str]| g(p).and_then(Json::as_f64).unwrap_or(0.0) as u64; + let mut s = vec![]; + s.push(format!("This account covers events {} to {}: {} and {} read, {} of them checkpoint samples.", u(&["window", "first"]), u(&["window", "last"]), + pl(u(&["window", "events"]), "event", "events"), pl(u(&["window", "lines"]), "line", "lines"), u(&["window", "checkpoints"]))); + let ins: Vec<(String, usize)> = g(&["inputs"]).and_then(Json::as_obj).map_or(vec![], |kv| kv.iter().map(|(k, v)| (k.clone(), v.as_f64().unwrap_or(0.0) as usize)).collect()); + let ins = by_count(&ins); + s.push(if ins.is_empty() { "No input was set in this window.".to_string() } else { + let parts: Vec = ins.iter().take(3).map(|(k, c)| format!("{k} ({})", pl(*c as u64, "event", "events"))).collect(); + format!("The inputs set most often were {}.", join(&parts)) }); + match g(&["surprise"]).filter(|x| !x.is_null()) { + None => s.push("This persona has no expectation block, so no surprise was scored.".into()), + Some(sp) => { + s.push(format!("The expectation scored {} and {} of them registered as surprising (z at least {}).", pl(u(&["surprise", "scored"]), "event", "events"), u(&["surprise", "registered"]), f6(DEV_Z))); + if let Some(t) = sp.get("top").and_then(Json::as_arr).and_then(|v| v.first()) { let f = |k: &str| f6(t.get(k).and_then(Json::as_f64).unwrap_or(0.0)); + s.push(format!("The largest surprise was event {}: z {}, the expectation predicted {} and the event measured {}.", t.get("n").and_then(Json::as_f64).unwrap_or(0.0) as u64, f("z"), f("pred"), f("self"))); } + } + } + if u(&["window", "checkpoints"]) == 0 { + s.push("No checkpoint line falls in this window, so the levels, moods, drives and habits are not reported.".into()); + } else { + let modal: Vec = trs.iter().filter(|t| t.id != "pursue").filter_map(|t| by_count(&t.taken).first().map(|(l, c)| format!("{} {} ({})", t.id, l, c))).take(3).collect(); + if !modal.is_empty() { s.push(format!("Across the checkpoint samples the levels taken most were {}.", join(&modal))); } + // the moods named valence and pressure first, then the others, at most two sentences + let mut ms: Vec<&Md> = mds.iter().filter(|m| !m.path.is_empty()).collect(); + ms.sort_by_key(|m| match m.id.as_str() { "valence" => 0, "pressure" => 1, _ => 2 }); + for m in ms.into_iter().take(2) { let f = |k: &str| f6(g(&["moods", m.id.as_str(), k]).and_then(Json::as_f64).unwrap_or(0.0)); + s.push(format!("The mood {} moved from {} to {} on a scale of 0 to 1 (min {}, max {}, {}).", m.id, f("start"), f("end"), f("min"), f("max"), pl(u(&["moods", m.id.as_str(), "turns"]), "turning point", "turning points"))); } + let hb: Vec<(String, usize)> = g(&["habits", "bound"]).and_then(Json::as_obj).map_or(vec![], |kv| kv.iter().map(|(k, v)| (k.clone(), v.as_f64().unwrap_or(0.0) as usize)).collect()); + s.push(match by_count(&hb).first() { None => "No habit bound in the checkpoint samples.".into(), + Some((h, c)) => format!("The habit {h} bound in {c} of {}, and {} bound in all.", pl(u(&["window", "checkpoints"]), "sample", "samples"), pl(hb.len() as u64, "habit", "habits")) }); + if let Some(Json::Obj(dv)) = g(&["drives"]) { let mut v: Vec<(&String, f64, u64)> = dv.iter().map(|(k, x)| (k, x.get("peak").and_then(Json::as_f64).unwrap_or(0.0), x.get("at").and_then(Json::as_f64).unwrap_or(0.0) as u64)).collect(); + v.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(core::cmp::Ordering::Equal).then(a.0.cmp(b.0))); + match v.first() { Some((k, x, n)) if *x > 0.0 => s.push(format!("The strongest want was {k} at {} (event {n}).", f6(*x))), + Some(_) => s.push("No drive's want rose above 0 in the checkpoint samples.".into()), None => {} } } + } + match g(&["credit"]).filter(|x| !x.is_null()) { + None => s.push("This persona has no learning block, so no credit was settled.".into()), + Some(_) => { + s.push(format!("Feedback settled the credit {} as reward and {} as correction ({} cancelled, {} zeroed by control lines).", pl(u(&["credit", "rewards"]), "time", "times"), + pl(u(&["credit", "corrections"]), "time", "times"), u(&["credit", "cancelled"]), u(&["credit", "zeroed"]))); + if let Some(l) = g(&["learned"]).filter(|x| !x.is_null()) { if l.get("net").is_some_and(|n| !n.is_null()) { + s.push(format!("The learned weights moved by {} in total between the first and the last of {}.", f6(l.get("moved").and_then(Json::as_f64).unwrap_or(0.0)), pl(u(&["learned", "samples"]), "sample", "samples"))); } } + } + } + // RFC 0016: one sentence on the self-started events, only with a declared stir (so 13 sentences at most then) + if let Some(Json::Obj(st)) = g(&["stirs"]) { let n: u64 = st.iter().map(|(_, c)| c.as_f64().unwrap_or(0.0) as u64).sum(); + let parts: Vec = st.iter().map(|(k, c)| format!("{k} ({})", c.as_f64().unwrap_or(0.0) as u64)).collect(); + s.push(if n == 0 { "The curiosity drive started no self-started event (stir) in this window.".into() } + else { format!("The curiosity drive started {} in this window, by class: {}.", pl(n, "self-started event (stir)", "self-started events (stirs)"), join(&parts)) }); } + // RFC 0015: one sentence on the rest passes, only when the window holds one + if let Some(cj) = g(&["consolidation"]) { let k = |x: &str| cj.get(x).and_then(Json::as_f64).unwrap_or(0.0); + s.push(format!("The rest pass ran {} in this window: {} merged, the remembered weights moved by {} in total.", pl(k("lines") as u64, "time", "times"), + pl(k("merged") as u64, "context", "contexts"), f6(k("moved")))); } + let c = |k: &str| u(&["controls", k]); + let rests = |n: u64| if n == 1 { "1 rest line was written".to_string() } else { format!("{n} rest lines were written") }; + s.push(if c("zero_credit") == 0 { format!("No control line was applied, and {}.", rests(c("rests"))) } + else { format!("Control lines applied: {} pause, {} resume, {} retire and {} erase; {}.", c("pause"), c("resume"), c("retire"), c("erase"), rests(c("rests"))) }); + s +} +/// `n` with the word for one or for many +fn pl(n: u64, one: &str, many: &str) -> String { format!("{n} {}", if n == 1 { one } else { many }) } +fn join(p: &[String]) -> String { + match p.len() { 0 => String::new(), 1 => p[0].clone(), n => format!("{} and {}", p[..n - 1].join(", "), p[n - 1]) } +} + +/// The reflection line after the line whose sha256 is `prev`: canonical JSON {prev, reflection} +pub fn line(prev: &str, r: &Reflection) -> String { persona::canon(&Json::Obj(vec![("prev".into(), Json::Str(prev.into())), ("reflection".into(), r.doc.clone())])) } +/// A reflection line's check (`live verify`): the document it carries is the reflection of the lines before it over the window +/// it names, byte for byte -> Ok, or what differs +pub fn check_line(before: &[&str], l: &str) -> Result<(), String> { + let j = json::parse(l).map_err(|e| format!("not JSON: {}", e.msg))?; + let doc = j.get("reflection").ok_or("no reflection")?; + let w = Window::of(doc).ok_or("the reflection names no window (window.last_n, window.since_checkpoint)")?; + let r = reflect_window(before, w).map_err(|e| format!("the reflection cannot be recomputed: {}", e.msg))?; + if persona::canon(doc) != r.text() { return Err(format!("the reflection differs from the lines before it (recomputed digest {})", r.digest)); } + Ok(()) +} + +#[cfg(all(test, feature = "std"))] +mod tests { + use super::*; + /// The words the summary never contains (the project's language law), written reversed so the source does not carry them + const FORBIDDEN_REVERSED: [&str; 4] = ["luos", "sleef", "evila", "suoicsnoc"]; + fn fixture(name: &str) -> String { std::fs::read_to_string(format!("{}/../probbit-cli/tests/fixtures/{name}", env!("CARGO_MANIFEST_DIR"))).unwrap() } + #[test] + fn deterministic_canonical_and_lawful() { + for f in ["monitor/tutor-expect.strand", "monitor/tutor-week.strand", "strands/expectation-0.13.0.strand", "strands/sensed-credit-0.9.0-dev.strand"] { + let t = fixture(f); let ls = crate::strand::lines(&t); + for w in [Window::All, Window::Last(5), Window::Last(10_000)] { + let a = reflect_window(&ls, w).unwrap(); let b = reflect_window(&ls, w).unwrap(); + assert_eq!(a.text(), b.text(), "{f}"); assert_eq!(persona::canon(&json::parse(&a.text()).unwrap()), a.text()); + let mut no = a.doc.clone(); if let Json::Obj(v) = &mut no { v.retain(|(k, _)| k != "digest"); } + assert_eq!(persona::digest_of(&persona::canon(&no)), a.digest); + let s = a.summary(); assert!((6..=12).contains(&s.len()), "{f}: {} sentences", s.len()); + for x in &s { let low = x.to_lowercase(); for w in FORBIDDEN_REVERSED { let w: String = w.chars().rev().collect(); assert!(!low.contains(&w), "{f}: {x}"); } } + } + } + } + /// The z the fold recomputes is the z each checkpoint's stance carries (the expectation's own, at that event) + #[test] + fn z_matches_the_checkpoints() { + let t = fixture("monitor/tutor-expect.strand"); let ls = crate::strand::lines(&t); + let r = reflect_window(&ls, Window::All).unwrap(); + let mut zs: Vec<(u64, f64)> = vec![]; + for l in &ls { if l.starts_with("{\"checkpoint\":") { let j = json::parse(l).unwrap(); + if let Some(z) = j.get("stance").and_then(|s| s.get("deviation")).and_then(|d| d.get("z")).and_then(Json::as_f64) { zs.push((j.get("checkpoint").unwrap().as_f64().unwrap() as u64, z)); } } } + assert!(!zs.is_empty()); + for (n, z) in zs { let r1 = reflect_window(&ls[..ls.iter().position(|l| l.starts_with(&format!("{{\"checkpoint\":{n},"))).unwrap()], Window::Last(1)).unwrap(); + let got = r1.doc.get("surprise").and_then(|s| s.get("top")).and_then(Json::as_arr).and_then(|v| v.first()).and_then(|e| e.get("z")).and_then(Json::as_f64); + assert_eq!(got, Some(z), "event {n}"); } + assert!(r.doc.get("surprise").is_some()); + } + #[test] + fn line_checks_and_refuses_a_forgery() { + let t = fixture("monitor/tutor-expect.strand"); let ls = crate::strand::lines(&t); + let r = reflect(&ls, Some(10)).unwrap(); let l = line(&persona::digest_of(ls[ls.len() - 1]), &r); + assert!(is_reflection_line(&l)); assert!(check_line(&ls, &l).is_ok()); + let forged = l.replacen("This account covers", "This account spans", 1); + assert!(check_line(&ls, &forged).unwrap_err().contains("differs")); + let moved = l.replacen("\"last_n\":10", "\"last_n\":9", 1); + assert!(check_line(&ls, &moved).is_err()); + } +} diff --git a/probbit-persona/src/rest.rs b/probbit-persona/src/rest.rs new file mode 100644 index 0000000..0560470 --- /dev/null +++ b/probbit-persona/src/rest.rs @@ -0,0 +1,253 @@ +//! RFC 0015 the rest pass (memory consolidation): at rest, when a host decides (standby, night, idle), the expectation's memory +//! (RFC 0009) is consolidated from what the events since the last pass left in it, and one `consolidation` line logs what moved. +//! Mechanism only, and exact: everything the pass reads is a fold of the logged lines inside the expectation's canonical memory +//! (engine 0.14.0 on), so a checkpoint carries it and `live verify` re-runs the pass from the lines before the line. +//! +//! The window (per remembered context, since the last pass): `a` its visits decayed by the expectation's half-life (the part of +//! its count the window added), `b` the same visits decayed by the rest pass's `half_life`, `k` the visits, `zs` the sum of the +//! visits' one-sided surprise z+ = r6(max(z, 0)) (z as the deviation computed it: the logged prediction against the learned +//! normal before the event; a cold event adds 0). The pass, in order: +//! 1. re-fold: each count becomes r6(max(0, count - a + b)) (the window re-weighted with the rest half-life; the default +//! half-life is the expectation's own, and the count stays); +//! 2. surprise: with `surprise_gain` g > 0, each count is weighed by f = 1 + g min(zs/k, 1.5)/1.5 (contexts that kept +//! surprising keep more weight) and the counts scaled back to their sum before (r6 each); +//! 3. merge: with `bucket` B < 3, the contexts whose key is the same with each declared number input in B bins (instead of the +//! waking key's thirds) become one: the largest count's inputs (ties: key order), the counts summed; +//! 4. the learned deltas times `decay` (r6 each; 1 leaves the state as it is). +//! +//! The curiosity fold (RFC 0016) and the learned normal are left as they are. The line: `{at, consolidation {after, before, +//! decay, events, merged {count, keys}, moved}, n, prev, state}` with the memory's digests before and after (sha256 of its +//! checkpoint form), the window's event count, the merged-away keys, sum |count after - count before| over every key, and the +//! state digest after. +use crate::json::Json; +use crate::persona::{self, canon, digest_of, r6, ExpectMem, Kind as InKind, Persona, State, DEV_Z}; +#[allow(unused_imports)] +use std::prelude::rust_2021::*; + +/// The window of the rest pass inside the expectation's memory: events since the last pass, per context [a, b, k, zs] +#[derive(Clone, Debug, Default, PartialEq)] +pub struct RestWin { pub since: u64, pub w: Vec<(String, [f64; 4])> } +impl RestWin { + /// The window as the memory's `window` key: {combos: per remembered combo in key order [a, b, k, zs], since} + pub(crate) fn to_json(&self, keys: &[&str]) -> Json { + let row = |k: &str| self.w.iter().find(|e| e.0 == k).map_or([0.0; 4], |e| e.1); + Json::Obj(vec![("combos".into(), Json::Arr(keys.iter().map(|k| Json::Arr(row(k).iter().map(|x| Json::Num(*x)).collect())).collect())), + ("since".into(), Json::Num(self.since as f64))]) + } + /// A checkpoint's `window` for the memory's combos (`keys`, key order) -> the window; Err(what is wrong) + pub(crate) fn read(j: &Json, keys: &[String]) -> Result { + let kv = j.as_obj().filter(|kv| kv.len() == 2).ok_or("window: an object {combos, since}")?; + let f = |k: &str| kv.iter().find(|(x, _)| x == k).map(|(_, v)| v).ok_or(format!("window: no {k}")); + let since = f("since")?.as_f64().filter(|x| *x >= 0.0 && x.fract() == 0.0 && *x <= 9_007_199_254_740_992.0).ok_or("window.since: a whole number >= 0")? as u64; + let rows = f("combos")?.as_arr().filter(|r| r.len() == keys.len()).ok_or("window.combos: one row per combo")?; + let mut w = vec![]; + for (k, r) in keys.iter().zip(rows) { + let x: Vec = r.as_arr().filter(|r| r.len() == 4).map(|r| r.iter().filter_map(Json::as_f64).filter(|x| x.is_finite() && *x >= 0.0 && r6(*x) == *x).collect()).unwrap_or_default(); + if x.len() != 4 { return Err("window.combos: each [a, b, k, zs], numbers >= 0 at 6 decimals".into()); } + if x.iter().any(|v| *v != 0.0) { w.push((k.clone(), [x[0], x[1], x[2], x[3]])); } } + Ok(RestWin { since, w }) + } + /// One event folded in (after the memory's own step): every row decays (`d` the expectation's factor, `db` the pass's), the + /// event's context gains a visit and its z+ + pub(crate) fn step(&mut self, key: &str, d: f64, db: f64, zp: f64) { + for e in &mut self.w { e.1[0] *= d; e.1[1] *= db; } + match self.w.iter_mut().find(|e| e.0 == key) { Some(e) => { e.1[0] += 1.0; e.1[1] += 1.0; e.1[2] += 1.0; e.1[3] += zp; } None => self.w.push((key.to_string(), [1.0, 1.0, 1.0, zp])) } + self.since += 1; + } + /// The canonical rounding and the memory's prune: rows at 6 decimals, a context the memory dropped is dropped here + pub(crate) fn settle(&mut self, combos: &[(String, f64, Json)]) { + for e in &mut self.w { for x in e.1.iter_mut() { *x = r6(*x); } } + self.w.retain(|e| combos.iter().any(|c| c.0 == e.0)); + } +} +/// The merge key: the waking key (`combo_key`) with each declared number input in `bins` bins instead of thirds +fn merge_key(p: &Persona, raw: &[(String, Json)], bins: u8) -> String { + if bins >= 3 { return persona::combo_key(p, raw); } + let mut kv: Vec<(String, Json)> = raw.iter().filter(|(k, v)| k != "elapsed_hours" && !matches!(v, Json::Null | Json::Bool(false)) && !matches!(v, Json::Num(x) if *x == 0.0)) + .map(|(k, v)| (k.clone(), match (p.input(k), v) { + (Some(i), Json::Num(x)) if i.kind == InKind::Number && i.max > i.min => Json::Str(format!("bin{}of{bins}", (((x - i.min) / (i.max - i.min)) * f64::from(bins)).floor().clamp(0.0, f64::from(bins - 1)) as i64)), + _ => v.clone() })).collect(); + kv.sort_by(|a, b| a.0.cmp(&b.0)); canon(&Json::Obj(kv)) +} +/// The pass on the memory alone -> the line's `consolidation` object; Err without a rest block or before the canonical fold. +/// Leaves the window empty (the next pass reads the events after this one). +#[inline(never)] +pub fn consolidate_mem(p: &Persona, mem: &mut ExpectMem) -> Result { + let ex = p.expect.as_ref().ok_or("the persona has no expectation block")?; + let rp = ex.rest.as_ref().ok_or("the persona declares no expectation.rest block (the rest pass is off)")?; + if !mem.canon { return Err("a rest pass needs a strand of engine 0.14.0 or later".into()); } + let before = digest_of(&canon(&mem.to_json())); + let b0: Vec<(String, f64)> = mem.combos.iter().map(|c| (c.0.clone(), c.1)).collect(); + let win = mem.win.take().unwrap_or_default(); + // 1. the window re-folded with the rest half-life + for c in &mut mem.combos { if let Some(e) = win.w.iter().find(|e| e.0 == c.0) { c.1 = r6((c.1 - e.1[0] + e.1[1]).max(0.0)); } } + // 2. contexts that kept surprising keep more weight (the total kept) + if rp.surprise_gain > 0.0 { + let f: Vec = mem.combos.iter().map(|c| win.w.iter().find(|e| e.0 == c.0).filter(|e| e.1[2] > 0.0) + .map_or(1.0, |e| 1.0 + rp.surprise_gain * (e.1[3] / e.1[2]).min(DEV_Z) / DEV_Z)).collect(); + let t: f64 = mem.combos.iter().map(|c| c.1).sum(); let s: f64 = mem.combos.iter().zip(&f).map(|(c, x)| c.1 * x).sum(); + if s > 0.0 { for (c, x) in mem.combos.iter_mut().zip(&f) { c.1 = r6(c.1 * x * t / s); } } + } + // 3. near-duplicate contexts merged (the same key with the numbers in `bucket` bins) + let mut merged: Vec = vec![]; + if rp.bucket < 3 { + let mk: Vec = mem.combos.iter().map(|c| merge_key(p, c.2.as_obj().unwrap_or(&[]), rp.bucket)).collect(); + let mut keep = vec![true; mem.combos.len()]; + for i in 0..mem.combos.len() { + if !keep[i] { continue; } + let g: Vec = (i..mem.combos.len()).filter(|&j| keep[j] && mk[j] == mk[i]).collect(); + if g.len() < 2 { continue; } + // the group's head: the largest count (ties: key order) + let h = *g.iter().min_by(|&&x, &&y| mem.combos[y].1.partial_cmp(&mem.combos[x].1).unwrap_or(core::cmp::Ordering::Equal).then(mem.combos[x].0.cmp(&mem.combos[y].0))).unwrap_or(&i); + let sum = g.iter().fold(0.0, |a, &j| a + mem.combos[j].1); + for &j in &g { if j != h { keep[j] = false; merged.push(mem.combos[j].0.clone()); } } + mem.combos[h].1 = r6(sum); + } + let mut it = keep.iter(); mem.combos.retain(|_| *it.next().unwrap_or(&true)); + } + mem.combos.retain(|c| c.1 > 0.0); + mem.win = Some(RestWin::default()); + merged.sort(); + let after = digest_of(&canon(&mem.to_json())); + let mut moved = 0.0; + for (k, x) in &b0 { moved += (mem.combos.iter().find(|c| &c.0 == k).map_or(0.0, |c| c.1) - x).abs(); } + for c in &mem.combos { if !b0.iter().any(|b| b.0 == c.0) { moved += c.1; } } + let s = |x: &str| Json::Str(x.to_string()); + Ok(Json::Obj(vec![("after".into(), s(&after)), ("before".into(), s(&before)), ("decay".into(), Json::Num(rp.decay)), ("events".into(), Json::Num(win.since as f64)), + ("merged".into(), Json::Obj(vec![("count".into(), Json::Num(merged.len() as f64)), ("keys".into(), Json::Arr(merged.into_iter().map(Json::Str).collect()))])), + ("moved".into(), Json::Num(r6(moved)))])) +} +/// Step 4: the learned deltas times the pass's `decay` (r6 each), in place (no state by value on a frame); 1 leaves the state +#[inline(never)] +pub fn decay_learned(p: &Persona, st: &mut State) { + let Some(rp) = p.expect.as_ref().and_then(|e| e.rest.as_ref()) else { return }; + if rp.decay >= 1.0 { return; } + for (_, d) in st.learned.iter_mut() { for x in d.iter_mut() { let y = r6(rp.decay * *x); *x = if y == 0.0 { 0.0 } else { y }; } } + st.seal(p); +} +/// The `consolidation` line after the line whose sha256 is `prev` (canonical JSON: at, consolidation, n, prev, state) +pub fn line(prev: &str, at: &str, n: u64, c: Json, state: &str) -> String { + let s = |x: &str| Json::Str(x.to_string()); + canon(&Json::Obj(vec![("at".into(), s(at)), ("consolidation".into(), c), ("n".into(), Json::Num(n as f64)), ("prev".into(), s(prev)), ("state".into(), s(state))])) +} +/// The first field (dotted path) where the line `got` differs from the line the replay wrote, `want` +pub fn first_field(want: &str, got: &str) -> String { + fn walk(a: &Json, b: &Json, path: &str) -> Option { + match (a, b) { + (Json::Obj(x), Json::Obj(y)) => { + for (k, v) in x { let p = if path.is_empty() { k.clone() } else { format!("{path}.{k}") }; + match y.iter().find(|(j, _)| j == k) { None => return Some(p), Some((_, w)) => if let Some(d) = walk(v, w, &p) { return Some(d); } } } + y.iter().find(|(k, _)| !x.iter().any(|(j, _)| j == k)).map(|(k, _)| if path.is_empty() { k.clone() } else { format!("{path}.{k}") }) } + _ => (canon(a) != canon(b)).then(|| path.to_string()) } + } + match (crate::json::parse(want), crate::json::parse(got)) { (Ok(a), Ok(b)) => walk(&a, &b, "").unwrap_or_else(|| "the bytes".into()), _ => "the line".into() } +} + +#[cfg(test)] +mod tests { + use crate::json::{self, Json}; + use crate::persona::{build, canon, init, run_program, Flags}; + use crate::strand::{self, Clock, Live}; + fn run(prog: &Json, f: &Flags) -> Json { run_program(prog, f, 1, 100, 0) } + const DOC: &str = r#"{"probbit_persona":1,"identity":{"name":"R","version":"1","seed":3}, + "traits":[{"id":"verbosity","levels":["terse","short","full"],"logw":[0.2,0.5,0.1]},{"id":"initiative","levels":["wait","offer","lead"],"logw":[0.3,0.2,0]}], + "moods":[{"id":"valence","levels":["down","even","up"],"inertia":0.5,"half_life_hours":6}], + "inputs":[{"id":"night","kind":"flag","effects":{"verbosity":-0.5}},{"id":"ping","kind":"flag","effects":{"valence":0.6}},{"id":"odd","kind":"number","max":3,"effects":{"valence":-0.8,"verbosity":0.6}},{"id":"praise","kind":"flag"},{"id":"criticism","kind":"flag"}], + "habits":[{"id":"quiet_at_night","when":{"night":true},"then":{"verbosity":["terse"],"initiative":["wait","offer"]}}], + "learning":{"from":["praise","criticism"],"traits":["verbosity"],"rate":0.5,"step_cap":0.2,"total_cap":1}, + "expectation":{"half_life_events":8,"max_combos":6REST}}"#; + const REST: &str = r#","rest":{"half_life":24,"bucket":2,"surprise_gain":1,"decay":0.5}"#; + fn doc(r: &str) -> Json { json::parse(&DOC.replace("REST", r)).unwrap() } + fn ev(s: &str) -> Json { json::parse(s).unwrap() } + fn e(k: usize) -> Json { ev(&match k % 5 { 0 => r#"{"ping":true,"elapsed_hours":0.25}"#.to_string(), 1 => format!(r#"{{"odd":{},"elapsed_hours":0.25}}"#, (k * 7) % 4), + 2 => (if k % 3 == 0 { r#"{"praise":true,"elapsed_hours":0.25}"# } else { r#"{"criticism":true,"elapsed_hours":0.25}"# }).to_string(), 3 => format!(r#"{{"odd":{}.5,"night":true,"elapsed_hours":0.5}}"#, k % 3), _ => r#"{"night":true,"elapsed_hours":0.25}"#.to_string() }) } + /// A life of `n` events, a rest pass every `every` events (0: never), a checkpoint every `cp` (0: never) -> (text, individual) + fn life(r: &str, n: usize, every: usize, cp: usize) -> (String, Box) { + let d = doc(r); let p = build(&d).unwrap(); + let (mut lv, header) = Live::start(p.clone(), &d, init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.14.0"); + let mut text = format!("{header}\n"); + for k in 0..n { let (s, l) = lv.event(&e(k), &run).unwrap(); text.push_str(&l); text.push('\n'); + if cp > 0 && lv.n % cp as u64 == 0 { text.push_str(&lv.checkpoint(&s)); text.push('\n'); } + if every > 0 && lv.n % every as u64 == 0 { text.push_str(&lv.consolidate(&format!("2026-10-28T03:{:02}:00Z", k % 60)).unwrap()); text.push('\n'); } } + (text, Box::new(lv)) + } + #[test] + fn the_block_is_read_strictly_and_off_by_default() { + let r = |x: &str| match build(&doc(x)) { Err(e) => format!("{}: {}", e.path, e.msg), Ok(_) => "ok".into() }; + assert_eq!(r(""), "ok"); assert_eq!(r(REST), "ok"); assert_eq!(r(r#","rest":{}"#), "ok"); + assert!(r(r#","rest":{"bucket":4}"#).starts_with("expectation.rest.bucket"), "{}", r(r#","rest":{"bucket":4}"#)); + assert!(r(r#","rest":{"decay":1.5}"#).starts_with("expectation.rest.decay")); + assert!(r(r#","rest":{"nap":1}"#).starts_with("expectation.rest")); + // without the block: the pass is refused, nothing is written, the memory has no window + let (text, mut lv) = life("", 12, 0, 0); + assert_eq!(lv.consolidate("t").err().map(|e| e.path), Some("rest".to_string())); + assert!(!text.contains("consolidation") && !canon(&lv.expect.as_ref().unwrap().to_json()).contains("window")); + // the defaults: nothing moves (the window re-folded with the expectation's own half-life, no gain, no merge, no decay) + let (_, mut lv) = life(r#","rest":{}"#, 30, 0, 0); let st0 = lv.st.digest.clone(); + let l = lv.consolidate("t").unwrap(); let j = json::parse(&l).unwrap(); let c = j.get("consolidation").unwrap(); + assert_eq!(c.get("moved").and_then(Json::as_f64), Some(0.0), "{l}"); assert_eq!(lv.st.digest, st0); + assert_eq!(c.get("merged").and_then(|m| m.get("count")).and_then(Json::as_f64), Some(0.0)); + } + #[test] + fn the_pass_replays_verifies_resumes_and_refuses_forged_lines() { + for cp in [0, 7] { + let (text, lv) = life(REST, 60, 10, cp); + assert_eq!(text.matches(r#""consolidation":"#).count(), 6); + let v = strand::verify(&text, &run).unwrap_or_else(|(i, w)| panic!("line {i}: {w}")); + assert_eq!(v.get("events").and_then(Json::as_f64), Some(60.0), "{}", canon(&v)); + let ls = strand::lines(&text); + let (mut r, s0) = strand::Replay::open_lines(&ls).unwrap(); + for i in s0..ls.len() { r.step_at(&ls, i, &run).unwrap_or_else(|(i, w)| panic!("line {i}: {w}")); } + assert_eq!(canon(&r.live.expect.as_ref().unwrap().to_json()), canon(&lv.expect.as_ref().unwrap().to_json())); + assert_eq!(r.live.st.digest, lv.st.digest); + let rs = strand::resume(&text, &lv.p, lv.st.clone(), Clock::Fixed).unwrap(); + assert_eq!((rs.n, rs.head().to_string(), canon(&rs.expect.as_ref().unwrap().to_json())), (lv.n, lv.head().to_string(), canon(&lv.expect.as_ref().unwrap().to_json()))); + if cp > 0 { strand::verify_from_checkpoint(&text, &run).unwrap_or_else(|(i, w)| panic!("from checkpoint: line {i}: {w}")); + assert!(text.contains(r#""window":{"#), "the checkpoint carries the window"); + // the resume from a record behind the last line: the state an event logs before the last rest line + let k = ls.iter().rposition(|l| l.contains(r#""consolidation":"#)).unwrap() - 1; + let rp = strand::Replay::open_lines(&ls[..=k]).map(|(mut r, s0)| { for i in s0..=k { r.step_at(&ls, i, &run).unwrap(); } r.live.st.clone() }).unwrap(); + let rr = strand::resume_from_record(&text, &lv.p, Box::new(rp), Clock::Fixed, &run).unwrap_or_else(|e| panic!("resume from record: {}", e.msg)); + assert_eq!((rr.n, rr.head().to_string(), rr.st.digest.clone(), canon(&rr.expect.as_ref().unwrap().to_json())), (lv.n, lv.head().to_string(), lv.st.digest.clone(), canon(&lv.expect.as_ref().unwrap().to_json()))); } + // the pass moved something: merges (bucket 2), weights, the learned deltas halved + let cs: Vec = ls.iter().filter(|l| l.contains(r#""consolidation":"#)).map(|l| json::parse(l).unwrap()).collect(); + assert!(cs.iter().any(|c| c.get("consolidation").and_then(|c| c.get("moved")).and_then(Json::as_f64).unwrap_or(0.0) > 0.0)); + eprintln!("R15 lines: {}", cs.iter().map(|c| canon(c.get("consolidation").unwrap())).collect::>().join("\n")); + // forged lines: each field edited -> refused at that line, naming the field + let li = ls.iter().position(|l| l.contains(r#""consolidation":"#)).unwrap(); + let line = ls[li]; let j = json::parse(line).unwrap(); + let edits: Vec<(&str, String)> = vec![ + ("consolidation.moved", { let m = j.get("consolidation").unwrap().get("moved").and_then(Json::as_f64).unwrap(); line.replacen(&format!(r#""moved":{}"#, canon(&Json::Num(m))), r#""moved":9.5"#, 1) }), + ("consolidation.decay", line.replacen(r#""decay":0.5"#, r#""decay":1"#, 1)), + ("consolidation.events", line.replacen(r#""events":10"#, r#""events":11"#, 1)), + ("n", line.replacen(r#""n":10"#, r#""n":9"#, 1)), + ("state", { let s = j.get("state").and_then(Json::as_str).unwrap(); line.replacen(s, &format!("sha256:{}", "0".repeat(64)), 1) }), + ("consolidation.after", { let s = j.get("consolidation").unwrap().get("after").and_then(Json::as_str).unwrap(); line.replacen(s, &format!("sha256:{}", "1".repeat(64)), 1) }), + ]; + for (field, forged) in edits { + assert_ne!(forged, line, "{field}: the edit applies"); + let t2 = text.replacen(line, &forged, 1); + let (i, why) = strand::verify(&t2, &run).err().unwrap_or_else(|| panic!("{field}: a forged line verifies")); + assert_eq!(i, li + 1, "{field}: {why}"); assert!(why.ends_with(&format!("at {field}")), "{field}: {why}"); + } + } + } + #[test] + fn the_reflection_reports_the_rest_pass_and_old_engines_refuse_it() { + let (text, _) = life(REST, 30, 10, 0); + let ls = strand::lines(&text); + let r = crate::reflect::reflect_window(&ls, crate::reflect::Window::Last(30)).unwrap(); + let s = canon(&r.doc); + assert!(s.contains(r#""consolidation":{"lines":3"#) && s.contains("The rest pass ran 3 times in this window"), "{s}"); + // without a rest line in the window, no key and no sentence + let (t0, _) = life(REST, 30, 0, 0); + let r0 = crate::reflect::reflect_window(&strand::lines(&t0), crate::reflect::Window::Last(30)).unwrap(); + assert!(!canon(&r0.doc).contains("consolidation") && !canon(&r0.doc).contains("rest pass")); + // a consolidation line in an old engine's strand is refused (the line kind is engine-gated) + let d = doc(REST); let p = build(&d).unwrap(); + let (mut lv, header) = Live::start(p.clone(), &d, init(&p, Some(2), true, &run), Clock::Fixed, "probbit 0.13.0"); + let (_, l) = lv.event(&e(0), &run).unwrap(); + assert!(lv.consolidate("t").is_err(), "the fold before 0.14.0 has no window: refused"); + let _ = (header, l); + } +} diff --git a/probbit-cli/src/router.rs b/probbit-persona/src/router.rs similarity index 94% rename from probbit-cli/src/router.rs rename to probbit-persona/src/router.rs index 8223558..4764c6f 100644 --- a/probbit-cli/src/router.rs +++ b/probbit-persona/src/router.rs @@ -1,18 +1,23 @@ //! `probbit decide`, the router front-end: the router document -> `probbit_decide::Problem` (the strict input contract), and the decision //! pipeline (the exact tiers, then the gated sampler and the polish) as one function that returns the document and its exit code. //! main.rs parses the flags and prints it; probbit-wasm runs the same code in a browser. +#[allow(unused_imports)] +use std::prelude::rust_2021::*; use crate::json::{self, num, obj, str as jstr, Json}; -use crate::{config_echo, exact_cap, sys, tui}; +use crate::hooks::{self as tui, config_echo}; +use crate::run::exact_cap; +pub use crate::run::item_bars; +use crate::sys; use probbit_core::rt::Instant; use probbit_decide::*; const VERSION: &str = env!("CARGO_PKG_VERSION"); -pub(crate) struct Named { pub(crate) p: Problem, pub(crate) tasks: Vec, pub(crate) workers: Vec } +pub struct Named { pub p: Problem, pub tasks: Vec, pub workers: Vec } /// JSON problem -> Problem. See README "Problem format"; the strict contract (types, unknown fields, duplicates, empty domains, /// structured errors) is docs/probbit-ir-json.md "Input contract". -pub(crate) fn from_json(j: &Json) -> Result { +pub fn from_json(j: &Json) -> Result { use json::{arr, at, fields, ix, limit, opt, req, schema, text, value, weight}; fields(j, "", &["workers", "tasks", "affinity", "comment"])?; if let Some(c) = opt(j, "comment") { text(c, "comment")?; } @@ -92,13 +97,13 @@ fn components_tier(n: &Named, lowered: &probbit_ir::Model, exact_limit: u64, fr_ Some((obj(std::mem::take(out)), 0)) } /// The decision flags of `probbit decide` (main.rs reads them from the command line, probbit-wasm from a `flags` object). -pub(crate) struct Opts { pub(crate) budget: f64, pub(crate) seed: u64, pub(crate) exact_limit: u64, pub(crate) polish_ms: f64, pub(crate) polish_sweeps: usize, - pub(crate) fr_states: usize, pub(crate) sweeps: usize, pub(crate) mode: String, pub(crate) chains: usize, pub(crate) threads: usize, pub(crate) cpu_pct: u32, - pub(crate) xms: Option } +pub struct Opts { pub budget: f64, pub seed: u64, pub exact_limit: u64, pub polish_ms: f64, pub polish_sweeps: usize, + pub fr_states: usize, pub sweeps: usize, pub mode: String, pub chains: usize, pub threads: usize, pub cpu_pct: u32, + pub xms: Option } /// `probbit decide` on a parsed document: the exact tiers, then the gated sampler and the polish. -> (the decision document, its exit /// code: 0 answer, 1 infeasible, 3 refused / declined; the per-task error bars for `--summary`). `mem` = (--mem-limit-mb, rows per /// chain), read only when the sampler runs. -pub(crate) fn decide(n: &Named, o: &Opts, mem: &dyn Fn() -> (usize, usize)) -> (Json, i32, Vec) { +pub fn decide(n: &Named, o: &Opts, mem: &dyn Fn() -> (usize, usize)) -> (Json, i32, Vec) { let (mut doc, code, bars) = decide_inner(n, o, mem); crate::run::release_contract(&mut doc); (doc, code, bars) } @@ -180,11 +185,8 @@ fn decide_inner(n: &Named, o: &Opts, mem: &dyn Fn() -> (usize, usize)) -> (Json, ("ms", num(t0.elapsed().as_secs_f64() * 1e3))]); (obj(out), if verdict == "refused" { 3 } else { 0 }, item_bars(&g)) } -/// Per item, the error bar the gate compares with `tv_tol`: z x max(MCSE short, MCSE long) (`Gate::released_tasks`). -pub(crate) fn item_bars(g: &Gate) -> Vec { g.sig_tv.iter().zip(&g.sig_tv_long).map(|(&s, &l)| GATE.z * if s.is_nan() || l.is_nan() { f64::NAN } else { s.max(l) }).collect() } - /// The demo's task templates: (text, PII, production DB, judge fit per worker in `DEMO_W` order). -pub(crate) const DEMO_TPL: [(&str, bool, bool, [f64; 6]); 8] = [ +pub const DEMO_TPL: [(&str, bool, bool, [f64; 6]); 8] = [ ("draft reply: refund dispute (PII)", true, false, [2.0, 1.6, 1.0, 0.6, -1.0, 1.2]), ("summarize 40-page contract (PII)", true, false, [2.2, 1.8, 0.8, 0.9, -1.0, 0.5]), ("refactor auth middleware", false, false, [1.6, 1.2, 0.0, -0.5, 2.0, -1.0]), @@ -194,7 +196,7 @@ pub(crate) const DEMO_TPL: [(&str, bool, bool, [f64; 6]); 8] = [ ("fix flaky CI test", false, false, [1.0, 0.8, 0.2, -0.5, 1.8, -2.0]), ("investor update draft", false, false, [1.8, 1.4, 0.6, 0.2, -1.0, 0.8])]; /// The demo document for `n` tasks (seeded): the bytes of `probbit demo` since 0.1. -pub(crate) fn demo_doc(n: usize, seed: u64, hard: bool) -> Json { +pub fn demo_doc(n: usize, seed: u64, hard: bool) -> Json { const W: [(&str, usize); 6] = [("opus", 2), ("sonnet", 2), ("luna-pro", 3), ("local-gemma", 2), ("codex", 2), ("human", 3)]; const CUST: [&str; 8] = ["acme", "globex", "initech", "umbrella", "hooli", "stark", "wayne", "wonka"]; const TPL: [(&str, bool, bool, [f64; 6]); 8] = DEMO_TPL; diff --git a/probbit-cli/src/run.rs b/probbit-persona/src/run.rs similarity index 66% rename from probbit-cli/src/run.rs rename to probbit-persona/src/run.rs index bbb75f5..6e43966 100644 --- a/probbit-cli/src/run.rs +++ b/probbit-persona/src/run.rs @@ -2,6 +2,8 @@ //! Instructions (`--op`): `decide` (default: exact enumeration if the feasible set is <= --exact-limit, else the exact //! frontier DP if the program allows it (--frontier-states, 0 = off), else 4 gated chains), //! `exact` (enumeration only), `sample` (chains + gate, never enumerate). What-if = `"clamp"` on a variable. +#[allow(unused_imports)] +use std::prelude::rust_2021::*; use crate::json::{num, obj, str as jstr, InErr, Json}; use probbit_ir::*; @@ -20,7 +22,97 @@ pub struct Prog { pub m: Model, pub vars: Vec, pub values: Vec, /// JSON (probbit_ir 1) -> Model. See docs/probbit-ir-json.md; strict: every field type-checked, unknown fields, duplicate ids / value /// names and empty domains rejected, with a structured error (its "Input contract"). -pub fn from_json(j: &Json) -> Result { +pub fn from_json(j: &Json) -> Result { from_json_parts(j)?.into_prog() } +/// A program read and lowered, before the compile pass (`compile_parts`) and the model: what `from_json` builds the model from. +/// RFC 0009 (the expectation's members): programs that differ only in their vars' `h` share these, so a host reads the text once +/// and builds each model with its own `h` (`Parts::model_with`). +#[derive(Clone)] +pub struct Parts { pub vars: Vec, pub values: Vec, h: Vec, allowed: Vec, clamp: Vec>, pairs: Vec, caps: Vec, start: Option> } +impl Parts { + /// The program's own `h` (n x k), as its text gives it + pub fn h(&self) -> &[f64] { &self.h } + /// T3-c: the pairs of a text read without its `pairs` (the members' skeleton: the persona's couplings, read as `from_json` + /// reads the text's) + pub fn set_pairs(&mut self, pairs: Vec) { self.pairs = pairs; } + /// T3-c: the program's caps as read and lowered (before the compile pass), with its variables and values + pub fn into_caps(self) -> (Vec, Vec, Vec) { (self.vars, self.values, self.caps) } + /// The program's own model: the compile pass and `Model::new` on its `h`, as `from_json` + #[inline(always)] + pub fn into_prog(self) -> Result { + let Parts { vars, values, mut h, allowed, clamp, pairs, caps, start } = self; let (n, k) = (vars.len(), values.len()); + let (pairs, caps, compiled) = compile_parts(k, &mut h, pairs, caps); + let mut m = Model::new(n, k, h, allowed, clamp, pairs, caps).map_err(|e| crate::json::value("", e))?; m.start = start; + Ok(Prog { m, vars, values, compiled }) + } + /// The model of the same program with `h` (n x k, by variable then value index) in place of its own: each weight checked as + /// `json::weight` checks a weight of the text (a program whose text carries such a weight is refused), then the compile pass + /// and `Model::new` as `from_json` runs them + pub fn model_with(&self, mut h: Vec) -> Result { + if h.len() != self.h.len() || h.iter().any(|x| !x.is_finite() || x.abs() > crate::json::MAX_WEIGHT) { return Err(crate::json::value("vars", "a weight beyond the finite weight limit")); } + let (pairs, caps, _) = compile_parts(self.values.len(), &mut h, self.pairs.clone(), self.caps.clone()); + let mut m = Model::new(self.vars.len(), self.values.len(), h, self.allowed.clone(), self.clamp.clone(), pairs, caps).map_err(|e| crate::json::value("", e))?; m.start = self.start.clone(); + Ok(m) + } +} +/// RFC 0009 (the expectation's members, T3-b): one model for every program of an event that differs from A_t only in `h`. The +/// compile pass's folds (each separable table, in the pass's order: per value the row term into its first variable, then the +/// column term into its second) are kept apart, so a member's model is this one with the member's `h`, the folds replayed in +/// the same order: the same floats `compile_parts` makes, with no copy of the pairs and no second model held +pub struct Members { pub m: Model, folds: Vec<(usize, usize, Vec, Vec)>, + /// T3-c: the components tier's split of the model (built on the first member it answers; dropped when the caps change) + split: Split } +impl Members { + /// The model with `h` (n x k) in place of its own, as `Parts::model_with` builds it: each weight checked as the text's, the + /// folds added in the pass's order, then `Model::new`'s finite check + pub fn set_h(&mut self, mut h: Vec) -> Result<(), InErr> { + let bad = || crate::json::value("vars", "a weight beyond the finite weight limit"); + if h.len() != self.m.h.len() || h.iter().any(|x| !x.is_finite() || x.abs() > crate::json::MAX_WEIGHT) { return Err(bad()); } + let k = self.m.k; + for (i, j, row, col) in &self.folds { for a in 0..k { h[i * k + a] += row[a]; h[j * k + a] += col[a]; } } + if h.iter().any(|x| !x.is_finite()) { return Err(crate::json::value("", "h must contain only finite weights")); } + self.m.h = h; Ok(()) + } + /// T3-c: the model with `caps` (a member's rules lowered, `compiled`: already through the compile pass) in place of its + /// own: the pass's never-binding caps dropped as `compile_parts` drops them, then `Model::new` on the same parts, moved (no + /// copy of the pairs): the model `from_json` builds for the member's text. Err: `Model::new` refused them (the model is + /// then gone: the caller reads no member from it again) + #[inline(never)] + pub fn set_caps(&mut self, caps: Vec, compiled: bool) -> Result<(), InErr> { + let caps = if compiled { caps } else { compile_parts(self.m.k, &mut [], vec![], caps).1 }; + let m = &mut self.m; let (start, collective, cluster, cycles) = (m.start.take(), m.collective, m.cluster, m.cycles); + use core::mem::take; + let mut nm = Model::new(m.n, m.k, take(&mut m.h), take(&mut m.allowed), take(&mut m.clamp), take(&mut m.pairs), caps).map_err(|e| crate::json::value("", e))?; + nm.start = start; nm.collective = collective; nm.cluster = cluster; nm.cycles = cycles; self.m = nm; self.split = Split::default(); Ok(()) + } + /// `enumerated` on the model, the components tier with the members' one split (`probbit_ir::exact_components_split`: the + /// members differ only in `h` while the caps stay) + pub fn enumerated(&mut self, exact_limit: u64, sweeps: usize) -> Option>> { enumerated_in(&mut self.m, Some(&mut self.split), exact_limit, sweeps) } +} +impl Parts { + /// The members' model (`Members`): this program's model built from its parts (moved, as `into_prog`: no copy of the pairs), + /// its folds apart, with its variables, values and own `h` -> None when the folds read here are not the pass's (checked: + /// replayed on the program's own `h` they give the model's `h` bit for bit, and the pairs add up) + pub fn into_members(self) -> Option<(Vec, Vec, Vec, Members)> { + let k = self.values.len(); let mut folds = vec![]; let mut dropped = 0; + for p in &self.pairs { match &p.c { + Coupling::Potts(w) if *w == 0.0 => dropped += 1, + Coupling::Table(t) if t.iter().all(|&x| x == 0.0) => dropped += 1, + Coupling::Table(t) if t.len() == k * k && (0..k).all(|a| (0..k).all(|b| t[a * k + b] == t[a * k] + (t[b] - t[0]))) => + folds.push((p.i, p.j, (0..k).map(|a| t[a * k]).collect(), (0..k).map(|a| t[a] - t[0]).collect())), + _ => {} } } + let np = self.pairs.len(); + let Parts { vars, values, h, allowed, clamp, pairs, caps, start } = self; let h0 = h.clone(); let mut h = h; + let (pairs, caps, _) = compile_parts(k, &mut h, pairs, caps); + let mut m = Model::new(vars.len(), k, h, allowed, clamp, pairs, caps).ok()?; m.start = start; + if m.pairs.len() + folds.len() + dropped != np { return None; } + let want: Vec = m.h.iter().map(|x| x.to_bits()).collect(); let mut s = Members { m, folds, split: Split::default() }; + s.set_h(h0.clone()).ok()?; + (s.m.h.iter().map(|x| x.to_bits()).collect::>() == want).then_some((vars, values, h0, s)) + } +} +/// `from_json` up to the compile pass (inlined: `from_json` keeps one frame) +#[inline(always)] +pub fn from_json_parts(j: &Json) -> Result { use crate::json::{arr, at, count, fields, ix, keyed, limit, names, number, opt, req, schema, text, value, weight}; fields(j, "", &["probbit_ir", "values", "vars", "pairs", "caps", "all_different", "implies", "tables", "precedes", "linear", "start", "comment"])?; if let Some(c) = opt(j, "comment") { text(c, "comment")?; } @@ -183,9 +275,35 @@ pub fn from_json(j: &Json) -> Result { for (c, cp) in caps.iter().enumerate() { let load: u128 = cp.members.iter().enumerate().filter(|&(_, &(i, v))| x[i] == v).map(|(t, _)| cp.w(t) as u128).sum(); if load > cp.limit as u128 { return Err(value("start", format!("infeasible: lowered cap #{c} holds {load} > limit {}", cp.limit))); } } Some(x) } }; - let (pairs, caps, compiled) = compile_parts(k, &mut h, pairs, caps); - let mut m = Model::new(n, k, h, allowed, clamp, pairs, caps).map_err(|e| value("", e))?; m.start = start; - Ok(Prog { m, vars, values, compiled }) + drop(vindex); // no_std: hashbrown's map drop is not `may_dangle`, so the borrow of `values` ends here, not at the scope end + Ok(Parts { vars, values, h, allowed, clamp, pairs, caps, start }) +} + +/// RFC 0009 (the expectation's members): the marginals `run` would put in a `decide` answer of `persona::run_program` (no +/// deadline, no --exact-ms, frontier states `FRONTIER_MAX_STATES`, `sweeps` of fixed work) when its first exact tier answers it: +/// over `exact_limit` raw plans the components tier (`exact_components_until(m, limit, FRONTIER_MAX_STATES, None)`), within it +/// the whole-program enumeration (`exact_within(m, 5, limit, None, None)`). Some(Some(marginals, n x k)) for an exact verdict +/// whose numbers are all finite (what `run_program`'s finite check reads: the plan's log weight, log Z, the top plans' +/// probabilities, the marginals of the allowed values); Some(None) for an answer without an exact verdict (infeasible, or a number +/// not finite); None when `run` would go on to another tier (that one declining), or would stop its exact tiers on the wall +/// clock (`sweeps` 0 over the limit): the caller runs the program. +pub fn enumerated(m: &mut Model, exact_limit: u64, sweeps: usize) -> Option>> { enumerated_in(m, None, exact_limit, sweeps) } +fn enumerated_in(m: &mut Model, split: Option<&mut Split>, exact_limit: u64, sweeps: usize) -> Option>> { + m.collective = true; m.cluster = true; m.cycles = true; // as `persona::run_program` sets them before `run` + let space: f64 = (0..m.n).map(|i| m.cand_count(i) as f64).product(); + let fin = |m: &Model, marg: &[f64]| (0..m.n * m.k).all(|q| !m.allowed[q] || marg[q].is_finite()); + if space > exact_limit as f64 { + if sweeps == 0 { return None; } + let c = match split { Some(s) => exact_components_split(m, s, exact_limit, FRONTIER_MAX_STATES, None)?, None => exact_components_until(m, exact_limit, FRONTIER_MAX_STATES, None)? }; + if c.infeasible { return Some(None); } + let ok = c.map_logw.is_finite() && c.logz.is_finite() && (c.map_logw - c.logz).exp().is_finite() && fin(m, &c.marg); + return Some(ok.then_some(c.marg)); + } + let e = exact_within(m, 5, exact_limit, None, None).0?; + if e.n_feasible == 0 { return Some(None); } + let lw = m.logw(&e.top[0].1); + let ok = lw.is_finite() && e.logz.is_finite() && e.top.iter().all(|(p, _)| p.is_finite()) && fin(m, &e.marg); + Some(ok.then_some(e.marg)) } /// R19.6 (P2.1): `phases` on every `probbit run` answer, `[{"phase": name, "ms": wall}]` in order: parse (stdin JSON), compile @@ -245,13 +363,14 @@ fn ids(p: &Prog, mask: &[bool], want: bool) -> Json { Json::Arr(p.vars.iter().en /// Preserve the legacy full candidate `plan`, but make its release status explicit. /// `released_plan` is a projection of that same feasible candidate, not a separately /// solved complete plan; escalated variables still require a joint completion. -pub(crate) fn release_contract(doc: &mut Json) { +pub fn release_contract(doc: &mut Json) { let Some(Json::Obj(plan)) = doc.get("plan") else { return; }; let verdict = doc.get("verdict").and_then(Json::as_str).unwrap_or(""); let status = match verdict { "exact" | "diagnostics_passed" => "released", "partial" => "partial", _ => "diagnostic" }; let released: std::collections::HashSet<&str> = doc.get("released").and_then(Json::as_arr) .map_or_else(Default::default, |a| a.iter().filter_map(Json::as_str).collect()); let projection = Json::Obj(plan.iter().filter(|(id, _)| status != "diagnostic" && released.contains(id.as_str())).cloned().collect()); + drop(released); // as above: the set borrows `doc` if let Json::Obj(v) = doc { v.push(("plan_status".into(), jstr(status))); v.push(("released_plan".into(), projection)); } } @@ -336,7 +455,7 @@ fn run_inner(p: &Prog, op: &str, budget: f64, seed: u64, exact_limit: u64, polis // budget, and if a chain finds no start the exact search runs again, past its gap budget, to the end of --budget-ms: its // first plan is also the proof of infeasibility. An earlier version gave the exact tier half the budget BEFORE sampling, which // cost feasible programs sampler time (200-job schedules at 5 s: 174 -> 97 and 157 -> 0 released). `--sweeps N`: no fallback. - let fallback = op == "decide" && sweeps == 0 && !m.is_partition(); crate::tui::tier("sampler"); + let fallback = op == "decide" && sweeps == 0 && !m.is_partition(); crate::hooks::tier("sampler"); let s = match sample_on_starting(m, chains, threads, sweeps, if sweeps > 0 { None } else { Some(budget) }, seed, false, cpu_pct, mem.1, if fallback { 0.5 } else { 1.0 }) { Some(s) => s, None => { // Matching (quota-shaped programs) proves infeasibility; the bounded search for other programs does not if m.is_partition() { out.extend([("verdict", jstr("infeasible")), ("reason", jstr("no assignment satisfies every rule (capacitated matching)")), ("ms", ms(t0))]); return (obj(out), 1); } @@ -358,9 +477,9 @@ fn run_inner(p: &Prog, op: &str, budget: f64, seed: u64, exact_limit: u64, polis let hint = if op == "decide" { "no feasible start found within the search budget, and the bounded exact search (run again to the end of --budget-ms) found no plan (not a proof of infeasibility); --op exact searches exhaustively and can prove infeasibility, or raise --budget-ms" } else { "no feasible start found within the search budget (not a proof of infeasibility); try --op decide" }; out.extend([("verdict", jstr("refused")), ("reason", jstr(hint)), ("ms", ms(t0))]); return (obj(out), 3); } }; - let sample_s = ts.elapsed().as_secs_f64(); let tg = probbit_core::rt::Instant::now(); crate::tui::tier("gate"); + let sample_s = ts.elapsed().as_secs_f64(); let tg = probbit_core::rt::Instant::now(); crate::hooks::tier("gate"); let g = gate_stats(m, &s); let gate_ms = tg.elapsed().as_secs_f64() * 1e3; let rel = g.released_tasks(&GATE); let whole = g.diagnostics_passed(&GATE); - let nrel = rel.iter().filter(|&&r| r).count(); crate::tui::gate_seen(g.rhat, g.tv_bound(&GATE), if whole { m.n } else { nrel }, m.n); crate::tui::tier("polish"); *bars = crate::item_bars(&g); let verdict = if whole { "diagnostics_passed" } else if nrel > 0 { "partial" } else { "refused" }; + let nrel = rel.iter().filter(|&&r| r).count(); crate::hooks::gate_seen(g.rhat, g.tv_bound(&GATE), if whole { m.n } else { nrel }, m.n); crate::hooks::tier("polish"); *bars = item_bars(&g); let verdict = if whole { "diagnostics_passed" } else if nrel > 0 { "partial" } else { "refused" }; let mask: Vec = if whole { vec![true; m.n] } else { rel }; // plan polish (as `probbit decide`): anneal the chains' best plan through beta 2 -> 32; never worse than that plan // --polish-sweeps N = fixed-work polish (deterministic); --polish-ms is wall-clock, so its plan can vary run to run @@ -377,9 +496,106 @@ fn run_inner(p: &Prog, op: &str, budget: f64, seed: u64, exact_limit: u64, polis ("telemetry", obj([vec![("chains", num(chains as f64)), ("threads", num(threads.clamp(1, chains) as f64)), ("sweeps", num(s.sweeps as f64)), ("site_updates_per_s", num(if sample_s > 0.0 { (s.sweeps as f64 * m.n as f64 / sample_s).round() } else { 0.0 })), ("sample_ms", num(sample_s * 1e3)), ("gate_ms", num(gate_ms)), ("polish_ms", num(if polish_sweeps > 0 { 0.0 } else { polish_ms })), ("polish_sweeps", num(polish_sweeps as f64)), ("process_cpu_ms", crate::sys::usage().map_or(Json::Null, |u| num(u.0))), ("peak_rss_mb", crate::sys::usage().map_or(Json::Null, |u| num(u.1))), ("nice", crate::sys::nice().map_or(Json::Null, |v| num(v as f64))), ("cpu_limit_pct", num(cpu_pct as f64)), - ("mem_limit_mb", num(mem.0 as f64)), ("traj_rows", num(s.traj.iter().map(|tr| tr.len() / m.n).sum::() as f64))], crate::exact_cap(exact_ms, reached), + ("mem_limit_mb", num(mem.0 as f64)), ("traj_rows", num(s.traj.iter().map(|tr| tr.len() / m.n).sum::() as f64))], exact_cap(exact_ms, reached), if auto && reached { vec![("exact_budget_reached", Json::Bool(true))] } else { vec![] }, - vec![("exact_phase_ms", num((ts - t0).as_secs_f64() * 1e3)), ("polish_wall_ms", num(polish_wall))], crate::config_echo()].concat())), ("ms", ms(t0))]); + vec![("exact_phase_ms", num((ts - t0).as_secs_f64() * 1e3)), ("polish_wall_ms", num(polish_wall))], crate::hooks::config_echo()].concat())), ("ms", ms(t0))]); let code = if verdict == "refused" { 3 } else { 0 }; (obj(out), code) } +/// Telemetry fields of a sampled answer when --exact-ms was given (none otherwise): the cap and whether the exact tiers hit it. +pub fn exact_cap(xms: Option, reached: bool) -> Vec<(&'static str, Json)> { + xms.map_or(vec![], |x| vec![("exact_ms", num(x)), ("exact_ms_reached", Json::Bool(reached))]) +} +/// Per item, the error bar the gate compares with `tv_tol`: z x max(MCSE short, MCSE long) (`Gate::released_tasks`). +pub fn item_bars(g: &Gate) -> Vec { g.sig_tv.iter().zip(&g.sig_tv_long).map(|(&s, &l)| GATE.z * if s.is_nan() || l.is_nan() { f64::NAN } else { s.max(l) }).collect() } + +/// RFC 0011 (research): the `amplitude` tier, an exact simulated qubit register run on a CPU (`probbit_decide::amp`). Round 2: +/// a variable with d candidate values (allowed and clamp) takes ceil(log2 d) qubits (`amp::Encoding`; the unused codes keep +/// amplitude 0), and the program's log weight enters as one general diagonal phase layer (E = -log w per basis state), so +/// pair tables between multi-level variables need no decomposition. The program is split into its connected components +/// (two variables are linked by a pair or by sharing a capacity rule, as the `components` tier splits it) and each component +/// runs its own register: the phase is a sum over components and the mixer acts per variable, so the joint state is exactly +/// the product of the components' states. Eligible when every variable has an allowed value and the largest component has at +/// most `q_max` encoded qubits; Err(why) otherwise, and the caller runs the existing tiers. Each register runs the default +/// sweep from the uniform superposition over its valid codes; the rules are a projection: the Born probabilities of the +/// plans that break one are set to 0 and the rest renormalised (per component; the feasible mass is the product). Plan = per +/// component its most probable feasible plan (probabilities at 6 decimals, then the lowest basis index); marginals exact. The +/// receipt: qubits (encoded, summed), the components' qubits, the digests of the encoding, the circuit, the schedule and the +/// final amplitudes (6 decimals; the components' texts joined), the mixer's and the schedule's names, the feasible mass. +/// Round 4, `amplify` (`engine.amplify: true`): no projection; each component's swept state is rotated onto its feasible plans +/// by exact amplitude amplification (`amp::amplify_exact`: the rules as the oracle, the diffusion about the swept state, one +/// phase-matched final iteration), so the register itself holds feasible mass 1 (up to 1e-12) and the plan and marginals are +/// read from its Born probabilities; the receipt adds `amplify` (iterations and the final phases per component, the mass +/// before and after) and `feasible_mass` is the mass after. The relative amplitudes inside the feasible plans are the swept +/// state's, so the numbers equal the projection's (docs/rfc/0011-amplitude-register.md, "Amplification"). +#[cfg(feature = "decide")] +pub fn amplitude(prog: &Json, q_max: usize, amplify: bool) -> Result { amplitude_with(prog, q_max, amplify, &probbit_decide::amp::Evolution::Sweep(probbit_decide::amp::DEFAULT_SCHEDULE)) } +/// Round 5: `amplitude` with the evolution `ev` in place of the default sweep (`engine.schedule`; the receipt names it) +#[cfg(feature = "decide")] +pub fn amplitude_with(prog: &Json, q_max: usize, amplify: bool, ev: &probbit_decide::amp::Evolution) -> Result { + use probbit_decide::amp; + let p = from_json(prog).map_err(|e| format!("program {}: {}", e.path, e.msg))?; let m = &p.m; let (n, k) = (m.n, m.k); + let cand: Vec> = (0..n).map(|i| (0..k).filter(|&v| m.ok(i, v)).collect()).collect(); + if let Some(i) = cand.iter().position(Vec::is_empty) { return Err(format!("{} has no allowed value", p.vars[i])); } + // the components: pairs and capacity rules link variables + let mut par: Vec = (0..n).collect(); + fn find(par: &mut [usize], mut x: usize) -> usize { while par[x] != x { par[x] = par[par[x]]; x = par[x]; } x } + fn join(par: &mut [usize], a: usize, b: usize) { let (a, b) = (find(par, a), find(par, b)); if a != b { par[a.max(b)] = a.min(b); } } + for q in &m.pairs { join(&mut par, q.i, q.j); } + for c in &m.caps { if let Some(&(i0, _)) = c.members.first() { for &(i, _) in &c.members { join(&mut par, i0, i); } } } + let mut slot = vec![usize::MAX; n]; let mut mems: Vec> = vec![]; + for i in 0..n { let r = find(&mut par, i); if slot[r] == usize::MAX { slot[r] = mems.len(); mems.push(vec![]); } mems[slot[r]].push(i); } + let comp_of: Vec = (0..n).map(|i| slot[find(&mut par, i)]).collect(); + let encs: Vec = mems.iter().map(|mem| amp::Encoding { levels: mem.iter().map(|&i| cand[i].len()).collect() }).collect(); + let qs: Vec = encs.iter().map(amp::Encoding::qubits).collect(); let qtop = qs.iter().copied().max().unwrap_or(0); + if qtop > q_max.min(amp::Q_MAX_HOST) { return Err(format!("a component of {qtop} encoded qubits, q_max {q_max}")); } + let s = ev; let r6 = |x: f64| (x * 1e6).round() / 1e6; + let (mut enc_t, mut circ_t, mut state_t) = (String::new(), String::new(), String::new()); + let mut bp = vec![0usize; n]; let mut mg = vec![0.0; n * k]; let mut mass_all = 1.0; + let (mut am_it, mut am_ph, mut am_before) = (vec![], vec![], 1.0); + for (c, mem) in mems.iter().enumerate() { + // the component's sub-model: its variables, pairs and capacity rules (local indices), as the components tier builds it + let mut loc = vec![usize::MAX; n]; for (q, &i) in mem.iter().enumerate() { loc[i] = q; } + let nn = mem.len(); let mut h = Vec::with_capacity(nn * k); let mut al = Vec::with_capacity(nn * k); + for &i in mem { h.extend_from_slice(&m.h[i * k..i * k + k]); al.extend_from_slice(&m.allowed[i * k..i * k + k]); } + let pairs: Vec = m.pairs.iter().filter(|q| comp_of[q.i] == c).map(|q| Pair { i: loc[q.i], j: loc[q.j], c: q.c.clone() }).collect(); + let caps: Vec = m.caps.iter().filter(|q| q.members.first().is_some_and(|&(i, _)| comp_of[i] == c)) + .map(|q| Cap { members: q.members.iter().map(|&(i, v)| (loc[i], v)).collect(), limit: q.limit, weights: q.weights.clone() }).collect(); + let sub = Model::new(nn, k, h, al, mem.iter().map(|&i| m.clamp[i]).collect(), pairs, caps).map_err(|e| format!("component {c}: {e}"))?; + let enc = &encs[c]; let nb = 1usize << qs[c]; + let plan_of = |codes: &[usize]| -> Vec { codes.iter().zip(mem).map(|(&cd, &i)| cand[i][cd]).collect() }; + let e: Vec = (0..nb).map(|x| enc.decode(x).map_or(f64::NAN, |cd| -sub.logw(&plan_of(&cd)))).collect(); + let mut reg = s.run(enc, &e); + if amplify { + let ok: Vec = (0..nb).map(|x| enc.decode(x).is_some_and(|cd| sub.violations(&plan_of(&cd)) == 0)).collect(); + let Some(a) = amp::amplify_exact(&mut reg, |x| ok[x]) else { + // round 5: None after a rotation (good mass near 1) = the landing missed its bar: decline, never panic + let m: f64 = reg.probabilities().iter().zip(&ok).filter(|(_, &g)| g).map(|(p, _)| p).sum(); + return Err(if m > 0.5 { format!("component {c}: amplification missed its 1 - 1e-12 landing") } else { format!("component {c}: no feasible plan has weight in the register") }); }; + let pr = reg.probabilities(); + let best = (0..nb).fold(usize::MAX, |bx, x| if enc.decode(x).is_some() && pr[x] > 0.0 && (bx == usize::MAX || r6(pr[x]) > r6(pr[bx])) { x } else { bx }); + for (&i, v) in mem.iter().zip(plan_of(&enc.decode(best).expect("a plan with weight has valid codes"))) { bp[i] = v; } + for (x, &w) in pr.iter().enumerate() { if let Some(cd) = enc.decode(x) { for (&i, v) in mem.iter().zip(plan_of(&cd)) { mg[i * k + v] += w; } } } + mass_all *= a.mass_after; am_before *= a.mass_before; am_it.push(num(a.iterations as f64)); + am_ph.push(Json::Arr(if a.iterations == 0 { vec![] } else { vec![num(r6(a.phases.0)), num(r6(a.phases.1))] })); + for t in [&mut enc_t, &mut circ_t, &mut state_t] { t.push_str(&format!("component {c}\n")); } + enc_t.push_str(&enc.text()); circ_t.push_str(®.circuit_text()); state_t.push_str(®.canonical()); + continue; + } + let pr: Vec = reg.probabilities().iter().enumerate().map(|(x, &w)| match enc.decode(x) { Some(cd) if sub.violations(&plan_of(&cd)) == 0 => w, _ => 0.0 }).collect(); + let mass: f64 = pr.iter().sum(); if mass <= 0.0 || !mass.is_finite() { return Err(format!("component {c}: no feasible plan has weight in the register")); } + let best = (0..nb).fold(usize::MAX, |bx, x| if pr[x] > 0.0 && (bx == usize::MAX || r6(pr[x] / mass) > r6(pr[bx] / mass)) { x } else { bx }); + for (&i, v) in mem.iter().zip(plan_of(&enc.decode(best).expect("a feasible plan has valid codes"))) { bp[i] = v; } + for (x, &w) in pr.iter().enumerate() { if w > 0.0 { for (&i, v) in mem.iter().zip(plan_of(&enc.decode(x).expect("valid"))) { mg[i * k + v] += w / mass; } } } + mass_all *= mass; + for t in [&mut enc_t, &mut circ_t, &mut state_t] { t.push_str(&format!("component {c}\n")); } + enc_t.push_str(&enc.text()); circ_t.push_str(®.circuit_text()); state_t.push_str(®.canonical()); + } + let d = |t: &[u8]| jstr(&format!("sha256:{}", amp::hex(&::sha256(t)))); + let amp_r = if amplify { vec![("amplify", obj(vec![("iterations", Json::Arr(am_it)), ("phases", Json::Arr(am_ph)), ("mass_before", num(r6(am_before))), ("mass_after", num(r6(mass_all)))]))] } else { vec![] }; + Ok(obj(vec![("verdict", jstr("exact")), ("tier", jstr("amplitude")), ("plan", plan(&p, &bp)), ("plan_logw", num(m.logw(&bp))), ("violations", num(m.violations(&bp) as f64)), + ("marginals", marginals(&p, &mg)), ("released", ids(&p, &vec![true; n], true)), ("escalated", Json::Arr(vec![])), + ("amplitude", obj(vec![("qubits", num(qs.iter().sum::() as f64)), ("components", Json::Arr(qs.iter().map(|&q| num(q as f64)).collect())), + ("encoding", d(enc_t.as_bytes())), ("mixer", jstr(amp::MIXER)), ("circuit", d(circ_t.as_bytes())), ("schedule", d(s.text().as_bytes())), ("schedule_name", jstr(&s.name())), + ("state", d(state_t.as_bytes())), ("feasible_mass", num(r6(mass_all)))].into_iter().chain(amp_r).collect()))])) +} diff --git a/probbit-persona/src/strand.rs b/probbit-persona/src/strand.rs new file mode 100644 index 0000000..c4609b5 --- /dev/null +++ b/probbit-persona/src/strand.rs @@ -0,0 +1,1114 @@ +//! The strand writer and its checks (docs/persona.md §5.7), with no file I/O: the pure half of `probbit live`. Events come in as +//! JSON objects of inputs; the clock stamps each one's `elapsed_hours` (the time since the previous event, quantised to 1e-6 h), +//! so moods decay by their half-lives while nobody is talking to it, and feedback moves the learned deltas (§2.8). Every event +//! appends one line to the strand: the inputs as the turn used them, the stance's and the new state's digests, and the sha256 of +//! the line before. The strand's header carries the persona document, the initial state and the engine version, so `verify` +//! replays the whole life from the strand alone and names the earliest line that differs, or the earliest line with a field or +//! line kind this engine does not know (`unsupported`, a newer engine's strand: not verified, and not a divergence). +//! +//! Control lines (pause, resume, retire: appended by `probbit live control`, never by an event; no credit crosses them) and +//! checkpoint lines (the whole state every K events, so a replay can start there) are written here; `verify` replays all three. +//! The writer returns each line as a `String` and the caller stores it (the CLI appends to a file under a writer lock, +//! probbit-cli/src/live.rs; a firmware writes flash). Every function here takes the strand as text, so it runs on `no_std`. +use crate::json::{self, InErr, Json}; +use crate::persona::{self, perr, Engine, Persona, State}; +#[allow(unused_imports)] +use std::prelude::rust_2021::*; + +/// The strand format +pub const FORMAT: f64 = 1.0; +/// The clock. `Real`: a monotonic clock started with the run; each event's elapsed hours are the time since the previous event +/// (the opening event: since the start). `Fixed`: time comes from the events, each carrying its own `elapsed_hours` (default 0), so a +/// run is a pure function of its events. +pub enum Clock { Real(probbit_core::rt::Instant), Fixed } +/// Hours quantised to 1e-6 h (3.6 ms): the value the turn uses is the value the strand logs +fn q6(h: f64) -> f64 { persona::r6(h) } + +/// A strand's life as its control lines leave it: events are answered only while `Active` +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Status { Active, Paused, Retired } +impl Status { + pub fn name(self) -> &'static str { match self { Status::Active => "active", Status::Paused => "paused", Status::Retired => "retired" } } + /// The status a control line leads to: pause an active individual, resume a paused one, retire either; nothing after retire + pub fn after(self, what: &str) -> Result { + match (what, self) { + (_, Status::Retired) => Err(refusal(Status::Retired)), + ("pause", Status::Active) => Ok(Status::Paused), ("resume", Status::Paused) => Ok(Status::Active), ("retire", _) => Ok(Status::Retired), + // RFC 0004 amendment A2: `erase` returns the learned weights to zero and leaves the status as it was + ("erase", s) => Ok(s), + ("pause", _) => Err(perr("control", "already paused")), ("resume", _) => Err(perr("control", "not paused (resume follows a pause)")), + _ => Err(perr("control", "pause | resume | retire")) } + } +} +/// The refusal of an event (or a control line) by a paused or retired individual: code `paused` / `retired` (`live` exits 4) +fn refusal(s: Status) -> InErr { + let msg = if s == Status::Retired { "the individual is retired: every event is refused, for good (probbit live control ... retire)" } + else { "the individual is paused: every event is refused until `probbit live control STRAND resume`" }; + InErr { code: s.name(), path: "strand".into(), msg: msg.into() } +} +/// The default number of events between two checkpoint lines +pub const CHECKPOINT_EVERY: u64 = 1000; + +/// A live individual: the persona, the current state, the clock, the strand's chain and its status (control lines) +pub struct Live { pub p: Persona, pub st: State, clock: Clock, last: f64, prev: String, pub n: u64, pub status: Status, pub controls: u64, pub checkpoints: u64, pub rests: u64, + /// RFC 0009: the expectation's memory (a fold of the logged events) and the last event's exact trajectory (derived, never logged) + pub expect: Option, pub traj: Option, + /// The strand's engine is before 0.14.0 (DEV_R6_SINCE): the stance's `deviation` keeps full precision, as that engine wrote it + pub dev_full: bool, + /// RFC 0009: the members' exact marginals by program (`persona::ExpectCache`, on by default; off gives the same bytes) + pub expect_cache: persona::ExpectCache, + /// RFC 0014: the rest probes of `integrity` (the fresh individual's once per seed; the individual's when its state moved) + pub integ: persona::integrity::Cache } +/// The engine from which the stance's `deviation` (on checkpoint lines) is rounded to 6 decimals: a transcendental's last bits +/// differ between platforms' libm (macOS, glibc, the `libm` crate of a `nostd` build), and 6 decimals make the line's bytes, +/// and so the chain, the same on all of them +pub const DEV_R6_SINCE: (u64, u64, u64) = (0, 14, 0); +/// A strand header's engine `probbit X.Y.Z` (a `-pre` or `+build` suffix ignored) -> (X, Y, Z); None for any other name +pub fn engine_version(engine: &str) -> Option<(u64, u64, u64)> { + let v = engine.strip_prefix("probbit ")?.split(['-', '+']).next()?; + let mut it = v.split('.').map(|x| x.parse::().ok()); + let r = (it.next()??, it.next()??, it.next()??); + if it.next().is_some() { None } else { Some(r) } +} +/// A line with its stance's `deviation` self, pred and input at 6 decimals, as canonical JSON (None when it is not JSON): a +/// checkpoint of a strand before 0.14.0 verifies when it is the replay's at the 6 decimals 0.14.0 writes (the libm's last bits +/// set aside); the stance digest excludes `deviation` and the event line's receipt is checked exactly +fn dev_r6(l: &str) -> Option { + fn field<'a>(j: &'a mut Json, k: &str) -> Option<&'a mut Json> { match j { Json::Obj(v) => v.iter_mut().find(|(x, _)| x == k).map(|(_, y)| y), _ => None } } + let mut j = json::parse(l).ok()?; + if let Some(d) = field(&mut j, "stance").and_then(|s| field(s, "deviation")) { + for k in ["self", "pred", "input"] { if let Some(Json::Num(x)) = field(d, k) { *x = persona::r6(*x); } } } + Some(persona::canon(&j)) +} +/// One line of a strand without its line ending: strands are written with `\n`; a `\r\n` copy (Windows text mode, an editor) +/// reads the same, since a canonical JSON line never ends in `\r` itself +fn bare(l: &str) -> &str { l.strip_suffix('\r').unwrap_or(l) } +impl Live { + /// Start a life from `st` (an `init` state or a stored one) -> (the individual, the strand's header line). The header keeps + /// the persona document in its own key order (the compiled program follows it, and `engine.program` with it) and the + /// state as canonical JSON (keys sorted), so a state read from a file and the same state from `init` give one header; the + /// header is compact JSON, not key-sorted. + pub fn start(p: Persona, doc: &Json, st: State, clock: Clock, engine: &str) -> (Live, String) { + let (lv, header) = Live::start_boxed(Box::new(p), doc, Box::new(st), clock, engine); (*lv, header) + } + /// `start` on the heap: the persona and the state come in boxed and the individual goes out boxed, so the setup path of a + /// small-stack target holds none of them by value (docs/embedding.md "Stack budget"). The header is `start`'s, byte for byte. + #[inline(never)] + pub fn start_boxed(p: Box, doc: &Json, st: Box, clock: Clock, engine: &str) -> (Box, String) { + let mut b = Vec::new(); + let lv = Live::start_to(p, doc, st, clock, engine, HEADER_CHUNK, &mut |c: &[u8]| b.extend_from_slice(c)); + // the pieces are the bytes of one UTF-8 text cut at fixed offsets: whole again, it is that text + (lv, String::from_utf8(b).unwrap_or_default()) + } + /// `start_boxed` with the header handed to `out` in pieces of at most `chunk` bytes (the last one shorter) instead of returned + /// whole: a host writes each piece to its store and may yield between pieces (a flash write of the whole line can outlast a + /// watchdog). The pieces end to end are `start`'s header (without its `\n`); a piece may end inside a UTF-8 character. + #[inline(never)] + pub fn start_to(p: Box, doc: &Json, st: Box, clock: Clock, engine: &str, chunk: usize, out: &mut dyn FnMut(&[u8])) -> Box { + let prev = write_header(&p, doc, &st, engine, chunk, out); + let dev_full = engine_version(engine).is_some_and(|v| v < DEV_R6_SINCE); + let expect = p.expect.as_ref().map(|_| new_mem(dev_full)); + Live::boxed(p, st, clock, prev, 0, expect, dev_full) + } + /// The individual on the heap from its parts (one place builds a `Live`: `start_to` and `resume_boxed`), written in place: + /// the persona and the state are copied from their boxes into the new one, so no frame holds a `Live` (`Box::new` of the + /// value built it on the stack first: a 1.7 KB frame on Xtensa) + #[inline(never)] + fn boxed(p: Box, st: Box, clock: Clock, prev: String, n: u64, expect: Option, dev_full: bool) -> Box { + use core::mem::MaybeUninit; use core::ptr::{addr_of_mut, copy_nonoverlapping}; + // every field is written below: this pattern stops compiling when a field is added and not written + let _all = |l: &Live| { let Live { p: _, st: _, clock: _, last: _, prev: _, n: _, status: _, controls: _, checkpoints: _, rests: _, expect: _, traj: _, dev_full: _, expect_cache: _, integ: _ } = l; }; + let mut b: Box> = Box::new(MaybeUninit::uninit()); + let l = b.as_mut_ptr(); + // SAFETY: `l` points to an allocation for one `Live`; each field is written once, through raw field pointers (no reference + // to the uninitialised value); the persona and the state are moved by a bitwise copy out of their boxes, which are then + // freed as `MaybeUninit` (their contents are not dropped: they live on in the new box) + unsafe { + copy_nonoverlapping(&*p as *const Persona, addr_of_mut!((*l).p), 1); + copy_nonoverlapping(&*st as *const State, addr_of_mut!((*l).st), 1); + drop(Box::from_raw(Box::into_raw(p) as *mut MaybeUninit)); drop(Box::from_raw(Box::into_raw(st) as *mut MaybeUninit)); + addr_of_mut!((*l).clock).write(clock); addr_of_mut!((*l).last).write(0.0); addr_of_mut!((*l).prev).write(prev); addr_of_mut!((*l).n).write(n); + addr_of_mut!((*l).status).write(Status::Active); addr_of_mut!((*l).controls).write(0); addr_of_mut!((*l).checkpoints).write(0); addr_of_mut!((*l).rests).write(0); + addr_of_mut!((*l).expect).write(expect); addr_of_mut!((*l).traj).write(None); addr_of_mut!((*l).dev_full).write(dev_full); + addr_of_mut!((*l).expect_cache).write(persona::ExpectCache::new(true)); + addr_of_mut!((*l).integ).write(persona::integrity::Cache::new()); + Box::from_raw(Box::into_raw(b) as *mut Live) + } + } + /// One event: stamp its elapsed hours, run the turn, chain the strand line -> (the stance document, the strand line). A bad + /// event is an error and changes nothing (no turn, no line); so is every event while the individual is paused or retired. + pub fn event(&mut self, ev: &Json, eng: Engine) -> Result<(Json, String), InErr> { + use crate::trace::{enter, Phase}; + crate::trace::begin(); + if self.status != Status::Active { return Err(refusal(self.status)); } + let Json::Obj(kv) = ev else { return Err(perr("event", "must be a JSON object of inputs")) }; + // RFC 0016: a self-started event carries `stir: `, one of the classes the persona's stir block declares + let stir = match (self.p.cur.as_ref().and_then(|c| c.stir.as_ref()), ev.get(persona::curiosity::STIR_KEY)) { + (Some(sp), Some(x)) => { let cl = x.as_str().filter(|c| sp.classes().contains(c)).ok_or_else(|| perr("event.stir", format!("a declared stir class: {}", sp.classes().join(" | "))))?; Some(cl.to_string()) } + _ => None }; + let mut kv: Vec<(String, Json)> = kv.iter().filter(|(k, _)| k != "elapsed_hours").cloned().collect(); + let given = ev.get("elapsed_hours"); + let (elapsed, now) = match &self.clock { + Clock::Real(t0) => { if given.is_some() { return Err(perr("event.elapsed_hours", "the clock stamps it (--clock fixed reads it from the events)")); } + let now = t0.elapsed().as_secs_f64() / 3600.0; (q6((now - self.last).max(0.0)), now) } + Clock::Fixed => { let e = match given { None => 0.0, Some(Json::Num(x)) if x.is_finite() && *x >= 0.0 => *x, _ => return Err(perr("event.elapsed_hours", "a number >= 0")) }; (q6(e), self.last) } }; + kv.push(("elapsed_hours".into(), Json::Num(elapsed))); + let inputs = Json::Obj(kv); + // RFC 0016: the drive's level the turn reads (and every member of the expectation with it), from the fold before the event + let cdoc = self.expect.as_ref().and_then(|m| persona::cur_doc(&self.p, m)); + self.st.cur = self.expect.as_ref().and_then(|m| persona::cur_level(&self.p, m)); + enter(Phase::Expect); + // RFC 0009: the deviation and the trajectory are read from the state before the event (effects {}: the turn is as without + // them). The trajectory before the turn (nothing of the turn held while it solves). With the cache reusing the turn's own + // program (the default) the members but A_t run before the turn (their marginals held as rows, nothing of the turn held) + // and A_t's mixture after it, which hands A_t and its exact answer; else the whole deviation before it, as before (A_t + // built and solved by the deviation) + let traj = persona::trajectory(&self.p, &self.st, &inputs); + let want = self.expect.is_some() && self.expect_cache.reuses(); + let (early, held) = { let cache = &mut self.expect_cache; match self.expect.as_ref() { + Some(m) if want => (None, persona::deviation_pre(&self.p, &self.st, &inputs, m, eng, cache)), + Some(m) => (persona::deviation_at(&self.p, &self.st, &inputs, m, eng, cache, None), None), + None => (None, None) } }; + let (mut stance, ns, at) = persona::turn_at(&self.p, &self.st, &inputs, eng, want)?; + let sd = persona::stance_sha(&stance); + // RFC 0016: `drives.curiosity` = the level the turn read and its two inputs (and the stir's class on a self-started event); + // outside the stance digest, as `deviation` (the line's receipt carries it and `verify` recomputes it) + let cdoc = cdoc.map(|mut d| { if let (Some(cl), Json::Obj(v)) = (&stir, &mut d) { v.push(("stir".into(), Json::Str(cl.clone()))); } d }); + if let (Some(cd), Json::Obj(v)) = (&cdoc, &mut stance) { match v.iter_mut().find(|(k, _)| k == "drives") { + Some((_, Json::Obj(dv))) => dv.push(("curiosity".into(), cd.clone())), + _ => v.push(("drives".into(), Json::Obj(vec![("curiosity".into(), cd.clone())]))) } } + enter(Phase::Expect); + let cache = &mut self.expect_cache; + let dev = if want { self.expect.as_ref().and_then(|m| persona::deviation_post(&self.p, &self.st, &inputs, m, eng, cache, held, at)) } else { early }; + if self.expect.is_some() { if let Json::Obj(v) = &mut stance { v.push(("deviation".into(), dev.as_ref().map_or(Json::Null, |d| d.to_json(self.dev_full)))); } } + // §2.18: the glyph reads the finished document, `deviation` included (the surprise modifier) + persona::set_glyph(&self.p, &mut stance); + // RFC 0014 (0.14.0+): the drift from the declared self, from the state after the turn (effects {}: outside both digests) + let integ = if self.dev_full { None } else { persona::integrity::compute_cached(&self.p, &ns, &stance, &mut self.integ, eng)? }; + if let (Some(ig), Json::Obj(v)) = (&integ, &mut stance) { v.push(("integrity".into(), ig.doc.clone())); } + // RFC 0002 part 2: a rest line due before this event is its own line, chained before the event line (the turn applied it) + enter(Phase::Line); + let (rest, prev) = match persona::rest_line_digest(&self.p, &self.st, &inputs) { + Some((r, rd)) => { let l = rest_line(&self.prev, r, &rd); let d = persona::digest_of(&l); (Some(l), d) } + None => (None, self.prev.clone()) }; + let mut lj = vec![("n".into(), Json::Num((self.n + 1) as f64)), ("inputs".into(), inputs.clone()), ("stance".into(), Json::Str(sd)), + ("state".into(), Json::Str(ns.digest.clone())), ("prev".into(), Json::Str(prev))]; + if let Some(d) = &dev { lj.push(("deviation".into(), d.receipt())); } + if let Some(cd) = &cdoc { lj.push(("curiosity".into(), cd.clone())); } + if let Some(g) = stance.get("glyph") { lj.push(("glyph".into(), g.clone())); } + if let Some(ig) = &integ { lj.push(("integrity".into(), ig.doc.clone())); } + let line = persona::canon(&Json::Obj(lj)); + enter(Phase::Fold); + if let Some(m) = self.expect.as_mut() { persona::expect_step(&self.p, m, &inputs, dev.as_ref().and_then(persona::Deviation::logged_pred)); } + self.traj = traj; + enter(Phase::Line); + self.n += 1; self.last = now; self.prev = persona::digest_of(&line); self.st = *ns; + let line = match &rest { Some(r) => { self.rests += 1; format!("{r}\n{line}") } None => line }; + crate::trace::end(); + Ok((stance, line)) + } + /// A control line (pause, resume or retire) `by` a person or a sensor (`human[:id]`, `env[:id]`) for `reason`, stamped `at`: + /// the status moves, the credit is zeroed (no feedback after the line credits a stance before it) and the line is chained -> + /// the line. Nothing else changes: moods, drives, learned weights and the clock are the individual's as before. + pub fn control(&mut self, what: &str, by: &str, reason: &str, at: &str) -> Result { + let to = self.status.after(what)?; + let line = control_line(&self.prev, what, by, reason, at)?; + self.st = self.st.zero_credit(&self.p); if what == "erase" { self.st = self.st.erased(&self.p); } self.status = to; self.controls += 1; self.prev = persona::digest_of(&line); + Ok(line) + } + /// A checkpoint line after event n: the event count, the whole state after it and that event's stance document (from 0.14.0, + /// with an expectation block, also the expectation's memory: `expect`), chained -> the line. A replay can start there (the + /// state) and a board can be drawn there (the stance) without the events before. + pub fn checkpoint(&mut self, stance: &Json) -> String { + let mut kv = vec![("checkpoint".into(), Json::Num(self.n as f64)), ("prev".into(), Json::Str(self.prev.clone())), ("stance".into(), stance.clone()), + ("state".into(), self.st.to_json(&self.p))]; + // from 0.14.0 the expectation's memory (the canonical fold) rides on the line, so a replay and a resume can start here + if let Some(m) = self.expect.as_ref().filter(|m| m.is_canonical()) { kv.push(("expect".into(), m.to_json())); } + let line = persona::canon(&Json::Obj(kv)); + self.checkpoints += 1; self.prev = persona::digest_of(&line); line + } + /// RFC 0015 the rest pass (memory consolidation), when a host decides (standby, night, idle), stamped `at` (1-64 characters, + /// the host's clock): the expectation's memory consolidated from the window since the last pass, the learned deltas times the + /// pass's `decay`, and one `consolidation` line chained -> the line. Refused without an `expectation.rest` block, before engine + /// 0.14.0, and once retired. In place: no `State` or `Live` by value on a frame (the boxed individual of a small-stack target + /// calls it as it is; `consolidate_boxed` is this call). The expectation's cache is emptied (a changed memory: its rows are + /// rebuilt), which changes no byte. + #[inline(never)] + pub fn consolidate(&mut self, at: &str) -> Result { + if self.status == Status::Retired { return Err(refusal(self.status)); } + if at.is_empty() || at.len() > 64 { return Err(perr("rest.at", "a time stamp, 1-64 characters (RFC 3339 by default)")); } + let mem = self.expect.as_mut().ok_or_else(|| perr("rest", "the persona has no expectation block (the rest pass consolidates its memory)"))?; + let c = persona::rest::consolidate_mem(&self.p, mem).map_err(|m| perr("rest", m))?; + persona::rest::decay_learned(&self.p, &mut self.st); + let line = persona::rest::line(&self.prev, at, self.n, c, &self.st.digest); + self.expect_cache.invalidate(); self.prev = persona::digest_of(&line); + Ok(line) + } + /// `consolidate` for the boxed individual of a `no_std` body (docs/embedding.md): the same call, the same line + #[inline(never)] + pub fn consolidate_boxed(lv: &mut Box, at: &str) -> Result { lv.consolidate(at) } + /// The sha256 of the strand's last line (the header's before any event) + pub fn head(&self) -> &str { &self.prev } + /// RFC 0016: is a self-started idle event due `idle_hours` after the last event? Exactly from the drive's fold and the idle + /// hours: None without a stir block, below the threshold, inside the rate cap, or asleep without `when_asleep`. A host that + /// acts on it feeds `{"stir": class}` with its own readings as an event; a stir it ignores leaves no line. + pub fn stir_due(&self, idle_hours: f64) -> Option { + if self.status != Status::Active { return None; } + persona::stir_due(&self.p, self.expect.as_ref()?, idle_hours) + } +} +/// The header piece size `start_boxed` collects with (any size gives the same bytes) +pub const HEADER_CHUNK: usize = 1024; +/// A sink that cuts the text into pieces of `cap` bytes for `out` and hashes it on the way +struct Pieces<'a> { buf: Vec, cap: usize, out: &'a mut dyn FnMut(&[u8]), h: persona::Sha256 } +impl Pieces<'_> { + fn put(&mut self, mut b: &[u8]) { + self.h.update(b); + while !b.is_empty() { let k = (self.cap - self.buf.len()).min(b.len()); self.buf.extend_from_slice(&b[..k]); b = &b[k..]; + if self.buf.len() == self.cap { (self.out)(&self.buf); self.buf.clear(); } } + } +} +impl persona::Sink for Pieces<'_> { + fn push(&mut self, c: char) { let mut t = [0u8; 4]; self.put(c.encode_utf8(&mut t).as_bytes()) } + fn push_str(&mut self, t: &str) { self.put(t.as_bytes()) } +} +/// The strand's header line for `p`, `doc`, `st` and `engine` (`Live::start`'s, without its `\n`), handed to `out` in pieces of +/// at most `chunk` bytes (a 0 is taken as 1) -> its sha256 (`sha256:` + hex), the first line's `prev`. Neither the line nor a +/// copy of the document is held whole: the buffer is one piece. +pub fn write_header(p: &Persona, doc: &Json, st: &State, engine: &str, chunk: usize, out: &mut dyn FnMut(&[u8])) -> String { + write_header_seg(p, doc, st, engine, None, chunk, out) +} +/// `write_header` of a segment (docs/persona.md §5.7, "Segments"): the header with one more key at its end, `segment` = {index, +/// previous, sealed} (canonical JSON); None writes `write_header`'s bytes +pub fn write_header_seg(p: &Persona, doc: &Json, st: &State, engine: &str, segment: Option<&Json>, chunk: usize, out: &mut dyn FnMut(&[u8])) -> String { + use persona::Sink; + let cap = chunk.max(1); + let mut w = Pieces { buf: Vec::with_capacity(cap), cap, out, h: persona::Sha256::new() }; + let s = |x: &str| Json::Str(x.to_string()); + // the keys in `Live::start`'s order; values through the canonical writer (the state with its keys sorted, the rest as given) + w.push_str("{\"probbit_strand\":"); persona::write_to(&Json::Num(FORMAT), false, &mut w); + w.push_str(",\"engine\":"); persona::write_to(&s(engine), false, &mut w); + w.push_str(",\"persona\":"); persona::write_to(&Json::Obj(vec![("name".into(), s(&p.name)), ("version".into(), s(&p.version)), ("digest".into(), s(&p.digest))]), false, &mut w); + w.push_str(",\"seed\":"); persona::write_to(&Json::Num(st.seed as f64), false, &mut w); + w.push_str(",\"state\":"); persona::write_to(&st.to_json(p), true, &mut w); + w.push_str(",\"document\":"); persona::write_to(doc, false, &mut w); + if let Some(sg) = segment { w.push_str(",\"segment\":"); persona::write_to(sg, true, &mut w); } + w.push('}'); + if !w.buf.is_empty() { (w.out)(&w.buf); } + format!("sha256:{}", persona::hex(&w.h.finish())) +} +/// A control line after the line whose sha256 is `prev` (canonical JSON: at, by, control, prev, reason). `by` names who: a +/// person or a sensor, never the individual or the clock; the reason is 1-500 characters. +pub fn control_line(prev: &str, what: &str, by: &str, reason: &str, at: &str) -> Result { + if !["pause", "resume", "retire", "erase"].contains(&what) { return Err(perr("control", "pause | resume | retire")); } + if !matches!(persona::src_kind(by), Some("human" | "env")) { return Err(perr("control.by", "who: human[:id] or env[:id] (id: 1-64 of A-Z a-z 0-9 _ . - @ / :); the individual cannot control itself")); } + if reason.trim().is_empty() || reason.chars().count() > 500 { return Err(perr("control.reason", "1-500 characters")); } + if at.is_empty() || at.len() > 64 { return Err(perr("control.at", "a time stamp, 1-64 characters (RFC 3339 by default)")); } + let s = |x: &str| Json::Str(x.to_string()); + Ok(persona::canon(&Json::Obj(vec![("at".into(), s(at)), ("by".into(), s(by)), ("control".into(), s(what)), ("prev".into(), s(prev)), ("reason".into(), s(reason))]))) +} +/// A rest line (RFC 0002 part 2) after the line whose sha256 is `prev` (canonical JSON: prev, rest, state): what the line did +/// (downscale, prune, the pressure, the learned weights after it, each gate as kept) and the digest of the state after it. It is +/// written only together with the event it precedes, and that event's line chains to it. +fn rest_line(prev: &str, rest: Json, state: &str) -> String { + persona::canon(&Json::Obj(vec![("prev".into(), Json::Str(prev.into())), ("rest".into(), rest), ("state".into(), Json::Str(state.into()))])) +} +/// What kind of strand line `j` is +pub enum Kind { Event, Control, Checkpoint, Rest, Reflection, Consolidation } +pub fn kind(j: &Json) -> Kind { if j.get("control").is_some() { Kind::Control } else if j.get("checkpoint").is_some() { Kind::Checkpoint } else if j.get("rest").is_some() { Kind::Rest } else if j.get("reflection").is_some() { Kind::Reflection } else if j.get("consolidation").is_some() { Kind::Consolidation } else { Kind::Event } } +/// The divergence of a consolidation line in a strand of an engine before 0.14.0 (RFC 0015: consolidation lines are engine-gated) +pub const CONSOLIDATE_OLD: &str = "a consolidation line in a strand of an engine before 0.14.0 (the rest pass needs 0.14.0 or later)"; +/// The header's keys, as this engine writes them (a segment's header has one more at its end: `segment`) +const HEADER: [&str; 6] = ["probbit_strand", "engine", "persona", "seed", "state", "document"]; +/// The divergence of a segment whose second line is not the line its header links +pub const SEALED: &str = "the line after a segment's header is not its sealed line (its sha256 is not the header's `sealed`)"; +/// A header's `segment`: exactly {index: a whole number >= 1, previous: a file name (1-255 characters, no `/` or `\\`), sealed: a +/// sha256} +fn check_segment(sg: &Json) -> Result<(), String> { + let kv = sg.as_obj().filter(|kv| kv.len() == 3).ok_or("an object {index, previous, sealed}")?; + let f = |k: &str| kv.iter().find(|(x, _)| x == k).map(|(_, v)| v); + f("index").and_then(Json::as_f64).filter(|x| *x >= 1.0 && x.fract() == 0.0 && *x <= 9_007_199_254_740_992.0).ok_or("index: a whole number >= 1")?; + f("previous").and_then(Json::as_str).filter(|s| !s.is_empty() && s.len() <= 255 && !s.contains(['/', '\\'])).ok_or("previous: the previous segment's file name")?; + f("sealed").and_then(Json::as_str).filter(|s| s.len() == 71 && s.starts_with("sha256:") && s[7..].bytes().all(|c| c.is_ascii_hexdigit())).ok_or("sealed: a sha256")?; + Ok(()) +} +/// The file name of segment `k` of the strand `base` (§5.7): `NAME.strand` for 0, `NAME.000k.strand` after it (four digits at least); +/// a base without `.strand` gets `.000k` appended +pub fn segment_name(base: &str, k: u64) -> String { + if k == 0 { return base.to_string(); } + match base.strip_suffix(".strand") { Some(stem) => format!("{stem}.{k:04}.strand"), None => format!("{base}.{k:04}") } +} +/// The header of segment `index` that continues `lv` after its sealed line (whose sha256 is `sealed`; `previous` is the file name +/// of the segment it seals), for the persona document `doc` and the strand's `engine` -> the header line. The new file is this +/// header, then the sealed line itself (byte for byte), then the events: the chain goes on across the files unchanged. +pub fn segment_header(lv: &Live, doc: &Json, engine: &str, index: u64, previous: &str, sealed: &str) -> String { + let sg = Json::Obj(vec![("index".into(), Json::Num(index as f64)), ("previous".into(), Json::Str(previous.into())), ("sealed".into(), Json::Str(sealed.into()))]); + let mut b = Vec::new(); write_header_seg(&lv.p, doc, &lv.st, engine, Some(&sg), HEADER_CHUNK, &mut |c: &[u8]| b.extend_from_slice(c)); + String::from_utf8(b).unwrap_or_default() +} +/// The earliest key of a line after the header that its kind does not have (the line's keys in canonical order): a line kind +/// this engine does not know (a line without `control` or `checkpoint` is read as an event line, so `{"prev": ..., "nap": ...}` +/// gives `nap`) or a field it does not know on a known one +fn unknown_key(j: &Json) -> Option<&str> { + let known: &[&str] = match kind(j) { Kind::Event => &["curiosity", "deviation", "glyph", "inputs", "integrity", "n", "prev", "stance", "state"], Kind::Control => &["at", "by", "control", "prev", "reason"], + Kind::Checkpoint => &["checkpoint", "expect", "prev", "stance", "state"], Kind::Rest => &["prev", "rest", "state"], Kind::Reflection => &["prev", "reflection"], + Kind::Consolidation => &["at", "consolidation", "n", "prev", "state"] }; + j.as_obj()?.iter().map(|(k, _)| k.as_str()).find(|k| !known.contains(k)) +} +/// The divergence of a reflection line in a strand of an engine before 0.14.0 (RFC 0013: reflection lines are engine-gated) +pub const REFLECT_OLD: &str = "a reflection line in a strand of an engine before 0.14.0 (reflection lines need 0.14.0 or later)"; +/// The start of a replay's `why` that is not a divergence: the line has a field or a line kind this engine does not know (a +/// strand from a newer engine). Nothing from that line on was checked: the strand is not verified, and not shown to differ. +/// `verify` reports it as `unsupported` (exit 5), never as `diverges`; a changed value of a field this engine knows still +/// diverges. +pub const UNSUPPORTED: &str = "unsupported by this engine: "; +/// The `why` of an unsupported field: UNSUPPORTED, then its path (a persona field as the persona validator names it, a state +/// field under `state.`, otherwise its path in the strand line) +fn unsupported(path: &str) -> String { format!("{UNSUPPORTED}{path}") } +/// The field path of an unsupported `why`; None for a divergence +pub fn unsupported_field(why: &str) -> Option<&str> { why.strip_prefix(UNSUPPORTED) } +/// A persona, state or event error for a field this engine does not know (the validators' "unknown field" at its path) -> the +/// unsupported `why`; any other error -> `what` + the error, a divergence +fn read_err(e: InErr, what: &str) -> String { if e.msg.starts_with("unknown field") { unsupported(&e.path) } else { format!("{what}{}: {}", e.path, e.msg) } } +/// Now as RFC 3339 UTC to the second (the control lines' default `at`); `std` only (a firmware passes its own `at`) +#[cfg(feature = "std")] +pub fn utc_now() -> String { + let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_secs()); + let (z, t) = ((secs / 86_400) as i64 + 719_468, secs % 86_400); // days since 0000-03-01 (civil-from-days) + let (era, doe) = (z.div_euclid(146_097), z.rem_euclid(146_097)); let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365; + let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); let mp = (5 * doy + 2) / 153; let d = doy - (153 * mp + 2) / 5 + 1; + let m = if mp < 10 { mp + 3 } else { mp - 9 }; let y = yoe + era * 400 + i64::from(m <= 2); + format!("{y:04}-{m:02}-{d:02}T{:02}:{:02}:{:02}Z", t / 3600, t % 3600 / 60, t % 60) +} + +/// A strand replayed line by line: `verify` and `probbit monitor` (§5.8) share it, so both read a strand with one set of +/// rules. `open` checks the header and rebuilds the individual; each `step` replays one event line on the fixed clock and +/// checks it (`prev`, `n`, the stance and state digests, the bytes). After an error the replay is spent: the chain past the +/// line that differs cannot be checked. +pub struct Replay { pub live: Box, pub header: Json, pub doc: Json, pub line: usize, pub from: Option, last: Option, checkpoint_ready: bool, norms: Normals, at_cap: Vec, seen: (f64, Vec), viab: Option<(f64, Option)>, rest: Option } +/// A senses block's normals over the replay (§2.11): per sense the means after the last 50 events, the largest move of a mean +/// within 50 events so far, and the latest event's move (the monitor's alert: over drift_cap / 2) +#[derive(Default)] +pub struct Normals { window: Vec>, pub max_move: Vec, pub moved: Vec } +impl Normals { + fn start(p: &Persona, st: &State) -> Normals { + let n = persona::normals(p, st); + Normals { window: n.iter().map(|x| std::collections::VecDeque::from(vec![x.1])).collect(), max_move: vec![0.0; n.len()], moved: vec![0.0; n.len()] } + } + fn track(&mut self, p: &Persona, st: &State) { + for (k, (_, m, _, _)) in persona::normals(p, st).iter().enumerate() { let q = &mut self.window[k]; + let mv = persona::r6(q.iter().fold(0.0f64, |a, x| a.max((m - x).abs()))); + self.moved[k] = mv; self.max_move[k] = self.max_move[k].max(mv); q.push_back(*m); if q.len() > 50 { q.pop_front(); } } + } +} +impl Replay { + /// The header line (without its line ending) -> the replay, ready for event lines; Err((1, what differs)), or Err((1, the + /// unsupported `why`)) for a header key, persona field or state field this engine does not know + pub fn open(head: &str) -> Result { Replay::open_boxed(head).map(|r| *r) } + /// `open` on the heap (docs/embedding.md "Stack budget"): the header's checks of `open`, in its order with its messages, run as + /// the chain check runs them (the persona built on the heap, the state read into a box, the header written again piece by + /// piece against the line, never whole), and the individual is built in its box: no frame holds a `Live`, a `Persona` or the + /// header text. `open` is this call with the replay moved out of its box (the replay itself is small: the individual stays + /// boxed in it). + #[inline(never)] + pub fn open_boxed(head: &str) -> Result, (usize, String)> { + let (p, h, st0, prev) = open_head(head)?; + let doc = h.get("document").cloned().unwrap_or(Json::Null); + let dev_full = engine_version(h.get("engine").and_then(Json::as_str).unwrap_or("")).is_some_and(|v| v < DEV_R6_SINCE); + let expect = p.expect.as_ref().map(|_| new_mem(dev_full)); + let live = Live::boxed(p, st0, Clock::Fixed, prev, 0, expect, dev_full); + Ok(Box::new(Replay::at_header(live, h, doc))) + } + /// The replay of a life just started from its header + #[inline(never)] + fn at_header(live: Box, header: Json, doc: Json) -> Replay { + let norms = Normals::start(&live.p, &live.st); let at_cap = vec![0.0; persona::deficits(&live.p, &live.st).len()]; let seen = (0.0, vec![0.0; persona::gate_gains(&live.p, &live.st).len()]); + let viab = viab_start(&live); + Replay { live, header, doc, line: 1, from: None, last: None, checkpoint_ready: false, norms, at_cap, seen, viab, rest: None } + } + /// The replay from a checkpoint line (`cp`, without its line ending, at 1-based line `at`; `before` = the event line before it) + /// instead of the header: the header is read as by `open`; the checkpoint must follow `before` (prev), at its event count, with + /// its stance (the digest `before` logs) and its state (read as any state is, with the digest `before` logs); the chain goes on + /// from the checkpoint line. The lines before it are taken as written: a full `verify` replays them and checks the checkpoint. + pub fn open_at(head: &str, before: &str, cp: &str, at: usize) -> Result { + let mut r = Replay::open(head)?; + // RFC 0009: the expectation's memory is a fold of every event line, so a strand with the block replays from its header + // RFC 0009: before 0.14.0 the expectation's memory is a fold of every event line and no checkpoint carries it, so such a + // strand replays from its header; from 0.14.0 the checkpoint carries it + let canon = !r.live.dev_full; + if r.live.p.expect.is_some() && !canon { return Err((at, "the persona's expectation block reads every event before this checkpoint: verify this strand from its header".into())); } + let (n, stance, st, mem) = check_checkpoint(&r.live.p, canon, Some(before), cp, at)?; + r.start_at(n, stance, st, mem, cp, at); r.live.checkpoints = 1; + Ok(r) + } + /// The replay restarted at the checkpoint line `cp` (1-based line `at`): its count, stance, state and memory + fn start_at(&mut self, n: u64, stance: Json, st: Box, mem: Option, cp: &str, at: usize) { + let r = self; + r.live.st = *st; r.live.n = n; if mem.is_some() { r.live.expect = mem; } r.live.prev = persona::digest_of(cp); r.line = at; r.from = Some(n); r.last = Some(stance); r.checkpoint_ready = false; + r.norms = Normals::start(&r.live.p, &r.live.st); r.at_cap = vec![0.0; r.at_cap.len()]; r.seen = (0.0, vec![0.0; r.seen.1.len()]); r.viab = viab_start(&r.live); + } + /// The replay of a strand's `lines` from their start -> (the replay, the index of the first line still to step): the header, or + /// for a segment (a header with `segment`) the header and its sealed line (line 2), taken on the header's link: its sha256 is + /// the header's `sealed`, it is a valid checkpoint line and its state is the header's; the replay goes on from it + pub fn open_lines(lines: &[&str]) -> Result<(Replay, usize), (usize, String)> { + let mut r = Replay::open(lines.first().copied().unwrap_or(""))?; + let Some(sg) = r.header.get("segment").cloned() else { return Ok((r, 1)) }; + let canon = !r.live.dev_full; + if r.live.p.expect.is_some() && !canon { return Err((1, "a segment of a persona with an expectation block needs a strand of engine 0.14.0 or later".into())); } + let cp = lines.get(1).copied().unwrap_or(""); + if Some(persona::digest_of(cp).as_str()) != sg.get("sealed").and_then(Json::as_str) { return Err((2, SEALED.into())); } + let (n, stance, st, mem) = check_checkpoint(&r.live.p, canon, None, cp, 2)?; + if st.digest != r.live.st.digest { return Err((2, "the sealed line's state is not the segment header's".into())); } + r.start_at(n, stance, st, mem, cp, 2); + Ok((r, 2)) + } + /// The stance document of the last event replayed (or of the checkpoint the replay started from) + pub fn last_stance(&self) -> Option<&Json> { self.last.as_ref() } + /// The senses' normals over the replay so far (§2.11) + pub fn normals(&self) -> &Normals { &self.norms } + /// One line (without its line ending) -> the stance document an event line replays to (None for a control or checkpoint + /// line); Err((its 1-based line number, what differs)). A control line must be a valid move of the status and the line its + /// fields give; a checkpoint line must immediately follow an event while active and carry + /// the event count and the state the replay reached. The last stance remains available to + /// the monitor after controls/checkpoints, but that does not authorize another checkpoint. A chained line with a key its + /// kind does not have (a newer line kind, a newer field), or an event whose inputs have a field this engine does not know, + /// is Err((its line number, the unsupported `why`)). + pub fn step(&mut self, l: &str, eng: Engine) -> Result, (usize, String)> { + let (i, live) = (self.line + 1, &mut self.live); + let j = json::parse(l).map_err(|e| (i, format!("not JSON: {}", e.msg)))?; + if persona::canon(&j) != l { return Err((i, "not canonical JSON".into())); } + let head = self.rest.as_deref().map_or(live.head().to_string(), persona::digest_of); + if j.get("prev").and_then(Json::as_str) != Some(head.as_str()) { return Err((i, PREV.into())); } + if let Some(k) = unknown_key(&j) { return Err((i, unsupported(k))); } + if self.rest.is_some() && !matches!(kind(&j), Kind::Event) { return Err((i, "a rest line must be followed by its event line".into())); } + match kind(&j) { + // RFC 0002 part 2: a rest line is checked with the event after it (the replay recomputes it there) + Kind::Rest => { if live.status != Status::Active { return Err((i, format!("rest line while {}", live.status.name()))); } + self.rest = Some(l.to_string()); self.line = i; self.checkpoint_ready = false; return Ok(None) } + Kind::Control => { let f = |k: &str| j.get(k).and_then(Json::as_str).unwrap_or(""); + let line = live.control(f("control"), f("by"), f("reason"), f("at")).map_err(|e| (i, format!("control line: {}: {}", e.path, e.msg)))?; + if line != l { return Err((i, "the control line differs".into())); } + self.line = i; self.checkpoint_ready = false; return Ok(None) } + Kind::Checkpoint => { + if live.status != Status::Active { return Err((i, format!("checkpoint while {}", live.status.name()))); } + if !self.checkpoint_ready { return Err((i, "a checkpoint must immediately follow an event line".into())); } + if j.get("checkpoint").and_then(Json::as_f64) != Some(live.n as f64) { return Err((i, format!("the checkpoint is not at event {}", live.n))); } + let Some(stance) = self.last.as_ref() else { return Err((i, "a checkpoint follows an event line".into())) }; + let line = live.checkpoint(stance); + if line != l { + // before 0.14.0 the deviation is at full precision: equal at 6 decimals, the chain goes on from the logged bytes + if !(live.dev_full && dev_r6(&line) == dev_r6(l)) { return Err((i, "the checkpoint's state or stance differs from the replay".into())); } + live.prev = persona::digest_of(l); + } + self.line = i; self.checkpoint_ready = false; return Ok(None) } + // RFC 0013: a reflection line changes nothing (its document is checked against the lines before it: `step_at`) + Kind::Reflection => { if live.dev_full { return Err((i, REFLECT_OLD.into())); } + live.prev = persona::digest_of(l); self.line = i; self.checkpoint_ready = false; return Ok(None) } + // RFC 0015: the rest pass run again from the lines before; a forged line is refused naming its first field that differs + Kind::Consolidation => { if live.dev_full { return Err((i, CONSOLIDATE_OLD.into())); } + let line = live.consolidate(j.get("at").and_then(Json::as_str).unwrap_or("")).map_err(|e| (i, format!("consolidation line: {}: {}", e.path, e.msg)))?; + if line != l { return Err((i, format!("the consolidation line differs at {}", persona::rest::first_field(&line, l)))); } + self.line = i; self.checkpoint_ready = false; return Ok(None) } + Kind::Event => {} } + if j.get("n").and_then(Json::as_f64) != Some((live.n + 1) as f64) { return Err((i, format!("n is not {}", live.n + 1))); } + let ev = j.get("inputs").ok_or((i, "no inputs".to_string()))?; + // a needs block (RFC 0002): the stamped hours of events that found a deficit at its cap (it sat there since the event before) + let before = persona::deficits(&live.p, &live.st); let h = ev.get("elapsed_hours").and_then(Json::as_f64).unwrap_or(0.0); + let logged = self.rest.take(); + let (stance, both) = live.event(ev, eng).map_err(|e| (i, read_err(e, "")))?; + let line = match (both.split_once('\n'), logged) { + (Some((r, e)), Some(lr)) => { if r != lr { return Err((i - 1, "the rest line differs".into())); } e.to_string() } + (Some(_), None) => return Err((i, "a rest line is due before this event and missing".into())), + (None, Some(_)) => return Err((i - 1, "a rest line where none is due".into())), + (None, None) => both }; + for (k, (_, x, cap)) in before.iter().enumerate() { if *x >= cap - 5e-7 { self.at_cap[k] = persona::r6(self.at_cap[k] + h); } } + if j.get("stance").and_then(Json::as_str) != Some(persona::stance_sha(&stance).as_str()) { return Err((i, "the stance differs".into())); } + // RFC 0009: the logged deviation is recomputed (an edited receipt diverges here) + if json::parse(&line).ok().and_then(|x| x.get("deviation").cloned()) != j.get("deviation").cloned() { return Err((i, "the deviation differs".into())); } + // §2.18: the logged glyph is recomputed from the replayed stance (an edited glyph diverges here) + if stance.get("glyph") != j.get("glyph") { return Err((i, "the glyph differs".into())); } + // RFC 0014: the logged integrity is recomputed from the replayed state (an edited number diverges here) + if stance.get("integrity") != j.get("integrity") { return Err((i, "the integrity differs".into())); } + if j.get("state").and_then(Json::as_str) != Some(live.st.digest.as_str()) { return Err((i, "the state differs".into())); } + if line != l { return Err((i, "the line differs".into())); } + self.line = i; self.last = Some(stance.clone()); self.checkpoint_ready = true; self.norms.track(&self.live.p, &self.live.st); + // RFC 0002: the largest R̄ (Design C) and per learned trait the largest gain (Design B) after any event replayed + if let Some((r, _, _)) = persona::pace_rbar(&self.live.p, &self.live.st) { self.seen.0 = self.seen.0.max(r); } + for (k, (_, g, _)) in persona::gate_gains(&self.live.p, &self.live.st).iter().enumerate() { if let Some(m) = self.seen.1.get_mut(k) { *m = m.max(*g); } } + // RFC 0008 E: the smallest margin_min after any event replayed, and the event at which viability was lost + if let (Some((seen, lost_at)), Some((_, m, lost))) = (self.viab.as_mut(), persona::viability(&self.live.p, &self.live.st)) { + *seen = seen.min(m); if lost && lost_at.is_none() { *lost_at = Some(self.live.n); } } + Ok(Some(stance)) + } + /// `step` of `lines[i]`, a reflection line also checked against the lines before it (RFC 0013): every verify path steps so + pub fn step_at(&mut self, lines: &[&str], i: usize, eng: Engine) -> Result, (usize, String)> { + let r = self.step(lines[i], eng)?; + if r.is_none() && crate::reflect::is_reflection_line(lines[i]) { crate::reflect::check_line(&lines[..i], lines[i]).map_err(|why| (i + 1, why))?; } + Ok(r) + } + /// The end of the strand: a rest line (RFC 0002 part 2) is only ever written with the event after it, so a strand that ends + /// on one diverges at it + pub fn end(&self) -> Result<(), (usize, String)> { + if self.rest.is_some() { return Err((self.line, "a rest line must be followed by its event line".into())); } Ok(()) + } + /// `verify`'s summary of the lines replayed so far: `engine` is the strand header's (the engine that wrote it), `verified_by` + /// the engine that replayed it + pub fn summary(&self) -> Json { + let (s, live) = (|x: &str| Json::Str(x.to_string()), &self.live); + let mut v = vec![("ok".into(), Json::Bool(true)), ("events".into(), Json::Num(live.n as f64)), ("persona".into(), s(&live.p.name)), ("seed".into(), Json::Num(live.st.seed as f64)), + ("engine".into(), self.header.get("engine").cloned().unwrap_or(Json::Null)), ("verified_by".into(), Json::Str(engine())), ("final_state".into(), s(&live.st.digest)), + ("last_line".into(), s(live.head()))]; + // a strand with control or checkpoint lines says so (a strand without them gives the summary of before) + if live.controls > 0 { v.push(("controls".into(), Json::Num(live.controls as f64))); v.push(("status".into(), s(live.status.name()))); } + if live.checkpoints > 0 { v.push(("checkpoints".into(), Json::Num(live.checkpoints as f64))); } + // RFC 0002 part 2: rest lines replayed, the pressure now; the load now + if live.rests > 0 { v.push(("rest_lines".into(), Json::Num(live.rests as f64))); } + if let Some((sp, pend)) = persona::rest_pressure(&live.st) { v.push(("pressure".into(), Json::Obj(vec![("s".into(), Json::Num(sp)), ("pending".into(), Json::Bool(pend))]))); } + if let Some((l, _)) = persona::load_level(&live.p, &live.st) { v.push(("load".into(), Json::Obj(vec![("l".into(), Json::Num(l))]))); } + if let Some(n) = self.from { v.push(("from_checkpoint".into(), Json::Num(n as f64))); } + // a segment (§5.7): its index and the digest of the sealed line its replay started from, taken on the link + if let Some(sg) = self.header.get("segment") { v.push(("segment".into(), sg.get("index").cloned().unwrap_or(Json::Null))); v.push(("trusted_link".into(), sg.get("sealed").cloned().unwrap_or(Json::Null))); } + // a senses block (§2.11): per sense the normal now and the largest move of its mean within 50 events + let nm = persona::normals(&live.p, &live.st); + if !nm.is_empty() { v.push(("normals".into(), Json::Obj(nm.iter().enumerate().map(|(k, (id, m, n, _))| (id.clone(), Json::Obj(vec![("mean".into(), Json::Num(*m)), + ("n".into(), Json::Num(*n as f64)), ("max_move_50".into(), Json::Num(self.norms.max_move.get(k).copied().unwrap_or(0.0)))]))).collect()))); } + // a needs block (RFC 0002): per need its deficit now and the stamped hours it sat at its cap + let dx = persona::deficits(&live.p, &live.st); + if !dx.is_empty() { v.push(("needs".into(), Json::Obj(dx.iter().enumerate().map(|(k, (id, x, _))| (id.clone(), Json::Obj(vec![("deficit".into(), Json::Num(*x)), + ("at_cap_hours".into(), Json::Num(self.at_cap.get(k).copied().unwrap_or(0.0)))]))).collect()))); } + // RFC 0002 Design C: R̄ now and the largest after any event replayed; Design B: per learned trait the largest gain seen + if let Some((r, _, _)) = persona::pace_rbar(&live.p, &live.st) { v.push(("pace".into(), Json::Obj(vec![("rbar".into(), Json::Num(r)), ("max".into(), Json::Num(self.seen.0))]))); } + let gg = persona::gate_gains(&live.p, &live.st); + if !gg.is_empty() { v.push(("surprise".into(), Json::Obj(gg.iter().enumerate().map(|(k, (t, _, _))| (t.clone(), + Json::Obj(vec![("g_max_seen".into(), Json::Num(self.seen.1.get(k).copied().unwrap_or(0.0)))]))).collect()))); } + // RFC 0008 E: with a bound, recomputed from the logged readings and amounts + if let Some((seen, lost_at)) = self.viab { v.push(("viability".into(), Json::Obj(vec![("margin_min_seen".into(), Json::Num(seen)), ("lost_at".into(), lost_at.map_or(Json::Null, |n| Json::Num(n as f64)))]))); } + // a credit block (§2.8): the decisions still awaiting an outcome + if let Some(o) = persona::open_decisions(&live.p, &live.st) { v.push(("open".into(), Json::Arr(o.iter().map(|(t, _)| Json::Num(*t as f64)).collect()))); } + Json::Obj(v) + } +} +/// RFC 0008 E: the viability record a replay starts from: the margin_min of its first state, and lost_at (the event count when the +/// state is already lost, as after a checkpoint); none without a bound +fn viab_start(live: &Live) -> Option<(f64, Option)> { persona::viability(&live.p, &live.st).map(|(_, m, lost)| (m, lost.then_some(live.n))) } +/// Per event line its n and its block +pub type Blocks = Vec<(u64, String)>; +/// `live verify --blocks`: replay from the header -> (the summary, per event line its n and the block its stance renders, RFC 0008 E) +pub fn verify_blocks(text: &str, eng: Engine) -> Result<(Json, Blocks), (usize, String)> { + let lines = lines(text); + let replay = || -> Result<(Json, Blocks), (usize, String)> { + let (mut r, k) = Replay::open_lines(&lines)?; let mut out = vec![]; + for i in k..lines.len() { if let Some(doc) = r.step_at(&lines, i, eng)? { out.push((r.live.n, persona::block(&r.live.p, &doc))); } } + Ok((r.summary(), out)) }; + replay().map_err(|e| chain_first(&lines, e)) +} +/// A strand's text -> its lines without their line endings (a final newline ends the last line; it does not start another). +/// One leading UTF-8 byte-order mark is skipped, as `\r` before `\n` is: a Windows editor adds both on save, and neither is +/// part of the line text the chain hashes. +pub fn lines(text: &str) -> Vec<&str> { + let mut lines: Vec<&str> = text.strip_prefix('\u{feff}').unwrap_or(text).split('\n').map(bare).collect(); + if lines.last() == Some(&"") { lines.pop(); } + lines +} +/// `probbit live verify`: replay a strand from its header -> Ok(summary) when every line is as recorded; Err((1-based line +/// number, what differs)) at the earliest line that is not. Lines end in `\n` or `\r\n`; the chain hashes the line text alone. +/// A line with a field or line kind this engine does not know stops the replay as unsupported, unless a later line breaks the +/// chain (`chain_first`). +pub fn verify(text: &str, eng: Engine) -> Result { + let lines = lines(text); + let replay = || -> Result { + let (mut r, k) = Replay::open_lines(&lines)?; + for i in k..lines.len() { r.step_at(&lines, i, eng)?; } + r.end()?; Ok(r.summary()) }; + replay().map_err(|e| chain_first(&lines, e)) +} +/// The divergence the hash chain shows after an unsupported line: a strand this engine cannot replay to the end is still checked +/// for what the format alone says, so an edited, removed or reordered line after it (or the unsupported line itself, edited: +/// the next line's `prev`) diverges at the earliest line that breaks the chain (not JSON, not canonical JSON, or a `prev` that is +/// not the sha256 of the line before), as in a replay. `unsupported` is left only for a strand whose chain is whole; a +/// divergence passes unchanged. +fn chain_first(lines: &[&str], (line, why): (usize, String)) -> (usize, String) { + if unsupported_field(&why).is_none() { return (line, why); } + for i in line.max(1)..lines.len() { + let j = match json::parse(lines[i]) { Ok(j) => j, Err(e) => return (i + 1, format!("not JSON: {}", e.msg)) }; + if persona::canon(&j) != lines[i] { return (i + 1, "not canonical JSON".into()); } + if j.get("prev").and_then(Json::as_str) != Some(persona::digest_of(lines[i - 1]).as_str()) { return (i + 1, PREV.into()); } + } + (line, why) +} +/// The divergence of a line whose `prev` is not the sha256 of the line before +const PREV: &str = "prev is not the sha256 of the line before (a line before it was changed, removed or reordered)"; +/// The 0-based index of the strand's last checkpoint line followed by at least `after` lines, if any (a cheap scan: the key) +pub fn last_checkpoint(lines: &[&str], after: usize) -> Option { + (1..lines.len().saturating_sub(after)).rev().find(|&i| lines[i].starts_with(r#"{"checkpoint":"#)) +} +/// `verify --from-checkpoint`: replay from the last checkpoint line (the header read as always), not from the header +pub fn verify_from_checkpoint(text: &str, eng: Engine) -> Result { + let lines = lines(text); + // a segment's sealed line (line 2) is where its own replay starts anyway + let Some(k) = last_checkpoint(&lines, 0).filter(|&k| k >= 2) else { return verify(text, eng) }; + let replay = || -> Result { + let mut r = Replay::open_at(lines[0], lines[k - 1], lines[k], k + 1)?; + for i in k + 1..lines.len() { r.step_at(&lines, i, eng)?; } + r.end()?; Ok(r.summary()) }; + replay().map_err(|e| chain_first(&lines, e)) +} +/// `verify` from the header, keeping the stance documents of the given turns (what `--actions` compares with) +pub fn verify_keep(text: &str, eng: Engine, keep: &[u64]) -> Result<(Json, Vec<(u64, Json)>, Persona), (usize, String)> { + let lines = lines(text); + let replay = || -> Result<(Json, Vec<(u64, Json)>, Persona), (usize, String)> { + let (mut r, k) = Replay::open_lines(&lines)?; let mut kept = vec![]; + for i in k..lines.len() { if let Some(doc) = r.step_at(&lines, i, eng)? { + if let Some(t) = doc.get("turn").and_then(Json::as_f64).map(|x| x as u64).filter(|t| keep.contains(t)) { kept.push((t, doc)); } } } + r.end()?; Ok((r.summary(), kept, r.live.p.clone())) }; + replay().map_err(|e| chain_first(&lines, e)) +} +/// One line of a host's action log (`live verify --actions FILE`, §5.7): the decision's turn and, per trait the host acted on, +/// the level it acted on +pub struct Action { pub line: usize, pub turn: u64, pub levels: Vec<(String, String)> } +/// A host's action log: one JSON object per line, {"turn": T, "action": {trait: level, ...}} (blank lines skipped) +pub fn read_actions(text: &str) -> Result, InErr> { + let mut out = vec![]; + for (i, l) in text.split('\n').map(bare).enumerate() { if l.trim().is_empty() { continue; } let path = format!("actions.line{}", i + 1); + let bad = |m: &str| perr(&path, m.to_string()); + let j = json::parse(l).map_err(|e| perr(&path, format!("not JSON: {}", e.msg)))?; + let kv = j.as_obj().filter(|kv| kv.len() == 2 && kv.iter().all(|(k, _)| k == "turn" || k == "action")).ok_or_else(|| bad("an object {\"turn\": T, \"action\": {trait: level}}"))?; + let turn = kv.iter().find(|(k, _)| k == "turn").and_then(|(_, x)| x.as_f64()).filter(|x| *x >= 0.0 && x.fract() == 0.0 && *x <= 9_007_199_254_740_992.0) + .ok_or_else(|| bad("turn: a whole number >= 0 (the stance's turn)"))? as u64; + let act = kv.iter().find(|(k, _)| k == "action").and_then(|(_, x)| x.as_obj()).filter(|a| !a.is_empty()).ok_or_else(|| bad("action: an object trait -> level"))?; + let levels = act.iter().map(|(t, x)| x.as_str().map(|l| (t.clone(), l.to_string())).ok_or_else(|| bad("action: each level a string"))).collect::, InErr>>()?; + out.push(Action { line: i + 1, turn, levels }); } + Ok(out) +} +/// The earliest turn (then the earliest line) whose host action differs from the level its stance released -> (actions +/// checked, the difference: {turn, line, trait, action, released (null: not released or no such turn), why}); an action on a +/// trait the persona does not have is an input error +pub fn first_difference(p: &Persona, actions: &[Action], stances: &[(u64, Json)]) -> Result<(usize, Option), InErr> { + let mut order: Vec<&Action> = actions.iter().collect(); order.sort_by_key(|a| (a.turn, a.line)); + let s = |x: &str| Json::Str(x.to_string()); let mut checked = 0; + for a in order { checked += 1; let doc = stances.iter().find(|(t, _)| *t == a.turn).map(|x| &x.1); + for (t, level) in &a.levels { + let e = match doc { Some(d) => persona::doc_entry(p, d, t), None => None }; + if doc.is_some() && e.is_none() { return Err(perr(&format!("actions.line{}.action.{t}", a.line), "not a trait of the persona")); } + let released = e.as_ref().filter(|e| e.get("released") == Some(&Json::Bool(true))).and_then(|e| e.get("level")).and_then(Json::as_str).map(str::to_string); + let why = match (doc, &released) { (None, _) => "the strand has no stance with this turn", (_, None) => "the stance did not release this trait", + (_, Some(r)) if r != level => "the host acted on another level", _ => continue }; + return Ok((checked, Some(Json::Obj(vec![("turn".into(), Json::Num(a.turn as f64)), ("line".into(), Json::Num(a.line as f64)), ("trait".into(), s(t)), ("action".into(), s(level)), + ("released".into(), released.map_or(Json::Null, |r| s(&r))), ("why".into(), s(why))])))); } } + Ok((checked, None)) +} +/// `verify` as one document: the summary, or `failure`'s document +pub fn verify_doc(text: &str, eng: Engine) -> Json { verify(text, eng).unwrap_or_else(|(line, why)| failure(text, line, &why)) } +/// The document of a strand that does not verify at `line`: {ok: false, line, diverges: what differs}, or, for a field or line +/// kind this engine does not know, {ok: false, line, unsupported: the field's path, engine: this engine, strand_engine: the +/// header's `engine` (null if it names none)} +pub fn failure(text: &str, line: usize, why: &str) -> Json { + let mut v = vec![("ok".into(), Json::Bool(false)), ("line".into(), Json::Num(line as f64))]; + match unsupported_field(why) { + Some(f) => { v.push(("unsupported".into(), Json::Str(f.to_string()))); v.push(("engine".into(), Json::Str(engine()))); v.push(("strand_engine".into(), strand_engine(text))); } + None => v.push(("diverges".into(), Json::Str(why.to_string()))) } + Json::Obj(v) +} +/// The engine a strand's header names: its `engine` string, or null when the header is not JSON or has none +pub fn strand_engine(text: &str) -> Json { + json::parse(lines(text).into_iter().next().unwrap_or("")).ok().and_then(|h| h.get("engine").cloned()).filter(|e| matches!(e, Json::Str(_))).unwrap_or(Json::Null) +} +/// This binary's engine (the strand header's `engine`) +pub fn engine() -> String { format!("probbit {}", env!("CARGO_PKG_VERSION")) } + +/// Validate the stored chain and lifecycle before extending it. This is deliberately not +/// inference replay or authentication: verify still recomputes every event. In particular, a +/// checkpoint cannot reactivate a retired individual or hide an invalid earlier control. +pub fn check_chain(text: &str) -> Result { check_chain_head(text).map(|(s, _, _)| s) } +/// The header checks of `Replay::open` without a replay (the same checks, in its order, with its messages): the header read, its +/// persona built on the heap, its digest and initial state checked, and the header written again piece by piece against the line +/// (no `Live`, no whole header) -> the persona, the header, its initial state and the header's sha256 +#[inline(never)] +fn open_head(head: &str) -> Result<(Box, Json, Box, String), (usize, String)> { + let h = json::parse(head).map_err(|e| (1, format!("the header is not JSON: {}", e.msg)))?; + if h.get("probbit_strand").and_then(Json::as_f64) != Some(FORMAT) { return Err((1, "not a probbit strand (format 1)".into())); } + if let Some((k, _)) = h.as_obj().and_then(|kv| kv.iter().find(|(k, _)| !HEADER.contains(&k.as_str()) && k != "segment")) { return Err((1, unsupported(k))); } + if let Some(sg) = h.get("segment") { check_segment(sg).map_err(|e| (1, format!("the header's segment: {e}")))?; } + let doc = h.get("document").ok_or((1, "the header has no persona document".to_string()))?; + let p = persona::build_boxed(doc).map_err(|e| (1, read_err(e, "the persona document: ")))?; + if h.get("persona").and_then(|x| x.get("digest")).and_then(Json::as_str) != Some(p.digest.as_str()) { return Err((1, "the persona digest does not match the document".into())); } + let st0 = read_boxed(&p, h.get("state").unwrap_or(&Json::Null)).map_err(|e| (1, read_err(e, "the initial state: ")))?; + let (hb, mut at, mut same) = (head.as_bytes(), 0usize, true); + let digest = write_header_seg(&p, doc, &st0, h.get("engine").and_then(Json::as_str).unwrap_or(""), h.get("segment"), HEADER_CHUNK, &mut |c: &[u8]| { same = same && hb.get(at..at + c.len()) == Some(c); at += c.len(); }); + if !same || at != hb.len() { return Err((1, "the header is not as written".into())); } + Ok((p, h, st0, digest)) +} +/// A checkpoint line's own checks (`cp`, without its line ending, at 1-based line `at`; `before` = the line before it) for persona +/// `p`, with `open_at`'s messages: canonical JSON, its four fields, right after an event line (prev), at that event's count, with +/// the stance and the state (read as any state is) that event logs -> the count, the stance and the state. No replay starts here, +/// so it holds for a persona with an expectation block too: the chain check needs the line valid, not a replay start. +#[inline(never)] +fn check_checkpoint(p: &Persona, canon: bool, before: Option<&str>, cp: &str, at: usize) -> Result<(u64, Json, Box, Option), (usize, String)> { + let j = json::parse(cp).map_err(|e| (at, format!("not JSON: {}", e.msg)))?; + // a segment's sealed line (no `before`: the event before it is in the previous segment) is taken on its link + let b = match before { Some(before) => json::parse(before).map_err(|e| (at - 1, format!("not JSON: {}", e.msg)))?, None => Json::Null }; + if persona::canon(&j) != cp { return Err((at, "not canonical JSON".into())); } + if let Some(k) = unknown_key(&j) { return Err((at, unsupported(k))); } + // from 0.14.0 a persona with an expectation block also carries `expect` (the canonical fold's memory) + let with = canon && p.expect.is_some(); + if !j.as_obj().is_some_and(|kv| kv.len() == 4 + usize::from(with) && kv.iter().all(|(k, _)| ["checkpoint", "prev", "stance", "state"].contains(&k.as_str()) || (with && k == "expect"))) { + return Err((at, if with { "checkpoint fields must be checkpoint, expect, prev, stance and state" } else { "checkpoint fields must be checkpoint, prev, stance and state" }.into())); + } + // §2.18: an event line of a persona with glyphs also carries `glyph`; RFC 0009: of a persona with an expectation block, `deviation` + // RFC 0016: of a persona with a curiosity block, `curiosity` + let extra = |kv: &[(String, Json)]| kv.iter().filter(|(k, _)| k == "glyph" || k == "deviation" || k == "integrity" || k == "curiosity").count(); + if let Some(before) = before { + if !b.as_obj().is_some_and(|kv| kv.len() == 5 + extra(kv) && kv.iter().all(|(k, _)| ["n", "inputs", "prev", "stance", "state", "glyph", "deviation", "integrity", "curiosity"].contains(&k.as_str()))) { + return Err((at - 1, "a checkpoint must immediately follow an event line".into())); + } + if j.get("prev").and_then(Json::as_str) != Some(persona::digest_of(before).as_str()) { return Err((at, "prev is not the sha256 of the line before".into())); } + } + let n = j.get("checkpoint").and_then(Json::as_f64).filter(|x| *x >= 1.0 && x.fract() == 0.0).ok_or((at, "not a checkpoint line".to_string()))? as u64; + if before.is_some() && b.get("n").and_then(Json::as_f64) != Some(n as f64) { return Err((at, format!("the line before is not event {n}"))); } + let stance = j.get("stance").cloned().unwrap_or(Json::Null); + if before.is_some() && b.get("stance").and_then(Json::as_str) != Some(persona::stance_sha(&stance).as_str()) { return Err((at, "the checkpoint's stance is not the one event {n} logs".replace("{n}", &n.to_string()))); } + let st = read_boxed(p, j.get("state").unwrap_or(&Json::Null)).map_err(|e| (at, read_err(e, "the checkpoint's state: ")))?; + if before.is_some() && b.get("state").and_then(Json::as_str) != Some(st.digest.as_str()) { return Err((at, format!("the checkpoint's state is not the one event {n} logs"))); } + let mem = if with { Some(persona::ExpectMem::read(p, j.get("expect").unwrap_or(&Json::Null)).map_err(|e| (at, format!("the checkpoint's expect: {e}")))?) } else { None }; + Ok((n, stance, st, mem)) +} +#[inline(never)] +fn read_boxed(p: &Persona, j: &Json) -> Result, InErr> { State::read(p, j).map(Box::new) } +/// `check_chain` -> the status, the header's persona and the header (`resume_boxed` continues with them) +#[inline(never)] +fn check_chain_head(text: &str) -> Result<(Status, Box, Json), InErr> { + let ls = lines(text); + let err = |line: usize, why: String| perr("strand", format!("line {line}: {why}")); + let (hp, h, _, _) = open_head(ls.first().copied().unwrap_or("")).map_err(|(i, why)| err(i, why))?; + let canon = !engine_version(h.get("engine").and_then(Json::as_str).unwrap_or("")).is_some_and(|v| v < DEV_R6_SINCE); + let mut status = Status::Active; + let mut n = 0u64; + let mut from = 1; + // a segment: its second line is the sealed line, taken on its link (the header's `sealed`); the chain goes on from it + if let Some(sg) = h.get("segment") { + if hp.expect.is_some() && !canon { return Err(err(1, "a segment of a persona with an expectation block needs a strand of engine 0.14.0 or later".into())); } + let cp = ls.get(1).copied().unwrap_or(""); + if Some(persona::digest_of(cp).as_str()) != sg.get("sealed").and_then(Json::as_str) { return Err(err(2, SEALED.into())); } + let (cn, _, st, _) = check_checkpoint(&hp, canon, None, cp, 2).map_err(|(at, why)| err(at, why))?; + if Some(st.digest.as_str()) != h.get("state").and_then(|s| s.get("digest")).and_then(Json::as_str) { return Err(err(2, "the sealed line's state is not the segment header's".into())); } + n = cn; from = 2; + } + for i in from..ls.len() { + let j = json::parse(ls[i]).map_err(|e| err(i + 1, e.msg))?; + if persona::canon(&j) != ls[i] { return Err(err(i + 1, "not canonical JSON".into())); } + if j.get("prev").and_then(Json::as_str) != Some(persona::digest_of(ls[i - 1]).as_str()) { + return Err(err(i + 1, "prev is not the sha256 of the line before".into())); + } + if let Some(k) = unknown_key(&j) { return Err(err(i + 1, unsupported(k))); } + match kind(&j) { + Kind::Control => { + let f = |k: &str| j.get(k).and_then(Json::as_str).unwrap_or(""); + status = status.after(f("control")).map_err(|e| err(i + 1, e.msg))?; + let line = control_line(f("prev"), f("control"), f("by"), f("reason"), f("at")).map_err(|e| err(i + 1, e.msg))?; + if line != ls[i] { return Err(err(i + 1, "the control line differs".into())); } + } + Kind::Event => { + if status != Status::Active { return Err(err(i + 1, format!("event while {}", status.name()))); } + n += 1; + if j.get("n").and_then(Json::as_f64) != Some(n as f64) { return Err(err(i + 1, format!("n is not {n}"))); } + let Some(Json::Obj(_)) = j.get("inputs") else { return Err(err(i + 1, "no input object".into())) }; + for k in ["state", "stance"] { + if !j.get(k).and_then(Json::as_str).is_some_and(|s| s.len() == 71 && s.starts_with("sha256:") && s[7..].bytes().all(|c| c.is_ascii_hexdigit())) { + return Err(err(i + 1, format!("invalid {k} digest"))); + } + } + } + Kind::Reflection => { if !canon { return Err(err(i + 1, REFLECT_OLD.into())); } } + Kind::Consolidation => { + if !canon { return Err(err(i + 1, CONSOLIDATE_OLD.into())); } + if status == Status::Retired { return Err(err(i + 1, "consolidation line while retired".into())); } + if j.get("n").and_then(Json::as_f64) != Some(n as f64) { return Err(err(i + 1, format!("the consolidation line is not at event {n}"))); } + if !j.get("state").and_then(Json::as_str).is_some_and(|s| s.len() == 71 && s.starts_with("sha256:") && s[7..].bytes().all(|c| c.is_ascii_hexdigit())) { return Err(err(i + 1, "invalid state digest".into())); } + } + Kind::Rest => { + if status != Status::Active { return Err(err(i + 1, format!("rest line while {}", status.name()))); } + if !ls.get(i + 1).and_then(|x| json::parse(x).ok()).is_some_and(|x| matches!(kind(&x), Kind::Event)) { return Err(err(i + 1, "a rest line must be followed by its event line".into())); } + } + Kind::Checkpoint => { + if status != Status::Active { return Err(err(i + 1, format!("checkpoint while {}", status.name()))); } + // Full state validation plus links to the immediately preceding event; no inference and no replay start, so a + // persona with an expectation block (whose replay cannot start at a checkpoint) resumes after one as well + let (cn, _, _, _) = check_checkpoint(&hp, canon, Some(ls[i - 1]), ls[i], i + 1).map_err(|(at, why)| err(at, why))?; + if cn != n { return Err(err(i + 1, "checkpoint event count differs".into())); } + } + } + } + Ok((status, hp, h)) +} + +/// Continue a strand: `text` is the strand so far and `st` the individual after its last line -> the individual, ready for the +/// next event (no header to write). The header's persona must be `p` (the same digest); the run uses the header's document, the +/// key order the strand replays with. The lines before are not replayed here (`verify` does that). +pub fn resume(text: &str, p: &Persona, st: State, clock: Clock) -> Result { resume_boxed(text, p, Box::new(st), clock).map(|lv| *lv) } +/// `resume` on the heap: the state comes in boxed and the individual goes out boxed (docs/embedding.md "Stack budget") +#[inline(never)] +pub fn resume_boxed(text: &str, p: &Persona, st: Box, clock: Clock) -> Result, InErr> { + let bad = |m: String| perr("strand", m); + if !text.ends_with('\n') { return Err(bad("its last line is incomplete (no newline at the end)".into())); } + // the chain check opens the header (its persona built once, on the heap): resume continues with that persona + let (_, q, h) = check_chain_head(text)?; + let lines = lines(text); + if q.digest != p.digest { return Err(bad(format!("it is another persona's ({} != {})", &q.digest[..19], &p.digest[..19]))); } + let last = lines[lines.len() - 1]; + // the control lines after the last event or checkpoint line (the anchor; the header without either) set the status: an + // event or a checkpoint is only ever written while active + let (mut a, mut tail) = (lines.len() - 1, vec![]); + while a > 0 { let j = json::parse(lines[a]).map_err(|e| bad(format!("line {} is not JSON: {}", a + 1, e.msg)))?; + // RFC 0013: a reflection line changes nothing: the anchor is before it + if matches!(kind(&j), Kind::Reflection) { a -= 1; continue; } + if !matches!(kind(&j), Kind::Control) { break; } tail.push(j); a -= 1; } + let (n, want) = if a == 0 { (0, h.get("state").and_then(|s| s.get("digest")).and_then(Json::as_str).unwrap_or("").to_string()) } else { + let j = json::parse(lines[a]).map_err(|e| bad(format!("line {} is not JSON: {}", a + 1, e.msg)))?; + match kind(&j) { Kind::Checkpoint => (j.get("checkpoint").and_then(Json::as_f64).unwrap_or(0.0) as u64, j.get("state").and_then(|s| s.get("digest")).and_then(Json::as_str).unwrap_or("").to_string()), + _ => (j.get("n").and_then(Json::as_f64).unwrap_or(0.0) as u64, j.get("state").and_then(Json::as_str).unwrap_or("").to_string()) } }; + if st.digest != want { return Err(perr("state", format!("not the strand's last state ({} != {})", &st.digest[..st.digest.len().min(19)], &want[..want.len().min(19)]))); } + // RFC 0009: the expectation's memory, folded from every event line's inputs and logged deviation; from 0.14.0 from the last + // checkpoint's (the chain check read it), folding only the event lines after it + let dev_full = engine_version(h.get("engine").and_then(Json::as_str).unwrap_or("")).is_some_and(|v| v < DEV_R6_SINCE); + let expect = if q.expect.is_none() { None } else { Some(match last_checkpoint(&lines, 0).filter(|_| !dev_full) { + Some(k) => { let j = json::parse(lines[k]).map_err(|e| bad(format!("line {} is not JSON: {}", k + 1, e.msg)))?; + let m = persona::ExpectMem::read(&q, j.get("expect").unwrap_or(&Json::Null)).map_err(|e| bad(format!("line {}: the checkpoint's expect: {e}", k + 1)))?; + expect_fold(&q, m, &lines[k + 1..]) } + None => expect_fold(&q, new_mem(dev_full), &lines[1..]) }) }; + let mut lv = Live::boxed(q, st, clock, persona::digest_of(last), n, expect, dev_full); + for c in tail.iter().rev() { let what = c.get("control").and_then(Json::as_str).unwrap_or(""); + lv.status = lv.status.after(what).unwrap_or(Status::Retired); lv.st = lv.st.zero_credit(&lv.p); lv.controls += 1; } + Ok(lv) +} + +/// Continue a strand from a state record that may be behind it (docs/persona.md §5.7, "Resume from a record"): `st` is the state +/// some line logs (an event line's `state`, a checkpoint's, or the header's), not necessarily the last; a reboot between a line +/// and its record leaves the record k lines behind. The whole strand is chain-checked as by `resume`; then the walk starts at the +/// last checkpoint at or before the record's line (the header without one), and every event line after the record's line is +/// **applied** to the record: its turn run on the logged inputs (the main solve; the expectation's member solves are not run: the +/// line's logged deviation is folded as `resume` folds it), the stance and state digests and the line's bytes checked (its logged +/// `deviation` and `glyph` taken as written: `verify` recomputes them), rest and control lines applied as the live writer applies +/// them. -> the individual after the strand's last line, as a header replay reaches it, byte for byte. Refused, with the line, +/// when the record is no state the strand logs, a line differs from its application, or the persona is another one. +#[inline(never)] +pub fn resume_from_record(text: &str, p: &Persona, st: Box, clock: Clock, eng: Engine) -> Result, InErr> { + let bad = |m: String| perr("strand", m); + if !text.ends_with('\n') { return Err(bad("its last line is incomplete (no newline at the end)".into())); } + let (_, q, h) = check_chain_head(text)?; + let ls = lines(text); + // the record's line: the last whose logged state is the record (each logs its own: the state carries the turn) + let want = format!("\"{}\"", st.digest); + let a = (1..ls.len()).rev().find(|&i| ls[i].contains(&want)).unwrap_or(0); + let dev_full = engine_version(h.get("engine").and_then(Json::as_str).unwrap_or("")).is_some_and(|v| v < DEV_R6_SINCE); + // the walk starts at the last checkpoint at or before it (a segment's sealed line at the latest), whose memory it reads; the + // header otherwise (and for an expectation block before 0.14.0, whose checkpoints do not carry the memory) + let c = last_checkpoint(&ls[..=a.max(1).min(ls.len() - 1)], 0).filter(|_| !(dev_full && q.expect.is_some()) || h.get("segment").is_some()); + let (cp, from) = match c { Some(c) => (Some((ls[c], c + 1)), c + 1), None => (None, 1) }; + resume_walk(q, &h, ls[0], cp, ls[from..].iter().copied(), p, st, clock, eng) +} +/// The embedded resume (docs/embedding.md "Resume on a body"): continue a strand from its header line `head` (the persona +/// document, the engine and the initial state it carries), its last checkpoint line `cp` (None before the first: the walk starts +/// at the header) and the lines after that checkpoint, read one at a time from `trailing` (a host reads them from its store at the +/// checkpoint's offset), never the whole strand. `line` is the checkpoint's 1-based line number in the strand, for messages (0: +/// lines are counted from the checkpoint). The header is checked as `verify` checks it (built once, written again in pieces); the +/// checkpoint line by its own fields (canonical, its state and memory read, at an event count): its link to the line before it is +/// the host's (the offset it stored), as a segment's sealed line is taken on its link. Each trailing line is chain-checked as +/// `resume` checks it, and the record `st` may be any state logged from the checkpoint on (the event lines after it are applied, as +/// by `resume_from_record`). Cost: O(trailing lines), plus the header's check and the persona's build. +#[inline(never)] +#[allow(clippy::too_many_arguments)] +pub fn resume_boxed_from>(head: &str, cp: Option<&str>, line: usize, trailing: impl IntoIterator, p: &Persona, st: Box, clock: Clock, eng: Engine) -> Result, InErr> { + let (q, h, _, _) = open_head(head).map_err(|(i, why)| perr("strand", format!("line {i}: {why}")))?; + let dev_full = engine_version(h.get("engine").and_then(Json::as_str).unwrap_or("")).is_some_and(|v| v < DEV_R6_SINCE); + if cp.is_some() && dev_full && q.expect.is_some() { + return Err(perr("strand", "before engine 0.14.0 a checkpoint does not carry the expectation's memory: resume from the whole strand")); + } + if cp.is_none() && h.get("segment").is_some() { return Err(perr("strand", "a segment resumes from its sealed line (line 2), not from its header")); } + resume_walk(q, &h, head, cp.map(|c| (c, line.max(1))), trailing, p, st, clock, eng) +} +/// The walk both resumes share: from the header (`cp` None) or a checkpoint line (and its line number), over the lines after it. +/// Every line is chain-checked; the record `st` anchors at the line that logs its state; the event lines after that line are +/// applied (`apply_line`); the expectation's memory is the start's, folded over every event line. +#[inline(never)] +#[allow(clippy::too_many_arguments)] +fn resume_walk>(q: Box, h: &Json, head: &str, cp: Option<(&str, usize)>, trailing: impl IntoIterator, p: &Persona, st: Box, clock: Clock, eng: Engine) -> Result, InErr> { + let sw = crate::trace::Stopwatch::start(); + let mut tr = crate::trace::ResumeTrace::default(); + let err = |line: usize, why: String| perr("strand", format!("line {line}: {why}")); + if q.digest != p.digest { return Err(perr("strand", format!("it is another persona's ({} != {})", &q.digest[..19], &p.digest[..19]))); } + let dev_full = engine_version(h.get("engine").and_then(Json::as_str).unwrap_or("")).is_some_and(|v| v < DEV_R6_SINCE); + let canon = !dev_full; + let (mut n, mut prev_line, logged, mut mem, mut at) = match cp { + None => (0, head.to_string(), h.get("state").and_then(|s| s.get("digest")).and_then(Json::as_str).unwrap_or("").to_string(), q.expect.as_ref().map(|_| new_mem(dev_full)), 1), + Some((c, at)) => { let (cn, _, cst, cmem) = check_checkpoint(&q, canon, None, c, at).map_err(|(i, why)| err(i, why))?; + let mem = if q.expect.is_some() { cmem } else { None }; + (cn, c.to_string(), cst.digest.clone(), mem, at) } }; + let mut prev = persona::digest_of(&prev_line); + // the record: anchored once the walk reaches the line that logs its state; from there on it is the individual's state + let mut rec = Some(st); + let mut cur: Option> = if rec.as_ref().is_some_and(|s| s.digest == logged) { rec.take() } else { None }; + let (mut status, mut tail, mut rest) = (Status::Active, 0u64, None::); + for l in trailing { + let l = l.as_ref(); at += 1; tr.lines += 1; + let j = json::parse(l).map_err(|e| err(at, format!("not JSON: {}", e.msg)))?; + if persona::canon(&j) != l { return Err(err(at, "not canonical JSON".into())); } + if j.get("prev").and_then(Json::as_str) != Some(prev.as_str()) { return Err(err(at, "prev is not the sha256 of the line before".into())); } + if let Some(k) = unknown_key(&j) { return Err(err(at, unsupported(k))); } + if rest.is_some() && !matches!(kind(&j), Kind::Event) { return Err(err(at - 1, "a rest line must be followed by its event line".into())); } + match kind(&j) { + Kind::Rest => { if status != Status::Active { return Err(err(at, format!("rest line while {}", status.name()))); } rest = Some(l.to_string()); } + Kind::Reflection => { if !canon { return Err(err(at, REFLECT_OLD.into())); } } + // RFC 0015: the pass applied to the memory (and to the state once anchored) as the live writer applies it + Kind::Consolidation => { + if !canon { return Err(err(at, CONSOLIDATE_OLD.into())); } + if status == Status::Retired { return Err(err(at, "consolidation line while retired".into())); } + let m = mem.as_mut().ok_or_else(|| err(at, "a consolidation line for a persona without an expectation block".into()))?; + let c = persona::rest::consolidate_mem(&q, m).map_err(|why| err(at, why))?; + let logged = j.get("state").and_then(Json::as_str).unwrap_or(""); + match cur.as_mut() { + Some(s) => { persona::rest::decay_learned(&q, s); + let line = persona::rest::line(&prev, j.get("at").and_then(Json::as_str).unwrap_or(""), n, c, &s.digest); + if line != l { return Err(err(at, format!("the consolidation line differs at {}", persona::rest::first_field(&line, l)))); } } + None => if rec.as_ref().is_some_and(|s| s.digest == logged) { cur = rec.take(); } } } + Kind::Control => { let f = |k: &str| j.get(k).and_then(Json::as_str).unwrap_or(""); + status = status.after(f("control")).map_err(|e| err(at, e.msg))?; + let line = control_line(f("prev"), f("control"), f("by"), f("reason"), f("at")).map_err(|e| err(at, e.msg))?; + if line != l { return Err(err(at, "the control line differs".into())); } + // as the live writer: no credit crosses a control line; `erase` returns the learned weights to zero + if let Some(s) = cur.as_mut() { let mut z = Box::new(s.zero_credit(&q)); if f("control") == "erase" { z = Box::new(z.erased(&q)); } *s = z; } + tail += 1; } + Kind::Checkpoint => { + if status != Status::Active { return Err(err(at, format!("checkpoint while {}", status.name()))); } + let (cn, _, cst, cmem) = check_checkpoint(&q, canon, Some(&prev_line), l, at).map_err(|(i, why)| err(i, why))?; + if cn != n { return Err(err(at, "checkpoint event count differs".into())); } + if q.expect.is_some() && cmem.is_some() { mem = cmem; } + if cur.is_none() && rec.as_ref().is_some_and(|s| s.digest == cst.digest) { cur = rec.take(); } + tail = 0; } + Kind::Event => { + if status != Status::Active { return Err(err(at, format!("event while {}", status.name()))); } + n += 1; + if j.get("n").and_then(Json::as_f64) != Some(n as f64) { return Err(err(at, format!("n is not {n}"))); } + let Some(Json::Obj(_)) = j.get("inputs") else { return Err(err(at, "no input object".into())) }; + for k in ["state", "stance"] { + if !j.get(k).and_then(Json::as_str).is_some_and(|s| s.len() == 71 && s.starts_with("sha256:") && s[7..].bytes().all(|c| c.is_ascii_hexdigit())) { + return Err(err(at, format!("invalid {k} digest"))); + } + } + let logged = j.get("state").and_then(Json::as_str).unwrap_or(""); + match cur.as_mut() { + Some(s) => { let t0 = crate::trace::Stopwatch::start(); + // RFC 0016: the drive's level from the fold before this line, as the live writer set it + s.cur = mem.as_ref().and_then(|m| persona::cur_level(&q, m)); + *s = apply_line(&q, s, &j, l, rest.as_deref(), eng).map_err(|why| err(at, why))?; + tr.applied += 1; tr.applied_us += t0.us(); } + None => if rec.as_ref().is_some_and(|s| s.digest == logged) { cur = rec.take(); } } + rest = None; + if let Some(m) = mem.as_mut() { persona::expect_step(&q, m, j.get("inputs").unwrap_or(&Json::Null), logged_pred(&j)); } + tail = 0; } } + prev = persona::digest_of(l); prev_line.clear(); prev_line.push_str(l); + } + if rest.is_some() { return Err(err(at, "a rest line must be followed by its event line".into())); } + let Some(st) = cur else { + let d = rec.as_ref().map_or("", |s| s.digest.as_str()); + return Err(perr("state", format!("not a state this strand logs from where the resume starts ({}): neither the start's nor any line's after it", &d[..d.len().min(19)]))); + }; + let mut lv = Live::boxed(q, st, clock, prev, n, mem, dev_full); + lv.status = status; lv.controls = tail; + tr.total_us = sw.us(); crate::trace::resumed(tr); + Ok(lv) +} +/// One recorded event line applied to the state before it (`resume_from_record`): the line's inputs stamped as the live writer +/// stamps them on the fixed clock (its logged `elapsed_hours`), a rest line due before it recomputed against the logged one, the +/// turn run, and the line written again from what the turn gives (n, inputs, stance digest, state digest, prev) with the logged +/// `deviation` and `glyph` -> the state after it, or what differs. Without the expectation's member solves: the deviation and +/// the glyph that reads it are the line's. +#[inline(never)] +fn apply_line(p: &Persona, st: &State, j: &Json, l: &str, rest: Option<&str>, eng: Engine) -> Result, String> { + let inputs = stamp_logged(j)?; + rest_check(p, st, &inputs, rest)?; + // the turn from the smallest frame: its document and state only (the checks are their own frames, before and after) + let (stance, ns) = persona::turn_boxed(p, st, &inputs, false, eng, false).map_err(|e| read_err(e, ""))?; + line_check(j, l, inputs, &stance, &ns.digest)?; + Ok(ns) +} +/// A logged event's inputs stamped as the live writer stamps them on the fixed clock (its `elapsed_hours`, quantised, last) +#[inline(never)] +fn stamp_logged(j: &Json) -> Result { + let ev = j.get("inputs").unwrap_or(&Json::Null); + let Json::Obj(kv) = ev else { return Err("no input object".into()) }; + let mut kv: Vec<(String, Json)> = kv.iter().filter(|(k, _)| k != "elapsed_hours").cloned().collect(); + let e = match ev.get("elapsed_hours") { None => 0.0, Some(Json::Num(x)) if x.is_finite() && *x >= 0.0 => *x, _ => return Err("event.elapsed_hours: a number >= 0".into()) }; + kv.push(("elapsed_hours".into(), Json::Num(q6(e)))); Ok(Json::Obj(kv)) +} +/// A rest line due before the event (RFC 0002 part 2) recomputed against the logged one +#[inline(never)] +fn rest_check(p: &Persona, st: &State, inputs: &Json, rest: Option<&str>) -> Result<(), String> { + match (persona::rest_line_digest(p, st, inputs), rest) { + (Some((r, rd)), Some(lr)) => { let rp = json::parse(lr).ok().and_then(|x| x.get("prev").and_then(Json::as_str).map(str::to_string)).unwrap_or_default(); + if rest_line(&rp, r, &rd) != lr { return Err("the rest line differs".into()); } Ok(()) } + (Some(_), None) => Err("a rest line is due before this event and missing".into()), + (None, Some(_)) => Err("a rest line where none is due".into()), + (None, None) => Ok(()) } +} +/// The applied event's line against the logged one: its stance and state digests, then its bytes (n, inputs, stance, state, +/// prev as the turn gives them; the logged `deviation` and `glyph` as written) +#[inline(never)] +fn line_check(j: &Json, l: &str, inputs: Json, stance: &Json, state: &str) -> Result<(), String> { + let sd = persona::stance_sha(stance); + if j.get("stance").and_then(Json::as_str) != Some(sd.as_str()) { return Err("the stance differs".into()); } + if j.get("state").and_then(Json::as_str) != Some(state) { return Err("the state differs".into()); } + let mut lj = vec![("n".into(), j.get("n").cloned().unwrap_or(Json::Null)), ("inputs".into(), inputs), ("stance".into(), Json::Str(sd)), + ("state".into(), Json::Str(state.to_string())), ("prev".into(), j.get("prev").cloned().unwrap_or(Json::Null))]; + for k in ["deviation", "curiosity", "glyph", "integrity"] { if let Some(x) = j.get(k) { lj.push((k.into(), x.clone())); } } + if persona::canon(&Json::Obj(lj)) != l { return Err("the line differs".into()); } + Ok(()) +} +/// RFC 0009: an event line's logged prediction (none on a cold start), the value the expectation's memory folds +fn logged_pred(j: &Json) -> Option { j.get("deviation").filter(|d| d.get("cold").is_none()).and_then(|d| d.get("pred")).and_then(Json::as_f64) } + +/// The expectation's empty memory: the canonical fold from 0.14.0 (`dev_full` false), the fold before it otherwise +fn new_mem(dev_full: bool) -> persona::ExpectMem { if dev_full { persona::ExpectMem::new() } else { persona::ExpectMem::canonical() } } +/// RFC 0009: the expectation's memory `m` with the event lines of `lines` folded in, one line read at a time +#[inline(never)] +fn expect_fold(q: &Persona, mut m: persona::ExpectMem, lines: &[&str]) -> persona::ExpectMem { + for l in lines.iter() { if let Ok(j) = json::parse(l) { match kind(&j) { + Kind::Event => persona::expect_step(q, &mut m, j.get("inputs").unwrap_or(&Json::Null), logged_pred(&j)), + // RFC 0015: the rest pass on the memory (the chain check has read the line) + Kind::Consolidation => { let _ = persona::rest::consolidate_mem(q, &mut m); } + _ => {} } } } + m +} +/// The audit of a recorded life (`probbit persona bound PERSONA --audit STRAND`): the strand replayed line by line; after every +/// event and control line, per learned trait the resting TV between the individual and the same individual with its learned +/// weights at zero, as a share of its epsilon (the room used). An exceedance is impossible by the theorem; the audit shows it +/// on the recorded life. +pub fn bound_audit(text: &str, b: &persona::bound::Bound, eng: Engine) -> Result<(Json, bool), (usize, String)> { + let lines = lines(text); + let (mut r, k0) = Replay::open_lines(&lines)?; + let p = r.live.p.clone(); let (traits, cap) = persona::bound::learned_traits(&p).ok_or((1, "the strand's persona has no learning block: nothing to audit".to_string()))?; + let rest = b.rest_hours.unwrap_or_else(|| persona::bound::default_rest(&p)); let seed = r.live.st.seed; + let e = |x: InErr| (0usize, format!("{}: {}", x.path, x.msg)); + let st0 = persona::init(&p, Some(seed), true, eng); let p0 = persona::bound::rest_odds(&p, &st0, rest, eng).map_err(e)?; + let mut eps = vec![]; let mut rr = vec![]; + for (k, t) in traits.iter().enumerate() { let sc = persona::bound::scope(&p, b, t, rest, Some(&st0)); + let bs = persona::bound::bases(&p, &sc, &st0).ok_or((0, format!("{t}: more than {} enumerated history combinations: the audit needs an exact epsilon", persona::bound::MAX_HISTORY)))?; + eps.push(persona::bound::search(&p, k, &sc.coords, cap, &bs, &p0[k], rest, eng).map_err(e)?.0); rr.push((sc.d_r / 4.0).tanh()); } + let nt = traits.len(); let (mut maxr, mut at_n, mut cur, mut exceed) = (vec![0.0f64; nt], vec![0u64; nt], vec![0.0f64; nt], 0usize); + let (mut events, mut feedback, mut erasures) = (0u64, 0u64, 0u64); let mut last = persona::bound::learned_of(&r.live.st); + let measure = |st: &persona::State| -> Result, (usize, String)> { let a = persona::bound::rest_odds(&p, st, rest, eng).map_err(e)?; + let z = persona::bound::rest_odds(&p, &persona::bound::with_learned(&p, st, &persona::bound::learned_of(st).iter().map(|d| vec![0.0; d.len()]).collect::>()), rest, eng).map_err(e)?; + Ok(a.iter().zip(&z).map(|(x, y)| persona::bound::tv(x, y)).collect()) }; + for l in lines.iter().skip(k0) { if l.trim().is_empty() { continue; } + let was_event = r.step(l, eng)?.is_some(); if was_event { events += 1; } else if l.contains(r#""control":"erase""#) { erasures += 1; } + let now = persona::bound::learned_of(&r.live.st); if now != last { if was_event { feedback += 1; } last = now; } + let tvs = measure(&r.live.st)?; + for k in 0..nt { cur[k] = tvs[k]; if eps[k] > 0.0 && tvs[k] / eps[k] > maxr[k] { maxr[k] = tvs[k] / eps[k]; at_n[k] = r.live.n; } if tvs[k] > eps[k] + rr[k] + 1e-6 { exceed += 1; } } } + let s = |x: &str| Json::Str(x.to_string()); + let rows: Vec = traits.iter().enumerate().map(|(k, t)| Json::Obj(vec![("trait".into(), s(t)), ("epsilon".into(), Json::Num(eps[k])), + ("room_used_max".into(), Json::Num(persona::r6(maxr[k]))), ("at_event".into(), Json::Num(at_n[k] as f64)), ("room_used_now".into(), Json::Num(persona::r6(if eps[k] > 0.0 { cur[k] / eps[k] } else { 0.0 }))), + ("distance_to_ceiling".into(), Json::Num(persona::r6(eps[k] - cur[k]))), ("tv_now".into(), Json::Num(persona::r6(cur[k])))])).collect(); + let ok = exceed == 0; + Ok((Json::Obj(vec![("persona".into(), s(&p.name)), ("digest".into(), s(&p.digest)), ("seed".into(), Json::Num(seed as f64)), ("rest_hours".into(), Json::Num(rest)), + ("events".into(), Json::Num(events as f64)), ("feedback_events".into(), Json::Num(feedback as f64)), ("erasures".into(), Json::Num(erasures as f64)), + ("traits".into(), Json::Arr(rows)), ("exceedances".into(), Json::Num(exceed as f64)), ("ok".into(), Json::Bool(ok))]), ok)) +} diff --git a/probbit-cli/src/sys.rs b/probbit-persona/src/sys.rs similarity index 98% rename from probbit-cli/src/sys.rs rename to probbit-persona/src/sys.rs index 1001905..816c67e 100644 --- a/probbit-cli/src/sys.rs +++ b/probbit-persona/src/sys.rs @@ -1,4 +1,6 @@ //! Process resource usage without crates: getrusage(2) via FFI on 64-bit Unix (macOS and Linux struct layouts); None elsewhere. +#[allow(unused_imports)] +use std::prelude::rust_2021::*; // 64-bit only: the layouts below assume 64-bit time_t / long (32-bit Linux or BSD would read garbage telemetry) #[cfg(all(unix, target_pointer_width = "64"))] mod imp { diff --git a/probbit-persona/src/trace.rs b/probbit-persona/src/trace.rs new file mode 100644 index 0000000..9617ca9 --- /dev/null +++ b/probbit-persona/src/trace.rs @@ -0,0 +1,128 @@ +//! `turn_trace` (feature, off by default): where one event's time and bytes go, by phase, so a host measures the same breakdown +//! on its own hardware (docs/embedding.md "The turn's phases"). With the feature off every hook is an empty inline function. +//! +//! `Live::event` opens a trace and closes it on its way out; between, each phase is a segment of the installed clock +//! (`probbit_core::rt` on `nostd`). After the event `last()` holds per phase the microseconds spent in it and, when the host +//! installed a probe (`set_probe`), the largest value the probe returned at the end of one of its segments: a host reads its own +//! counter there (its allocator's peak since the last call, the stack's high-water). Turns outside `Live::event` are not traced. +//! The parse of the event line and the append of the strand line are the host's, before and after the call. +#[allow(unused_imports)] +use std::prelude::rust_2021::*; + +/// A phase of one event +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Phase { + /// the event's inputs read and stamped (elapsed hours) + Input = 0, + /// RFC 0009: the deviation and the trajectory (exact solves on the state before the event) + Expect = 1, + /// the steps before the compiler: provenance, payments, rest, senses / needs / drives / pace, credit, the surprise gate + Prep = 2, + /// the persona's program for the event (and for a yield or a hold) + Compile = 3, + /// the engine's calls + Solve = 4, + /// the engine's answer read back into a stance and the next state + Decode = 5, + /// the stance document: its blocks, credit, glyph and digest + Document = 6, + /// the strand line (and a rest line due before it) and its digest + Line = 7, + /// RFC 0009: the expectation's memory after the event + Fold = 8, +} +/// The number of phases +pub const PHASES: usize = 9; +/// The phases' names, in `Phase` order +pub const NAMES: [&str; PHASES] = ["input", "expect", "prep", "compile", "solve", "decode", "document", "line", "fold"]; + +/// One event's breakdown: per phase (indexed by `Phase as usize`) the microseconds and the largest probe value; the event's +/// total microseconds and the engine calls of the turn; RFC 0009: the expectation's programs answered from its cache +/// (`expect_cached`) and solved (`expect_solved`), and the part of `expect` spent building the members' programs +/// (`expect_build_us`) and solving them (`expect_solve_us`, cache lookups included) +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct TurnTrace { pub us: [u64; PHASES], pub bytes: [u64; PHASES], pub total_us: u64, pub solves: u32, pub expect_cached: u32, pub expect_solved: u32, pub expect_build_us: u64, pub expect_solve_us: u64 } + +/// A resume's walk (`strand::resume_from_record`, `strand::resume_boxed_from`): the lines read after its start, the event lines +/// applied to a record that was behind (each one turn without the expectation's member solves) and their microseconds, and the +/// whole call's microseconds (with the feature off: the counts, zero microseconds) +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ResumeTrace { pub lines: u64, pub applied: u64, pub applied_us: u64, pub total_us: u64 } + +#[cfg(feature = "turn_trace")] +mod on { + use super::*; + use probbit_core::rt::Instant; + use std::cell::RefCell; + struct S { open: Option<(Instant, u64)>, cur: Phase, acc: TurnTrace, last: TurnTrace, probe: Option u64> } + thread_local! { static T: RefCell = RefCell::new(S { open: None, cur: Phase::Input, acc: TurnTrace::default(), last: TurnTrace::default(), probe: None }); } + fn us(t0: Instant) -> u64 { (t0.elapsed().as_secs_f64() * 1e6) as u64 } + /// The breakdown of the last traced event on this thread (zeros before the first) + pub fn last() -> TurnTrace { T.with(|t| t.borrow().last) } + /// Install (or remove) the probe: called with the phase at the end of each of its segments; its value is kept per phase as + /// the largest seen in the event. Its own time is not charged to the phase. + pub fn set_probe(f: Option u64>) { T.with(|t| t.borrow_mut().probe = f) } + pub(crate) fn begin() { T.with(|t| { let mut t = t.borrow_mut(); t.acc = TurnTrace::default(); t.cur = Phase::Input; t.open = Some((Instant::now(), 0)); }) } + /// End the current segment and start one of `ph` -> the phase that was current (a call into the engine restores it after) + pub(crate) fn enter(ph: Phase) -> Phase { + T.with(|t| { let mut g = t.borrow_mut(); let Some((t0, at)) = g.open else { return ph }; + let (prev, probe) = (g.cur, g.probe); let now = us(t0); g.acc.us[prev as usize] += now.saturating_sub(at); + if ph == Phase::Solve { g.acc.solves += 1; } + drop(g); + let b = probe.map(|f| f(prev)); + let mut g = t.borrow_mut(); if let Some(b) = b { let x = &mut g.acc.bytes[prev as usize]; *x = (*x).max(b); } + g.open = Some((t0, us(t0))); g.cur = ph; prev }) + } + thread_local! { static RT: RefCell = RefCell::new(ResumeTrace::default()); } + /// The last resume's walk on this thread (zeros before the first) + pub fn last_resume() -> ResumeTrace { RT.with(|r| *r.borrow()) } + pub(crate) fn resumed(t: ResumeTrace) { RT.with(|r| *r.borrow_mut() = t) } + /// A clock read for the resume's counts + pub(crate) struct Stopwatch(Instant); + impl Stopwatch { pub(crate) fn start() -> Stopwatch { Stopwatch(Instant::now()) } pub(crate) fn us(&self) -> u64 { us(self.0) } } + pub(crate) fn expect_cache(hits: u32, solves: u32) { T.with(|t| { let mut g = t.borrow_mut(); if g.open.is_some() { g.acc.expect_cached += hits; g.acc.expect_solved += solves; } }) } + pub(crate) fn expect_split(build: u64, solve: u64) { T.with(|t| { let mut g = t.borrow_mut(); if g.open.is_some() { g.acc.expect_build_us += build; g.acc.expect_solve_us += solve; } }) } + pub(crate) fn end() { + let cur = T.with(|t| t.borrow().cur); enter(cur); + T.with(|t| { let mut g = t.borrow_mut(); if let Some((t0, _)) = g.open.take() { g.acc.total_us = us(t0); g.last = g.acc; } }) + } +} +#[cfg(feature = "turn_trace")] +pub use on::{last, last_resume, set_probe}; +#[cfg(feature = "turn_trace")] +pub(crate) use on::{begin, end, enter, expect_cache, expect_split, resumed, Stopwatch}; +#[cfg(not(feature = "turn_trace"))] +#[inline(always)] +pub(crate) fn begin() {} +#[cfg(not(feature = "turn_trace"))] +#[inline(always)] +pub(crate) fn enter(ph: Phase) -> Phase { ph } +#[cfg(not(feature = "turn_trace"))] +#[inline(always)] +pub(crate) fn end() {} +#[cfg(not(feature = "turn_trace"))] +#[inline(always)] +pub(crate) fn expect_cache(_hits: u32, _solves: u32) {} +#[cfg(not(feature = "turn_trace"))] +#[inline(always)] +pub(crate) fn expect_split(_build: u64, _solve: u64) {} +#[cfg(not(feature = "turn_trace"))] +#[inline(always)] +pub(crate) fn resumed(_t: ResumeTrace) {} +#[cfg(not(feature = "turn_trace"))] +pub(crate) struct Stopwatch; +#[cfg(not(feature = "turn_trace"))] +impl Stopwatch { #[inline(always)] pub(crate) fn start() -> Stopwatch { Stopwatch } #[inline(always)] pub(crate) fn us(&self) -> u64 { 0 } } + +impl TurnTrace { + /// As JSON: `{"us": {phase: µs}, "bytes": {phase: value}, "total_us": n, "solves": n, "expect_cached": n, "expect_solved": n, + /// "expect_build_us": n, "expect_solve_us": n}` + /// (phases in `Phase` order) + pub fn to_json(&self) -> crate::json::Json { + use crate::json::Json; + let row = |v: &[u64; PHASES]| Json::Obj(NAMES.iter().zip(v).map(|(k, x)| (k.to_string(), Json::Num(*x as f64))).collect()); + Json::Obj(vec![("us".into(), row(&self.us)), ("bytes".into(), row(&self.bytes)), ("total_us".into(), Json::Num(self.total_us as f64)), ("solves".into(), Json::Num(self.solves as f64)), + ("expect_cached".into(), Json::Num(self.expect_cached as f64)), ("expect_solved".into(), Json::Num(self.expect_solved as f64)), + ("expect_build_us".into(), Json::Num(self.expect_build_us as f64)), ("expect_solve_us".into(), Json::Num(self.expect_solve_us as f64))]) + } +} diff --git a/probbit-cli/src/yaml.rs b/probbit-persona/src/yaml.rs similarity index 86% rename from probbit-cli/src/yaml.rs rename to probbit-persona/src/yaml.rs index 81f7bf1..2e01310 100644 --- a/probbit-cli/src/yaml.rs +++ b/probbit-persona/src/yaml.rs @@ -8,7 +8,9 @@ //! `miniyaml.py`; this is a port of it). Differences, both on inputs it got wrong: a quoted scalar as a block-sequence item //! (`- "x"`) is read (miniyaml refused it), and a number that is not exact in a double (an integer beyond 2^53, `1e999`) is an //! error (miniyaml kept the integer exact, or read infinity). -use crate::json::Json; +#[allow(unused_imports)] +use std::prelude::rust_2021::*; +use crate::json::{Json, MAX_DEPTH}; pub struct YamlError { pub line: usize, pub msg: String } fn err(line: usize, msg: impl Into) -> Result { Err(YamlError { line, msg: msg.into() }) } @@ -131,7 +133,7 @@ fn scalar(tok: &str, ln: usize) -> Result { /// One-line flow collection `[...]` / `{...}` fn flow(t: &[char], ln: usize) -> Result { - struct F<'a> { t: &'a [char], pos: usize, ln: usize } + struct F<'a> { t: &'a [char], pos: usize, ln: usize, d: usize } impl F<'_> { fn ws(&mut self) { while self.pos < self.t.len() && self.t[self.pos] == ' ' { self.pos += 1; } } /// (token, quoted) @@ -145,7 +147,13 @@ fn flow(t: &[char], ln: usize) -> Result { let mut j = i; while j < t.len() && !stops.contains(t[j]) { j += 1; } self.pos = j; Ok((t[i..j].iter().collect::().trim_matches(space).to_string(), false)) } + /// A value; a collection more than `MAX_DEPTH` deep is an error (the recursion's stack is bounded) fn value(&mut self) -> Result { + self.ws(); let open = matches!(self.t.get(self.pos), Some('[' | '{')); + if open { if self.d >= MAX_DEPTH { return err(self.ln, format!("limit: flow collections nested deeper than {MAX_DEPTH}")); } self.d += 1; } + let v = self.item(); if open { self.d -= 1; } v + } + fn item(&mut self) -> Result { self.ws(); let ln = self.ln; if self.pos >= self.t.len() { return err(ln, "flow collection ends early"); } match self.t[self.pos] { '[' => { self.pos += 1; let mut out = vec![]; self.ws(); @@ -165,7 +173,7 @@ fn flow(t: &[char], ln: usize) -> Result { _ => { let (tok, quoted) = self.token(",]}")?; if quoted { Ok(Json::Str(tok)) } else { scalar(&tok, ln) } } } } } - let mut f = F { t, pos: 0, ln }; let v = f.value()?; f.ws(); + let mut f = F { t, pos: 0, ln, d: 0 }; let v = f.value()?; f.ws(); if f.pos != t.len() { return err(ln, format!("trailing text after flow collection: {:?}", t[f.pos..].iter().collect::())); } Ok(v) } @@ -196,17 +204,20 @@ fn split_key(s: &str, ln: usize, item: bool) -> Result, fn is_item(s: &str) -> bool { s == "-" || s.starts_with("- ") } /// The block whose lines start at rows[i] with exactly `indent` spaces -> (value, next index) -fn block(rows: &mut Vec<(usize, String, usize)>, mut i: usize, indent: usize) -> Result<(Json, usize), YamlError> { +/// One block collection at `indent`, `d` collections deep (the document's own is 0): deeper than `MAX_DEPTH` is an error, so the +/// recursion's stack is bounded (a document indented one more space per line used to recurse once per line) +fn block(rows: &mut Vec<(usize, String, usize)>, mut i: usize, indent: usize, d: usize) -> Result<(Json, usize), YamlError> { + if d >= MAX_DEPTH { return err(rows[i].2, format!("limit: collections nested deeper than {MAX_DEPTH}")); } if is_item(&rows[i].1) { let mut out = vec![]; while i < rows.len() && rows[i].0 == indent && is_item(&rows[i].1) { let ln = rows[i].2; let rest = rows[i].1[1..].to_string(); let inner = rest.trim_start_matches(' ').to_string(); if inner.is_empty() { - let v = if i + 1 < rows.len() && rows[i + 1].0 > indent { let (v, n) = block(rows, i + 1, rows[i + 1].0)?; i = n; v } else { i += 1; Json::Null }; + let v = if i + 1 < rows.len() && rows[i + 1].0 > indent { let (v, n) = block(rows, i + 1, rows[i + 1].0, d + 1)?; i = n; v } else { i += 1; Json::Null }; out.push(v); continue; } let col = indent + 1 + (rest.len() - inner.len()); let key = if inner.starts_with(['[', '{']) { None } else { split_key(&inner, ln, true)? }; - if key.is_some() { rows[i] = (col, inner, ln); let (v, n) = block(rows, i, col)?; i = n; out.push(v); } + if key.is_some() { rows[i] = (col, inner, ln); let (v, n) = block(rows, i, col, d + 1)?; i = n; out.push(v); } else { out.push(scalar(&inner, ln)?); i += 1; } } if i < rows.len() && rows[i].0 > indent { return err(rows[i].2, "bad indentation"); } @@ -220,9 +231,9 @@ fn block(rows: &mut Vec<(usize, String, usize)>, mut i: usize, indent: usize) -> if out.iter().any(|(x, _)| *x == k) { return err(ln, format!("duplicate key {k:?}")); } let rest = rest.trim_matches(space); let v = if !rest.is_empty() { i += 1; scalar(rest, ln)? } - else if i + 1 < rows.len() && rows[i + 1].0 > indent { let (v, n) = block(rows, i + 1, rows[i + 1].0)?; i = n; v } + else if i + 1 < rows.len() && rows[i + 1].0 > indent { let (v, n) = block(rows, i + 1, rows[i + 1].0, d + 1)?; i = n; v } // a sequence at the key's own indentation (common YAML style) - else if i + 1 < rows.len() && rows[i + 1].0 == indent && is_item(&rows[i + 1].1) { let (v, n) = block(rows, i + 1, indent)?; i = n; v } + else if i + 1 < rows.len() && rows[i + 1].0 == indent && is_item(&rows[i + 1].1) { let (v, n) = block(rows, i + 1, indent, d + 1)?; i = n; v } else { i += 1; Json::Null }; out.push((k, v)); } @@ -235,7 +246,26 @@ pub fn load(text: &str) -> Result { let mut rows = rows(text)?; if rows.is_empty() { return Ok(Json::Null); } if rows.len() == 1 && !rows[0].1.contains(':') && !rows[0].1.starts_with('-') { return scalar(&rows[0].1.clone(), rows[0].2); } - let base = rows[0].0; let (v, i) = block(&mut rows, 0, base)?; + let base = rows[0].0; let (v, i) = block(&mut rows, 0, base, 0)?; if i != rows.len() { return err(rows[i].2, "unexpected text (indentation?)"); } Ok(v) } + +#[cfg(test)] +mod tests { + use super::*; + /// Block collections: `MAX_DEPTH` deep is read, one more is a `limit` error (the recursion's stack is bounded) + #[test] + fn block_nesting_is_bounded() { + let doc = |n: usize| { let mut t = String::new(); for k in 0..n { t.push_str(&" ".repeat(k)); t.push_str(if k + 1 == n { "k: 1\n" } else { "k:\n" }); } t }; + let mut v = load(&doc(MAX_DEPTH)).unwrap_or_else(|e| panic!("{}", e.msg)); let mut d = 0; while let Some(x) = v.get("k") { v = x.clone(); d += 1; } assert_eq!(d, MAX_DEPTH); + let e = load(&doc(MAX_DEPTH + 1)).err().unwrap(); assert!(e.msg.starts_with("limit: ") && e.msg.contains("256"), "{}", e.msg); assert_eq!(e.line, MAX_DEPTH + 1); + } + /// Flow collections: `MAX_DEPTH` deep is read, one more is a `limit` error + #[test] + fn flow_nesting_is_bounded() { + let doc = |n: usize| format!("k: {}1{}\n", "[".repeat(n), "]".repeat(n)); + assert!(load(&doc(MAX_DEPTH)).is_ok()); + let e = load(&doc(MAX_DEPTH + 1)).err().unwrap(); assert!(e.msg.starts_with("limit: ") && e.msg.contains("256"), "{}", e.msg); + } +} diff --git a/probbit-wasm/Cargo.toml b/probbit-wasm/Cargo.toml index 0f59dd1..3d6155d 100644 --- a/probbit-wasm/Cargo.toml +++ b/probbit-wasm/Cargo.toml @@ -15,3 +15,4 @@ crate-type = ["cdylib", "rlib"] probbit-core = { path = "../probbit-core" } probbit-ir = { path = "../probbit-ir" } probbit-decide = { path = "../probbit-decide" } +probbit-persona = { path = "../probbit-persona" } diff --git a/probbit-wasm/src/lib.rs b/probbit-wasm/src/lib.rs index c6d3263..deaf956 100644 --- a/probbit-wasm/src/lib.rs +++ b/probbit-wasm/src/lib.rs @@ -5,7 +5,7 @@ //! optional `flags` object, as in `probbit mcp`: //! `{"workers": [...], "tasks": [...], "flags": {"sweeps": 3200, "polish_ms": 0}}`. //! -//! The code is the CLI's (probbit-cli's json.rs, run.rs, router.rs, evaluate.rs, persona.rs, yaml.rs and sys.rs, included below) run with +//! The code is the CLI's (the probbit-persona library: json, run, router, evaluate, persona, yaml and sys) run with //! `probbit_core::rt::set_sequential(true)`: every chain, gate pass and polish chain runs in order on the calling thread. At fixed //! work the documents equal the CLI's at any `--threads` (timings and `telemetry.threads` aside; tests/no_threads.rs). //! Not here: `--summary`, `--pretty`, `--top`, `--progress`, `--cpu-limit`, `--priority` and the config file (an unknown flag is @@ -16,42 +16,9 @@ //! input buffer is consumed); //! `probbit_out_ptr()` / `probbit_out_code()` -> the output bytes and the CLI's exit code (0 answer, 1 infeasible, 2 bad input or flag, //! 3 refused / declined). The clock is the import `probbit.now_ms` (`performance.now()`). No dependencies (no wasm-bindgen). -#[allow(dead_code)] -#[path = "../../probbit-cli/src/json.rs"] -mod json; -#[allow(dead_code)] -#[path = "../../probbit-cli/src/run.rs"] -mod run; -#[allow(dead_code)] -#[path = "../../probbit-cli/src/router.rs"] -mod router; -#[allow(dead_code)] -#[path = "../../probbit-cli/src/evaluate.rs"] -mod evaluate; -#[allow(dead_code, unused_imports)] -#[path = "../../probbit-cli/src/sys.rs"] -mod sys; -#[allow(dead_code)] -#[path = "../../probbit-cli/src/persona.rs"] -mod persona; -#[allow(dead_code)] -#[path = "../../probbit-cli/src/yaml.rs"] -mod yaml; -/// The CLI's `--top` monitor hooks (probbit-cli/src/tui.rs): nothing is drawn here. -mod tui { - pub fn tier(_: &'static str) {} - pub fn gate_seen(_: f64, _: f64, _: usize, _: usize) {} -} -use json::{num, str as jstr, InErr, Json}; +use probbit_persona::{evaluate, json, persona, router, run}; +use json::{str as jstr, InErr, Json}; use probbit_core::rt::Instant; -use router::item_bars; - -/// The telemetry fields of `--exact-ms` (as probbit-cli/src/main.rs). -fn exact_cap(xms: Option, reached: bool) -> Vec<(&'static str, Json)> { - xms.map_or(vec![], |x| vec![("exact_ms", num(x)), ("exact_ms_reached", Json::Bool(reached))]) -} -/// No configuration file in a browser (the CLI echoes probbit.json / $PROBBIT_CONFIG here). -fn config_echo() -> Vec<(&'static str, Json)> { vec![] } /// `probbit decide` on a router document (+ optional `flags`) -> (the decision JSON, the CLI's exit code). pub fn decide(input: &str) -> (String, i32) { call(input, "decide") } diff --git a/probbit-wasm/tests/no_threads.rs b/probbit-wasm/tests/no_threads.rs index 840082a..140cde7 100644 --- a/probbit-wasm/tests/no_threads.rs +++ b/probbit-wasm/tests/no_threads.rs @@ -3,7 +3,7 @@ //! sampled probbit-ir program and a sampled `evaluate` request. Only timings and `telemetry.threads` (1 vs 4) may differ. The //! CLI's own threads-1-vs-N equality is probbit-cli's `run_threads_and_chains_are_resource_controls`; this extends it to no threads. #[allow(dead_code)] -#[path = "../../probbit-cli/src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; use json::Json; diff --git a/probbit-wasm/tests/persona.rs b/probbit-wasm/tests/persona.rs index 7368e6d..dbad0b7 100644 --- a/probbit-wasm/tests/persona.rs +++ b/probbit-wasm/tests/persona.rs @@ -2,7 +2,7 @@ //! the example personas' goldens (examples/persona/golden/, the `probbit persona` documents) on every workday turn, and the same //! bytes through `probbit_call`. Documents are canonical JSON, so equal values are equal bytes. #[allow(dead_code)] -#[path = "../../probbit-cli/src/json.rs"] +#[path = "../../probbit-persona/src/json.rs"] mod json; use json::Json;