Repository navigation
HermitCrab synthesis: 25.6% of synthesis steps end in SurfaceFormMismatch; early-reject check not built (research lead) #603
Description
Claude-drafted.
25.6% of synthesis steps on the five real-grammar samples end in SurfaceFormMismatch, after the whole rule stack has run. Whether an intermediate-shape check could reject those candidates earlier is not measured.
Status: research lead, not built. There is no soundness argument yet.
What we know (C# hc.dll, traced samples, unit: synthesis steps and analysis events):
SurfaceFormMismatchis 25.6% of synthesis steps pooled (279,872 steps); 37.9% on Indonesian, 27.2% on Mbugwe, 16.9% on Amharic.- Synthesis is a small part of total work (synthesis steps plus analysis events): 1.0% Aweti, 1.8% Sena, 9.0% Mbugwe, 9.4% Indonesian, 14.7% Amharic. A perfect check would therefore remove well under 25.6% of total work (inferred).
- A check at each stratum boundary has no soundness argument: analysis shapes over-approximate (optional and underspecified nodes), and the earlier synthesis-side length bound (ledger row 8) lost valid parses.
Not known: how many SurfaceFormMismatch steps a sound check would cut, and whether any successful candidate would fail it.
Proposed change: count, do not prune. At each stratum boundary, record whether the check would fire and whether any successful candidate fails it, in steps per grammar. Build the check only if the share is material. Kill it if any success fails it on any corpus.
Related: #560 (E1, open). Retired record: PR #490 (closed). PanGloss: sillsdev/PanGloss#105 (pointer, open).
Next: run the boundary census on hc.dll and post steps cut per grammar plus any success that fails the check.
Evidence -- where the numbers come from
The census harness is test-only (branch research/failure-reason-stats at af97348b, master c2454760 with PR #480 merged; whether the branch still exists was not checked). A step is one rule-level synthesis trace event. Counts come from traced parses, which turn analysis merging off, so more analyses are explored than in an untraced run. 22.1% of steps have no terminal event and are unattributed.
| Corpus (sample) | SurfaceFormMismatch, % of synthesis steps |
Synthesis, % of total work |
|---|---|---|
| Pooled real grammars | 25.6% | |
| Aweti, every 2nd word | 1.0% | |
| Sena, every 100th | 1.8% | |
| Mbugwe, every 40th | 27.2% | 9.0% |
| Indonesian, all 70 | 37.9% | 9.4% |
| Amharic, first 28 | 16.9% | 14.7% |
Blank cells were not given in the record. Conformance suite: 6.9% of its steps, 22.9% synthesis share.
Sources -- pinned to 2eba2f5 (closed PR #490 record)
- E6 definition and kill rule: synthesis round L496-L505 (lines 496–505); "not run": L679-L681 (lines 679–681).
- Pooled table: L120-L125 (lines 120–125); per-grammar shares: L141-L144 (lines 141–144).
- Synthesis share of total work: L536-L547 (lines 536–547); text: L52-L55 (lines 52–55).
- Ledger: row 31, L152 (line 152); row 37 (E6), L158 (line 158); row 8 (length bound), L93 (line 93).
What would reopen this
A soundness argument for a stratum-boundary check, or a census where no success fails it and the cut share is material.
Activity
[-]HermitCrab synthesis: intermediate-shape check for SurfaceFormMismatch (research lead, not built)[/-][+]HermitCrab synthesis: 25.6% of synthesis steps end in SurfaceFormMismatch; early-reject check not built (research lead)[/+]