[johnml1135]

Start here: src/SIL.Machine.Morphology.HermitCrab/RequiredSlotMaterial.cs -- the class remarks (Presence, Order) give the soundness argument for dropping a partial; the rest is wiring.

Status: open, review required; head d3c5c06. CI was still running when this was written. #576 and #579 are folded into this branch (both show as merged); #580 (draft) is parked on top and #577 (draft) is parked.

HermitCrab's affix-template walk now drops a partial as soon as a required slot can no longer find the segments its affix inserts. Parses are unchanged; doomed partials stop paying for every optional slot between them and the required one. The slot's literals must appear in the order the rule inserts them and, for prefix-only or suffix-only slots, near the word edge; slots whose unmatched parts are unbounded runs keep being tested. Pruning is on by default; Morpher.PruneRequiredSlots = false turns it off (set it before parsing), and tracing prunes too: each prune is traced as FailureReason.UnreachableRequiredSlot with the slot (Trace.FailureObject).

The question a reviewer arrives with is "does it lose a parse?". Tests assert identical parses with pruning off and on for each hard case, and a shadow pass found 0 violations (carried over, see Evidence). A slot is tested only while every slot walked since is pure, ordered where order is used, and typed; untyped constraints and boundary-only inserts are handled conservatively. The question worth your time is the cost side: the look-ahead has its own unification cost, so whether it pays on real grammars.

Where to look

• Soundness of pruning: Apply_ImpureSlotBetween_DoesNotPruneForTheInnerRequiredSlot (purity gate), AnalysisAffixTemplateOrderTests (order and edge window SlotWalk.Lo/Hi), AnalysisAffixTemplateFusionTests (unbounded runs).
• RequiredSlotMaterial.RestoredSegments against AnalysisMorphologicalTransform.Untruncate: an unbounded run must restore nothing, or the purity test is unsound.
• Parallel and tracing: Apply_Parallel_ReturnsTheSameParsesAsSequential, Apply_TracingPrunedPartial_TracesTheUnreachableRequiredSlot.
• Public switch and conservative cases: Apply_PublicPruningSwitch_ControlsRuleAttempts, Compile_UntypedConstraints_DisablesTheSummaryConservatively, Apply_BoundaryOnlyInfix_DoesNotMakeReachUnbounded.

Not here: the performance follow-ups (repeated shape scans, narrowing, keeping shadow mode) are tracked in #598; #580 is parked; the stack was not re-measured on top of #569, which may reduce its value.

Verification at 91ee3da: HermitCrab tests 199/199 pass (Release); dotnet csharpier and comment hygiene clean; the HermitCrab project builds with 0 warnings. The full local_check.sh did not complete (cmake is missing for SentencePiece). The measurements below were not re-run at this head.

Next: review RequiredSlotMaterial.cs, and say whether the cost side justifies merging before #569 is measured.

---

Reading this a year from now -- start here

This branch is four commits on master c245476: 0bf9e64 (the required-slot prune, issue #561), 28cbee1 (was PR #576: slot literals in order and at reachable edges, issue #567), faa2499 (was PR #579: keep testing behind slots whose unmatched parts are unbounded runs) and 91ee3da (review fixes). #576 and #579 were separate stacked PRs and were folded in so the look-ahead is reviewed once. All of it runs under the internal RequiredSlotLookaheadMode (Off, Shadow, On); Morpher.PruneRequiredSlots is the public on/off view of it. The unification and census counters that produced the numbers below are not in this PR; they live on the research branches research/hc-rsl-census @ 6ad11702 and research/hc-ordered-census @ 4bb687f0.

Decisions, and why

• Net unification calls over the whole parse (look-ahead included) is the yardstick for the first commit, not the original plan's kill criterion. That criterion (look-ahead check unifications more than 3x the analysis events saved) tripped on Mbugwe at 3.19x (1,112,761 check unifications against 349,004 events saved) because it compared unlike units; on the same-unit net comparison Mbugwe was break-even.
• Shadow mode counts and prunes nothing, so soundness and cost can be checked on real grammars without changing parses. It and RequiredSlotCheckObserver (null by default) are kept for the soundness tests; whether to keep them is part of HermitCrab analysis: measure the required-slot look-ahead's repeated scans, narrowing and shadow diagnostics in steps #598.
• An unbounded quantifier counts as restoring zero segments because Untruncate writes nothing for it, so a slot whose unmatched parts are unbounded runs stays pure. Bounded runs, bare restored segments, alternations with an optional branch and ModifyFromInput stay impure.
• A part copied twice is unordered, so only the presence test runs behind it.
• Review fixes: tracing no longer bypasses pruning (d3c5c06, at maintainer request); a prune is traced as UnreachableRequiredSlot with the slot, which needs a FieldWorks change (EndUnapplyTemplate gains a failureObj parameter); an allomorph with an untyped constraint is impure and unprunable; boundary-only inserts do not make the edge reach unbounded.

Paths not taken

• Skipping slots that insert a single segment was tried on the first commit: it halved Sena's gain and did not reduce Mbugwe's check cost.
• A supply-set design (an optional restored node could stand for any required segment, so count distinct nodes) was not built here; it is perf(hermitcrab): keep the required-slot look-ahead testing behind slots that write back deleted stem segments #580, parked.

Evidence (carried over, not re-run at 91ee3da)

All figures are from the bodies of #575, #576 and #579 before the fold, measured on those PRs' own heads (before the review-fix commit; the #579 figures also before #522 removed the analysis memo). They are not from 91ee3da.

• Whole stack against master, from the perf(hermitcrab): keep the required-slot look-ahead testing behind slots whose unmatched parts are unbounded runs #579 table: events plus synthesis steps, Mbugwe 40.85% less, Sena 45.92% less, Aweti 11.50% less, Amharic 14.08% less, conformance 1.80% less, Indonesian equal. Unification calls including the look-ahead's own checks: Mbugwe 22.55% less, Sena 5.22% less, Aweti 9.27% less, Amharic 2.75% less, conformance 1.22% less, Indonesian equal. Words measured: Mbugwe 40, Sena 51, Aweti 99, Amharic 24, Indonesian 70, conformance 1,452.
• First commit alone (earlier Prune affix template partials whose required slots cannot find their material, in order and at the word edges #575 body), unification calls: Mbugwe and conformance within 0.4% of break-even, Sena 1.6% less, Aweti 4.5% less, stress grammar 37.9% less, Amharic 0.63% more (411,073 to 413,674 calls, 24 words). The order and edge commit turned Amharic into a net saving (3.4 points, per the Require slot literals in order and at reachable edges in the template look-ahead #576 body).
• Look-ahead's own check unifications: Mbugwe 995,434 to 1,133,568 (+13.88%) and Aweti 232,302 to 232,639 (+0.15%) from the unbounded-run commit; both are inside the totals above.
• Shadow runs over conformance, Indonesian, Aweti, Amharic, Mbugwe, Sena and the Bantu stress grammar: 0 violations, 0 completions under a fired check; per-word parses equal to master in traced and merged modes.
• Conformance runner, base vs branch (first and second commits): the same 4 known failures in both (truncate-morphotactic family), per-fixture verdicts identical.
• The fusion stress grammar is synthetic (it models vowel-fusion allomorphs in template slots, which none of the other stress grammars do); its percentages are not evidence about a real language. Mbugwe is the real one.

Deferred, and what would unblock it

• Mbugwe's remaining impure slots and Aweti's events behind impure slots (bounded runs, bare restored segments, modification): perf(hermitcrab): keep the required-slot look-ahead testing behind slots that write back deleted stem segments #580, parked until it justifies its code on real grammars.
• Look-ahead on the shared trailing-slot walk: perf(hermitcrab): run the required-slot look-ahead on the shared trailing-slot template walk #577 (draft), parked until perf(hermitcrab): share trailing analysis template slot batches #546 lands.
• Re-measuring on top of Extend MergeEquivalentAnalyses to cover templates and rules #569 (merge equivalent analyses across templates and rules), which shrinks the same search space.
• The three performance questions in HermitCrab analysis: measure the required-slot look-ahead's repeated scans, narrowing and shadow diagnostics in steps #598.

🤖 Generated with Claude Code

---

This change is

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.36493% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.55%. Comparing base (0cbf85f) to head (d3c5c06).

Files with missing lines	Patch %	Lines
...hine.Morphology.HermitCrab/RequiredSlotMaterial.cs	94.92%	4 Missing and 12 partials ⚠️
...Morphology.HermitCrab/AnalysisAffixTemplateRule.cs	88.29%	7 Missing and 4 partials ⚠️
src/SIL.Machine.Morphology.HermitCrab/Morpher.cs	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #575      +/-   ##
==========================================
+ Coverage   75.33%   75.55%   +0.22%     
==========================================
  Files         454      455       +1     
  Lines       38509    38915     +406     
  Branches     5335     5424      +89     
==========================================
+ Hits        29010    29404     +394     
  Misses       8364     8364              
- Partials     1135     1147      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnml1135]

On the design question: the overlap is intended, and keeping both checks is the cheaper choice.

• Same data, different points. For an anchored allomorph, the look-ahead's first and last inserted constraints are the same feature structures as LeftEdgeConstraint/RightEdgeConstraint. They were identical everywhere both exist, across every template allomorph in Sena, Mbugwe, Aweti and Amharic. But the prefilter judges one allomorph against the current edge, immediately before the matcher runs. The look-ahead judges a required slot that may sit behind optional slots, before any of them are walked.

• The prefilter's constraints are only valid for the next slot. Reading them for deeper slots loses parses. For example, with a template of [required suffix -ti] then [optional suffix -z], the word "sagtiz" is pruned, although unapplying -z first leaves "sagti", which parses. 15 of 50 template tests fail that way. Reach and Lo/Hi are what make deeper slots sound.

• Dropping the duplicated next-slot check costs work. In a C# prototype that skips the look-ahead whenever the prefilter covers the next slot, parses are unchanged, but rule attempts rise 6.5% on Sena and 10% on Amharic.

• Retired instructions agree. These are Windows hardware counters, measured on PanGloss (the Rust port, which has both mechanisms) on 2026-11-11, with 2 repeats against a no-change control. Each figure is the change with the mechanism turned off, so + means it saves work.

  Off	Sena	Swahili	Indonesian	Mbugwe	Aweti	Amharic
  the look-ahead	+4.2%	+1.2%	within noise	+7.1%	+20.2%	within noise
  only its next-slot check, where the edge checks cover it	within noise	+2.0%	within noise	within noise	+11.9%	within noise

  Removing the overlap never saves instructions.

• The prefilter covers less than the look-ahead. It does not see AnalysisRealizationalAffixProcessRule, and the look-ahead does.

A shared builder would save only compile-time data, roughly one frozen feature-structure clone per anchored allomorph. Its per-word cost is unification either way. The repeated edge search (most of the look-ahead's unifications on Sena) fits #598, which already tracks repeated shape scans. I have not measured it, so I'm not claiming a saving there.

[claude]

Verdict: approve. F11 and F12 are fixed in a6e0ade, and the branch is rebased onto master at 0cbf85f. No new findings.

Most important: nothing open. Your answer on the edge-prefilter overlap settles last round's design question: the prefilter's edge constraints are sound only for the next slot, and dropping the duplicate next-slot check costs rule attempts.

Counts: 0 Critical, 0 Important, 0 new Low.

Ran: dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests -c Release on merge commit 929ab61: 229 passed, 0 failed. Because the rebase rewrote history, I ran /code-review high over the whole PR. Its candidates either repeated F4/F5 (#598) or the design question you answered, or did not hold up on a check. gh pr checks 575: both builds and claude-review are pending, and the advisory comment hygiene reports passed.

Not verified: the strict comment hygiene scan (--agent-strict) was not run here. I could not fetch d957913 locally, so I did not diff the rebased PR against the last reviewed head.

Public API: Morpher.PruneRequiredSlots (new, on by default) is unchanged except for its doc. Target framework, package, and sillsdev/machine.py parity: None verified.

Findings: F1 addressed, F2 addressed, F3 addressed, F4 accepted (#598), F5 accepted (#598, unverified), F6 addressed, F7 addressed, F8 addressed, F9 addressed, F10 addressed, F11 addressed, F12 addressed.

Reviewed at a6e0ade

[jtmaxwell3]

In general, I'd like to move away from turning optimizations off when tracing, so that tracing doesn't become slow and so that bugs in the optimization code are easier to identify. I propose that RequiredSlotCannotUnapply trace failures with something like FailureReason.UnreachableRequiredSlot along with the name of the slot. This will require a change to FieldWorks. I think that the user can figure out the failure from the slot name and the failure description.

[johnml1135]

Thanks, done in d3c5c06: the look-ahead stays on while tracing, and the tracing guard is gone.

• When a required slot cannot unapply, RequiredSlotCannotUnapply now calls EndUnapplyTemplate(template, word, false, FailureReason.UnreachableRequiredSlot, slot) (new FailureReason.UnreachableRequiredSlot; the AffixTemplateSlot is the failure object, so slot.Name is available). TraceManager stores it on the TemplateAnalysisOutput node as Trace.FailureObject.
• ITraceManager.EndUnapplyTemplate gains an object failureObj parameter, matching Failed and the *NotApplied methods. This is a breaking change for implementers.
• Tests (AnalysisAffixTemplateRuleTests): a pruned partial is traced with the right reason, template and slot and with zero rule attempts; a present required slot traces no such failure and gives the same parses as pruning off; traced and untraced walks give the same parses and rule attempts for five forms. HermitCrab suite: 231/231 pass (Release); csharpier check clean.

FieldWorks change needed: Src/LexText/ParserCore/FwXmlTraceManager.cs implements ITraceManager. (1) Add object failureObj to EndUnapplyTemplate (line 72). (2) In it, when reason == FailureReason.UnreachableRequiredSlot, add a FailureReason element of a new type, e.g. type="unreachableRequiredSlot" carrying the slot name (((AffixTemplateSlot) failureObj).Name), next to the existing nonPartialRuleAfterFinalTemplate case. (3) Add the display text for the new type to the trace XSLT / strings that render FailureReason type=..., so the user sees the slot name with a description like "a required slot cannot be found in the word". The other ITraceManager implementers (tests, tool) are updated in this PR.

[claude]

No new findings. Reviewed at d3c5c06

[johnml1135]

FieldWorks follow-up for this PR: sillsdev/FieldWorks#1247 (draft), tracked as LT-22899.

It updates FieldWorks' two ITraceManager implementations, FwXmlTraceManager (Try A Word) and HcXmlTraceManager (HCSynthByGloss), for the new EndUnapplyTemplate(..., object failureObj) signature. Both now record FailureReason.UnreachableRequiredSlot with the slot name, and both trace stylesheets show it as a reason that names the required slot. It was built and tested against a local package packed from d3c5c06.

FieldWorks will not compile against a SIL.Machine release that contains this PR until that PR merges. It stays a draft until such a release exists. At that point it bumps SilMachineVersion and is marked ready.

[johnml1135]

Correction to the retired-instruction table in my earlier reply: two Amharic cells were wrong. The conclusion stands.

I re-ran the Windows hardware-counter trial with a tighter instrument. It now uses 5 repeats, a 95% Welch interval with a 0.5% minimum effect, buffered output and stricter per-process attribution. I also re-measured the original builds with it, to separate instrument changes from newer PanGloss code.

What was wrong: Amharic is not within noise in either row. The old 2-repeat figures were instrument noise.

• Look-ahead off: +3.2% (current code, 95% CI +2.9% to +3.5%; original builds +3.1%).
• Only its next-slot check off: +1.6% (current code, CI +1.5% to +1.8%; original builds +1.5%).

What holds. Figures are the change with the mechanism turned off, on current code; + means the mechanism saves work.

Off	Sena	Swahili	Indonesian	Mbugwe	Aweti	Amharic
the look-ahead	+2.5%¹	cost, size unresolved²	no change	+6.0%¹	+17.3%	+3.2%
only its next-slot check	no change	+2.6%²	no change	no change	+9.6%	+1.6%

¹ One word parses differently between the builds on current code, so it is left out of these two cells.
² Swahili is at the instrument's limit: loading the grammar costs about five times the parse of these 77 words. Look-ahead off reads +3.7% to +6.6% depending on repeats, and next-slot off is inconclusive at 10 repeats.

The Mbugwe and Aweti figures are smaller because newer PanGloss code parses faster; the original builds still reproduce the earlier +7.1% and +20.2%. Sena's original builds read +3.3% under the new instrument.

Removing the overlap still never saves instructions. With Amharic corrected, it measurably costs on Aweti and Amharic, and probably on Swahili.