Skip to content

Draw UK QRF stages from uniforms keyed on each record's id, and tie pension contributions to pay - #1332

Merged
juaristi22 merged 12 commits into
mainfrom
uk-keyed-draws-pension-coherence
Oct 21, 2026
Merged

juaristi22 merged 12 commits into
mainfrom
uk-keyed-draws-pension-coherence

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Prepares the next policyengine-uk bump (#1095). #1308's arm on policyengine-uk 2.134.5 failed the weight-ratio gate. The engine change moved few households, but the build redrew most of the spine. This PR makes each stage's draws move only for the records whose own inputs move, and ties pension contributions to the pay they come from. ORR's rail support row is already held out of the fit on main, by #1329.

Rebased onto main 211483f, after #1329 and #1295. This PR's own rail commit is dropped: #1329 holds the same row out under its own entry.

The bump itself is the stacked PR that follows. It merges after this PR and after the policyengine-uk release that carries the fixes of PolicyEngine/policyengine-uk#2523.

Why the weight ratio failed on 2.134.5

  • RegimeGatedQRF drew each target's quantile and sign uniforms from one positional stream (rng.random(n) per target), so a row's draw depended on its place in the frame, not on the row itself.
  • 2.134.5 changes household_net_income for some households. was_wealth reads it and redrew their wealth. high_value_home_support then copied a different set of households, and the frame's length changed.
  • So every later QRF stage (NTS travel, LCFS consumption, ETB services and the rest) drew different numbers for nearly every row, including households whose inputs had not changed.
  • Rows with low design coverage then rested on different households. ORR's rail support is one, and two HMRC band rows are others: property income for total income of £12,570 to £15,000, and dividends for £40,000 to £50,000. The solver's 10× cap took up the stretch, and the family-folded weight ratio went over its certified bound.

The record-level evidence stays local.

Summary

Keyed draws

  • fit_draw_next (microcosm-fit) takes caller-owned quantiles and sign_uniforms, both or neither, so a chain step can draw from keyed uniforms. The fit.qrf parity pins and the spec-engine golden re-derive. microcosm.fit.qrf is a kernel module the seed protocol attests, so the US inventory's seed protocol and compiled seed map digests re-pin, with the F0 coverage report, as they did for predict_from_uniforms in 1734b9e.
  • Every UK stage that imputes with RegimeGatedQRF now draws from stable_identity_uniforms, keyed on the record's own id and salted by stage and target:
    • was_wealth, the LCFS consumption chain and etb_services, on the household id;
    • uc_reporter_redraw, on the benefit unit id;
    • the SPI income stage's second forest, on person_source_id under the stage's own seed;
    • salary_sacrifice, on the person id.
  • A copied household's id is an offset of its source's id, so the keys do not move when the frame's length does.
  • test_support/microcosm_build/uk_keyed_draws.py checks the property on three stages. It reverses the recipients, changes one and adds one, and no other draw may move. With the positional stream restored, all three tests fail. The SPI bridge test also checks the second forest's keys.
  • The H2 fixture's two source files re-derive.

Pension contributions that cohere with pay

  • policyengine-uk relieves employee contributions through the net pay arrangement, which cannot take more than the pay. From 2.134.5 it also counts every contribution, the employer's included, as pension input for the annual allowance (Limit pension relief by earnings and charge the annual allowance policyengine-uk#2240).
  • The FRS records each job's pension deduction apart from the adult's usual earnings, so a deduction can exceed the pay recorded beside it. The FRS spine now caps employee contributions at employment income.
  • An SPI-channel row redraws its pay from the tape, while the second forest draws its contributions. The forest reads the drawn pay, so whether the row has an employer contribution follows that pay. But it can only draw amounts that FRS rows in the same leaf received, so it cannot scale them to pay above the FRS's range. After the second forest:
    • the row's employee contributions are capped at its pay;
    • the forest's draw decides whether the row has an employer contribution;
    • a member whose FRS twin has an employer rate, which the ASHE draw of UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069 set, takes that rate times its own pay;
    • any other member keeps the forest's amount, bounded by ASHE's top employer rate times its pay.
  • The twin's scheme record is not carried over. It describes the twin's job at the twin's pay, and the row's pay is redrawn. Carrying it put members at low drawn pay, where few people are members (María's ruling of 21 October, after Vahid's review).
  • salary_sacrifice refuses employee contributions above the pay that remains.
  • Each step's receipt gives its support as a band of distinct source households and withholds its weighted amounts as "<10" where fewer than ten carry the change.

Measurement

Two arms ran on this machine: national role, local staging only, feed abf0921.

  • C0, the control: main a90eec3, policyengine-uk 2.134.4.

  • C1: this PR at 4198bdd, before the review fixes. Its employer rule took the twin's scheme record both ways; the rule now leaves membership to the forest (c11ac14), and Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333's arm measures the final code.

  • Both arms pass all seven terminal gates. On C1, the weight-ratio gate passes with much more room than on C0.

  • The fit is slightly better: the final loss falls a little, and a few more rows sit within 10% of their targets.

  • 24 rows move by more than three points, nearly all toward their targets: income tax by region, self-employment counts, the State Pension bands and population by region among them. One row moves just outside the 10% line, and four move inside it.

  • The support floor admits one Universal Credit payment band row and drops another, as the draws move.

  • ORR's rail support is measured, not bound (on C1 by this PR's own entry, since replaced by UK pre-Budget data: measure stored values at their period, draw Carer's Allowance claims, and the #1278 items (#1324, #1319, #1278) #1329's).

  • The two fragile HMRC rows stay bound, and less of their fitted weight sits on households near the solver's cap. Across the frame, the share of weight near the cap falls by more than half.

  • Build time on a quiet machine is about the same: 631 seconds of stage time against 607.

  • The pension receipts sit in each stage's evidence: support as a band, and weighted amounts withheld where fewer than ten source households carry them.

Tests

  • Engine-free tier, with policyengine-uk hidden (its import and its metadata), on this branch before the employer rule took its final form: 20,117 tests. Three fail on this machine only: one US scorer test needs Python 3.14.7, and two Orrery tests need an npm package the run did not install. At 4198bdd, the touched engine-free tests pass (232).
  • Engine tier, every touched surface, before that change: 117 tests, no failures. At 4198bdd, the SPI, salary-sacrifice and FRS engine tests pass (26).
  • tools/ci_test_plan.py verify and the four --check tools pass, and the H2 fixture regenerates byte for byte at 4198bdd. tools/spec_engine_coverage.py --check passes at 1258b13.
  • At 1599324, after the rebase: the UK, shared, fit and graph engine-free tiers pass (10,955 tests; two Orrery tests need an npm package the run did not install), and so do the touched engine tests (28, H2 parity included) and the US pin tests with policyengine-us 2.2.1 (40). The four --check tools, the spec-engine coverage check and the H2 fixture check pass.

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

@juaristi22 thanks, this is my review at 1258b13ff.

The design is right. Keying each QRF draw on the record's own id, salted by stage and target, is the correct fix for the positional stream. The keys hold up: blake2b gives 64 bits, the salts differ by stage, target and quantile or sign, and high-value copies get their own offset ids, so they draw independently as before. The property tests do catch the old behaviour. One thing blocks merge: CI is red on engine-us, because one US spec hash pin was not updated. Before #1333 merges I would also like three things: the 2.134.5 invariance check, a clear statement that this PR redraws everything once, and an annual allowance check against HMRC, now that employer contributions follow pay. I also have one question about the SPI second forest, where clones of one FRS person now share uniforms.

What I ran

  • Fresh worktree at 1258b13ff, uv sync --all-packages --locked --extra uk.
  • Every touched test file: the engine-free UK, shared loader and test_qrf_stateless.py tests, plus tests/engine/uk/test_uk_spi_income.py. 436 tests, all pass.
  • I checked that a property test bites. Removing the two keyed-uniform arguments from impute_etb_services makes test_etb_draws_move_only_for_the_households_whose_inputs_move and test_etb_services_chain_order_and_records fail. I restored the file afterwards.
  • tools/ci_test_plan.py verify: ok.
  • Probes of stable_identity_uniforms: dtype sensitivity and how the key string is built.
  • CI at head: engine-us 3.13 and 3.14 fail; engine-free, engine-uk, integration-uk, lint and wheels pass. Main passed at the base a90eec3cb.
  • Not run: anything that needs the licensed national frame. That covers the C0 and C1 arms, the weight-ratio gate, the receipt counts and any comparison with HMRC figures.

Findings

  1. must: CI is red at head because one US pin was missed. packages/microcosm-build/tests/engine_workflow/us/test_us_multispine_pool_tool.py:345 still pins spec_sha256 at 740d1ff5.... The bundle now hashes to 19460fc168a6f9ed44e580d0e36777cce78bb33cb52b75630ee86203fa795684, which the PR already writes into docs/evidence/spec-engine/us-f0-coverage.json:2602,2612. The seed protocol re-pin covered inventory_coverage.py and test_spec_engine_loader.py but not this engine-workflow test, and the four --check tools do not reach that tier. It is a one-line fix. Please also extend the comment above the pin with the reason, as the loader test's comment does: the attested fit.qrf bytes changed.

  2. should: the SPI second forest keys on the twin's id, so clones of one FRS person share uniforms. spi_income.py:1221 keys stage 2 on person_source_id, the FRS twin's person id. The band-donor stage (clone index 2) and the SPI channel (clone index 1) both take their copies from the base FRS. Two copies of one person therefore get the same quantile and sign uniforms for every stage-2 target. Stage 1 does this on purpose and counts it (spi_rank_transport.py:510-513, shared_identities). Stage 2 now does the same without saying so, and its receipt records nothing. The comment at spi_income.py:1168-1169 ("each row draws from uniforms keyed on its own identity") does not hold for these rows. Under the positional stream they drew independently. With different stage-1 incomes the draws are not identical, but they are comonotone: both copies sit at the same quantile of contributions. Is that intended for contributions? If so, please count shared keys in the stage-2 receipt and fix the comment. If not, key on person_id. Clone ids are offsets of the source id (spi_support.py:1276-1282), so they are just as stable. The other keyed stages refuse duplicate ids (was_wealth.py, lcfs_consumption.py, etb_services.py), and stage 2 has no such guard.

  3. should: the measurement cannot show the property the PR exists for. C0 and C1 both run on policyengine-uk 2.134.4, and the redraw problem appears on 2.134.5. The fixture tests prove the local invariance. The release-level claim, that an engine change no longer redraws the spine, can only be tested on a 2.134.5 arm, and Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333's body says that arm is still to run. Please quote that result here once it exists, or say that this PR's evidence is the fixture tests. The C0 to C1 deltas (24 rows moving more than three points, weight near the cap halved) mix three effects that cannot be told apart: the one-off redraw, the pension rule and the ORR exclusion.

  4. should: this PR redraws every record once, and the changelog does not say so. Moving from the positional stream to keyed uniforms changes the draws of every household, benefit unit and person in the touched stages: wealth, consumption, ETB services, UC reports, SPI stage 2 and salary sacrifice. changelog.d/uk-keyed-qrf-draws.changed.md says what will stay still in future, but not that everything moves now. Anyone comparing the next release with the last one will see that shift. One sentence in the fragment would cover it.

  5. should: the annual allowance consequence of tying employer contributions to pay needs checking against HMRC. From 2.134.5 (pe-uk #2240, merged 16 October), employer contributions count as pension input. An SPI-channel row now takes its twin's ASHE rate times the tape's pay (pension_coherence.py:81-86). The twin is drawn by region stratum, not by income, so a row with £500k of pay and a defined-benefit twin at 25% gets £125k of employer input. Adjusted income includes that input, so at that level the tapered allowance is £10,000. #2536's HMRC test compares the model with HMRC's Private pension statistics (July 2026): for 2024-25, 30,440 people reporting an excess through Self Assessment and £672m of excess (Table 7), and £0.3bn of charge (Table 6). On the eFRS it fails by 8 to 12 times, and #2536 says employer contributions are 69% of payers' input there. Microcosm uses ASHE rates instead of three times the employee figure, so this is the data where that check can actually pass or fail. Please give the three figures from the C1 arm, or from Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333's arm on 2.134.5, before the bump merges. If they come out high, a pensionable-pay ceiling on the twin-rate rule would be the obvious lever.

  6. should: please show what the twin-rate rule does to participation and to the bound employer total. The rule zeroes the employer contribution wherever the twin has pay but no scheme, so SPI-channel membership follows the base FRS rate, not the higher-earner rate. ONS's workplace pension tables (ASHE, 2024 provisional, March 2026) publish participation by gross weekly earnings, so this can be checked. The zeroing also removes mass against the bound DWP employer total (see the employer_pension_contributions.py docstring). The employer_none_outside_twin_scheme receipt gives mass, not a share. Two numbers from C1 would settle it: the share of SPI-channel employees left with no employer contribution, and the DWP employer row's ratio to target before and after.

  7. should: is the FRS cap treating a symptom? frs_spine.py:1050-1070 caps the summed JOB.DEDUC1 × 52 at ADULT.INEARNS × 52. A deduction above pay could come from a reporting-period mismatch between the job tab and the adult's usual earnings, or from a blank INEARNS on a job that does record pay. The cap zeroes the contribution in the second case and only trims it in the first. Could the stage evidence (net_pay_contributions) split capped rows by whether INEARNS is zero? That would show whether the cap is fixing contributions or hiding missing pay.

  8. should: salary sacrifice and the UC reporter redraw have no invariance test. uk_keyed_draws.py is used only by the WAS, LCFS and ETB tests. test_uk_salary_sacrifice.py and test_support/microcosm_build/uk_uc_reporter_redraw.py only check that uniforms of the right shape are passed. The same reverse, change one, add one check would be cheap for both stages, and for SPI stage 2 too, once finding 2 is settled.

  9. should: overlaps and merge order. Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333 is stacked on this branch, so the order is Draw UK QRF stages from uniforms keyed on each record's id, and tie pension contributions to pay #1332 then Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333. UK pre-Budget data: measure stored values at their period, draw Carer's Allowance claims, and the #1278 items (#1324, #1319, #1278) #1329 is based on main and touches the same files: frs_spine.py, spi_income.py, calibration_measure_exclusions.json, test_uk_frs_spine.py, test_uk_measure_simulation.py and both parity fixture JSONs. Whichever merges second has to regenerate the H2 fixture again and fix the register count ("58 entries").

  10. nit: the salary-sacrifice guard (salary_sacrifice.py:366-371) depends on no stage between 16 and 42 lowering pay. Today only hmrc_spi_income_spine and salary_sacrifice write employment_income, so it holds. A one-line note naming that dependency would help the next person who adds a pay rewrite.

  11. nit: stable_identity_uniforms builds its key from f"{seed}:{salt}:{value}" (stochastic_assignment.py:34), so an int id 1001 and a float id 1001.0 give different draws. A string id with a colon can also collide with a different salt: salt a with id b:1 hashes the same as salt a:b with id 1. I checked both. UK ids are int64 throughout, so neither bites today, but an integer-dtype assertion where these stages build record_ids would stop a silent full redraw if a merge ever turns an id column into floats. The helper predates this PR.

What looks right

  • fit_draw_next takes both uniform arrays or neither, and leaves the chain's draw stream alone when they are supplied, so stages that are not keyed are unchanged.
  • The salts are distinct across stages and targets (uk_was_2018_20_wealth, uk_lcfs_2024_25_consumption, uk_etb_2024_services, uk_frs_only_spi_fill, uc_reporter_redraw, salary_sacrifice), each with its own :sign stream, so no two stages or targets share a uniform.
  • High-value home copies (stage 21) take every id plus k times the id multiplier, so each copy keys its own draws. That means the keying neither creates nor removes copies that look identical. Copies of one family still draw independently in the later stages, as they did under the positional stream.
  • The WAS, LCFS and ETB stages refuse duplicate or misaligned record_ids.
  • The twin's employer rate is capped at 30% by UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069's construction, and a twin with no pay falls back to ASHE's top rate times the row's pay, so a small twin pay cannot produce a runaway rate.
  • Each coherence receipt counts support on distinct source households and withholds weighted amounts as "<10" below ten, so the evidence discloses no small cells.
  • The ORR exclusion has a dated expiry and a stated route to rebinding it, and it records @juaristi22's ruling of 20 October.
  • The fixture changes are re-derived synthetic aggregates and contain no record-level data.

🤖 Generated with Claude Code

juaristi22 added a commit that referenced this pull request Oct 21, 2026
The US bundle's semantic hash covers microcosm.fit.qrf's attested bytes,
which fit_draw_next's caller-owned uniforms change, as the seed protocol
re-pin records. The engine-workflow tier pins the hash apart, and the
four --check tools do not reach it (Vahid's review of #1332, finding 1).
The value is the one the regenerated F0 coverage report carries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 21, 2026
stable_identity_uniforms formats each id into its key, so the integer
1001 and the float 1001.0 draw differently, and a merge that turned an
id column into floats would silently redraw every record. A repeated id
would give two records the same uniforms. integer_record_keys refuses
both, and every keyed UK stage (was_wealth, lcfs_consumption,
etb_services, uc_reporter_redraw and salary_sacrifice) passes its ids
through it (Vahid's review of #1332, nit 11).

The changelog fragment now says that the switch redraws these stages
once for every record (finding 4), and salary_sacrifice's guard names
its dependency: no stage between the pay caps and this one may lower pay
(nit 10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 21, 2026
Stage 2 keyed its uniforms on person_source_id, the FRS twin's id. The
SPI channel and the band donors copy one FRS person more than once, so
those copies shared every stage-2 uniform and drew comonotone values,
where the positional stream had drawn them apart. Stage 1 shares the
twin's identity on purpose and counts it; stage 2 now keys on the row's
own person_id, which is just as stable (a copy's id is an offset of its
source's), and refuses ids that are not unique integers (Vahid's review
of #1332, finding 2).

The bridge test checks the stage-2 keys, and the locality test now also
checks a stage-2 draw: dropping a recipient or reversing the recipients
moves no other row's draw. Keying stage 2 on the twin's id fails them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 21, 2026
Only the WAS, LCFS and ETB tests ran the shared locality check. Salary
sacrifice now runs it too: reversing the people, changing one and adding
another moves no other person's uniform, and a positional stream fails
it. The UC reporter redraw checks that each redrawn unit's uniforms are
the ones keyed on its own benefit unit id, which no row order can move
(Vahid's review of #1332, finding 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 21, 2026
The FRS spine caps each adult's net-pay deduction at their usual
earnings. A deduction beside no recorded pay may stand for pay the adult
tab left blank, not for a contribution above pay, so the stage evidence
now counts the two cases apart (above_recorded_pay and
beside_no_recorded_pay), each with banded support and amounts withheld
below ten source households (Vahid's review of #1332, finding 7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 21, 2026
…in's rate

The symmetric rule carried the FRS twin's scheme record onto the SPI row:
a member twin's rate applied to the row's own pay even where the forest
drew no contribution, and a twin outside any scheme cleared the forest's
draw. But the record describes the twin's job at the twin's pay, and the
row's pay is redrawn, often far from the twin's. On the arms it put
members at low drawn pay, where few people are members, so participation
by pay band moved away from the FRS profile that the forest reproduces
(Vahid's review of #1332, finding 6).

Now the forest's draw decides membership, since it reads the row's own
drawn pay. A member whose twin has an ASHE rate takes that rate times
its own pay, which still scales the amount to pay above the FRS's range;
any other member keeps the forest's amount, bounded by ASHE's top rate.
Nothing is cleared any more, so the receipt drops
employer_none_outside_twin_scheme. The rule test and the SPI stage test
fail under both earlier rules (María's ruling of 21 October).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

@vahid-ahmadi thanks. Pushed c11ac1471, six commits.

  1. must, fixed in c7fca10a5. The engine-workflow pin is 19460fc1…, the value the F0 report carries, and its comment names the attested fit.qrf bytes. With policyengine-us 2.2.1 locally, the multispine pool tool test and both inventory coverage tests pass (40 tests), and tools/spec_engine_coverage.py --check passes.
  2. should, fixed in fb9617c70. Stage 2 keys on the row's own person_id, so two copies of one FRS person draw apart, as under the positional stream. It refuses ids that are not unique integers, and the comment now says why stage 2 differs from stage 1. The bridge test checks the stage-2 keys, and the locality test checks a stage-2 draw. Keying stage 2 on person_source_id fails both.
  3. should, agreed. This PR's evidence for the property is the fixture tests. The release-level check comes from Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333's arm, compared with this PR's spine: the records whose draws change against the records whose engine inputs change. The C0 to C1 deltas mix the one-off redraw, the pension rule and the rail exclusion, and the body now says so.
  4. should, fixed in d7299b412: the fragment says the switch redraws these stages once for every record.
  5. should. C1 runs on 2.134.4, which does not count employer contributions as pension input, so it cannot answer this. Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333's arm, on the release that carries #2536, will give the three figures against Tables 6 and 7 before Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333 merges.
  6. should, fixed in c11ac1471. You were right to ask. Carrying the twin's scheme record put members at low drawn pay, where few people are members: the record describes the twin's job at the twin's pay, and the row's pay is redrawn. The forest's draw now decides membership, since it reads the row's own pay. On C0, where SPI rows carry the forest's membership, participation by weekly pay band follows the FRS's. The twin's rate still scales a member's amount to its own pay. Nothing is cleared any more, so the receipt drops employer_none_outside_twin_scheme. The DWP employer row fits within 0.1% of its target on both arms.
  7. should, fixed in b7efc8085. The receipt now splits capped rows into above_recorded_pay and beside_no_recorded_pay. On FRS 2024-25, deductions above a recorded pay rest on fewer than ten source households. Nearly all the capped mass sits beside no recorded pay, so the cap mostly removes contributions where the adult tab records no usual earnings. The cap stays here, since the engine cannot relieve a contribution without pay. Whether the job tab records pay for these adults is a separate question for a follow-up.
  8. should, fixed in 1fbadc048 and fb9617c70. Salary sacrifice runs the shared locality check, and a positional stream fails it. The UC reporter redraw checks that each unit's uniforms are keyed on its own benefit unit id, and SPI stage 2 is covered as in item 2.
  9. should. Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333 is stacked on this branch, so it follows this PR. Between this PR and UK pre-Budget data: measure stored values at their period, draw Carer's Allowance claims, and the #1278 items (#1324, #1319, #1278) #1329, whichever lands second regenerates the H2 fixture and fixes the register count ("58 entries").
  10. nit, fixed in d7299b412: the guard names its dependency on no later stage lowering pay.
  11. nit, fixed in d7299b412. integer_record_keys refuses ids that are not unique integers in every keyed stage. A test shows that 1001 and 1001.0 key different draws.

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

@juaristi22 thanks, this is my second review, at c11ac1471. It covers the delta from 1258b13ff and your reply.

Nothing in the delta blocks this PR once it is rebased on #1329. CI is green at head, both engine-us jobs included. The stage-2 keying is fixed and the integer-key guard is a good addition. Three things carry to #1333: the annual allowance figures (5), the first build of the final employer rule (6), and the release-level invariance check (3). pe-uk 2.142.0, released at 13:04 today, carries #2536, so all three can now come from one #1333 arm. One correction to your reply on item 2, and one follow-up I would like filed now (item 7).

What I ran

  • Fresh worktree at c11ac1471, uv sync --all-packages --locked --extra uk (policyengine-uk 2.134.4).
  • Every touched UK test file, plus the WAS, LCFS and ETB stage tests and tests/engine/uk/test_uk_spi_income.py: 352 tests, all pass.
  • Two mutations, in a separate worktree:
    • salary sacrifice keyed on row position: test_draws_move_only_for_the_people_whose_inputs_move fails.
    • SPI stage 2 keyed back on person_source_id: test_stage1_queries_only_drawn_recipients_with_the_bridge fails, but test_child_exclusion_keeps_the_other_recipients_draws passes (see item 2).
  • tools/ci_test_plan.py verify: ok.
  • A merge simulation of this branch over UK pre-Budget data: measure stored values at their period, draw Carer's Allowance claims, and the #1278 items (#1324, #1319, #1278) #1329's head (2279fe821), and over current main.
  • CI at head (run 39595974154, at c11ac1471): all ten jobs pass, including engine-us 3.13 and 3.14.
  • Not run: the US engine-workflow test locally (no policyengine-us in a UK venv; CI covers it), and anything that needs the licensed national frame, so no arm, receipt count or HMRC comparison.

Round-1 items

  1. must: fixed. c7fca10a5 pins 19460fc1… with a comment naming the fit.qrf bytes. Both engine-us jobs pass at head.
  2. should: fixed, with one correction. Stage 2 keys on person_id and refuses ids that are not unique integers (spi_income.py:1228-1231), and the comment says why stage 2 differs from stage 1. Your reply says keying on person_source_id fails both tests. In my run it fails only the bridge test. The locality test still passes, because its fixture has no two rows with one source id (the uniqueness guard would otherwise have raised). So the bridge test pins the key, and on real data the guard would catch twin keying, since copies share a source id. That is enough. If you want the locality test to cover it too, add a second copy of one FRS person to its fixture.
  3. should: agreed, carried to Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333. I accept that this PR's evidence is the fixture tests. The PR body now separates the three effects in the C0 to C1 deltas.
  4. should: fixed in d7299b412. The fragment states the one-off redraw.
  5. should: open, carried to Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333, and now larger. I accept that C1 on 2.134.4 cannot answer it. Two things have changed since round 1. First, 2.142.0 carries #2536, so Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333 can lock to it and give the three figures against HMRC's Tables 6 and 7 (30,440 people, £672m excess, £0.3bn charge). Second, the new rule under item 6 gives more SPI-channel rows an employer contribution at redrawn pay, so the exposure is larger than in round 1, not smaller. The twin is still chosen by region, not income, so a member on £500k with a 25% twin still gets £125k of employer input. 2.142.0 also adds defined_benefit_pension_share. Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333 should say whether Microcosm writes it or leaves the engine's default, because that changes the input amount these rows carry.
  6. should: fixed in code, not yet measured. Membership now follows the forest's draw on the row's own pay, which answers the participation point. The C0 evidence supports it, since C0 already carried the forest's membership. Two caveats:
    • No arm has run this rule. C1 ran the old one, so "the DWP employer row fits within 0.1% on both arms" is about the old rule. That row is also bound, so a close fit shows the solver doing its job, not that the rule is right. The useful number from Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333's arm is the SPI-channel employer total before calibration, against C0.
    • A member whose twin has no rate keeps the forest's amount, which does not follow the redrawn pay. No receipt counts those rows unless the ASHE ceiling bounds them. One more receipt entry, employer_kept_forest_amount, would show what share of SPI-channel members actually scale with pay.
  7. should: fixed as asked, and the answer confirms the worry. The receipt now splits capped rows. employment_income comes only from ADULT.INEARNS (frs_spine.py:565), so "nearly all the capped mass sits beside no recorded pay" means these adults have a job and a pension deduction but no employment income in the spine. That is missing pay, not just a capped contribution. I agree it does not belong in this PR, but please file the follow-up now and link it here, so it does not get lost. The question for it: does the job table record gross pay for these adults where the adult table does not?
  8. should: fixed. Salary sacrifice runs the shared locality check, and my mutation shows it bites. The UC reporter test checks the keys directly against stable_identity_uniforms of the unit ids, which is enough there.
  9. should: agreed. See the merge order below.
  10. nit: fixed.
  11. nit: fixed. integer_record_keys is used in all six keyed stages, and the test pins the 1001 against 1001.0 case.

New findings

  1. nit: the PR body is stale in two places. Under "Keyed draws" it still says the SPI second forest keys on person_source_id. Under "Tests" every count is from 4198bddfe or earlier, before the final employer rule. Please update both before merge, since the body is what the release notes will be read against.

Merge order

The order is #1329, then #1332, then #1333:

  1. UK pre-Budget data: measure stored values at their period, draw Carer's Allowance claims, and the #1278 items (#1324, #1319, #1278) #1329 merges first.
  2. Draw UK QRF stages from uniforms keyed on each record's id, and tie pension contributions to pay #1332 is rebased on main after UK pre-Budget data: measure stored values at their period, draw Carer's Allowance claims, and the #1278 items (#1324, #1319, #1278) #1329. It merges cleanly on current main. Over UK pre-Budget data: measure stored values at their period, draw Carer's Allowance claims, and the #1278 items (#1324, #1319, #1278) #1329's head, five files conflict: calibration_measure_exclusions.json, test_uk_frs_spine.py, test_uk_measure_simulation.py and the two parity source JSONs (cgt_asset_type_facts.json, property_purchase_targets.json). frs_spine.py and spi_income.py merge without conflicts, but both PRs change them, so please re-run their tests after the rebase. The rebase also regenerates the H2 fixture and fixes the register count ("58 entries").
  3. Lock policyengine-uk 2.145.0, follow its bus fares and bind thin target rows #1333 is rebased on that and locked to 2.142.0, which carries #2536. Its title still says 2.141.1. Its arm answers items 3, 5 and 6.

What looks right

  • integer_record_keys refuses floats, strings and repeats in one place, and every keyed stage now calls it.
  • The new employer rule is simpler than the old one and drops a receipt that is no longer needed.
  • The stage-2 comment says plainly why stage 1 and stage 2 key differently.
  • The salary-sacrifice guard now names the stages it depends on.
  • The receipt split keeps the same rule: support as a band, and weighted amounts withheld below ten source households.

🤖 Generated with Claude Code

juaristi22 and others added 11 commits October 21, 2026 15:05
fit_draw_next takes quantiles and sign_uniforms (both or neither), one in
[0, 1) per recipient row, and then draws as predict_from_uniforms does,
leaving the chain's draw stream where it was. Keyed on stable record ids,
a row's draw then depends on its own uniforms and features only: adding,
dropping or reordering other recipients moves no row's draw. Without them
the step consumes the positional stream as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six UK stages drew from a positional stream: was_wealth, the
lcfs_consumption chain, etb_services, uc_reporter_redraw, the SPI stage-2
fill and salary_sacrifice. A positional stream gives a row the uniform of
its position, so any change in the rows before it, or in the frame's
length, redraws it. On #1308's arms an engine bump moved a few households'
net income; was_wealth, which reads it, redrew them; the high-value
support copies then changed the frame's length, and every later
imputation redrew nearly every household, including households whose own
inputs had not moved.

Each draw now reads stable_identity_uniforms keyed on the household,
benefit unit or person id (person_source_id in the SPI fill, as stage 1
does) and a salt naming the stage and target. Copy ids are offsets of
their source's, so the keys hold across engines. Chains keep their order,
and every stage keeps its declared seed.

A shared check (test_support/microcosm_build/uk_keyed_draws.py) reverses
the recipients, changes one and adds another, and requires every other
recipient to draw what it drew before; the was_wealth, lcfs_consumption
and etb_services tests run it, and fail with the positional stream back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-uk relieves an employee's contributions through the net pay
arrangement, which cannot take more than the pay they come out of, and
from 2.134.5 it counts every contribution, the employer's included, as
pension input for the annual allowance. Two readings broke that link:

- the FRS records each job's pension deduction (JOB.DEDUC1) apart from
  the adult's usual earnings (ADULT.INEARNS), so a deduction could exceed
  the pay beside it;
- an SPI-channel row redraws its pay from the tape, while the stage-2
  forest draws its contributions. The forest reads the drawn pay, but it
  draws whether the row has an employer contribution apart from the
  twin's scheme record, and it can only draw amounts that FRS rows in the
  same leaf received, so it cannot scale them to pay above the FRS's
  range.

uk_runtime/pension_coherence.py caps employee contributions at pay, on
FRS rows as frs_spine reads them and on SPI rows after stage 2. An SPI
row takes its employer contribution as #1069 sets an FRS row's: its FRS
twin's scheme record times its own pay. A member twin carries the rate
its ASHE draw gave it, and a twin with pay outside any employer scheme
carries none, so the row's membership is the twin's both ways (María's
ruling of 20 October). Only a row whose twin has no pay, and so no
scheme record, keeps the forest's draw, bounded by ASHE's top rate times
its pay. Personal contributions stay as recorded: the engine limits their
relief itself.

Each step records a receipt in the stage evidence: support banded on
distinct source households, and weighted amounts withheld as "<10" where
one to nine households carry them. salary_sacrifice now refuses an
employee contribution above the pay that remains, which the conversion
can no longer produce; its tests keep their inputs inside that rule. New
tests pin the cap, the rule both ways, the receipts and the SPI stage's
wiring; the one-sided rule and an uncapped FRS deduction both fail them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 of the SPI income stage draws from uniforms keyed on
person_source_id too (42fb9f6), so the engine test that captured stage
1's query also saw stage 2's call and failed on its targets. The test
now records each stage's query by the targets it draws. It also checks
that stage 2 is queried over the whole age domain and that every row
draws from uniforms keyed on its own identity under the stage's seed,
which no test covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…chain step

fit_draw_next now takes caller-owned uniforms (c5276aa), and
microcosm.fit.qrf is one of the kernel modules the seed protocol attests,
so the protocol's content digest and the compiled seed stream map digest
legitimately move, as they did when predict_from_uniforms landed
(1734b9e). Nothing in the spec engine or the US spec changes, and main
still matches the old digests. The committed F0 coverage report is
regenerated from the re-pinned inventory, and
tools/spec_engine_coverage.py --check passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The US bundle's semantic hash covers microcosm.fit.qrf's attested bytes,
which fit_draw_next's caller-owned uniforms change, as the seed protocol
re-pin records. The engine-workflow tier pins the hash apart, and the
four --check tools do not reach it (Vahid's review of #1332, finding 1).
The value is the one the regenerated F0 coverage report carries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stable_identity_uniforms formats each id into its key, so the integer
1001 and the float 1001.0 draw differently, and a merge that turned an
id column into floats would silently redraw every record. A repeated id
would give two records the same uniforms. integer_record_keys refuses
both, and every keyed UK stage (was_wealth, lcfs_consumption,
etb_services, uc_reporter_redraw and salary_sacrifice) passes its ids
through it (Vahid's review of #1332, nit 11).

The changelog fragment now says that the switch redraws these stages
once for every record (finding 4), and salary_sacrifice's guard names
its dependency: no stage between the pay caps and this one may lower pay
(nit 10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 keyed its uniforms on person_source_id, the FRS twin's id. The
SPI channel and the band donors copy one FRS person more than once, so
those copies shared every stage-2 uniform and drew comonotone values,
where the positional stream had drawn them apart. Stage 1 shares the
twin's identity on purpose and counts it; stage 2 now keys on the row's
own person_id, which is just as stable (a copy's id is an offset of its
source's), and refuses ids that are not unique integers (Vahid's review
of #1332, finding 2).

The bridge test checks the stage-2 keys, and the locality test now also
checks a stage-2 draw: dropping a recipient or reversing the recipients
moves no other row's draw. Keying stage 2 on the twin's id fails them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the WAS, LCFS and ETB tests ran the shared locality check. Salary
sacrifice now runs it too: reversing the people, changing one and adding
another moves no other person's uniform, and a positional stream fails
it. The UC reporter redraw checks that each redrawn unit's uniforms are
the ones keyed on its own benefit unit id, which no row order can move
(Vahid's review of #1332, finding 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The FRS spine caps each adult's net-pay deduction at their usual
earnings. A deduction beside no recorded pay may stand for pay the adult
tab left blank, not for a contribution above pay, so the stage evidence
now counts the two cases apart (above_recorded_pay and
beside_no_recorded_pay), each with banded support and amounts withheld
below ten source households (Vahid's review of #1332, finding 7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in's rate

The symmetric rule carried the FRS twin's scheme record onto the SPI row:
a member twin's rate applied to the row's own pay even where the forest
drew no contribution, and a twin outside any scheme cleared the forest's
draw. But the record describes the twin's job at the twin's pay, and the
row's pay is redrawn, often far from the twin's. On the arms it put
members at low drawn pay, where few people are members, so participation
by pay band moved away from the FRS profile that the forest reproduces
(Vahid's review of #1332, finding 6).

Now the forest's draw decides membership, since it reads the row's own
drawn pay. A member whose twin has an ASHE rate takes that rate times
its own pay, which still scales the amount to pay above the FRS's range;
any other member keeps the forest's amount, bounded by ASHE's top rate.
Nothing is cleared any more, so the receipt drops
employer_none_outside_twin_scheme. The rule test and the SPI stage test
fail under both earlier rules (María's ruling of 21 October).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The keyed draws change the hermetic H2 fixture's CGT asset-type facts and
property purchase targets, and main (#1329, #1295) moved the same two
files, so they are regenerated on the rebased tree with
tools/graph_uk_spine_fixture.py; --check then passes. They are re-derived
synthetic aggregates with no record-level data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-keyed-draws-pension-coherence branch from c11ac14 to 1599324 Compare October 21, 2026 14:54
@juaristi22
juaristi22 merged commit 954257d into main Oct 21, 2026
10 checks passed
DTrim99 added a commit that referenced this pull request Oct 21, 2026
Main moved the seed protocol, the compiled seed map and the bundle
spec_sha256 (microcosm#104, #1124, #1301, #1332); this branch moved the
source-manifest, resource-semantics, tail and full-checkpoint digests.
The conflicts keep both sides' changes; the merged tree's identity values
are re-derived on Modal in the next commit, and the F0 attestation is
regenerated there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
DTrim99 added a commit that referenced this pull request Oct 22, 2026
Reran build_1073_scf_education_debt_receipt.py on Modal at 2a6e65a (run
20261022T153020Z-2a6e65a73), after the merge with main brought in #1332's
QRF change. Every fit figure is unchanged; only the receipt's commit, run
and date move.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
MaxGhenis pushed a commit that referenced this pull request Nov 6, 2026
* Impute federal student loan balances from the 2025 SCF (#1073)

Add the scf_education_debt runtime module. It carries federal education
debt from the 2025 SCF through two weighted QRFs and allocates it to the
SCF-style reference person and spouse as four person inputs:
federal_student_loan_balance and federal_parent_plus_loan_balance
(policyengine-us#9721), is_income_based_repayment_new_borrower and
federal_student_loan_graduate_share (policyengine-us#9724).

- Only loans flagged federal count; X7179 is spread over loans 1-6 in
  proportion to their balances; whose-education code 3 is Parent PLUS
  on the reference person, codes 6, 7 and -7 the reference person's own.
- scf_wealth declares the two pinned 2025 files as artifacts 3 and 4
  with a new scf_2025 vintage. Net worth, liquid assets and auto loans
  stay on 2022.
- The fiscal-refresh tool runs the stage only with --scf-education-debt.
  The four columns join the stored-input register with the pending-
  engine reason the parent ids carry; the coverage manifest is left
  until the engine pin defines them.
- Regenerate sources, vintages and calibration YAML and re-pin the
  frozen source_stages.json and calibration projection digests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Add SCF 2025 education-debt fit receipts (#1073)

Weighted fit of the stage against SCF 2025: source checks against the
Fed's Alternate Table 13 (19.92%, $25,500, matched exactly), a five-fold
family hold-out with a predictor ablation, and the stage imputed onto
the published default populace-us-2024-spm-20260915 at its weights.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* WIP: SCF-matching recipient definitions and #1073 host-fit diagnostics

Codes the CPS recipient side the way the SCF defines it: married includes
cohabiting partners, children count the partner's children, and the
reference person follows the householder-couple rule. These move Parent
PLUS from 21.5% to about 16% of imputed federal balances (SCF 13.8%) but
leave the borrower-household share at about 14.2% (SCF 18.2%).

Adds diagnosis and experiment scripts and their first receipts. The
predictor-shift and remedy runs still need to run (on Modal; local memory
is too low); the fit receipt and PR body are not updated yet.

Part of #1073.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Condition the education-debt QRFs on SCF-comparable predictors (#1073)

The first host fit drew federal education debt for 14.4% of CPS
households against 18.2% in SCF 2025, and gave Parent PLUS 21.5% of
balances against 13.8%, while the SCF hold-out reproduced the SCF. The
cause is covariate shift: the CPS host carries tax-record imputations of
interest and dividends and of Social Security and pensions, which put
42% and 18% of 25-34 households above zero against 16% and 6% of SCF
families. Mapping each to its SCF distribution at equal rank moves the
host's borrower share most of any predictor; all together reach the
hold-out level (receipts/1073-predictor-shift-diagnosis.json).

- SCF_EDUCATION_DEBT_CANDIDATE_PREDICTORS keeps every predictor both
  sides build; SCF_EDUCATION_DEBT_PREDICTORS, used by both QRFs, leaves
  out the two shifted income components, each with its reason in
  SCF_EDUCATION_DEBT_EXCLUDED_PREDICTORS.
- The recipient definition sums income over the reference person,
  spouse or partner and their children, the CPS counterpart of the SCF
  primary economic unit, on top of the SCF-matching codings.
- The stage spec drift check now covers both QRFs; the stage notes,
  module docstring and changelog say why.
- The receipt builder reports the build, the first receipt and the
  all-candidates setting side by side and records commit and run id.
- experiments/modal_1073.py runs the heavy steps on Modal, adds generate
  and spec steps, and collects every tracked file a run changed.

Re-pins the frozen source_stages.json digest; the generated mirrors are
regenerated on Modal in a follow-up commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Regenerate the US stage spec for the SCF-comparable education-debt predictors (#1073)

Output of tools/generate_us_bundle_from_constants.py (run on Modal at
b9f3d67, run 20261020T154855Z-b9f3d6719): both education-debt QRFs drop
interest_dividend_income and social_security_pension_income, the stage
notes follow the module, and the stage-asset digest moves.

The Modal experiment runner now strips the Modal runtime's PYTHONPATH from
step environments; inheriting it put Modal's own packages beside the venv's
and the worker-identity check refused the duplicate idna.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Make the spouse-routing test independent of draw chance (#1073)

test_a_spouse_draw_without_a_spouse_moves_to_the_reference_person checks
where a spouse's draw lands. With the synthetic donor's random incidence
and a 30-household frame, the SCF-comparable predictor set draws no debt
for any couple on 3 of 20 seeds (seed 11 among them), while the routing
assertion holds on all 20 for both predictor sets (Modal probe run
20261020T161732Z-5d047f687). Every donor family now borrows in this test.

The Modal runner gains --script for an uncommitted local probe, recorded
by sha256 in the run log.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Collect the e35c4b0 receipts, add a Modal launch entrypoint, bind the loan register entries (#1073)

- experiments/1073-scf-education-debt-receipts.md and its JSON are the
  receipt of Modal run 20261021T133110Z-e35c4b0f9 at e35c4b0 (build,
  first-receipt and all-candidates settings on the published default).
- experiments/modal_1073.py gains `launch`, which spawns the run and
  exits, so a stopped local client cannot cancel it, and `--script` for
  an uncommitted probe.
- test_support/microcosm_data/stored_inputs.py lists the four federal
  student loan inputs among the producer-bound register entries, next to
  the parent ids: no file examined for #1026 stores them, and
  test_us_stored_input_register.py binds them to their live producer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec-engine identity values the education-debt stage moves (#1073)

The scf_wealth education-debt declaration adds 145 configuration fields to
the US bundle (141 authored, 4 resolved). Modal probe run
20261021T145336Z-b1a461f19 printed every value below from the code; none
is derived by hand. It also diffed the field universe against origin/main:
all 159 added and 14 removed pointers belong to exactly one claim, and no
field is unclaimed or claimed twice.

- field_usage: 42,329 configuration / 32,545 authored / 9,784 resolved
  (was 42,184 / 32,404 / 9,780); claim pins source_stages 1,845 (the two
  2025 artifacts and six operations of scf_wealth), vintage_records 120
  and resolved_vintage_authorities 67 (scf_2025), calibration 316 (the four
  person outputs in the QRF tail-concentration list), each with its
  observed digest; ledger mode and generation-0 effect counts in the
  field-usage and coverage-tool tests.
- inventory_coverage: source_manifest, late_resource_semantics, tail and
  full_checkpoint digests; tail control fields 938 and resolved
  references 336.
- The pool tool's bundle spec_sha256 (a6f254a8...).
- experiments/modal_1073.py: the spec step first runs the bundle generator
  and the coverage attestation in --check mode.

The coverage tool's pointer-inventory digest and the regenerated
docs/evidence/spec-engine/us-f0-coverage.json follow in the next commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Pin the coverage pointer inventory and regenerate the F0 attestation (#1073)

Modal probe run 20261021T150509Z-fcbd704c9, with the previous commit's
field-usage and inventory pins in place, printed the coverage tool's
pointer-inventory digest (37ce78d6...) and regenerated
docs/evidence/spec-engine/us-f0-coverage.json with tools/spec_engine_coverage.py;
--check then reported 42329/42329 configuration fields and 41/41 inventory
checks. The bundle spec_sha256 stays a6f254a8....

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-pin the inventory test's resolved-reference and tail-control counts (#1073)

test_support/microcosm_build/spec_engine_inventory_coverage.py keeps its own
copy of the reviewed inventory counts. Modal run 20261021T151444Z-05856e6ca
reported the two the stage moves, matching inventory_coverage.py and probe
run 20261021T145336Z-b1a461f19: resolved references 336 (the scf_2025
vintage) and tail control fields 938 (the four outputs in the QRF
tail-concentration list). Everything else in that run passed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the merged tree's bundle digest and F0 attestation (#1073)

Modal run 20261021T183247Z-97bb3af92 regenerated the bundle mirrors on the
merge commit (no change) and re-derived every identity value there: the
field-usage counts and claim pins, the ledger counts, the inventory
digests and counts, and the pointer-inventory digest all match the
committed values. The only moved value is the bundle spec_sha256, now
90fa3260... (main's kernel changes plus this branch's stage), which the
pool tool pins. tools/spec_engine_coverage.py regenerated the attestation
and --check reported 42329/42329 configuration fields and 41/41 inventory
checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Pin the #1073 fit receipt to the merged commit

Reran build_1073_scf_education_debt_receipt.py on Modal at 2a6e65a (run
20261022T153020Z-2a6e65a73), after the merge with main brought in #1332's
QRF change. Every fit figure is unchanged; only the receipt's commit, run
and date move.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Describe #1073 as the student debt pilot in the changelog and module docstring

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Route loans for a child's education to co-resident adult children and swap X8000 codes on loans 4 and 6 (#1073)

Parent PLUS is drawn for itself before the own shares, the IBR new-borrower
share is labelled an upper bound, and the receipt benchmarks Parent PLUS and
borrowers against Federal Student Aid.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Keep the education-debt stage operations within the sources schema (#1073)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values for the child-loan routing (#1073)

Regenerated on Modal (run 20261023T143414Z-b93cc3ec0): sources.yaml, the
field-usage counts and source_stages claim pin, three inventory digests, the
coverage pointer-inventory digest, the bundle spec_sha256 and the F0
attestation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Rerun the #1073 receipts at the merged head and benchmark Parent PLUS against FSA

Modal run 20261023T152015Z-60255dd98 at 60255dd (tests, spec and receipt
steps all passed). Parent PLUS falls from 13.8% to 9.1% of SCF federal
balances and from 16.8% to 10.7% on the host; FSA's Direct Parent PLUS share
is 6.7%, a floor.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values on the merge with main (#1073)

Regenerated on Modal (run 20261102T162227Z-3c4cfe441): sources.yaml, the
field-usage counts (42,489 configuration fields, 32,704 authored and 9,785
resolved), the source_stages and calibration claim pins, four inventory
digests (source_manifest, late_resource_semantics, tail, full_checkpoint),
the resolved-reference and tail-control counts (337, 942), the coverage
pointer-inventory digest, the calibration projection digest and the bundle
spec_sha256. The source_stages.json byte digest is the merged file's. The F0
attestation is from run 20261102T163748Z-c012b92c9 (write, then --check and
the generator's --check: 42,489/42,489 configuration fields, 41/41
inventory checks).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Rerun the #1073 receipts on the merge with main

Modal run 20261102T172355Z-3297aefd8 at 3297aef. No reported number moved:
the fit JSON differs from the 60255dd receipt only in its provenance and in
the last digit or two of 14 floats (floating-point summation order).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values on the merge with main (#1073)

#1444 changes the attested kernel us_runtime/housing_inputs.py. Running the
documented sequence locally (repin --legacy-only, the bundle generator, the
repinner, the coverage tool) moved:

- seed_map and seed_protocol in inventory_coverage.py EXPECTED_HASHES, to
  main's values (07fe9086..., 4ff99232...), which the merge had already
  carried into the coverage evidence's map/protocol/implementation digests;
- the bundle spec_sha256, be81ebbc... -> 7d2aaa4d..., in
  test_us_multispine_pool_tool.py and both spec_sha256 fields of
  docs/evidence/spec-engine/us-f0-coverage.json.

Nothing else moved: source_stages.json, the bundle YAML, the field-usage
counts (42,489 configuration fields, 32,704 authored, 9,785 resolved), the
source_manifest, late_resource_semantics, tail, full_checkpoint and take_up
digests, and the resolved-reference and tail-control counts (337, 942) are
unchanged. The coverage tool reports 42489/42489 configuration fields and
41/41 inventory checks.

The repinner's first pass builds its inventory report against the stale seed
pins and wrote covered_item_count == 40 into test_spec_engine_coverage_tool.py;
a second pass restored 41, so that file is unchanged here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LwQq6Jmpqc2fSZAM219btF

* Count #1073's module in the transport battery's binding identity (#1073)

engine-free failed on the merge ref with test_the_batterys_binding_identity_
adds_only_the_leaf_and_its_imports: assert 221 == (197 + 23). The battery's
binding closure reaches the us_runtime package, which imports
us_runtime.scf_education_debt (us_runtime/__init__.py:786), so the merged
tree holds one module more than main's base of 197. Diffing the closure on
main (220) and on the merge (221) shows that module as the only addition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LwQq6Jmpqc2fSZAM219btF

* Hold graduate child loans and the dependent limit to the loan rules (#1073)

A loan for a child's education that no listed child fits is Parent PLUS only
when it financed undergraduate study (34 CFR 685.200(c)(2)(i)); a graduate
one is now the head's own loan, as codes 6, 7 and -7 are. On the host, a
child share drawn for a household without an adult child splits the same
way: its graduate part joins the head's own balance, the rest is Parent PLUS.

The dependent undergraduate aggregate limit follows the loan year: $17,250
before July 1, 1993, $23,000 from then (Pub. L. 102-325) and $31,000 from July
1, 2008 (Pub. L. 110-227), with 1993 and 2008 loans counted as on or after
July 1. The docstring, stage notes and changelog say the over-the-limit rule
leans toward Parent PLUS, and the receipt sizes a $57,500 reading.

The loader checks local SCF files against the pinned member SHA-256s. The
receipt counts the loans each rule moves, the X8000 swap by loan position,
the host's moved child shares and the host's children-in-law. Citations:
20 U.S.C. 1087vv(d)(1), 34 CFR 685.209(b)(13)(ii), program 16.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Rerun the #1073 receipts for the graduate-loan and dependent-limit rules

Modal run 20261103T172225Z-78928589e at 7892858. The receipt's loans-moved
table header is relabelled ("Rule") in the builder and in the committed note,
which otherwise is the run's output.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Update the #1073 changelog figures to the new receipt

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values for the new stage notes (#1073)

The scf_wealth stage notes changed in 7892858, so the documented repin
sequence ran on Modal at that commit (run 20261103T172238Z-78928589e):
repin --legacy-only, the bundle generator, the repinner (twice), the
coverage tool, then each tool's --check, all passing. Moved: the frozen
source_stages.json digest (de488b42...), sources.yaml, the source_manifest,
late_resource_semantics and full_checkpoint inventory digests, the bundle
spec_sha256 (4fd75d58...) and the F0 attestation. The field-usage counts,
claim pins and inventory counts did not move. ruff format turns the
repinner's single-quoted literals, and 0caf490's two seed pins, into
double quotes; the values are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values on the merge with main (#1073)

Ran the documented sequence locally on the merge with main 31c8939:
repin --legacy-only, the bundle generator, the repinner, the coverage tool,
then the repinner once more (the second pass changed nothing). Moved:
- the field-usage counts: 42,394 configuration fields (was 42,489),
  32,609 authored (was 32,704); compiler_semantic 27,673,
  front_end_validation 349, no_generation0_effect 3,614. Resolved bindings
  (9,785), legacy_behavior (14,269 / 38,780) and identity_only (103) did not
  move. The counts are the merge base's (42,305) plus this branch's 184
  minus the 95 authored fields main dropped;
- the coverage pointer-inventory digest (f6832e8d...);
- the bundle spec_sha256 (ce8b1865...) in test_us_multispine_pool_tool.py and
  both spec_sha256 fields of us-f0-coverage.json;
- the F0 attestation's documentation_sha256 and claim_count (48: main's
  #1484 retired the catalog unit waiver and its claim).
Nothing else moved: source_stages.json and its frozen digest, sources.yaml,
and the inventory_coverage.py digests are unchanged. The merge commit kept
main's seed_map and seed_protocol, which the coverage tool accepts. The
coverage tool reports 42394/42394 configuration fields and 41/41 inventory
checks. ruff format turns the repinner's single-quoted literals into double
quotes; the values are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Count the two modules #1342 adds to the transport bindings' identity (#1073)

The merged head's binding identity holds 223 modules: 200 without the
leaf's 23. This branch added us_runtime.scf_education_debt (198, b1218b0),
and main's #1342 adds microcosm.calibrate.hierarchy_labels and
microcosm.diagnostics.catalogs, which the bindings reach through
microcosm.calibrate's package __init__. Main fails the same assertion at
31c8939 (222 == 197 + 23); #1495, still open, counts them there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sum an only child's loans against the dependent limit, and pin the boundaries (#1073)

Review of 2026-11-03 (second round):

- The dependent undergraduate limit caps a student's total. When exactly
  one listed child fits the family's federal undergraduate loans for a
  child, those loans are read year by year, oldest first: a year's loans
  within the limit on their own are the parent's once the child's total
  through that year would pass that year's limit, and the parent's loans
  do not count toward it. With two or more fitting children each loan is
  still compared with the limit alone, which leans toward the child; the
  docstring, stage notes and changelog say so, and the receipt counts
  those loans. _DEPENDENT_LIMIT_SUMS_AN_ONLY_CHILDS_LOANS switches the
  build reading off for the receipt's per-loan reading, and
  _scf_education_loan_holders returns what the rule read (_LoanHolders).
- Undergraduate means programs 10-12 and 16 (_UNDERGRADUATE_PROGRAM_CODES,
  checked against the 2025 codebook, which lists 10-16 and 0), so a loan
  with any other code is never parents-only.
- The moved-dollar columns compute part / sums * total
  (_part_of_drawn_total), so a subnormal share sum gives no NaN.
- The $46,000 pre-July-2008 limit for a dependent student whose parent
  cannot get PLUS joins the docstring, notes and changelog.

Tests: routing cases on the age-16 boundary (code 25, 2014, $9,000) and
the age-24 boundary (code 30, 2021, $40,000), and an unlisted program
code; test_the_dependent_limit_sums_an_only_fitting_childs_loans (two
$20,000 2016 loans against one $40,000 loan, oldest-first years, two
fitting children, the per-loan reading); the split test now draws shares
that sum to 0.6 and a non-new child loan for a new head, and asserts the
head's flag comes out False; a direct test of the subnormal guard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Count the summed-limit and pre-2008 loans in the #1073 receipt

- child_education_loan_moves counts and sizes the code-3 undergraduate
  loans, each within its year's limit, that the child's total through
  their year cannot take, by holder under the build reading and under a
  new per-loan reading (per_loan_limit, also in the readings table), and
  the code-3 undergraduate loans borrowed before 2008 that fit a listed
  child who could still have been dependent, by where they fall against
  their year's limit and $31,000.
- The reading for a parent who cannot get PLUS uses $46,000 before July
  2008 and $57,500 from then (limit_when_a_parent_cannot_get_plus).
- host_children_in_law also counts a PECOHAB partner of an adult child.
- The host draw check compares the four balance columns, placed on the
  build's people, as well as the weighted total.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Rerun the #1073 receipts for the summed dependent limit

Modal run 20261104T190152Z-63776f2ce at 63776f2. Summing an only fitting
child's loans moves 27 pooled code-3 loans (5 in implicate 1; $3.79B,
0.38% of SCF federal balances) to Parent PLUS: the SCF's Parent PLUS share
goes from 8.99% to 9.36%. Of the 49 loans each within its year's limit but
over the child's remaining room, 22 stay the child's in families with two
or more fitting children. No code-3 undergraduate loan before 2008 fits a
dependent-age child. The refit second QRF puts the host's Parent PLUS share
at 10.7% (was 9.7%) and the hold-out's at 6.8% (was 6.5%); household
totals are unchanged. The repeated host draw matches the build in its
total and in all four balance columns. With PECOHAB partners, 0.97% of
host households have an adult child's spouse or partner present. The
changelog's figures follow the receipt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values for the summed-limit stage notes (#1073)

The scf_wealth stage notes changed in b4f9260, so the documented repin
sequence ran on Modal at 63776f2 (run 20261104T190206Z-63776f2ce): repin
--legacy-only, the bundle generator, the repinner (twice), the coverage
tool, ruff format on the changed files, then each tool's --check, all
passing (42,394/42,394 configuration fields, 41/41 inventory checks).
Moved: the frozen source_stages.json digest (098c90d5...), sources.yaml,
the source_manifest, late_resource_semantics and full_checkpoint inventory
digests, the bundle spec_sha256 (cd762197...) and the F0 attestation. The
field-usage counts, claim pins and inventory counts did not move. ruff
format turns the seed_map and seed_protocol pins' single quotes into double
quotes; the values are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Pin the summing rule's parts and note the Perkins lean (#1073)

Third review of #1246. test_the_summing_rule_for_an_only_fitting_child
pins nine parts of the summed dependent limit (oldest first, the parent's
years out of the total, every kept year in it, a loan over the limit out
of its year's group, a loan at the limit counted, private loans out,
pooled loans deciding how many children fit, and the 1993-2007 and
pre-1993 limits), and
test_over_the_total_skips_a_year_the_child_could_not_be_dependent pins
the dependency check in the running comparison. Both are the reviewer's.

The module docstring's within-limit line gains the exception for an only
fitting child whose total through that year exceeds that year's limit,
and its lean bullet, the stage notes and the changelog note that the SCF
lists Perkins loans among federal student loans while 34 CFR 685.203(e)
caps only Direct, Stafford and SLS loans. The receipt builder's docstring
says federal for the pre-2008 count and describes over_the_total as the
family's running total. The stage notes' spec identity values are
re-derived after the merge with main that follows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values after the merge with main (#1073)

The documented sequence in tools/repin_us_spec_engine_contracts.py, run
on Modal (20261105T212737Z-525506fef): repin --legacy-only, the bundle
generator, the repinner twice, the coverage tool, ruff format, then every
tool's --check. What moved:

- the frozen source_stages.json SHA-256 (the stage notes' Perkins line):
  sources.yaml, the generator, the test-support bundle and the core
  contract test;
- the inventory digests full_checkpoint, late_resource_semantics and
  source_manifest; seed_map and seed_protocol keep main's values,
  requoted;
- the bundle spec_sha256 in the multispine pool test;
- the F0 coverage attestation (42,394/42,394 fields, 41/41 checks).

The field counts and claim pins did not move.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Nov 7, 2026
…folio (#1074) (#1357)

* Impute federal student loan balances from the 2025 SCF (#1073)

Add the scf_education_debt runtime module. It carries federal education
debt from the 2025 SCF through two weighted QRFs and allocates it to the
SCF-style reference person and spouse as four person inputs:
federal_student_loan_balance and federal_parent_plus_loan_balance
(policyengine-us#9721), is_income_based_repayment_new_borrower and
federal_student_loan_graduate_share (policyengine-us#9724).

- Only loans flagged federal count; X7179 is spread over loans 1-6 in
  proportion to their balances; whose-education code 3 is Parent PLUS
  on the reference person, codes 6, 7 and -7 the reference person's own.
- scf_wealth declares the two pinned 2025 files as artifacts 3 and 4
  with a new scf_2025 vintage. Net worth, liquid assets and auto loans
  stay on 2022.
- The fiscal-refresh tool runs the stage only with --scf-education-debt.
  The four columns join the stored-input register with the pending-
  engine reason the parent ids carry; the coverage manifest is left
  until the engine pin defines them.
- Regenerate sources, vintages and calibration YAML and re-pin the
  frozen source_stages.json and calibration projection digests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Add SCF 2025 education-debt fit receipts (#1073)

Weighted fit of the stage against SCF 2025: source checks against the
Fed's Alternate Table 13 (19.92%, $25,500, matched exactly), a five-fold
family hold-out with a predictor ablation, and the stage imputed onto
the published default populace-us-2024-spm-20260915 at its weights.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* WIP: SCF-matching recipient definitions and #1073 host-fit diagnostics

Codes the CPS recipient side the way the SCF defines it: married includes
cohabiting partners, children count the partner's children, and the
reference person follows the householder-couple rule. These move Parent
PLUS from 21.5% to about 16% of imputed federal balances (SCF 13.8%) but
leave the borrower-household share at about 14.2% (SCF 18.2%).

Adds diagnosis and experiment scripts and their first receipts. The
predictor-shift and remedy runs still need to run (on Modal; local memory
is too low); the fit receipt and PR body are not updated yet.

Part of #1073.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Condition the education-debt QRFs on SCF-comparable predictors (#1073)

The first host fit drew federal education debt for 14.4% of CPS
households against 18.2% in SCF 2025, and gave Parent PLUS 21.5% of
balances against 13.8%, while the SCF hold-out reproduced the SCF. The
cause is covariate shift: the CPS host carries tax-record imputations of
interest and dividends and of Social Security and pensions, which put
42% and 18% of 25-34 households above zero against 16% and 6% of SCF
families. Mapping each to its SCF distribution at equal rank moves the
host's borrower share most of any predictor; all together reach the
hold-out level (receipts/1073-predictor-shift-diagnosis.json).

- SCF_EDUCATION_DEBT_CANDIDATE_PREDICTORS keeps every predictor both
  sides build; SCF_EDUCATION_DEBT_PREDICTORS, used by both QRFs, leaves
  out the two shifted income components, each with its reason in
  SCF_EDUCATION_DEBT_EXCLUDED_PREDICTORS.
- The recipient definition sums income over the reference person,
  spouse or partner and their children, the CPS counterpart of the SCF
  primary economic unit, on top of the SCF-matching codings.
- The stage spec drift check now covers both QRFs; the stage notes,
  module docstring and changelog say why.
- The receipt builder reports the build, the first receipt and the
  all-candidates setting side by side and records commit and run id.
- experiments/modal_1073.py runs the heavy steps on Modal, adds generate
  and spec steps, and collects every tracked file a run changed.

Re-pins the frozen source_stages.json digest; the generated mirrors are
regenerated on Modal in a follow-up commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Regenerate the US stage spec for the SCF-comparable education-debt predictors (#1073)

Output of tools/generate_us_bundle_from_constants.py (run on Modal at
b9f3d67, run 20261020T154855Z-b9f3d6719): both education-debt QRFs drop
interest_dividend_income and social_security_pension_income, the stage
notes follow the module, and the stage-asset digest moves.

The Modal experiment runner now strips the Modal runtime's PYTHONPATH from
step environments; inheriting it put Modal's own packages beside the venv's
and the worker-identity check refused the duplicate idna.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Make the spouse-routing test independent of draw chance (#1073)

test_a_spouse_draw_without_a_spouse_moves_to_the_reference_person checks
where a spouse's draw lands. With the synthetic donor's random incidence
and a 30-household frame, the SCF-comparable predictor set draws no debt
for any couple on 3 of 20 seeds (seed 11 among them), while the routing
assertion holds on all 20 for both predictor sets (Modal probe run
20261020T161732Z-5d047f687). Every donor family now borrows in this test.

The Modal runner gains --script for an uncommitted local probe, recorded
by sha256 in the run log.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Collect the e35c4b0 receipts, add a Modal launch entrypoint, bind the loan register entries (#1073)

- experiments/1073-scf-education-debt-receipts.md and its JSON are the
  receipt of Modal run 20261021T133110Z-e35c4b0f9 at e35c4b0 (build,
  first-receipt and all-candidates settings on the published default).
- experiments/modal_1073.py gains `launch`, which spawns the run and
  exits, so a stopped local client cannot cancel it, and `--script` for
  an uncommitted probe.
- test_support/microcosm_data/stored_inputs.py lists the four federal
  student loan inputs among the producer-bound register entries, next to
  the parent ids: no file examined for #1026 stores them, and
  test_us_stored_input_register.py binds them to their live producer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec-engine identity values the education-debt stage moves (#1073)

The scf_wealth education-debt declaration adds 145 configuration fields to
the US bundle (141 authored, 4 resolved). Modal probe run
20261021T145336Z-b1a461f19 printed every value below from the code; none
is derived by hand. It also diffed the field universe against origin/main:
all 159 added and 14 removed pointers belong to exactly one claim, and no
field is unclaimed or claimed twice.

- field_usage: 42,329 configuration / 32,545 authored / 9,784 resolved
  (was 42,184 / 32,404 / 9,780); claim pins source_stages 1,845 (the two
  2025 artifacts and six operations of scf_wealth), vintage_records 120
  and resolved_vintage_authorities 67 (scf_2025), calibration 316 (the four
  person outputs in the QRF tail-concentration list), each with its
  observed digest; ledger mode and generation-0 effect counts in the
  field-usage and coverage-tool tests.
- inventory_coverage: source_manifest, late_resource_semantics, tail and
  full_checkpoint digests; tail control fields 938 and resolved
  references 336.
- The pool tool's bundle spec_sha256 (a6f254a8...).
- experiments/modal_1073.py: the spec step first runs the bundle generator
  and the coverage attestation in --check mode.

The coverage tool's pointer-inventory digest and the regenerated
docs/evidence/spec-engine/us-f0-coverage.json follow in the next commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Pin the coverage pointer inventory and regenerate the F0 attestation (#1073)

Modal probe run 20261021T150509Z-fcbd704c9, with the previous commit's
field-usage and inventory pins in place, printed the coverage tool's
pointer-inventory digest (37ce78d6...) and regenerated
docs/evidence/spec-engine/us-f0-coverage.json with tools/spec_engine_coverage.py;
--check then reported 42329/42329 configuration fields and 41/41 inventory
checks. The bundle spec_sha256 stays a6f254a8....

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-pin the inventory test's resolved-reference and tail-control counts (#1073)

test_support/microcosm_build/spec_engine_inventory_coverage.py keeps its own
copy of the reviewed inventory counts. Modal run 20261021T151444Z-05856e6ca
reported the two the stage moves, matching inventory_coverage.py and probe
run 20261021T145336Z-b1a461f19: resolved references 336 (the scf_2025
vintage) and tail control fields 938 (the four outputs in the QRF
tail-concentration list). Everything else in that run passed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the merged tree's bundle digest and F0 attestation (#1073)

Modal run 20261021T183247Z-97bb3af92 regenerated the bundle mirrors on the
merge commit (no change) and re-derived every identity value there: the
field-usage counts and claim pins, the ledger counts, the inventory
digests and counts, and the pointer-inventory digest all match the
committed values. The only moved value is the bundle spec_sha256, now
90fa3260... (main's kernel changes plus this branch's stage), which the
pool tool pins. tools/spec_engine_coverage.py regenerated the attestation
and --check reported 42329/42329 configuration fields and 41/41 inventory
checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Pin the #1073 fit receipt to the merged commit

Reran build_1073_scf_education_debt_receipt.py on Modal at 2a6e65a (run
20261022T153020Z-2a6e65a73), after the merge with main brought in #1332's
QRF change. Every fit figure is unchanged; only the receipt's commit, run
and date move.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Describe #1073 as the student debt pilot in the changelog and module docstring

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Route loans for a child's education to co-resident adult children and swap X8000 codes on loans 4 and 6 (#1073)

Parent PLUS is drawn for itself before the own shares, the IBR new-borrower
share is labelled an upper bound, and the receipt benchmarks Parent PLUS and
borrowers against Federal Student Aid.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Keep the education-debt stage operations within the sources schema (#1073)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values for the child-loan routing (#1073)

Regenerated on Modal (run 20261023T143414Z-b93cc3ec0): sources.yaml, the
field-usage counts and source_stages claim pin, three inventory digests, the
coverage pointer-inventory digest, the bundle spec_sha256 and the F0
attestation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Rerun the #1073 receipts at the merged head and benchmark Parent PLUS against FSA

Modal run 20261023T152015Z-60255dd98 at 60255dd (tests, spec and receipt
steps all passed). Parent PLUS falls from 13.8% to 9.1% of SCF federal
balances and from 16.8% to 10.7% on the host; FSA's Direct Parent PLUS share
is 6.7%, a floor.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Calibrate the SCF-imputed student loan balances to Federal Student Aid portfolio facts (#1074)

A new runtime module, fsa_student_loan_targets, binds Federal Student Aid's
portfolio tables (the six packages/ed_fsa packages of chronicle#347) to the
SCF education-debt stage's person columns, federal_student_loan_balance and
federal_parent_plus_loan_balance: the balance and borrower count of each
state and DC, of five borrower age bands and of nine debt-size bands, and
the Parent PLUS balance beside the balance of every other loan type. Every
family binds at FY2026 Q2 (31 March 2026), the latest quarter all four bound
tables publish, as a stock unaged at the 2024 build period (target aging's
new declared stock_at_source_reference_date alignment). Not Reported is
allocated to the reported locations, Puerto Rico and Other leave with their
share, and the age, debt-size and loan-type rows bind as shares of the one
in-scope control ($1,679.1B, 44.31M borrowers). Repayment plans, the
portfolio summary and loan-type recipient counts stay diagnostic, each with
its reason in the receipt.

compile_us_fiscal_target_registry binds the facts only with
student_loan_targets, which the fiscal-refresh release tool sets with
--scf-education-debt; the release materializes the rows from the stored
person columns, guards their restated age and balance bands, records every
fact's disposition in us_source_coverage.json and keeps the opt-in family
off the target-parity contract. Default compiles and releases are unchanged.
The ED Federal Student Aid provider joins the calibration label registries.

experiments/fit_1074_fsa_calibration.py refits the published default with
and without the family (plus L2-anchored arms) on Modal through
experiments/modal_1074.py, against the pinned feed plus chronicle#347's FSA
rows, now built at its merge commit on Chronicle main.

Squashes #1357's earlier history onto #1246's review-round head.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Refit the #1074 receipt on #1246's child-loan routing and the merged chronicle#347 (#1074)

Modal run 20261023T191527Z-4b6bfce9a at 4b6bfce. chronicle#347's export
at its merge commit (acbedb5) is byte-identical to the PR head's (sha256
58774e11...), so the candidate feed and the test fixture are unchanged.

With loans for a child's education on co-resident adult children, the host
gains young borrowers (1,246 borrower records under 25, from 561) and its
Parent PLUS share falls from 16.8% to 10.7%, but holds the same borrowers
(21.2M) and borrower households (16.4%) at the published weights. Arm B now
meets 95% of the FSA targets within 5% (92% before) and every age band;
B + L2 78% (76%). Borrower households' share of weight after calibration is
33.7% (B) and 32.3% (B + L2), against 33.0% and 31.5% before and 18.2% of
SCF families: the routing moves balances within households, so it does not
shrink the doubling. Non-FSA capped error: B 3.62% (3.66%), B + L2 3.85%
(3.83%), A unchanged at 3.30%.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-derive the spec identity values on the merge with main (#1073)

Regenerated on Modal (run 20261102T162227Z-3c4cfe441): sources.yaml, the
field-usage counts (42,489 configuration fields, 32,704 authored and 9,785
resolved), the source_stages and calibration claim pins, four inventory
digests (source_manifest, late_resource_semantics, tail, full_checkpoint),
the resolved-reference and tail-control counts (337, 942), the coverage
pointer-inventory digest, the calibration projection digest and the bundle
spec_sha256. The source_stages.json byte digest is the merged file's. The F0
attestation is from run 20261102T163748Z-c012b92c9 (write, then --check and
the generator's --check: 42,489/42,489 configuration fields, 41/41
inventory checks).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Rerun the #1073 receipts on the merge with main

Modal run 20261102T172355Z-3297aefd8 at 3297aef. No reported number moved:
the fit JSON differs from the 60255dd receipt only in its provenance and in
the last digit or two of 14 floats (floating-point summation order).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Add down-weighted borrower-count arms to the #1074 fit (#1074)

Six arms bind the 67 FSA balance targets at full loss weight and the 65
borrower-count targets (state, age band, debt-size band) at 0, 0.1 and 0.25
of their weight, renormalized to mean 1, with and without the L2 anchor. At 0
the counts only validate. The existing arms are unchanged and the receipt
keeps their tables; a new section compares the arms on the FSA balance and
count fit, the non-FSA surface, the share of households with debt, total
borrowers, the Parent PLUS share and the borrower age profile against FSA.

The receipt's reference-date sentence now says what the stored balances are:
the March 2026 stock written at the 2024 build period, which
policyengine-us#10415 (merged 2026-10-30) holds at its stored value in every
year. The runner and fit docstrings say the candidate keeps the 51aa40f feed,
the pin before main's cf9ece2 re-pin, which adds only the B4923C facts that
bind no target.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Refit the #1074 receipt with the down-weighted borrower-count arms (#1074)

Modal run 20261102T174003Z-88d7a950d at 88d7a95. The merge with #1246's
updated head moved nothing: all 4,363 numbers of the existing arms are
identical to the 4b6bfce receipt.

Binding the balances with the borrower counts at 0.25, 0.1 and 0 of their
weight lowers the share of households with debt from 33.7% to 32.7%, 31.7%
and 30.2% (B + L2: 32.3% to 28.2%, 26.7% and 25.7%; SCF 18.2%), and the
borrowers from 44.26M to 42.98M, 41.71M and 39.75M (B + L2: 42.67M to 34.11M
at 0). Balance targets within 5%: 99% in B, 96% in each down-weighted arm (B +
L2 94%, 88% to 87%); count targets within 5%: 92% to 43%, 31% and 17%. The
solver reaches the balances with larger balances per borrower, so the bands
under $10K fall short. Non-FSA capped error 3.45% to 3.50% (B 3.62%, A
3.30%); Parent PLUS 6.8% to 6.9% throughout.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Restamp the FSA fixture v3 and refuse a debt build with no FSA facts (#1074)

The fixture's 244 rows now carry chronicle.consumer_fact.v3, as every row
of the feed microcosm#1506 pins (Chronicle 41b2fb2b) does, so each matches
one of that feed's FSA rows byte for byte; the stamp is the only change.

The release tool refuses a --scf-education-debt build on a feed with no
ed_fsa facts before the compile (_check_us_feed_carries_fsa_facts): the
plan was empty there, so the build compiled no FSA target and raised
nothing. The us_source_coverage.json receipt reuses the check's plan.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Keep FSA loan-type rows diagnostic and refuse changed FSA tables

Both loan-type dollar rows leave the loss, so the family binds 130 targets.
Originating a Direct Consolidation Loan discharges the loans it consolidates
(34 CFR 685.220(a)), so FSA's Parent PLUS row covers only unconsolidated loans,
while the stage's column holds every loan for a child's education. Every other
family pins the total, so a target on either side of FSA's split would set the
same split.

The plan refuses a location, age or debt-size table whose rows differ from the
set FSA publishes at the reference quarter. The feed-refusal test reads _main's
syntax tree, and two Hypothesis properties pin the family totals and the band
partition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Refuse repeated FSA location rows and restate the Parent PLUS reason

_require_reviewed_rows compared sets of row IDs, so a repeated Puerto Rico
or Other row passed and changed the location control. It now compares
sorted lists and names any repeat, and a new test repeats each of those
two rows.

FSA's notes don't say whether its Parent PLUS row leaves out loans the
Secretary paid off through consolidation (34 CFR 685.220(f)), so the
docstring and the loan-type reason no longer say it does. Every other
family pins the total, so either loan-type row would still constrain the
split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Count the FSA targets module in the transport bindings' identity

us_runtime.fiscal_targets imports the new fsa_student_loan_targets module, so the
transport bindings' identity reaches one more module. test_the_batterys_binding_identity_adds_only_the_leaf_and_its_imports
names it in BINDING_MODULES_ADDED_AFTER_BASE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Ready the FSA fit for the 130-target rerun: L2 grid, income tables, release fit table (#1074)

Max's review of 2026-11-05, on top of #1525's diagnostic loan-type rows:

- The fit runs the review's grid: A and B at lambda 0, record L2 0.03 and
  chi-square L2 0.03 (calibrate's l2_basis), and B with the borrower counts
  at x0, at lambda 0 and record 0.03. --extra-chi-square-lambda adds A and B
  at another lambda. Every receipt table covers every arm.
- It reads FSA's Parent PLUS figure from the diagnostic loan-type facts,
  writes strict JSON (null, allow_nan=False) and refuses a feed that binds no
  FSA target.
- The receipt adds every household-income quintile for every arm, at each
  arm's own cut points and at the release arm's applied to every arm.
- A --scf-education-debt release writes us_scf_education_debt_calibrated_fit.json,
  the stage's fit table on the exported frame at its shipped weights, before
  post-export scoring; a test pins the write and its place in _main.
- A missing age counts in the youngest band (FSA_MISSING_AGE), now named,
  documented and tested.
- The doc and changelog cite 34 CFR 685.220(f) and give FSA's filing of a
  consolidated Parent PLUS loan as an inference, say the materializer groups
  the stored, unaged age, and drop the re-pin as a future step (#1506).
- The Modal runner builds the feed the commit pins by default and gains
  fit_smoke and lint steps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Tidy the FSA receipt's signs and the missing-age note (#1074)

The receipt writes a flat Q5 - Q1 as +0.0, not -0.0, and a negative income
as -$945,807. The FSA_MISSING_AGE comment says the plan, not FSA, spreads the
"not reported" ages, and the doc's Parent PLUS share reads as a share of the
loan-type dollar rows. The fit_smoke run 20261105T214517Z-66b37d37c showed both
signs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Limit the 1074 runner's format check to this PR's files (#1074)

Run 20261105T215640Z-f83241311 passed ruff check and every test, and its
format check flagged only eleven us_runtime modules this PR leaves untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Say the FSA loan-type table is diagnostic in the parity manifest (#1074)

families' reviewed-exclusion reason no longer lists the loan-type table among
the tables a --scf-education-debt build binds. The reason changes in the
builder and in the eleven manifest entries alike.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Refit the #1074 receipt on main: 130 targets and the L2 grid (#1074)

Modal run 20261106T142002Z-8a9d335a7 at 8a9d335, on the feed main pins
(Chronicle 41b2fb2b, sha256 1d45a89c..., 40,642 rows, rebuilt as pinned):
6,102 non-FSA and 130 FSA targets, none dropped, every arm of the review's
grid solved. B binds 95% of the FSA targets within 5% at ESS 2,932 and 33.9%
of households with debt; B + record L2 76% at 4,484 and 32.4%; B +
chi-square L2 59% at 6,222 and 28.0%. Chi-square 0.03 moves the fit, so no
second lambda. The doc's measured limits take the new levels.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Write the calibrated SCF fit after the export cleanup (#1074)

test_us_post_export_scoring executes _main's export and cleanup statements
on their own, without args, so the calibrated fit-table write inside that
span failed it with a NameError. The write now follows the ladder cleanup,
still before any scoring or terminal gate, and the builder test pins that
order.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Describe the Parent PLUS column as the stage's classification and drop the "moved" clause (#1074)

Since #1246 the stage gives a loan for a child's education to a fitting
co-resident adult child, keeps it as the head's Parent PLUS only past the
dependent limit or when no child fits an undergraduate loan, and makes a
graduate loan with no fitting child the head's own. The module docstring,
the loan-type reason and the doc now say the column holds the balances the
stage classifies as Parent PLUS. The calibrated fit table reports shares at
the shipped weights, not the FSA family's effect, so the doc and the release
tool's comment no longer say it reports what the family moved.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Max Ghenis <mghenis@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants