[macqueen09]

Summary

2026-10-19 multi-model follow-up: Neo/Nano/Mini size matrices are complete; Pro is partially measured. N4096 is not numerically accepted. The current Neo deployment graph specializes the single-cohort path even at large E; verified 8192-edge runtime batching is a separate diagnostic, not the current compiled deployment behavior. See the new tables below. Keep Draft/default-off.

Add a default-off experimental first-order SO(2) input-gradient path for frozen DPA4-Neo inference. Bound temporary gradient storage without changing weights, representation, precision or GEMM scheduling.

本PR是湖南大学提交的。

This Draft targets personal master at 668b60cc2a9559f79abb9c33acd959833376344f, not an upstream merge request or validation against subsequent upstream commits.

Implementation and scope

• Intended large-E runtime algorithm: complete the reverse mixing chain for an 8192-edge cohort before advancing; retain matrix-column parallelism, full saved activations and full returned input gradients. Actual make_fx/AOT deployment currently specializes the single-cohort branch at trace time, including measured Neo N4096. This dispatch defect is not fixed in this documentation update.
• Small E (<=16384): a single-cohort triton_op with compiled wrap_triton launches. Reuse only internally owned residual/output storage; never overwrite caller inputs or alias GEMM input/output across tiles.
• Separate alpha backward and full-graph saved/output strides from cohort scratch strides, including tails.
• Four existing native launch sites explicitly provide zero stride defaults to avoid optional arguments becoming None under Inductor.
• Opt in before import/compilation with DP_SO2_EDGE_COHORT_INFER=1; default zero. Checked contiguous FP32 Neo layout only (F2/Cf32/lmax3/ROW320, three layers/two gated/apply_alpha). Weight gradients, higher-order and extra saved-output gradients retain the existing path.
• No GEMM tile, gate/recompute/point schedules, neighbor list, weight or precision changes. Follow-up below changes measurement tools/evidence, not the production kernels.

Primary capacity: includes the first peak

RTX 5090 GPU0; driver595.91.07, PyTorch2.11.0+cu128/Triton3.6.0. Strict level2 mixing FP32, TF32/AMP off.
Neo checkpoint DPA4-Neo-OMat24-v20260627.pt SHA256 8d8b2b02b9575cb9e0af420cc1e8b1d9ffcbc04752c2203d29f48ccd0cd5f07d.
Normal ASE energy/force/virial/stress API; N64 diamond, E~10114, seed0/0.02 Angstrom perturbations.

Retain the earlier three independent cold processes per arm, empty explicit compiler caches, counters covering model load/first JIT/warmup/timed inference. System/driver caches not purged. Worst-of-three, decimal MB; first and full-window peaks coincide. These are not the dual-model timing experiment's memory.

Metric	BASE	ECS	Change
First/full allocated peak MB	498.199552	480.156672	−3.6216%
First/full reserved peak MB	528.482304	507.510784	−3.9683%
Steady allocated peak MB (secondary)	397.536256	379.493376	−4.5387%

No new cold-memory rerun is implied. Same production source hashes; no use of warm reset peaks to replace first capacity.

Controlled latency follow-up: no arbitrary 3% noise rule

The previous independent-cold P50 12.157611→12.313776ms (+1.2845%) remains in the raw history, but cannot be attributed entirely to ECS. Cache audit found 41/109 common non-SO2 autotune programs with different actual launch selections in the first candidate; baseline self-runs also changed selections. max_autotune=False does not disable all generic pointwise/reduction autotuning.

New main test: same GPU/context and normal full API, frozen cache/autotune choices, four independent processes counterbalancing BASE/ECS initialization order and PYTHONHASHSEED0/1. Each has24 ABBA blocks, each round8 warmups/50 samples: 96 blocks, 9600 samples per arm. Monitor foreign processes/clocks/power/temperature; no global settings or clocks changed.

Full API statistic	BASE	ECS
Pooled raw P50 ms	11.683581	11.708142
Pooled raw P90 ms	11.911459	11.932993

• Paired mean log-ratio effect: +0.07954% latency, process-level t95 interval [−0.38661%, +0.54788%].
• Ratio of pooled P50s: +0.2102%; different estimator, not a contradiction.
• Process effects: +0.2999%, −0.1536%, +0.3656%, −0.1924%, associated with initialization order. Exact low-level cause not uniquely isolated.
• Clock-matched post-stratification retains90/96 blocks: +0.08607%, t95[−0.38733%,+0.56172%]. All96 remain primary. Clocks were dynamic, not locked; monitor differences disclosed in report.
• Same-implementation A/A, six resident-pair controls, three serialized single-process pairs and compiled-graph-only replay are separately reported, not pooled as interchangeable samples.
• New controlled result does not support a fixed +1.28% ECS cost in this setup. It does not establish exact zero overhead or stable speedup. Four independent processes, not9600 independent experimental replicates.
• A possible stride simplification was tested twice (21 cases each, input-gradient bitwise equivalent); no benefit in the scoped microbenchmark, so not adopted.

Actual Neo energy inference uses make_fx→aot_module_simplified→compile_fx_inner, bypassing Dynamo; no claim that inference Dynamo guards caused the difference.

Accuracy and regression

Elementwise atol1e-5/rtol1e-4; relative denominator max(abs(reference),1e-8), never relaxed; finite required.

Earlier cold matrix: BASE self3 pairs, ECS self3, all9 cross-pairs, all6 processes versus fresh pure PyTorch passed; worst error/tolerance0.352841. Pure PyTorch has three self-repeats in one GPU1 process, not a new multi-process Torch matrix.

New four-process main test: three frames each, energy/force/virial/stress versus paired BASE and the fresh Torch reference all pass; worst error/tolerance 0.376674.
Full model tolerance equivalence is not promised bitwise equality.

60 focused CUDA tests re-passed (14warnings,50.53s), including tails/zero/tiny-alpha, caller ownership, native static/dynamic compilation and threshold guards. Training/higher-order guards are not complete training acceptance. Full-tree Ruff check/format and diff-check passed; no new native build/full backend suite claimed.

Multi-model and N4096 update (2026-10-19)

Snapshot: 95 process results, 91 complete, 4 OOM; 117 planned. Neo/Nano/Mini matrices and 9 true-runtime Neo diagnostics complete; Pro is in progress, so pending/partial cells are not acceptance. Production kernels unchanged. Three independent processes per model/size/mode, three identical perturbed-carbon frames, three repeats/frame. TORCH = compiled pure PyTorch level0, not eager/exact truth. All modes of each model/size use the same physical GPU. Fixed atol1e-5/rtol1e-4, all outputs finite required; values below are worst error/tolerance, <=1 within criterion, not a percentage. BASE self-failure never excuses candidate failure.

模型与尺寸总表

模型	原子数	Torch自身	BASE自身	PR候选自身	PR vs BASE	PR vs Torch	PR实际路线
Neo	512	0.685	0.714	0.863	0.959	1.030	single（非8192运行时分批）
Neo	1000	0.498	0.551	0.502	0.594	0.570	single（非8192运行时分批）
Neo	4096	OOM/缺参照	2.854	2.854	3.239	OOM/缺参照	single（非8192运行时分批）
Nano	512	8.177	5.504	5.704	6.556	8.634	回退原生，不能算ECS验证
Nano	1000	7.570	9.229	6.968	8.142	8.053	回退原生，不能算ECS验证
Nano	4096	2.816	2.557	2.127	2.438	3.052	回退原生，不能算ECS验证
Mini	512	2.326	1.541	2.770	2.164	2.636	回退原生，不能算ECS验证
Mini	1000	0.800	0.726	1.328	1.286	1.028	回退原生，不能算ECS验证
Mini	4096	18.006	8.724	8.598	10.488	14.464	回退原生，不能算ECS验证
Pro	64	0.075（1/3，未齐）	0.073（1/3，未齐）	0.121（2/3，未齐）	0.095（1/3，未齐）	0.144（1/3，未齐）	回退原生，不能算ECS验证
Pro	512	OOM/缺参照	待测	待测	待测	OOM/缺参照	回退原生，不能算ECS验证
Pro	4096	待测	待测	待测	待测	待测	回退原生，不能算ECS验证

真正8192边运行时分批：与PR单批路径分开

模型	原子数	自身	对BASE	对Torch	对PR单批	已观察真实cohort
Neo	512	1.052	1.047	1.037	1.104	[(80898, 8192, 10)]
Neo	1000	0.416	0.603	0.544	0.511	[(158002, 8192, 20)]
Neo	4096	3.002	3.527	OOM/缺参照	3.168	[(647170, 8192, 80)]

4096原子位力最大绝对误差范围（eV）

范围是各对输出max_abs的min~max，非置信区间；所有比较仅对同一帧进行。

模型	Torch自身	BASE自身	PR自身	PR vs BASE	PR vs Torch
Neo	OOM/缺参照	2.275e-05 ~ 2.513e-04	2.126e-05 ~ 1.705e-04	2.113e-05 ~ 2.482e-04	OOM/缺参照
Nano	9.505e-05 ~ 6.979e-04	5.914e-05 ~ 4.544e-04	6.502e-05 ~ 7.514e-04	7.473e-05 ~ 6.500e-04	9.420e-05 ~ 1.060e-03
Mini	5.062e-05 ~ 4.385e-04	4.477e-05 ~ 3.680e-04	3.166e-05 ~ 3.210e-04	4.190e-05 ~ 3.580e-04	7.984e-04 ~ 1.493e-03
Pro	待测	待测	待测	待测	待测

Nano/Mini/Pro are native fallbacks, not demonstrated ECS acceleration/support. Pro is official Pro-MPtrj v2026.06.05, not OMat24-Pro; original checkpoint MD5/SHA256 verified. All three Pro modes use the disclosed legacy metadata-only fitting alias, with strict tensor loading; original weights unchanged. Neo4096 TORCH OOMed in all3 processes. The 9-call accuracy samples include compilation and are not latency/P50 evidence. This matrix is pinned to PR base668b60cc, not latest upstream master.

Historical N4096 speed/capacity observation — not accepted current-PR performance

2026-10-18 runtime prototype, GPU0, one process/arm, 8 warmups/30 synchronized wall samples, same normal ASE energy/force/virial/stress API. GB=1e9 bytes. Includes the first call in the full-window memory high-water mark, excludes compile from timing. Different experiment from the controlled N64 study above.

Strategy	Frame P50 ms	Time reduction	Steady allocated GB	Steady reduction	Full-window allocated peak incl. first GB	Full-peak reduction
Historical BASE	156.495125	—	23.970400	—	24.061593	—
Historical runtime ECS	152.996582	2.2356%	22.256657	7.1494%	22.256657	7.5013%

P90: 156.614295→153.179559ms. Candidate first-call-only peak22.255608GB; full-window22.256657GB is the primary capacity metric. Historical virial error/tolerance1.394 failed, TORCH OOM; only one process/arm. These observations are not validated stable speedup or production acceptance. Current PR single-path precision, true-runtime diagnostics, and old prototype performance must not be interchanged.

• Full protocol/model identity/dispatch explanation and limitations
• Compact multi-model snapshot, per-output accuracy ranges, machine-readable summary
• Historical N4096 per-call timing/memory/accuracy JSON

Complete original matrix NPZ arrays and its orchestration harness are not included in this documentation update; published aggregate/route evidence is not a claim of an end-to-end independently auditable matrix archive. Existing public benchmark commands are linked in the supplement. No model weights, compiler caches or private reports committed.

Mega-kernel extent

This borrows bounded-workset/dependency scheduling ideas; it is not a persistent cross-layer mega-kernel and does not integrate HazyResearch's device taskqueue/loader/consumer/storer/controller runtime. Representative traces show 657 GPU launches/frame in both arms, including32 SO2 stack launches. Lifetime reduction can reduce peak storage without reducing launch count or latency. No automatic speedup claim.

Reproduction and raw evidence

• Controlled protocol, all diagnostics and reproducible commands
• Controlled summary JSON
• Compressed per-call raw JSON, including Event/wall samples, clocks and accuracy
• Original first-peak protocol and cold raw evidence.
• Public entrypoints in source/tests/pt_expt/kernels: benchmark_so2_edge_cohort.py (torch/base/ecs), benchmark_so2_edge_cohort_same_context.py, paired/serial controls and analyzers. Explicit root/checkpoint/reference/cache/output arguments. No model weights or compiler caches committed.

Existing work and outstanding gates

Internal residual reuse overlaps deepmodeling#6058; gains are not additive or wholly novel. Loop tiling/lifetime scheduling are established ideas; no general compiler novelty claim.

• Multi-model/multi-size acceptance and incremental comparison to existing inplace changes. Neo/Nano/Mini repeatability matrices are now recorded; Pro remains partial; non-Neo uses fallback and failures remain unresolved.
• Earlier N512/N4096 strict-accuracy failures remain unresolved, not reclassified as pass.
• Actual large-E make_fx/AOT branch specialization/dispatch, PT2/C++/CUDA Graph acceptance. Dynamo dynamic unit tests alone do not validate AOT bucket deployment.
• Wider training/higher-order validation, required CI/native build and other hardware.
• Exact initialization-order mechanism and comparable-tree performance PR comparisons.

Latest full-model acceptance covers Neo N64 only. Keep default-off Draft status.