Repository navigation
[Feature] Add GLM-5.3-Flash F3: Kimi Delta Attention (KDA) - #2106
Merged
jayhenry merged 5 commits intoOct 15, 2026
Merged
Conversation
jayhenry
added this pull request to stack #2112
September 23, 2026 06:54
This was referenced Sep 23, 2026
jayhenry
commented
Oct 8, 2026
| from fla.ops.kda import fused_recurrent_kda as _fused_recurrent_kda | ||
| from fla.ops.kda.gate import fused_kda_gate as _fused_kda_gate | ||
|
|
||
| class FusedRMSNormGated(_FLAFusedRMSNormGated): |
Collaborator
Author
There was a problem hiding this comment.
是否可以和 Qwen3.5 的合并?
jayhenry
commented
Oct 8, 2026
| batch_size, seq_len, _ = hidden_states.shape | ||
| assert batch_size == 1, "KDA currently supports packed batch size 1" | ||
|
|
||
| cu_seqlens = seq_ctx.cu_seq_lens_q |
Collaborator
Author
There was a problem hiding this comment.
这样做和使用 mixed_qkv相比,哪个好?
jayhenry
commented
Oct 8, 2026
| cu_seqlens = seq_ctx.cu_seq_lens_q | ||
| projection_size = self.num_heads * self.head_dim | ||
|
|
||
| q = self._sp_short_conv( |
Collaborator
Author
There was a problem hiding this comment.
直接让 self.q_conv1d 等支持 SP 是否更好?
This was referenced Oct 8, 2026
jayhenry
force-pushed
the
feat/glm53flash-f3-kda
branch
from
October 9, 2026 06:32
dbb1256 to
9bc5db3
Compare
jayhenry
force-pushed
the
feat/glm53flash-f3-kda
branch
from
October 12, 2026 19:18
ecdcdd5 to
64ab664
Compare
jayhenry
force-pushed
the
feat/glm53flash-f3-kda
branch
from
October 14, 2026 16:25
64ab664 to
6d36bf0
Compare
jayhenry
force-pushed
the
feat/glm53flash-f3-kda
branch
from
October 14, 2026 17:17
6d36bf0 to
1eec2f5
Compare
Adds xtuner/v1/module/attention/kda.py (KDAConfig / KimiDeltaAttention) following doc/xtuner_glm5p3flash_design.md F3, with module structure and parameter names matching the published checkpoint layout (separate q/k/v_conv1d, low-rank f_a_proj->f_b_proj forget gate with fp32 A_log/dt_bias, low-rank g_a_proj->g_b_proj output gate). Critically, the chunk_kda/fused_recurrent_kda call convention differs from the ~/github/xtuner-ncp-k3 reference: the installed fla 0.4.2's chunk_kda does not accept A_log/dt_bias/use_beta_sigmoid_in_kernel, so those get silently swallowed into **kwargs without raising if passed the ncp-k3 way, producing a KDA that trains but computes the wrong gate. The gate is now computed externally via fla.ops.kda.gate.fused_kda_gate before calling the kernel, matching HF's own Glm5NextTextLinearAttention reference forward. A regression test asserts these three kwargs stay absent from chunk_kda's signature. Relaxes TransformerConfig.linear_attention to GatedDeltaNetConfig | KDAConfig | None and the corresponding attention_config type annotations in dense/moe decoder layers and model builders (mypy-strict plumbing only, no behavior change for existing GatedDeltaNet layers). Verified with tests/model/test_glm53_kda.py (5 tests, single/2-GPU): gate parity against fla's naive reference, module-level parity against HF's Glm5NextTextLinearAttention (weights bridged per the published<->HF key mapping), packed multi-document isolation, and Ulysses SP parity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Compare the chunked KDA forward output and every module gradient against HF's Glm5NextTextLinearAttention on GPU, including the split causal convolutions that HF fuses into one conv1d.
GLM-5.3 HF parity tests require the glm5_next API introduced in Transformers 5.17. The upgrade also changes Qwen video interpolation baselines. Use the public video processor API, preserve fp32 vision RoPE, and mirror HF gated norm rounding in parity mode.
Co-authored-by: 纪焘 (Tao Ji) <taoji@fudan.edu.cn> Co-authored-by: Kuang <128125671+Knlife@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
jayhenry
force-pushed
the
feat/glm53flash-f3-kda
branch
from
October 15, 2026 03:55
1eec2f5 to
d8d52d5
Compare
jayhenry
removed this pull request from stack #2112
October 15, 2026 03:58
jayhenry
added this pull request to stack #2179
October 15, 2026 03:59
jayhenry
force-pushed
the
feat/glm53flash-f3-kda
branch
from
October 15, 2026 04:21
d8d52d5 to
00e96c4
Compare
YanhuiDua
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack #2179 (bottom to top):
fix/deepep-layout-event→mainfeat/glm53flash-materialize-full-f0→fix/deepep-layout-eventfeat/glm53flash-f3-kda→feat/glm53flash-materialize-full-f0← you are herefeat/glm53flash-f4-mhc→feat/glm53flash-f3-kdafeat/glm53flash-f5-nope-dsa→feat/glm53flash-f4-mhcfeat/glm53flash-f1-vl-data→feat/glm53flash-f5-nope-dsafeat/glm53flash-f2-vision-tower→feat/glm53flash-f1-vl-datafix/mtp-local-future-embeddings→feat/glm53flash-f2-vision-towerfeat/glm53flash-f6-text-moe→fix/mtp-local-future-embeddingsSummary
Stack layer 3/9 of GLM-5.3-Flash support (base: layer 2, F0 + materialize_full).
The KDA compile, precision, kernel-gradient and sequence-parallel fixes previously in F6 are now included in this layer. The shared
fp32_linearutility is here because the KDA gate uses it.Adds
xtuner/v1/module/attention/kda.py(KDAConfig/KimiDeltaAttention) with module structure and parameter names matching the published checkpoint layout (separateq/k/v_conv1d, low-rankf_a_proj->f_b_projforget gate with fp32A_log/dt_bias, low-rankg_a_proj->g_b_projoutput gate).Root cause of a real correctness bug found while implementing: the installed
fla0.4.2'schunk_kda/fused_recurrent_kdacall convention differs from thextuner-ncp-k3reference — it doesn't acceptA_log/dt_bias/use_beta_sigmoid_in_kernel, and silently swallows them into**kwargswithout raising, producing a KDA that trains but computes the wrong gate. Fixed by computing the gate externally viafla.ops.kda.gate.fused_kda_gatebefore calling the kernel, matching HF's ownGlm5NextTextLinearAttentionreference forward. A regression test pins these three kwargs' absence fromchunk_kda's signature so this can't silently regress.Also relaxes
TransformerConfig.linear_attentiontoGatedDeltaNetConfig | KDAConfig | None(mypy-strict plumbing only, no behavior change for existing GatedDeltaNet layers).The Read the Docs configs now mock
einops, which KDA imports while Sphinx loads XTuner with docs-only dependencies. English and Chinese HTML builds passed locally with Python 3.12 and Sphinx 6.2.1.Test Plan
tests/model/test_glm53_kda.py(5 tests, single/2-GPU): gate parity against fla's naive reference, module-level parity against HF'sGlm5NextTextLinearAttention(weights bridged per the published<->HF key mapping), packed multi-document isolation, and Ulysses SP parity.