Skip to content

[Fix] Preserve GLM-5.3 vision RoPE frequencies during meta initializa… - #2138

Merged
jayhenry merged 1 commit into
InternLM:feat/glm53flash-f2-vision-towerfrom
xs1997zju:fix/glm53-vision-rope-meta
Oct 8, 2026
Merged

jayhenry merged 1 commit into
InternLM:feat/glm53flash-f2-vision-towerfrom
xs1997zju:fix/glm53-vision-rope-meta

Conversation

@xs1997zju

@xs1997zju xs1997zju commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

This fixes uninitialized vision RoPE frequencies when GLM-5.3-Flash is constructed on the meta device and loaded from a checkpoint. inv_freq is nonpersistent: the loader materializes the meta buffer with to_empty but cannot restore its values from the checkpoint. Image patch embeddings agree with HF, while the first ViT block already diverges.

Create the FP32 frequency tensor explicitly on CPU, following the existing text rotary initialization approach. Add a meta-construction regression and a frequency assertion to the real-checkpoint loading test. No checkpoint keys or language-model behavior change.

Stack placement: targets feat/glm53flash-f2-vision-tower (#2110), so the fix can propagate into #2111. The diff contains only the vision initialization fix and its regression tests.

Validation:

  • Meta construction and existing FP32/BF16 image, video, and multi-image forward checks: 6 passed.
  • Local reduced GLM-5.3-Flash checkpoint, single-GPU BF16 eager vision, 20 MedPix images: all 347 vision/projector parameters agree with HF; after the fix, patch embedding, all 24 ViT blocks, post norm, and projected image features agree elementwise with native HF.
  • Full VLM on 20 paired image/text inputs: image features agree elementwise with HF; text logits and NLL are unchanged. Against HF, the maximum sampled image-input logits relative L2 falls from 17.03% to 12.00%. Remaining language-path differences are outside this fix.
  • No training/backward, FlashAttention, or post-fix SP2 validation is claimed for the real-checkpoint experiment. The existing unwaited-collective shutdown warning remains.

@jayhenry
jayhenry merged commit df2efc8 into InternLM:feat/glm53flash-f2-vision-tower Oct 8, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants