[leejet]

Summary

• Fuse direct F16 output into the existing RMSNorm → Mul → RoPE path, eliminating the intermediate F32 output and separate conversion.
• Fuse weighted RMSNorm with MOSS modulation and optional shift, preserving the original reduction order and separately rounded arithmetic.
• Reuse existing ggml extension interfaces and shared kernel launch logic for F32 output and BF16 packing. No ggml changes or SM86-only restriction.

Adapted from @mirek190's work:

• leejet/ggml#18: direct RMSNorm/RoPE F16 output.
• leejet/ggml#20: normalization/modulation portion, producing F32 output.

The SiTU-to-Q8 packing fusion from leejet/ggml#19, and the Q8 packing and residual-chain fusion from #20, are outside this change. Existing native SiTU support remains enabled in both benchmark configurations.

Testing

Hardware: NVIDIA GeForce RTX 4090 24 GB, NVIDIA driver 596.36.

Build: Windows, CUDA 13.4, nvcc 13.4.92, Release.

DiT files:

Model	DiT file
MOSS-Image SFT	MOSS-Image-SFT-Q8_0.gguf

Sampling throughput in it/s; higher is better:

Model	Resolution	sd.cpp before	sd.cpp after
MOSS-Image SFT	1024×1024	1.54	1.59

Throughput improves by 2.90%, calculated from unrounded averages.

Configuration:

• CUDA backend, batch 1, CFG 3, Euler, discrete scheduler, seed 42.
• 20 steps per run; discard the first two steps and take the arithmetic mean of the printed it/s values for steps 3–20.
• Five fresh CLI runs per configuration after one excluded full-image warmup, with balanced execution order. All 20 steps are included.
• Same Q8_0 DiT weights, Qwen3.5-9B-Q4_K_M.gguf text encoder, flux2_ae.safetensors VAE, and tokenizer_qwen3.5.json.
• CPU offloading enabled; automatic flash attention selected CUDA MMA.
• Prompt: a lovely cat holding a sign says 'moss-image.cpp'.

Checklist

• I have read and confirmed this PR follows the contribution guidelines.

Summary by CodeRabbit

• Performance
  • Improved GPU execution for RMS normalization with affine and modulation operations, including residual paths.
  • Expanded fusion support for certain normalization, multiplication, and rotary embedding sequences.
• Compatibility
  • Updated image and video model modulation to use the same operation as before through a shared path.

[coderabbitai]

No actionable comments were generated in the recent review.

Walkthrough

The CUDA RMS-affine fusion now supports modulation operations, and MOSS uses ggml_ext_modulate for block and final video modulation. CUDA RoPE fusion also recognizes an additional sequence and adjusts FP16 output selection.

Changes: RMS-affine modulation

Layer / File(s)	Summary
Modulation matching and operation description src/core/cuda/rms_affine.cu	The optimizer and matcher recognize RMS_NORM–MUL–modulation sequences. Shared operation description distinguishes modulation, residual, and self-residual operations.
Kernel execution and workspace src/core/cuda/rms_affine.cu	The grouped kernel handles modulation and optional shift, as well as residual and self-residual operations. Execution checks overlap and uses shared launch configuration and operation description.
Fusion registration and MOSS call sites src/core/cuda/rms_affine.cu, src/model/diffusion/moss_image.hpp	A separate modulation kernel descriptor is registered. MOSS block and video paths call ggml_ext_modulate for scale-and-shift operations.

CUDA RoPE fusion

Layer / File(s)	Summary
RoPE matching and output selection src/core/cuda/rope.cu	The fusion checks that the RMSNorm input and weight differ, recognizes a four-node sequence ending in a contiguous F16 copy, and selects FP16 output for fused sequences longer than three nodes.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant Moss as MOSS forward
  participant Modulate as ggml_ext_modulate
  participant Matcher as CUDA RMS-affine matcher
  participant Kernel as CUDA RMS-affine kernel
  Moss->>Modulate: Apply scale and shift
  Matcher->>Matcher: Match RMS_NORM–MUL–modulation sequence
  Matcher->>Kernel: Dispatch fused operation

Loading

Possibly related PRs

• leejet/stable-diffusion.cpp#2265: Adds the initial CUDA RMS-affine residual matcher and kernel that this change extends.
• leejet/stable-diffusion.cpp#2258: Adds ggml_ext_modulate and CUDA modulation matching used by this change.
• leejet/stable-diffusion.cpp#2235: Adds the CUDA RMS-affine residual fusion that this change continues.

Merge Risk: Minimal · up to d7653

No actionable merge-blocking issue was established. The RoPE copy’s exact rounding equivalence remains unverified, but there is no demonstrated output mismatch.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title names two changes that the changeset contains: a CUDA fusion for MOSS modulation (rms_affine.cu, moss_image.hpp) and direct RoPE F16 output (rope.cu). It is concise, follows the conventional…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description includes a clear summary, related upstream discussions, detailed testing and performance data, and a completed checklist. It satisfies the repository template.

Full details: Title check

Explanation

The title names two changes that the changeset contains: a CUDA fusion for MOSS modulation (rms_affine.cu, moss_image.hpp) and direct RoPE F16 output (rope.cu). It is concise, follows the conventional commit style, and is specific enough for a teammate scanning history. It does not cover the broader kernel and optimizer changes, which is acceptable.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 12 functions across 3 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

Generate docstrings

• Commit to this branch
• Create a new PR

Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 2330980c-e66c-457b-a2f6-3454974cfa8a

Commits

Reviewing files that changed from the base of the PR and between d308544 and d765348.

Files selected for processing: 3

• src/core/cuda/rms_affine.cu
• src/core/cuda/rope.cu
• src/model/diffusion/moss_image.hpp

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.