[LeoBorcherding]

Adds NVIDIA's Cosmos3-Nano-Sim-Transfer to sd.cpp: give it a control video (edge maps) and a prompt, and it streams out a photoreal video one latent frame at a time.

Why: it's the fastest Cosmos3 path (4 distilled steps, no CFG, KV cache), and it lets Unsloth Studio restyle a video on one GPU without Python (the Unsloth Studio side: unslothai/unsloth#16975).

Input | edges | output, 832x480, 121 frames:

Clips are CC0 / US-government public domain from Wikimedia Commons (sources).

What's in it

• src/model/diffusion/cosmos3.hpp: the 16B Mixture-of-Transformers (text and generation paths, interleaved mRoPE, GQA). Text K/V and the clean control/RGB history stay in the runner cache between graphs, stored as [d, tokens, kv_heads] so flash attention reads them without a copy.
• src/pipeline/cosmos3_transfer.cpp: the causal loop (seed the control frame, 4-step denoise, refresh the clean frame), the reference's prompt decoration and seeding, Wan2.2 VAE in and out.
• src/runtime/edge_control.h: a gradient edge filter (blur, Sobel, thinning, threshold) for --ref-video RGB input. Canny frames go in with --control-video.
• scripts/unsloth/ggml-patches/0014: the CUDA concat used to launch one kernel per outer slice, about 1.5M launches per Wan VAE decode. Now one launch when all three tensors are contiguous.
• scripts/convert_cosmos3.py merges the transformer shards. OpenMDW-1.1 notice in licenses/, NOTICE and the file headers.

Speed and quality

RTX PRO 6000 Blackwell (Colab G4), Linux, CUDA 13.0, 832x480, 121 frames, NVIDIA's example 2 (edge), seed 2025, full KV history, flash attention. PSNR is against NVIDIA's reference (cosmos-framework 98c35ed, bf16) on the same VM.

	e6a98b1	a8345e5 (head)	reference, eager PyTorch
transformer, 31 latent frames	15.9 s (7.6 fps of output)	12.8 s (9.4 fps)	11.3 s (10.7 fps)
VAE decode	14.3 s	10.2 s
rally clip, sd-cli end to end	35.6 s	27.6 s
PSNR vs reference, mean / worst frame	37.9 / 37.0 dB	37.9 / 37.0 dB	43.2 / 40.9 dB (two reference runs, same seed)

a8345e5 is bit-identical to e6a98b1 (latent max diff 0.0); it only removes copies. The port is still about 13% slower than NVIDIA's eager PyTorch reference on the same GPU; the gap is the attention over the growing history. NVIDIA quotes 8.3 fps for vLLM-Omni on this GPU. With the tiny autoencoder (--tae taew2_2.safetensors) the rally clip takes 15.4 s, but PSNR drops to 29.1 dB, so it stays opt-in. Per latent frame still grows from 238 ms to 471 ms as the history fills; that is attention over the history.

The model was trained on Canny edges. A soft gradient map gets copied as a glowing outline, so the built-in filter thins and thresholds (cosmos3_edge_threshold=0.2). Edge study.

Memory and speed

Measured at e6a98b1 and a8345e5 against cosmos-framework 98c35ed (bf16, eager), back to back on one Colab G4 (RTX PRO 6000, 96 GB), two runs each. Peak VRAM is 39.9 GiB at both commits, set in the VAE decode, and 0.9 GiB under the reference's 40.8 GiB. Timed with a lean wrapper (no tensor dumps), the reference sampled the 31 latent frames in 11.3 s (10.7 fps), ahead of this port's 13.0 s (9.3 fps). An earlier reference run that also dumped tensors took 17.0 s; that number is no longer used. vLLM-Omni's 8.3 fps is NVIDIA's figure, not run here.

Tests

• scripts/edge_control_test.cpp: passes.
• Reference comparison through the SD_COSMOS3_DEBUG_DIR hook (reads the reference's noise, tokens and control latents): prompt ids identical, initial noise within 2.2e-6, first velocity cosine 0.99975, final frames in the table above.
• Four clips with Canny, soft and thinned control: no NaN, no failures.
• Codex: no major issues at a8345e5 on the fork mirror (Cosmos3-Nano-Sim-Transfer: video-to-video from edge control, streamed with a KV cache LeoBorcherding/stable-diffusion.cpp#2).

Not run

• Anything but CUDA on Linux (one RTX PRO 6000). No Metal, Vulkan, ROCm or CPU. The concat patch only changes the CUDA backend.
• 720p, clips over 121 frames (so the bounded KV window is untested), q8_0 GGUF weights, and blur, depth or segmentation control.
• No CI on this fork PR.