Lossless speculative decoding for Apple Silicon (MLX): 2× faster Qwen3 generation on a Mac with a trained hidden-state drafter. MIT.
-
Updated
Oct 22, 2026 - Python
#
draft-model
Here are 20 public repositories matching this topic...
On-policy distillation for speculative decoding: train the draft model on the target's own rollouts, no offline feature cache. A fork of deepseek-ai/DeepSpec with DeepSeek-V4-Flash (DSpark) support.
inference-acceleration llm-training llm-inference speculative-decoding deepseek draft-model on-policy-distillation
-
Updated
Aug 3, 2026 - Python
Speculative decoding runtime with rejection sampling, adaptive gamma strategy, and provable correctness guarantees. Achieves 1.41x speedup on CPU with Qwen2-0.5B/1.5B pair. Draft model generates candidates, target model verifies in single forward pass. 31/31 tests passing.
machine-learning transformers pytorch language-models rejection-sampling inference-optimization llm-inference speculative-decoding adaptive-gamma draft-model
-
Updated
Jul 29, 2026 - Python
Multi-Future Drafting (MFD): mixture block drafters and an exact mixture tree for lossless speculative decoding. Inference code, SGLang patch and released drafters.
-
Updated
Nov 11, 2026 - Python
MiniCPM5-2B on a GTX 1080 (Pascal, sm_61): from 33 to 83 tokens/s with llama.cpp kernels, KV f16 and an FR-Spec draft, bit-identical where it matters
pascal gpu cuda nvidia cuda-kernels inference-optimization kv-cache llm llama-cpp ggml local-llm llm-inference local-ai speculative-decoding minicpm sm61 draft-model openbmb minicpm5 gtx-1080
-
Updated
Oct 24, 2026 - Python
Draft-target model speculative decoding verification engine with rejection sampling, acceptance rate telemetry, and dynamic speedup ratio estimation.
mcp zero-dependency sampling edge-ai inference-acceleration llm-serving speculative-decoding agentic-ai model-context-protocol mcp-server draft-model
-
Updated
Sep 28, 2026 - Python
Find the draft model (or n-gram lookup) that speeds up your local LLM: speculative-decoding sweeps for llama.cpp and vLLM with tok/s, acceptance rate, tokens per joule and peak VRAM, plus a pick per VRAM budget.
-
Updated
Nov 2, 2026 - Python
Retrained EAGLE-3 draft head for Spark-X2.6-4B: draft acceptance on Chinese prompts. Rebuilt 48K draft vocabulary, trained on the deployed Q6_K weights.
-
Updated
Oct 31, 2026 - Python
Nanbeige4.3-33B on one M5 Ultra with TensorFold native Metal: the bf16 release as published, with its bf16 Nanbeige4.1-3B drafter (replies identical, 1.3-7x)
metal llama structured-output apple-silicon openai-api bf16 llm-inference speculative-decoding tool-calling draft-model m5-ultra nanbeige tensorfold
-
Updated
Nov 3, 2026 - Python
Analytical benchmark for speculative decoding at batch sizes 1-64. Finds the gamma crossover where speculative decoding becomes slower than greedy, and shows that batch size does not degrade performance when continuous refill is active.
python simulation throughput llm-inference speculative-decoding continuous-batching draft-model mlsystems batched-inference gamma-tuning
-
Updated
Jul 19, 2026 - Python
CLI for building and testing DFlash-style speculative decoding draft models.
cuda transformers mlx huggingface apple-silicon vllm llm-inference speculative-decoding draft-model dflash
-
Updated
Jun 2, 2026 - Python
Tree-based speculative decoding benchmarked against linear under equal verification budget — branch factor, depth, and draft quality sweep with fair node-level comparison.
python benchmarking pytorch transformer medusa llm-inference speculative-decoding draft-model autoregressive-decoding tree-attention specinfer
-
Updated
Jul 17, 2026 - Python
Simulates speculative decoding to find the optimal speculation length K across 576 configurations (3 draft models x 8 K values x 6 acceptance rates x 4 cost ratios). Key findings: 6.06x max speedup, breakeven at cost_ratio=0.25, optimal K grows from 1-3 at 50% acceptance to 7-15 at 95% acceptance.
simulation gpu optimization inference transformer throughput cpp20 llm speculative-decoding draft-model
-
Updated
Jul 10, 2026 - Python
Calibrated simulation benchmark for real-time LLM request routing, comparing complexity signals, output-length awareness, cost savings, and quality-risk trade-offs.
performance-engineering benchmark simulation latency routing inference cost-model systems gpu-memory request-routing perplexity kv-cache llm long-context llm-serving machine-learning-infrastructure model-routing draft-model
-
Updated
Jul 25, 2026 - Python
Speculative-draft training pipeline (DFLASH/DSPARK/DFLASH2), orchestrating the vllm-project/speculators engine end to end, with preset configs for Qwen-family models
-
Updated
Sep 28, 2026 - Python
Measures real speculative decoding speedup using the official HuggingFace assistant_model API across 4 model pairs and output lengths up to 512 tokens. Best result: distilgpt2->gpt2-medium achieves 1.747x speedup at 512 output tokens. Validates that cost_ratio and output_length are the key parameters.
python gpu transformers inference pytorch throughput llm speculative-decoding draft-model rtx-2070 assisted-generation
-
Updated
Jul 13, 2026 - Python
MTP drafter for lossless speculative decoding of Qwen3.8-27B-AEON on MLX. Weights on Hugging Face.
-
Updated
Oct 2, 2026 - Python
multiple tokens, and a verifier filters them using the main model’s confidence. Focuses on speed–accuracy tradeoffs, visualization, and modular design for easy benchmarking and research.
visualization benchmarking acceleration research rejection-sampling modular-design llm-inference speculative-decoding token-verification verifier-guided-decoding draft-model efficient-generation speed-accuracy-tradeoff
-
Updated
Nov 9, 2025 - Jupyter Notebook
Measure speculative-decoding acceptance rate and expected speedup for a target/draft model pair
-
Updated
Sep 7, 2026 - Python
Train a speculative-decoding draft model from a target LLM's text outputs alone. Released draft for Qwen3.5-4B: 1.58 tokens per target pass, 1.43x projected speedup.
-
Updated
Oct 25, 2026 - Python
Add this topic to your repo
To associate your repository with the draft-model topic, visit your repo's landing page and select "manage topics."