continuous-batching

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server and Mac app for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Release-gated with Claude Code, Codex CLI, Aider, Hermes and DeepSeek Harness.

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

A high-performance API server that provides OpenAI-compatible endpoints for MLX models. Developed using Python and powered by the FastAPI framework, it provides an efficient, scalable, and user-friendly solution for running MLX-based vision and language models locally with an OpenAI-compatible interface.

A light llama-like llm inference framework based on the triton kernel.

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

An Efficient and Versatile Inference Engine for Distributed LLM Serving

面向开发者与初学者的 nano-vLLM 交互式源码教程 - 通过 13 个 HTML 互动实验 + 13 章中文教程理解大模型推理引擎

Reproducible llama.cpp CPU inference profiling and a deterministic LLM serving simulator with continuous batching, KV cache, prefix caching, and workload-driven latency analysis.

A lightweight, educational LLM inference engine for studying continuous batching, paged KV cache, chunked prefill, and online serving.

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

An AI inference engine from scratch. Like nanoGPT, but for serving.

🌱 A tiny, readable LLM serving engine with vLLM/SGLang-style features.

Miniature LLM serving runtime with continuous batching, chunked prefill, KV-cache management, preemption, prefix caching, and real Llama-family execution.

Mini-SGLang port for Ascend NPUs with FIA, paged KV cache, ragged continuous batching, and request lifecycle safety.

Continuous batching for TTS — like vLLM, but for voice. Serve 10+ simultaneous text-to-speech requests on a single GPU.

vLLM-style continuous batching for iPhone. Native Swift on MLX, no Python. The fastest multi-agent inference stack on iOS.

A mini-vLLM: a from-scratch LLM inference server with a paged KV cache and continuous batching (GPT-2, OpenAI-compatible streaming API).

Real PyTorch inference server comparing FCFS, EagerContBatch, and ChunkedPrefill on RTX 2070. Key finding: ChunkedPrefill is not a throughput optimizer but a fairness mechanism — it reduces worst-case decode stalls from 58.6ms to 34.1ms while EagerContBatch maximizes mean TTFT and throughput.

.NET wrapper library for llama.cpp that provides batch processing (Continuous batching) and context sequence cache management

An educational LLM inference engine built from scratch: custom decoding loop, KV cache, continuous batching, paged memory with preemption, prefix caching, speculative decoding, OpenAI-compatible API, real benchmarks.