LLM Inference Guide

A practical guide for deploying and benchmarking open-source LLMs using two inference engines: llama.cpp (CPU/GGUF) and vLLM (GPU/Kubernetes). Each subfolder is self-contained with its own setup guide and benchmark results.

Repository Structure

Inference/
├── Llama.CPP/          # llama.cpp server on a cloud VM (CPU inference, GGUF models)
└── vLLM/               # vLLM on GKE with NVIDIA L4 GPU (Kubernetes deployment)

When to Use Which Engine

	llama.cpp	vLLM
Hardware	CPU (or Apple Silicon)	NVIDIA GPU
Model format	GGUF (quantized)	HuggingFace (fp16 / w4a16)
Setup complexity	Low — runs on any VM	Medium — requires Kubernetes + GPU node
Best for	Low-cost CPU inference, dev/testing	Production GPU inference, high throughput

Models Benchmarked

• llama.cpp — Meta-Llama-3.1-8B-Instruct (Q4_K_M and F16 GGUF)
• vLLM — gemma-3-12b-it (w4a16 quantized via RedHatAI)

Benchmarking Tool

Both engines are benchmarked using GuideLLM, which measures throughput, latency (TTFT, ITL), and token generation rates under configurable request rates.

---

See each subfolder's README.md for full setup and replication steps.