self

self is a simple Dockerized wrapper around llama.cpp, stable-diffusion.cpp, and ggmlc. It makes models easy to run and manage through a typed HTTP API, downloading them directly from Hugging Face.

It supports:

• Decision making
• Audio processing
• Text processing
• Embedding generation
• Vision input
• Image generation
• Image editing

Model registry

The goal is get new models up and running quickly with minimal configuration, using established open-source tools.

This is mostly optimized for my personal use and development workflow and hardware (rtx 5090).

Run with Docker

Images are published to GHCR for Linux hosts:

Tag	Runtime
latest	NVIDIA CUDA 13
cuda-12	NVIDIA CUDA 12
cpu	CPU-only

CUDA images require a host NVIDIA driver and the NVIDIA Container Toolkit. They exit at startup if no NVIDIA GPU is accessible. Use the cpu tag on hosts without a GPU.

CUDA 13:

docker run --rm --gpus all \
  -p 8080:8080 \
  -v self-models:/models \
  ghcr.io/<owner>/<repo>:latest

CPU-only:

docker run --rm \
  -p 8080:8080 \
  -v self-models:/models \
  ghcr.io/<owner>/<repo>:cpu

The /models volume keeps downloaded models between container runs.

Build an image

Docker builds the server and native engines inside a builder stage. No local Go, CUDA, or engine build is required:

docker buildx build --file docker/Dockerfile \
  --build-arg CUDA_VARIANT=cuda-13 \
  --build-arg ENGINE_VARIANT=cuda-13.4 \
  --tag self:latest .

For a CPU-only image, set both build arguments to cpu. To build directly from a public Git repository and ref without checking it out locally:

docker build -f docker/Dockerfile \
  --build-arg CUDA_VARIANT=cuda-13 \
  --build-arg ENGINE_VARIANT=cuda-13.4 \
  --tag self:latest \
  https://github.com/K0IN/self.git#main

For a CPU-only image, set both build arguments to cpu. CUDA 12 uses CUDA_VARIANT=cuda-12 and ENGINE_VARIANT=cuda-12.8.

Configuration

The container entrypoint accepts serve options after the image name:

Option	Default	Purpose
serve <model>	kev:0.5b	Model reference to download and serve; append @quant to select a quant
--quant <name>	Model default	Quantization, such as q4, q8, or fp16
--device <name>	auto	cpu, cuda, cuda:N, vulkan, or vulkan:N
--queue-size <n>	64	Maximum queued requests
--preprocess-concurrency <n>	8	Concurrent image preprocessing jobs
--verbose	off	Show engine and request logs
--allow-http-images	off	Allow non-HTTPS image URLs
--allow-private-images	off	Allow private or loopback image addresses

Environment variables can override container defaults:

Variable	Purpose
AI_SERVER_HOST	Listen host
AI_SERVER_PORT	Listen port
AI_SERVER_MODELS	Model directory; keep this as /models when using the volume
HF_TOKEN	Token for gated Hugging Face repositories
HF_ENDPOINT	Hugging Face mirror endpoint

For example:

docker run --rm --gpus all -p 9000:9000 -v self-models:/models \
  -e AI_SERVER_PORT=9000 \
  ghcr.io/<owner>/<repo>:latest serve decider-vision:2b@q4

Models

Models are downloaded directly from Hugging Face on first use, you can find the available models in the Model registry.

To see how fast a model runs on your machine, run self benchmark kev:0.5b. It writes a report you can share as a pull request, see benchmarks/.

HTTP API

See the full OpenAI-style API reference for endpoint examples, request and response fields, image inputs, Python and JavaScript clients, and error responses. It has one page per modality: decision, audio, and text, embeddings and speech to text.