Skip to content

Supported Models

llama.cpp serves models in the GGUF format — the quantized single-file format used by the llama.cpp ecosystem. Any model that upstream llama-server can load is supported: Llama, Qwen, Mistral, Gemma, Phi, DeepSeek, and the many other architectures llama.cpp implements, in any quantization (Q4_K_M, Q5_K_M, Q8_0, F16, …).

The container ships no baked-in model — you supply a GGUF at launch. There are three ways to provide one:

  1. Mount a local GGUF — a file on EC2, or a model.tar.gz staged via ModelDataUrl on Amazon SageMaker AI.
  2. Download from HuggingFace at startup — point the server at a HuggingFace repo and file, and it downloads the GGUF the first time the container boots. This path needs network access.
  3. Pass a model URL — any URL llama-server accepts (the build enables libcurl).

Getting GGUF Models

Thousands of ready-to-run GGUF models are already published on the HuggingFace Hub. The ggml-org and bartowski collections are good places to start. If you would rather convert your own weights, the llama.cpp repository ships the tooling to do it: convert_hf_to_gguf.py turns a HuggingFace model into a GGUF file, and llama-quantize shrinks it to the quantization you want.

Choose a quantization that fits your hardware. A smaller quant such as Q4_K_M uses less memory and runs faster, at a small cost to quality. A larger one such as Q8_0 or F16 keeps more of the original quality but needs more RAM or VRAM.

Choosing a Hardware Target

Target Image repository / tag Notes
x86 CPU llama-cpp:server-cpu-v1 Runs on any modern x86 instance and picks the best microarchitecture backend at runtime
x86 GPU (CUDA) llama-cpp:server-cuda-v1 Needs an NVIDIA GPU — offload layers with --n-gpu-layers (see Configuration)
Graviton (ARM64) CPU llama-cpp-arm64:server-cpu-v1 Built for the Graviton3 (Neoverse-V1) baseline and forward-compatible with newer Graviton generations. --n-gpu-layers does not apply here

The -sagemaker- tags are the Amazon SageMaker AI counterparts of the same three targets.

Specifying the Model

  • EC2 — pass the model as a llama-server argument. Mount a local GGUF and point --model at it, or fetch from HuggingFace with --hf-repo / --hf-file. See EC2 Deployment.
  • Amazon SageMaker AI — the entrypoint resolves the model in this order (see SageMaker Deployment):
  • SM_LLAMA_CPP_MODEL — an explicit GGUF path inside the container.
  • /opt/ml/model — the first *.gguf staged via ModelDataUrl is auto-detected (searched up to two directory levels deep).
  • SM_LLAMA_CPP_HF_REPO / SM_LLAMA_CPP_HF_FILE — download the GGUF from HuggingFace at startup.

For a multi-part (sharded) GGUF, provide every shard and point the server at the first one (…-00001-of-0000N.gguf). llama.cpp finds and loads the rest automatically.

Offline / Air-Gapped

The container runs with no network access only when the GGUF is provided locally — a mounted file on EC2 or a model.tar.gz on Amazon SageMaker AI. The HuggingFace-download path (--hf-repo / SM_LLAMA_CPP_HF_REPO) requires runtime network egress and is not air-gapped.

Full Reference