Serving GGUF Models using llama.cpp DLC¶
Production-ready Docker images for serving quantized GGUF models with llama.cpp on AWS. Built on Amazon Linux 2023 with ongoing security patching.
Run large language models efficiently on x86 CPUs, NVIDIA GPUs, or AWS Graviton (ARM64) through the upstream llama-server OpenAI-compatible API.
Images¶
The images ship for three hardware targets — x86 CPU, x86 NVIDIA GPU (CUDA), and Graviton (ARM64, CPU) — each in an EC2 and an
Amazon SageMaker AI flavor. The x86 images live in the llama-cpp repository, and the Graviton images live in llama-cpp-arm64. Every image serves on
port 8080.
| Platform | Architecture | Device | Image |
|---|---|---|---|
| EC2 | x86_64 | CPU | public.ecr.aws/deep-learning-containers/llama-cpp:server-cpu-v1 |
| EC2 | x86_64 | GPU (CUDA) | public.ecr.aws/deep-learning-containers/llama-cpp:server-cuda-v1 |
| EC2 | ARM64 (Graviton) | CPU | public.ecr.aws/deep-learning-containers/llama-cpp-arm64:server-cpu-v1 |
| Amazon SageMaker AI | x86_64 | CPU | public.ecr.aws/deep-learning-containers/llama-cpp:server-sagemaker-cpu-v1 |
| Amazon SageMaker AI | x86_64 | GPU (CUDA) | public.ecr.aws/deep-learning-containers/llama-cpp:server-sagemaker-cuda-v1 |
| Amazon SageMaker AI | ARM64 (Graviton) | CPU | public.ecr.aws/deep-learning-containers/llama-cpp-arm64:server-sagemaker-cpu-v1 |
All images are also available on the ECR Public Gallery (llama-cpp, llama-cpp-arm64). For private ECR URIs, see Image Access.
What's Included¶
Each image is a from-source build of the upstream llama.cpp project (tag b10433):
llama-server— the OpenAI-compatible HTTP inference server (the default entrypoint)llama-cliandllama-bench— the interactive CLI and the benchmarking tool, onPATHfor one-off use- libcurl-enabled build (
LLAMA_CURL=ON) — load a model directly from a HuggingFace repo at startup - Portable CPU dispatch — the x86 CPU image bundles every microarchitecture backend (SSE4.2 through AVX-512/AMX) and picks the fastest one at runtime, while the Graviton image is built for the Graviton3 (Neoverse-V1) baseline and is forward-compatible with newer Graviton generations
- CUDA 13.0.2 runtime (GPU image only) with automatic
cuda-compatfor forward compatibility
API Endpoints¶
llama-server exposes the upstream OpenAI-compatible API on port 8080:
| Endpoint | Purpose |
|---|---|
POST /v1/chat/completions |
Chat completions (OpenAI-compatible) |
POST /v1/completions |
Text completions (OpenAI-compatible) |
POST /v1/embeddings |
Embeddings (OpenAI-compatible) |
GET /v1/models |
Advertise the served model id |
GET /health |
Readiness health check |
POST /invocations |
SageMaker alias → /v1/chat/completions |
GET /ping |
SageMaker readiness alias → /health |
On Amazon SageMaker AI the container sits behind nginx, which maps GET /ping to /health and POST /invocations to /v1/chat/completions. Every other
path is proxied straight through to llama-server, so the full /v1/* API stays reachable. See EC2 Deployment and
Amazon SageMaker AI Deployment for examples, and Configuration for every launch option.
How We Build¶
These images are curated builds tracking the llama.cpp project:
- Built from upstream releases — images are built from a pinned llama.cpp build tag, each gated by our regression test suite before publication.
- Regression-tested — validated against quantized GGUF models on EC2 and Amazon SageMaker AI on every release. See Supported Models.
- Security-patched — continuously maintained with security patches from AWS on an Amazon Linux 2023 base.