Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Large Language Model (LLM) inference on Graviton CPUs with llama.cpp

Introduction

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide variety of hardware. It’s a plain C/C++ implementation without any dependencies. It supports quantized general matrix multiply-add (GEMM) kernels for faster inference and reduced memory use. The quantized GEMM kernels are optimized for AWS Graviton processors using Arm Neon and SVE based matrix multiply-accumulate (MMLA) instructions. This document covers how to run llama.cpp efficiently for LLM inference on AWS Graviton based Amazon EC2 Instances.

There are two ways to get llama.cpp running on Graviton:

  1. AWS Deep Learning Container. AWS publishes a production-ready Graviton (ARM64) llama.cpp image on the Amazon ECR Public Gallery. It ships the upstream llama-server with an OpenAI-compatible API, so you can serve a quantized GGUF model with a single docker run. Start here if you want a maintained image without building anything.
  2. Build from source. Compile llama.cpp with -mcpu=native for full control over build flags and the CLI tools. Start here if you need a custom build or want to run llama-cli/llama-bench directly.

Serve llama.cpp on Graviton with the AWS Deep Learning Container

The llama.cpp Deep Learning Container (DLC) is a from-source build of upstream llama.cpp for the Graviton3 (Neoverse-V1) baseline, forward-compatible with newer Graviton generations. Built on Amazon Linux 2023 and validated against quantized GGUF models before each release, it serves the upstream llama-server on port 8080.

AWS publishes the Graviton (ARM64) image in the llama-cpp-arm64 repository on the Amazon ECR Public Gallery. This guide uses the CPU image for Amazon EC2:

public.ecr.aws/deep-learning-containers/llama-cpp-arm64:server-cpu-v1

Prerequisites

Launch a Graviton3(E)-, Graviton4-, or Graviton5-based EC2 instance (for example, c7g, m7g, r8g, or c8g) with Docker installed. The server is unauthenticated by default and binds 0.0.0.0, so run it inside a private network (security group / VPC).

Serve a model from Hugging Face

The container forwards any llama-server arguments appended to docker run. Fetch a quantized GGUF from Hugging Face at startup with --hf-repo / --hf-file:

docker run -d -p 8080:8080 \
  public.ecr.aws/deep-learning-containers/llama-cpp-arm64:server-cpu-v1 \
  --hf-repo Qwen/Qwen2.5-0.5B-Instruct-GGUF \
  --hf-file qwen2.5-0.5b-instruct-q4_0.gguf \
  --ctx-size 4096

llama-server binds the socket only after the model has loaded, so /health refuses connections until the model is resident. Wait for readiness, then call the OpenAI-compatible API:

until curl -sf http://localhost:8080/health > /dev/null; do sleep 5; done

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "In one sentence, what is llama.cpp?"}]
  }'

Serve a local GGUF model

To serve a local GGUF instead, mount it and point --model at the mount:

docker run -d -p 8080:8080 \
  -v /path/to/models:/models:ro \
  public.ecr.aws/deep-learning-containers/llama-cpp-arm64:server-cpu-v1 \
  --model /models/qwen2.5-0.5b-instruct-q4_0.gguf \
  --ctx-size 4096

Require an API key

Set LLAMA_API_KEY to require a bearer token on every request:

docker run -d -p 8080:8080 \
  -e LLAMA_API_KEY=my-secret-key \
  public.ecr.aws/deep-learning-containers/llama-cpp-arm64:server-cpu-v1 \
  --hf-repo Qwen/Qwen2.5-0.5B-Instruct-GGUF \
  --hf-file qwen2.5-0.5b-instruct-q4_0.gguf

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello!"}]}'

Pass any llama-server flag (--ctx-size, --parallel, --threads, --batch-size, …) as a container argument. The image also bundles llama-cli and llama-bench; override the entrypoint to run them, e.g. docker run --rm --entrypoint llama-bench <image> ....

How to build llama.cpp on Graviton CPUs

Building from source gives you full control over the build flags and direct access to the llama.cpp CLI tools, and works on other hardware platforms too. This section provides the instructions on how to build llama.cpp from sources and how to install python bindings.

Prerequisites

Graviton3(E) (e.g. c7g/m7g/r7g, c7gn and Hpc7g Instances), Graviton4 (e.g. r8g Instances), and Graviton5 (e.g. m9g instances) CPUs support BFloat16 format and MMLA instructions for machine learning (ML) acceleration. These hardware features are enabled starting with Linux Kernel version 5.10. So, it is highly recommended to use the AMIs based on Linux Kernel 5.10 and beyond for the best LLM inference performance on Graviton Instances. Use the following queries to list the AMIs with the recommended Kernel versions.

# For Kernel 5.10 based AMIs list
aws ec2 describe-images --owners amazon --filters "Name=architecture,Values=arm64" "Name=name,Values=*kernel-5.10*" --query 'sort_by(Images, &CreationDate)[].Name'

# For Kernel 6.x based AMIs list
aws ec2 describe-images --owners amazon --filters "Name=architecture,Values=arm64" "Name=name,Values=*kernel-6.*" --query 'sort_by(Images, &CreationDate)[].Name'

Build llama.cpp from source

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# build with cmake
mkdir build
cd build
cmake .. -DCMAKE_CXX_FLAGS="-mcpu=native" -DCMAKE_C_FLAGS="-mcpu=native"
cmake --build . -v --config Release -j `nproc`

Install llama.cpp python bindings

CMAKE_ARGS="-DCMAKE_CXX_FLAGS='-mcpu=native' -DCMAKE_C_FLAGS='-mcpu=native'" pip3 install --no-cache-dir llama-cpp-python

Run LLM inference with llama.cpp

llama.cpp provides a set of tools to (1) convert model binary file into GPT-Generated Unified Format (GGUF), (2) quantize single and half precision format models into one of the quantized formats, and (3) run LLM inference locally. For the steps on how to convert model binary into GGUF format and how to quantize them into low precision formats, please check llama.cpp README.

The following instructions use Meta Llama-3 8B parameter model from Hugging Face models repository to demonstrate LLM inference performance on AWS Graviton based EC2 Instances. The model is already availble in multiple quantized formats which can be directly run on AWS Graviton processors.

# Download the model from Hugging Face model repo.
cd llama.cpp
wget https://huggingface.co/SanctumAI/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/meta-llama-3-8b-instruct.Q4_0.gguf

Using llama-cli

# Now, launch llama-cli with the above model and a sample input prompt. The following command is using 64 threads.
# Change -t argument for running inference with lower thread count. On completion, the script prints throughput and latency metics
# for prompt encoding and response generation.
./build/bin/llama-cli -m meta-llama-3-8b-instruct.Q4_0.gguf -p "Building a visually appealing website can be done in ten simple steps:" -n 512 -t 64

# Launch the model in conversation (chatbot) mode using this command
./build/bin/llama-cli -m meta-llama-3-8b-instruct.Q4_0.gguf -p "You are a helpful assistant" -cnv --color

Using llama.cpp python binding

Note: Set the n_threads to number of vcpus explicitly while creating the Llama object. This is required to use all cores(vcpus) on Graviton instances. Without this set, the python bindings use half of the vcpus and the performance is not the best.

import json
import argparse

from llama_cpp import Llama

parser = argparse.ArgumentParser()
parser.add_argument("-m", "--model", type=str, default="../models/7B/ggml-models.bin")
args = parser.parse_args()

# for example, for a .16xlarge instance, set n_threads=64
llm = Llama(model_path=args.model,
            n_threads=64)

output = llm(
"Question: How to build a visually appealing website in ten steps? Answer: ",
max_tokens=512,
echo=True,
)

Run DeepSeek R1 LLM Inference on AWS Graviton

DeepSeek R1 is an open-source LLM for conversational AI, coding, and problem-solving tasks. The model can be readily deployed on AWS Graviton-based Amazon EC2 Instances for inference use cases. We recommend using ollama service for the inference deployment. Ollama service is built on top of llama.cpp which is highly optimized to achieve the best performance on AWS Graviton processors. This section shows how to install ollama service and run DeepSeek R1 inference.

# Launch Graviton3 or Graviton4 based EC2 instance (for example, c7g, m7g, c8g or m8g instances)

# Download and install the ollama service
curl -fsSL https://ollama.com/install.sh | sh
ollama --version
systemctl is-active ollama.service

# If the output is active, the service is running, and you can skip the next step. If it’s not, start it manually
sudo systemctl start ollama.service

# DeepSeek has released multiple versions of the R1 model, sizes from 1.5B parameters to 70B parameters.
# ollama supports deepseek-r1:1.5b/7b/8b/14b/32b/70b models
# Download and run the 8b model using the following command
ollama run deepseek-r1:8b

# To benchmark the prompt evaluation rate and response genetation rate, launch ollama with verbose option.
# At the end of the inference, the script prints the token eval and the token generation rates
ollama run deepseek-r1:8b "<prompt>" --verbose

Additional Resources

Please refer to

  1. Best-in-class LLM performance on Arm Neoverse V1 based AWS Graviton3 CPUs to know the LLM inference performance measured on AWS Graviton3 based EC2 Instances.
  2. Running Llama 3 70B on the AWS Graviton4 CPU with Human Readable Performance for LLM inference performance on AWS Graviton4 based EC2 Instances.
  3. Intro to Llama on Graviton for a step by step guide on how to deploy an LLM model on AWS Graviton-based EC2 Instances. Note: This guide refers to llama.cpp version from July 2024. If you are using the latest llama.cpp version, please replace the Q4_0_4_8 and Q4_0_8_8 with Q4_0 format.
  4. Run LLMs on CPU with Amazon SageMaker Real-time Inference for running LLMs for real-time inference using AWS Graviton3 and Amazon SageMaker.
  5. llama.cpp Deep Learning Container on the Amazon ECR Public Gallery for the production-ready, actively maintained Graviton (ARM64) llama-server image.