LLM Serving using the Ray LLM DLC¶
Production-ready Docker image for online LLM serving with Ray Serve LLM and vLLM on AWS. Built on Amazon Linux 2023 with ongoing security patching.
This image pairs Ray Serve's HTTP serving layer with the vLLM inference engine, so one tag turns a model into a live LLM endpoint. The
build_openai_app helper from the ray[llm] extra runs vLLM engines as Ray Serve deployments and exposes OpenAI-compatible endpoints; vLLM supplies
the continuous batching, paged KV-cache, and tensor parallelism. The same image runs on a single GPU on EC2 and scales to multi-node
tensor-parallel serving on
Amazon EKS via KubeRay.
Images¶
| Platform | Variant | Image |
|---|---|---|
| EC2 / EKS | GPU | public.ecr.aws/deep-learning-containers/ray:serve-llm-cuda |
Ray LLM shares the ray repository with the Ray Serve DLC (serve-ml prefix) and the Ray Train DLC
(train-ml prefix), using the serve-llm tag prefix. The image is also available on the
ECR Public Gallery. For private ECR URIs, see Image Access.
What's Included¶
The image bundles the full serving stack so you can stand up an LLM endpoint without building a custom image:
- Ray 2.58.0 with the
default,serve, andllmextras —build_openai_appserves OpenAI-compatible endpoints backed by vLLM - vLLM 0.26.0 — inference engine with continuous batching, paged KV-cache, and tensor parallelism
- PyTorch 2.11.0 with
torchvision0.26.0 andtorchaudio2.11.0 — CUDA 13.0 wheels - CUDA 13.0.2 with cuDNN, plus NCCL for multi-GPU collectives
- Transformers 5.14.1 — tokenizers, processors, and model configs vLLM loads from
- FastAPI 0.133.1 with Uvicorn (via Ray Serve) — the async HTTP stack
- FlashInfer — GPU sampling kernels, with CUDA headers pre-installed for JIT
- EFA with OpenMPI and the AWS NCCL OFI plugin — for multi-node tensor-parallel serving
- boto3, awscli — AWS SDK, for pulling models from S3
- Python 3.13 — in a venv at
/opt/venv(PATHalready set)
For distributed training with Ray Train, use the Ray Train DLC. For non-LLM Ray Serve model serving, use the Ray Serve DLC.
Ports¶
| Port | Purpose |
|---|---|
| 8000 | Ray Serve HTTP — the OpenAI-compatible endpoint (/v1/chat/completions, /v1/completions) |
| 8265 | Ray dashboard and job-submission API |
| 6379 | GCS — where workers connect the head in a multi-node cluster |
The image declares ports 6379 and 8265 via EXPOSE. The serving port — 8000 by default, set by the Serve config's http_options.port —
is not, so publish it yourself: pass -p 8000:8000 to docker run on EC2, or declare a containerPort: 8000 in your Kubernetes
manifest. If you change http_options.port, publish that port instead.
FI_PROVIDER=efa and NCCL_DEBUG=INFO are set in the image, and the base image writes NCCL_SOCKET_IFNAME exclusions to /etc/nccl.conf so NCCL
auto-detects the right interface on any host. Our multi-node EKS test relies on this auto-detection and sets no interface override; set
NCCL_SOCKET_IFNAME explicitly only if auto-detection picks the wrong interface on your platform.
CUDA Forward Compatibility¶
The entrypoint detects host NVIDIA driver versions older than the bundled cuda-compat layer and automatically prepends /usr/local/cuda/compat to
LD_LIBRARY_PATH. No flag or env var needed — the check runs on every container start, then the entrypoint execs the command you passed.
How We Build¶
These images are curated builds tracking the Ray and vLLM projects:
- Built from upstream releases — Ray and vLLM are installed from upstream wheels, each build gated by our test suite before publication.
- Security-patched — continuously maintained with security patches from AWS on an Amazon Linux 2023 base.