Skip to content

Configuration

AWS Lambda controls ENTRYPOINT and CMD, so you configure these images with environment variables rather than command-line arguments. Set them in the Dockerfile with ENV, or on the function. Variables set on the function take precedence, so one image can serve several models.

To switch models, update MODEL_ID on the function and publish a new version. No rebuild is needed. UpdateFunctionConfiguration replaces the whole variable map rather than merging into it, so pass the variables you already set, such as VLLM_LOAD_FORMAT, along with the new MODEL_ID.

vLLM (vllm:cuda)

Read by the engine extension and passed to vllm serve.

Variable Default Maps to
MODEL_ID Qwen/Qwen2.5-0.5B-Instruct Positional model: Hugging Face ID, local path, or s3:// prefix
SERVED_MODEL_NAME Unset --served-model-name, and the default model field in requests
VLLM_LOAD_FORMAT Unset --load-format. Set runai_streamer with an s3:// MODEL_ID to stream weights into GPU memory
VLLM_GPU_MEM_UTIL 0.8 --gpu-memory-utilization
VLLM_MAX_MODEL_LEN Unset (model default) --max-model-len
VLLM_TP_SIZE Visible GPU count --tensor-parallel-size
VLLM_SERVER_ARGS Unset Any further vllm serve flags, shell-quoted
VLLM_SERVER_PORT 8000 --port (loopback only); also read by the handler
VLLM_SERVER_TIMEOUT 600 Seconds to wait for the engine to be ready. Raise to just under the function Timeout for large models
HF_TOKEN Unset Hugging Face token. Raises download rate limits and reaches private or gated models

SGLang (sglang:cuda)

Read by the engine extension and passed to python -m sglang.launch_server.

Variable Default Maps to
MODEL_ID Qwen/Qwen2.5-0.5B-Instruct --model-path: Hugging Face ID, local path, or s3:// prefix
SERVED_MODEL_NAME Unset --served-model-name, and the default model field in requests
SGLANG_LOAD_FORMAT Unset --load-format. Set runai_streamer with an s3:// MODEL_ID
SGLANG_MEM_FRACTION 0.8 --mem-fraction-static
SGLANG_ATTENTION_BACKEND flashinfer --attention-backend
SGLANG_MAX_TOTAL_TOKENS Unset --max-total-tokens
SGLANG_TP_SIZE Visible GPU count --tp-size
SGLANG_SERVER_ARGS Unset Any further sglang.launch_server flags, shell-quoted
SGLANG_SERVER_PORT 8000 --port (loopback only); also read by the handler
SGLANG_SERVER_TIMEOUT 600 Seconds to wait for the engine to be ready. Raise to just under the function Timeout for large models
HF_TOKEN Unset Hugging Face token. Raises download rate limits and reaches private or gated models

MODEL_ID has a default so that the image serves without configuration, but it points at a small test model. Set it for any real workload.

Tensor parallelism defaults to the number of visible GPUs, which is 1, because Lambda assigns one GPU to each execution environment. Multi-GPU and multi-node serving are not supported, so leave *_SERVER_ARGS and *_TP_SIZE at their defaults unless an execution environment is granted more than one GPU.

The handler takes its per-request timeout from the invocation deadline, so set the function's Timeout rather than a variable.

Engine extension (vllm:cuda, sglang:cuda)

Variable Default Effect
ENGINE_EXTENSION on Set to off for functions that run their own server. The extension registers without starting an engine
ENGINE_HEALTH_INTERVAL 10 Seconds between /health probes once serving
ENGINE_HEALTH_TIMEOUT 10 vLLM, 30 SGLang Seconds allowed for each probe. SGLang's health check can run a one-token generation, so it uses the longer default
ENGINE_HEALTH_FAILURES 3 Consecutive failed probes before the environment is recycled

Baked-in paths

/tmp is the only writable directory, so the images redirect everything that writes.

Variable Value Images
HOME /tmp All
XDG_CONFIG_HOME /tmp/.config All
XDG_CACHE_HOME /tmp/.cache All
TORCH_HOME /tmp/torch PyTorch, vLLM, SGLang
HF_HOME /tmp/huggingface PyTorch, vLLM, SGLang
CUPY_CACHE_DIR /tmp/cupy_cache CuPy
CUDA_HOME /usr/local/cuda CuPy, vLLM, SGLang
NVIDIA_DRIVER_CAPABILITIES compute,utility,video on PyTorch, compute,utility elsewhere All

Anything downloaded into these paths counts toward EphemeralStorage. Size it for the model when you are not streaming from S3.

Function settings that matter

Set these on the function rather than in the image. For the full list, see the GPU-accelerated Event Functions in the Lambda Developer Guide.

Setting Notes
AcceleratorMemorySize GPU memory, in GB: 3, 6, 12, 16, 24, 48. This is a minimum, and Lambda selects the smallest GPU that meets it
PerExecutionEnvironmentMaxConcurrency Default 4, maximum 48. Leave this at the default for vLLM and SGLang, because the server batches requests itself
ExecutionEnvironmentMemoryGiBPerVCpu Default 2. Raise it when MemorySize would otherwise claim every vCPU on the instance
EphemeralStorage 512 MB to 32 GB. This only needs to hold the model if you are not streaming it from S3
Timeout Maximum duration of one invocation. Raise it for longer invokes, and to 900 for vLLM and SGLang to give the model time to load at startup. Keep VLLM_SERVER_TIMEOUT / SGLANG_SERVER_TIMEOUT at or below it

Full reference