Skip to content

User Guide

Choose a framework to get started:

  • vLLM — serve large language models on EC2, EKS, or Amazon SageMaker AI
  • vLLM-Omni — serve multimodal models (TTS, image, video, audio, omni-chat)
  • TEI — serve text embedding, reranker, and classification models with Text Embeddings Inference on Amazon SageMaker AI
  • Ray — deploy any ML model with Ray Serve (NLP, vision, audio, tabular)
  • Ray Train — distributed training with Ray Train on any EKS cluster via KubeRay, or on EC2
  • WhisperX — transcribe, align, and diarize speech on EC2 or Amazon SageMaker AI
  • llama.cpp — serve quantized GGUF models on x86 CPU, NVIDIA GPU, or Graviton (ARM64) with the OpenAI-compatible llama-server API
  • PyTorch — distributed training with EFA, NCCL, flash-attn, and DeepSpeed pre-installed
  • TensorFlow Training — training on Amazon SageMaker AI with EFA-capable multi-node support on Amazon Linux 2023
  • TensorFlow Inference — serve TensorFlow SavedModels with TensorFlow Serving on Amazon SageMaker AI
  • Base — lightweight CUDA + Python images for building your own AI/ML container