Running Heterogeneous workloads¶¶
Kubernetes has support for heterogeneous clusters where you can have a mixture of Linux and Windows nodes in the same cluster. Within that cluster, you can have a mixture of Pods that run on Linux and Pods that run on Windows. You can even run multiple versions of Windows in the same cluster. However, there are several factors (as mentioned below) that will need to be accounted for when making this decision.
Assigning PODs to Nodes Best practices¶
In order to keep Linux and Windows workloads on their respective OS-specific nodes, you need to use some combination of node selectors and taints/tolerations. The main goal of scheduling workloads in a heterogeneous environment is to avoid breaking compatibility for existing Linux workloads.
Ensuring OS-specific workloads land on the appropriate container host¶
Users can ensure Windows containers can be scheduled on the appropriate host using nodeSelectors. All Kubernetes nodes today have the following default labels:
kubernetes.io/os = [windows|linux] kubernetes.io/arch = [amd64|arm64|...]
If a Pod specification does not include a nodeSelector like
"kubernetes.io/os": windows, the Pod may be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux.
In Enterprise environments, it's not uncommon to have a large number of pre-existing deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, like Helm charts. In these situations, you may be hesitant to make changes to a deployment's nodeSelectors. The alternative is to use Taints.
If you are using EKS, eksctl offers ways to apply taints through clusterConfig:
NodeGroups: - name: windows-ng amiFamily: WindowsServer2022FullContainer ... labels: nodeclass: windows2022 taints: os: "windows:NoSchedule"
Adding a taint to all Windows nodes, the scheduler will not schedule pods on those nodes unless they tolerate the taint. Pod manifest example:
nodeSelector: kubernetes.io/os: windows tolerations: - key: "os" operator: "Equal" value: "windows" effect: "NoSchedule"
Handling multiple Windows build in the same cluster¶
The Windows container base image used by each pod must match the same kernel build version as the node. If you want to use multiple Windows Server builds in the same cluster, then you should set additional node labels, nodeSelectors or leverage a label called windows-build.
Kubernetes 1.17 automatically adds a new label node.kubernetes.io/windows-build to simplify the management of multiple Windows build in the same cluster. If you're running an older version, then it's recommended to add this label manually to Windows nodes.
This label reflects the Windows major, minor, and build number that need to match for compatibility. Below are values used today for each Windows Server version.
It's important to note that Windows Server is moving to the Long-Term Servicing Channel (LTSC) as the primary release channel. The Windows Server Semi-Annual Channel (SAC) was retired on August 9, 2022. There will be no future SAC releases of Windows Server.
|Product Name||Build Number(s)|
|Server full 2022 LTSC||10.0.20348|
|Server core 2019 LTSC||10.0.17763|
It is possible to check the OS build version through the following command:
The KERNEL-VERSION output matches the Windows OS build version.
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-10-2-235.ec2.internal Ready <none> 23m v1.24.7-eks-fb459a0 10.10.2.235 18.104.22.168 Windows Server 2022 Datacenter 10.0.20348.1607 containerd://1.6.6 ip-10-10-31-27.ec2.internal Ready <none> 23m v1.24.7-eks-fb459a0 10.10.31.27 22.214.171.124 Windows Server 2019 Datacenter 10.0.17763.4131 containerd://1.6.6 ip-10-10-7-54.ec2.internal Ready <none> 31m v1.24.11-eks-a59e1f0 10.10.7.54 126.96.36.199 Amazon Linux 2 5.10.173-154.642.amzn2.x86_64 containerd://1.6.19
The example below applies an additional nodeSelector to the pod manifest in order to match the correct Windows-build version when running different Windows node groups OS versions.
nodeSelector: kubernetes.io/os: windows node.kubernetes.io/windows-build: '10.0.20348' tolerations: - key: "os" operator: "Equal" value: "windows" effect: "NoSchedule"
Simplifying NodeSelector and Toleration in Pod manifests using RuntimeClass¶
You can also make use of RuntimeClass to simplify the process of using taints and tolerations. This can be accomplished by creating a RuntimeClass object which is used to encapsulate these taints and tolerations.
Create a RuntimeClass by running the following manifest:
apiVersion: node.k8s.io/v1beta1 kind: RuntimeClass metadata: name: windows-2022 handler: 'docker' scheduling: nodeSelector: kubernetes.io/os: 'windows' kubernetes.io/arch: 'amd64' node.kubernetes.io/windows-build: '10.0.20348' tolerations: - effect: NoSchedule key: os operator: Equal value: "windows"
Once the Runtimeclass is created, assign it using as a Spec on the Pod manifest:
apiVersion: apps/v1 kind: Deployment metadata: name: iis-2022 labels: app: iis-2022 spec: replicas: 1 template: metadata: name: iis-2022 labels: app: iis-2022 spec: runtimeClassName: windows-2022 containers: - name: iis
Managed Node Group Support¶
To help customers run their Windows applications in a more streamlined manner, AWS launched the support for Amazon EKS Managed Node Group (MNG) support for Windows containers on December 15, 2022. To help align operations teams, Windows MNGs are enabled using the same workflows and tools as Linux MNGs. Full and core AMI (Amazon Machine Image) family versions of Windows Server 2019 and 2022 are supported.
Following AMI families are supported for Managed Node Groups(MNG)s.
AWS Official Documentation: https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html
To better understand how Pod Networking (CNI) works, check the following link: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html
AWS Blog on Deploying Managed Node Group for Windows on EKS: https://aws.amazon.com/blogs/containers/deploying-amazon-eks-windows-managed-node-groups/