Assigning PODs to Nodes Best practices¶
In order to keep Linux and Windows workloads on their respective OS-specific nodes, you need to use some combination of node selectors and taints/tolerations. The main goal of scheduling workloads in a heterogeneous environment is to avoid breaking compatibility for existing Linux workloads.
Ensuring OS-specific workloads land on the appropriate container host¶
Users can ensure Windows containers can be scheduled on the appropriate host using nodeSelectors. All Kubernetes nodes today have the following default labels:
kubernetes.io/os = [windows|linux] kubernetes.io/arch = [amd64|arm64|...]
If a Pod specification does not include a nodeSelector like
"kubernetes.io/os": windows, the Pod may be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux.
In Enterprise environments, it's not uncommon to have a large number of pre-existing deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, like Helm charts. In these situations, you may be hesitant to make changes to a deployment's nodeSelectors. The alternative is to use Taints.
If you are using EKS, eksctl offers ways to apply taints through clusterConfig:
NodeGroups: - name: windows-ng amiFamily: WindowsServer2019FullContainer ... labels: nodeclass: windows2019 taints: os: "windows:NoSchedule"
Adding a taint to all Windows nodes, the scheduler will not schedule pods on those nodes unless they tolerate the taint. Pod manifest example:
nodeSelector: kubernetes.io/os: windows tolerations: - key: "os" operator: "Equal" value: "windows" effect: "NoSchedule"
Handling multiple Windows build in the same cluster¶
The Windows container base image used by each pod must match the same kernel build version as the node. If you want to use multiple Windows Server builds in the same cluster, then you should set additional node labels, nodeSelectors or leverage a label called windows-build.
Kubernetes 1.17 automatically adds a new label node.kubernetes.io/windows-build to simplify the management of multiple Windows build in the same cluster. If you're running an older version, then it's recommended to add this label manually to Windows nodes.
This label reflects the Windows major, minor, and build number that need to match for compatibility. Below are values used today for each Windows Server version.
|Product Name||Build Number(s)|
|Server core 2019 LTSC||10.0.17763|
|Server core 2004 SAC||10.0.19041|
|Server core 20H2 SAC||10.0.19042|
It is possible to check the OS build version through the following command:
kubectl get pods -o wide
The KERNEL-VERSION output matches the Windows OS build version.
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-172-31-20-44.ec2.internal Ready <none> 42d v1.18.9-eks-d1db3c 172.31.20.44 22.214.171.124 Windows Server 2019 Datacenter 10.0.17763.1697 docker://19.3.13 ip-172-31-44-38.ec2.internal Ready <none> 42d v1.18.9-eks-d1db3c 172.31.44.38 126.96.36.199 Amazon Linux 2 4.14.209-160.339.amzn2.x86_64 docker://19.3.6 ip-172-31-5-245.ec2.internal Ready <none> 31d v1.18.9-eks-d1db3c 172.31.5.245 188.8.131.52 Windows Server Datacenter 10.0.19041.685 docker://19.3.14
The example below applies an additional nodeSelector to the pod manifest in order to match the correct Windows-build version when running different Windows node groups OS versions.
nodeSelector: kubernetes.io/os: windows node.kubernetes.io/windows-build: '10.0.17763' tolerations: - key: "os" operator: "Equal" value: "windows" effect: "NoSchedule"
Simplifying NodeSelector and Toleration in Pod manifests using RuntimeClass¶
You can also make use of RuntimeClass to simplify the process of using taints and tolerations. This can be accomplished by creating a RuntimeClass object which is used to encapsulate these taints and tolerations.
Create a RuntimeClass by running the following manifest:
apiVersion: node.k8s.io/v1beta1 kind: RuntimeClass metadata: name: windows-2019 handler: 'docker' scheduling: nodeSelector: kubernetes.io/os: 'windows' kubernetes.io/arch: 'amd64' node.kubernetes.io/windows-build: '10.0.17763' tolerations: - effect: NoSchedule key: os operator: Equal value: "windows"
Once the Runtimeclass is created, assign it using as a Spec on the Pod manifest:
apiVersion: apps/v1 kind: Deployment metadata: name: iis-2019 labels: app: iis-2019 spec: replicas: 1 template: metadata: name: iis-2019 labels: app: iis-2019 spec: runtimeClassName: windows-2019 containers: - name: iis
AWS Official Documentation: https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html
To better understand how Pod Networking (CNI) works, check the following link: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html