Windows Networking¶
Windows Container Networking Overview¶
Windows containers are fundamentally different than Linux containers. Linux containers use Linux constructs like namespaces, the union file system, and cgroups. On Windows, those constructs are abstracted from containerd by the Host Compute Service (HCS). HCS acts as an API layer that sits above the container implementation on Windows. Windows containers also leverage the Host Network Service (HNS) that defines the network topology on a node.
From a networking perspective, HCS and HNS make Windows containers function like virtual machines. For example, each container has a virtual network adapter (vNIC) that is connected to a Hyper-V virtual switch (vSwitch) as shown in the diagram above.
IP Address Management¶
A node in Amazon EKS uses it's Elastic Network Interface (ENI) to connect to an AWS VPC network. Presently, only a single ENI per Windows worker node is supported. The IP address management for Windows nodes is performed by VPC Resource Controller which runs in control plane. More details about the workflow for IP address management of Windows nodes can be found here.
The number of pods that a Windows worker node can support is dictated by the size of the node and the number of available IPv4 addresses. You can calculate the IPv4 address available on the node as below: - By default, only secondary IPv4 addresses are assigned to the ENI. In such a case:
Total IPv4 addresses available for Pods = Number of supported IPv4 addresses in the primary interface - 1
- If the cluster has been configured for high pod density by enabling prefix delegation feature then-
Here, instead of allocating secondary IPv4 addresses, VPC Resource Controller will allocate
Total IPv4 addresses available for Pods = (Number of supported IPv4 addresses in the primary interface - 1) * 16/28 prefixesand therefore, the overall number of available IPv4 addresses will be boosted 16 times.
Using the formula above, we can calculate max pods for an Windows worker noded based on a m5.large instance as below: - By default, when running in secondary IP mode-
- When usingprefix delegation-
For more information on how many IP addresses an instance type can support, see IP addresses per network interface per instance type.
Another key consideration is the flow of network traffic. With Windows there is a risk of port exhaustion on nodes with more than 100 services. When this condition arises, the nodes will start throwing errors with the following message:
"Policy creation failed: hcnCreateLoadBalancer failed in Win32: The specified port already exists."
To address this issue, we leverage Direct Server Return (DSR). DSR is an implementation of asymmetric network load distribution. In other words, the request and response traffic use different network paths. This feature speeds up communication between pods and reduces the risk of port exhaustion. We therefore recommend enabling DSR on Windows nodes.
DSR is enabled by default in Windows Server SAC EKS Optimized AMIs. For Windows Server 2019 LTSC EKS Optimized AMIs, you will need to enable it during instance provisioning using the script below and by using Windows Server 2019 Full or Core as the amiFamily in the eksctl nodeGroup. See eksctl custom AMI for additional information.
nodeGroups:
- name: windows-ng
instanceType: c5.xlarge
minSize: 1
volumeSize: 50
amiFamily: WindowsServer2019CoreContainer
ssh:
allow: false
<powershell>
[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
(Get-Content $EKSBootstrapScriptFile).replace('"--proxy-mode=kernelspace",', '"--proxy-mode=kernelspace", "--feature-gates WinDSR=true", "--enable-dsr",') | Set-Content $EKSBootstrapScriptFile
& $EKSBootstrapScriptFile -EKSClusterName "eks-windows" -APIServerEndpoint "https://<REPLACE-EKS-CLUSTER-CONFIG-API-SERVER>" -Base64ClusterCA "<REPLACE-EKSCLUSTER-CONFIG-DETAILS-CA>" -DNSClusterIP "172.20.0.10" -KubeletExtraArgs "--node-labels=alpha.eksctl.io/cluster-name=eks-windows,alpha.eksctl.io/nodegroup-name=windows-ng-ltsc2019 --register-with-taints=" 3>&1 4>&1 5>&1 6>&1
</powershell>
DSR enablement can be verified following the instructions in the Microsoft Networking blog and the Windows Containers on AWS Lab.
Container Network Interface (CNI) options¶
The AWSVPC CNI is the de facto CNI plugin for Windows and Linux worker nodes. While the AWSVPC CNI satisfies the needs of many customers, still there may be times when you need to consider alternatives like an overlay network to avoid IP exhaustion. In these cases, the Calico CNI can be used in place of the AWSVPC CNI. Project Calico is open source software that was developed by Tigera. That software includes a CNI that works with EKS. Instructions for installing Calico CNI in EKS can be found on the Project Calico EKS installation page.
Network Polices¶
It is considered a best practice to change from the default mode of open communication between pods on your Kubernetes cluster to limiting access based on network polices. The open source Project Calico has strong support for network polices that work with both Linux and Windows nodes. This feature is separate and not dependent on using the Calico CNI. We therefore recommend installing Calico and using it for network policy management.
Instructions for installing Calico in EKS can be found on the Installing Calico on Amazon EKS page.
In addition, the advice provided in the Amazon EKS Best Practices Guide for Security - Network Section applies equally to EKS clusters with Windows worker nodes, however, some features like "Security Groups for Pods" are not supported by Windows at this time.