We've Moved to the AWS Docs! 🚀
This content has been updated and relocated to improve your experience. Please visit our new site for the latest version: AWS EKS Best Practices Guide on the AWS Docs
Bookmarks and links will continue to work, but we recommend updating them for faster access in the future.
Windows Networking¶
Windows Container Networking Overview¶
Windows containers are fundamentally different than Linux containers. Linux containers use Linux constructs like namespaces, the union file system, and cgroups. On Windows, those constructs are abstracted from containerd by the Host Compute Service (HCS). HCS acts as an API layer that sits above the container implementation on Windows. Windows containers also leverage the Host Network Service (HNS) that defines the network topology on a node.
From a networking perspective, HCS and HNS make Windows containers function like virtual machines. For example, each container has a virtual network adapter (vNIC) that is connected to a Hyper-V virtual switch (vSwitch) as shown in the diagram above.
IP Address Management¶
A node in Amazon EKS uses it's Elastic Network Interface (ENI) to connect to an AWS VPC network. Presently, only a single ENI per Windows worker node is supported. The IP address management for Windows nodes is performed by VPC Resource Controller which runs in control plane. More details about the workflow for IP address management of Windows nodes can be found here.
The number of pods that a Windows worker node can support is dictated by the size of the node and the number of available IPv4 addresses. You can calculate the IPv4 address available on the node as below: - By default, only secondary IPv4 addresses are assigned to the ENI. In such a case:
Total IPv4 addresses available for Pods = Number of supported IPv4 addresses in the primary interface - 1
- If the cluster has been configured for high pod density by enabling prefix delegation feature then-
Here, instead of allocating secondary IPv4 addresses, VPC Resource Controller will allocate
Total IPv4 addresses available for Pods = (Number of supported IPv4 addresses in the primary interface - 1) * 16/28 prefixesand therefore, the overall number of available IPv4 addresses will be boosted 16 times.
Using the formula above, we can calculate max pods for an Windows worker noded based on a m5.large instance as below: - By default, when running in secondary IP mode-
- When usingprefix delegation-
For more information on how many IP addresses an instance type can support, see IP addresses per network interface per instance type.
Another key consideration is the flow of network traffic. With Windows there is a risk of port exhaustion on nodes with more than 100 services. When this condition arises, the nodes will start throwing errors with the following message:
"Policy creation failed: hcnCreateLoadBalancer failed in Win32: The specified port already exists."
To address this issue, we leverage Direct Server Return (DSR). DSR is an implementation of asymmetric network load distribution. In other words, the request and response traffic use different network paths. This feature speeds up communication between pods and reduces the risk of port exhaustion. We therefore recommend enabling DSR on Windows nodes.
DSR is enabled by default in Windows Server SAC EKS Optimized AMIs. For Windows Server 2019 LTSC EKS Optimized AMIs, you will need to enable it during instance provisioning using the script below and by using Windows Server 2019 Full or Core as the amiFamily in the eksctl nodeGroup. See eksctl custom AMI for additional information.
nodeGroups:
- name: windows-ng
instanceType: c5.xlarge
minSize: 1
volumeSize: 50
amiFamily: WindowsServer2019CoreContainer
ssh:
allow: false
<powershell>
[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
(Get-Content $EKSBootstrapScriptFile).replace('"--proxy-mode=kernelspace",', '"--proxy-mode=kernelspace", "--feature-gates WinDSR=true", "--enable-dsr",') | Set-Content $EKSBootstrapScriptFile
& $EKSBootstrapScriptFile -EKSClusterName "eks-windows" -APIServerEndpoint "https://<REPLACE-EKS-CLUSTER-CONFIG-API-SERVER>" -Base64ClusterCA "<REPLACE-EKSCLUSTER-CONFIG-DETAILS-CA>" -DNSClusterIP "172.20.0.10" -KubeletExtraArgs "--node-labels=alpha.eksctl.io/cluster-name=eks-windows,alpha.eksctl.io/nodegroup-name=windows-ng-ltsc2019 --register-with-taints=" 3>&1 4>&1 5>&1 6>&1
</powershell>
DSR enablement can be verified following the instructions in the Microsoft Networking blog and the Windows Containers on AWS Lab.
If preserving your available IPv4 addresses and minimizing wastage is crucial for your subnet, it is generally recommended to avoid using prefix delegation mode as mentioned in Prefix Mode for Windows - When to avoid. If using prefix delegation is still desired, you can take steps to optimize IPv4 address utilization in your subnet. See Configuring Parameters for Prefix Delegation for detailed instructions on how to fine-tune the IPv4 address request and allocation process. Adjusting these configurations can help you strike a balance between conserving IPv4 addresses and pod density benefits of prefix delegation.
When using the default setting of assigning secondary IPv4 addresses, there are currently no supported configurations to manipulate how the VPC Resource Controller requests and allocates IPv4 addresses. More specifically, minimum-ip-target and warm-ip-target are only supported for prefix delegation mode. Also take note that in secondary IP mode, depending on the available IP addresses on the interface, the VPC Resource Controller will typically allocate 3 unused IPv4 addresses on the node on your behalf to maintain warm IPs for faster pod startup times. If you would like to minimize IP wastage of unused warm IP addresses, you could aim to schedule more pods on a given Windows node such that you use as much IP address capacity of the ENI as possible. More explicitly, you could avoid having warm unused IPs if all IP addresses on the ENI are already in use by the node and running pods. Another workaround to help you resolve constraints with IP address availability in your subnet(s) could be to explore increasing your subnet size or separating your Windows nodes into their own dedicated subnets.
Additionally, it's important to note that IPv6 is not supported on Windows nodes at the moment.
Container Network Interface (CNI) options¶
The AWSVPC CNI is the de facto CNI plugin for Windows and Linux worker nodes. While the AWSVPC CNI satisfies the needs of many customers, still there may be times when you need to consider alternatives like an overlay network to avoid IP exhaustion. In these cases, the Calico CNI can be used in place of the AWSVPC CNI. Project Calico is open source software that was developed by Tigera. That software includes a CNI that works with EKS. Instructions for installing Calico CNI in EKS can be found on the Project Calico EKS installation page.
Network Polices¶
It is considered a best practice to change from the default mode of open communication between pods on your Kubernetes cluster to limiting access based on network polices. The open source Project Calico has strong support for network polices that work with both Linux and Windows nodes. This feature is separate and not dependent on using the Calico CNI. We therefore recommend installing Calico and using it for network policy management.
Instructions for installing Calico in EKS can be found on the Installing Calico on Amazon EKS page.
In addition, the advice provided in the Amazon EKS Best Practices Guide for Security - Network Section applies equally to EKS clusters with Windows worker nodes, however, some features like "Security Groups for Pods" are not supported by Windows at this time.