Last week Azure CNI Powered by Cilium went GA, which means everybody can use this as a supported service/eBPF data plane on the Azure Kubernetes (AKS) platform. You can read the Microsoft Tech Community announcement at techcommunity.microsoft.com.
It was a long wait, but finally AKS can natively profit from the power of eBPF with Cilium on both performance networking and security perspective. First feature that went GA is IPAM support. Additionally you are able to manage and enforce network access with kubernetes network policies.
During this blog I will show my first impressions how you can get started and some experimentation on applying some Cilium Enterprise features. Let’s first start with an overview of the cilium project and eBPF.
Overview
Cilium is an Open Source Software (OSS) project, since 2015. Cilium is built on top of a kernel feature called eBPF. In short eBPF is a Linux kernel technology that allows sandboxed programs to run verified code against kernel subsystems.
Primarily for the CNI we profit from the eBPF data path for network program types like tc (traffic control – Ingress/Egress) and XDP (eXpress Data Path). For example many large companies already use XDP for high performance networking like Facebook that built Katran which is their network load balancer (NLB) of choice. Another good example is Walmart with their OSS L3AF project.
Both topics require a blog on their own, but if you are interested you can read more information about eBPF at epbf.io.
First Impressions
Now let’s start our first steps to deploy an Azure Kubernetes Service (AKS) cluster. For this you can follow the Microsoft documentation here. For convenience I will use Cloud Shell, but the Azure CLI should also work.
Let’s start with the prerequisites for AKS. Start creating a resource group called “aks-cilium-playground” and creating a VNet with two subsets, one for the actual nodes and the other for running pods. Pods are the part that are responsible for running your actual workloads.
Below the sample snippets to create the Azure resources.
az network vnet create -g aks-cilium-playground --location eastus --name cilium-vnet --address-prefixes 10.0.0.0/8 -o none
az network vnet subnet create -g aks-cilium-playground --vnet-name cilium-vnet --name nodesubnet --address-prefixes 10.240.0.0/16 -o none
az network vnet subnet create -g aks-cilium-playground --vnet-name cilium-vnet --name podsubnet --address-prefixes 10.241.0.0/16 -o noneNow that we have implemented all prerequisites we can deploy the actual AKS Cluster. I will deploy my cluster in the US East region, but that is not mandatory.
Further take notice that you have to provide your SubscriptionId. Optionally you may need to generate your SSH Key Pair.
Below the sample snippet to create the AKS Cluster.
az aks create -n cilium-playground -g aks-cilium-playground -l eastus --max-pods 250 --network-plugin azure --vnet-subnet-id /subscriptions/<subscriptionId>/resourceGroups/aks-cilium-playground/providers/Microsoft.Network/virtualNetworks/cilium-vnet/subnets/nodesubnet --pod-subnet-id /subscriptions/<subscriptionId>/resourceGroups/aks-cilium-playground/providers/Microsoft.Network/virtualNetworks/cilium-vnet/subnets/podsubnet --network-dataplane cilium [--generate-ssh-keys]After approx 5 minutes the cluster becomes available to connect. From the Azure CLI, ensure that you have set the right Subscription and get the Managed Kubernetes cluster credentials to be imported in your local kubeconfig using ‘az aks get-credentials‘. You may also want to use Connect from the Azure Portal.
Great, let’s first start identifying what Cilium components are running within the ‘kube-system’ namespace.
Run the kubectl CLI command below.
kubectl get pods -n kube-system | grep cilium
cilium-2xzlr 1/1 Running 0 5m20s
cilium-operator-6745694fc-68hhv 1/1 Running 0 5m20s
cilium-operator-6745694fc-lcch7 1/1 Running 0 5m20s
cilium-rx92x 1/1 Running 0 5m20s
cilium-xjprw 1/1 Running 0 5m20sYou will identify a running Cilium DaemonSet and the actual Cilium Operator Deployment. Remember this for later, since we are going to dive further into the DaemonSet containers.
Most operators come with Custom Resources (CRs), let’s look if you have such Custom Resource Definitions (CRDs) registered.
kubectl get crds | grep “cilium.io”
ciliumclusterwidenetworkpolicies.cilium.io 2023-06-02T18:15:03Z
ciliumendpoints.cilium.io 2023-06-02T18:15:02Z
ciliumexternalworkloads.cilium.io 2023-06-02T18:15:02Z
ciliumidentities.cilium.io 2023-06-02T18:15:02Z
ciliumnetworkpolicies.cilium.io 2023-06-02T18:15:02Z
ciliumnodes.cilium.io 2023-06-02T18:15:02ZYou will see some familiar resources available like Cilium Network Policies and Cilium Endpoints. In our case the Cilium operator is responsible for IP Address management (IPAM) within the AKS cluster.
Find out more details about the Cilium Operator at docs.cilium.io.
Now it’s time to dive further into the DaemonSet details. The DaemonSet is the part that runs the actual ‘cilium-agent’ container on every node within the cluster. Let’s first look at what is part of the DaemonSet, configured probes and potential init-containers.
Below the kubectl CLI sample to execute.
kubectl describe ds/cilium -n kube-systemWithin the container we can execute several commands to provide us information for availability, debugging and troubleshooting purposes. The first command we will look at is cilium. Cilium is a helpful command with various interesting submodules available.
Let’s have a look at the submodules that are available.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium
CLI for interacting with the local Cilium Agent
Usage:
cilium [command]
Available Commands:
bpf Direct access to local BPF maps
cleanup Remove system state installed by Cilium at runtime
completion Output shell completion code
config Cilium configuration options
debuginfo Request available debugging information from agent
encrypt Manage transparent encryption
endpoint Manage endpoints
fqdn Manage fqdn proxy
help Help about any command
identity Manage security identities
ip Manage IP addresses and associated information
kvstore Direct access to the kvstore
lrp Manage local redirect policies
map Access userspace cached content of BPF maps
metrics Access metric status
monitor Display BPF program events
node Manage cluster nodes
nodeid List node IDs and associated information
policy Manage security policies
prefilter Manage XDP CIDR filters
preflight cilium upgrade helper
recorder Introspect or mangle pcap recorder
service Manage services & loadbalancers
status Display status of daemon
version Print version information
Flags:
--config string config file (default is $HOME/.cilium.yaml)
-D, --debug Enable debug messages
-h, --help help for cilium
-H, --host string URI to server-side API
Use "cilium [command] --help" for more information about a command.Let’s first lookup the Cilium version which has been deployed.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium version
Client: 1.12.8 87d85617d1 2023-03-15T19:28:16+01:00 go version go1.18.10 linux/amd64
Daemon: 1.12.8 87d85617d1 2023-03-15T19:28:16+01:00 go version go1.18.10 linux/amd64Another one which is helpful for troubleshooting is Cilium status.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium status
KVStore: Ok Disabled
Kubernetes: Ok 1.25 (v1.25.6) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [eth0 10.240.0.4 (Direct Routing)]
Host firewall: Disabled
CNI Chaining: none
Cilium: Ok 1.12.8 (v1.12.8-87d85617d1)
NodeMonitor: Disabled
Cilium health daemon: Ok
IPAM: IPv4: delegated to plugin,
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: Disabled
Controller Status: 45/45 healthy
Proxy Status: No managed proxy redirect
Global Identity Range: min 256, max 65535
Hubble: Disabled
Encryption: Disabled
Cluster health: Probe disabledHere you can learn that IPAM for IPv4 is delegated to the cilium plugin.
Unfortunately Encryption and Hubble are disabled, which are part of Isovalent Cilium Enterprise. Overall we can see that the Cilium health daemon is Ok.
For these specific health checks you also have a separate API available. You may recognise some overlapping information with ‘cilium status’ output.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it – cilium-health get
Daemon uptime: 5m8.314442604s
Node load: 0.55 0.61 0.59
KVStore: Ok Disabled
Kubernetes: Ok 1.25 (v1.25.6) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [eth0 10.240.0.4 (Direct Routing)]
Host firewall: Disabled
CNI Chaining: none
Cilium: Ok 1.12.8 (v1.12.8-87d85617d1)
NodeMonitor: Disabled
Cilium health daemon: Ok
IPAM: IPv4: delegated to plugin,
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: Disabled
Controller Status: 68/68 healthy
Proxy Status: No managed proxy redirect
Global Identity Range: min 256, max 65535
Hubble: Disabled
Encryption: DisabledLet’s dive further into more troubleshooting techniques. To have some actual load and create some traffic we will first deploy a demo workload. I will make use of the Sockshop provided by Weaveworks. Why? Just love geeky techie socks.
To get the application code just clone the repository and apply the actual complete application YAML.
It will create a namespace called sock-shop for you with all required deployments and services. If you want to see all objects, just run ‘kubectl get all’.
# Credits to Weaveworks for providing the Awesome Sockshop
git clone https://github.com/microservices-demo/microservices-demo.git
cd microservices-demo/deploy/kubernetes
kubectl apply -f complete-demo.yaml
kubectl get all -n sock-shopOptionally you may want to expose the ‘front-end’ service to the Internet by changing the ‘NodePort’ to LoadBalancer type using ‘kubectl edit’. Just replace NodePort for LoadBalancer in the front-end configuration.
Now you can open the Sockshop using your favorite browser.
You have just added a workload and introduced some traffic we can inspect with other features that are provided by Cilium. For example we can look up the exposed services like our front-end ‘LoadBalancer’.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium service list
ID Frontend Service Type Backend
1 10.0.0.1:443 ClusterIP 1 => 22.74.137.170:443 (active)
2 10.0.0.10:53 ClusterIP 1 => 10.241.0.53:53 (active)
2 => 10.241.0.15:53 (active)
3 10.0.196.193:443 ClusterIP 1 => 10.241.0.25:4443 (active)
2 => 10.241.0.7:4443 (active)
4 10.0.218.78:443 ClusterIP 1 => 10.241.0.16:8443 (active)
2 => 10.241.0.30:8443 (active)
5 10.0.67.179:443 ClusterIP 1 => 10.241.0.23:9449 (active)
6 10.0.100.55:80 ClusterIP 1 => 10.241.0.26:80 (active)
7 10.0.195.187:27017 ClusterIP 1 => 10.241.0.42:27017 (active)
8 10.0.119.123:80 ClusterIP 1 => 10.241.0.9:80 (active)
9 10.0.177.223:3306 ClusterIP 1 => 10.241.0.33:3306 (active)
10 10.0.141.51:80 ClusterIP 1 => 10.241.0.40:8079 (active)
11 10.240.0.4:30001 NodePort 1 => 10.241.0.40:8079 (active)
12 0.0.0.0:30001 NodePort 1 => 10.241.0.40:8079 (active)
13 10.0.166.237:80 ClusterIP 1 => 10.241.0.6:80 (active)
14 10.0.130.189:27017 ClusterIP 1 => 10.241.0.27:27017 (active)
15 10.0.23.60:80 ClusterIP 1 => 10.241.0.41:80 (active)
16 10.0.7.87:80 ClusterIP 1 => 10.241.0.11:80 (active)
17 10.0.75.25:5672 ClusterIP 1 => 10.241.0.22:5672 (active)
18 10.0.75.25:9090 ClusterIP 1 => 10.241.0.22:9090 (active)
19 10.0.23.29:6379 ClusterIP 1 => 10.241.0.39:6379 (active)
20 10.0.25.248:80 ClusterIP 1 => 10.241.0.14:80 (active)
21 10.0.101.30:80 ClusterIP 1 => 10.241.0.49:80 (active)
22 10.0.255.230:27017 ClusterIP 1 => 10.241.0.17:27017 (active)
23 33.44.161.238:80 LoadBalancer 1 => 10.241.0.40:8079 (active)This list provides you some details on the frontend IP and which backend target Endpoints that are currently active. Important detail you may see are the differences in Ip Address for each Service Type.
Another one is looking up the cached BPF maps in kernel user-space.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium map list
Name Num entries Num errors Cache enabled
cilium_policy_01101 0 0 false
cilium_lxc 11 0 true
cilium_ipcache 39 0 true
cilium_lb4_backends_v2 0 0 true
cilium_policy_00062 0 0 false
cilium_policy_00843 0 0 false
cilium_lb4_reverse_nat 23 0 true
cilium_policy_00153 0 0 false
cilium_policy_03135 0 0 false
cilium_policy_00591 0 0 false
cilium_policy_03868 0 0 false
cilium_policy_00753 0 0 false
cilium_policy_00045 0 0 false
cilium_policy_03594 0 0 false
cilium_lb4_services_v2 23 0 true
cilium_lb_affinity_match 0 0 true
cilium_lb4_affinity 0 0 false
cilium_lb4_source_range 0 0 true
cilium_policy_00199 0 0 false
cilium_policy_00601 0 0 false
cilium_node_map 0 0 false
cilium_metrics 0 0 false
You can see which maps are currently open, number of entries, errors that occurred and if caching is enabled for that specific map. Interesting are the cilium_ipcache and the cilium_lb4_services_v2. Do you recognise the number of services?
Let’s look deeper in the actual workload traffic and monitor packets. You can also specify a specific type like ‘drop’ or only ‘L7’ packets. Read more about this here.
See a monitor example below that shows actual packets from the CNI backbone.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium monitor
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
-> endpoint 3594 flow 0x58f55a72 , identity host->3041 state new ifindex 0 orig-ip 10.240.0.4: 10.240.0.4:46076 -> 10.241.0.9:80 tcp SYN
-> stack flow 0xe94e2331 , identity 3041->host state reply ifindex 0 orig-ip 0.0.0.0: 10.241.0.9:80 -> 10.240.0.4:46076 tcp SYN, ACK
-> endpoint 3594 flow 0x58f55a72 , identity host->3041 state established ifindex 0 orig-ip 10.240.0.4: 10.240.0.4:46076 -> 10.241.0.9:80 tcp ACK
-> endpoint 3594 flow 0x1f7b2e1a , identity host->3041 state new ifindex 0 orig-ip 10.240.0.4: 10.240.0.4:46074 -> 10.241.0.9:80 tcp SYN
-> stack flow 0x8f2673e7 , identity 3041->host state reply ifindex 0 orig-ip 0.0.0.0: 10.241.0.9:80 -> 10.240.0.4:46074 tcp SYN, ACK
-> endpoint 3594 flow 0x1f7b2e1a , identity host->3041 state established ifindex 0 orig-ip 10.240.0.4: 10.240.0.4:46074 -> 10.241.0.9:80 tcp ACK
-> endpoint 3594 flow 0x1f7b2e1a , identity host->3041 state established ifindex 0 orig-ip 10.240.0.4: 10.240.0.4:46074 -> 10.241.0.9:80 tcp ACK
-> endpoint 3594 flow 0x58f55a72 , identity host->3041 state established ifindex 0 orig-ip 10.240.0.4: 10.240.0.4:46076 -> 10.241.0.9:80 tcp ACK
-> stack flow 0x8f2673e7 , identity 3041->host state reply ifindex 0 orig-ip 0.0.0.0: 10.241.0.9:80 -> 10.240.0.4:46074 tcp ACK
-> stack flow 0x8f2673e7 , identity 3041->host state reply ifindex 0 orig-ip 0.0.0.0: 10.241.0.9:80 -> 10.240.0.4:46074 tcp ACK, FIN
-> stack flow 0xe94e2331 , identity 3041->host state reply ifindex 0 orig-ip 0.0.0.0: 10.241.0.9:80 -> 10.240.0.4:46076 tcp ACK
-> stack flow 0xe94e2331 , identity 3041->host state reply ifindex 0 orig-ip 0.0.0.0: 10.241.0.9:80 -> 10.240.0.4:46076 tcp ACK, FIN
-> endpoint 3594 flow 0x58f55a72 , identity host->3041 state established ifindex 0 orig-ip 10.240.0.4: 10.240.0.4:46076 -> 10.241.0.9:80 tcp ACK, FIN
-> endpoint 3594 flow 0x1f7b2e1a , identity host->3041 state established ifindex 0 orig-ip 10.240.0.4: 10.240.0.4:46074 -> 10.241.0.9:80 tcp ACK, FINExperimenting with unofficial Azure CNI Powered by Cilium features
Disclaimer
During this chapter you are going to experiment with Cilium features which are officially not supported by Microsoft opinionated GA release of Azure CNI Powered by Cilium. Additional to that there is no warranty or guarantee that it works, keeps working or becomes part of the Azure CNI extension. Below is only for research purposes and shall not be a reason for experimenting on Production systems or run production workloads.
People that want to leverage these cilium features like Cilium network policies and node-to-node transparent encryption must either Bring Your Own CNI (AKS BYO CNI) or upgrade to Isovalent Cilium Enterprise. Upgrading towards Enterprise Cilium is possible and explained after this chapter.
The intention here is to give you an idea what is possible with Cilium and let you decide if it’s valuable for you and want to implement your own CNI.
Cilium Network Policies
As already mentioned you can also can add Kubernetes Network Policy resources. This is great, but if you want to leverage Cilium Network Policies and do they work as expected. Let’s start with a simple block “ block-default-deny-access” to try out.
You may want to model your policy at editor.networkpolicy.io . This is a visual editor provided by Isovalent. Just choose default deny for both ingress and egress, set the namespace, name and voila you have your policy available.
Using the example policy below we are going to apply a default block on ingress and egress traffic within namespace sock-shop.
kubectl apply -f block-default-deny-access.yaml -n sock-shop
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: block-default-deny-access
namespace: sock-shop
specs:
- endpointSelector: {}
egress:
- {}
ingress:
- {}Try to open the sock-shop again with your favorite browser and you will see ingress traffic from the Internet is now blocked as configured.
You may want to validate if an internal namespace curl works!
kubectl run curl --image=radial/busyboxplus:curl -it --rm -n sock-shop --restart=Never -- curl http://front-endLet’s dive into a more advanced L3/L4 Cilium Network Policy. This will be used to harden the Sockshop by implementing this policy you are.
- Only allowing ingress traffic in the sock-shop namespace from pods/services in the sock-shop namespace.
- Allowing Egress from both world (internet) and cluster entities. This is helpful, since some pods require connectivity.
- Ensuring that you the world ingress for front-end services.
kubectl apply -f harden-sock-shop.yaml -n sock-shop
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: harden-sock-shop-access
namespace: sock-shop
specs:
- endpointSelector:
matchLabels: {}
egress:
- toEntities:
- world
- cluster
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": sock-shop
- endpointSelector:
matchLabels:
name: front-end
ingress:
- fromEntities:
- world
egress:
- toEndpoints:
- matchLabels: {}Now refresh the page again and you will notice traffic is coming in again. Great, you now have tried out some policies, let’s dive into another topic.
Node-to-Node Transparent Encryption
One of the most powerful features of Cilium is providing Node-to-Node Transparent Encryption. Let’s first look if this is enabled.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium encrypt status
Encryption: DisabledYou will notice that It’s disabled since it’s unsupported. Also there is currently no official way of adjusting the AKS Helm configuration, but we can experiment by overriding certain parameters in the ConfigMap. Let’s give this a try.
This is only for experimentation and totally not advised on AKS Production clusters until it’s supported. After a redeployment the configuration will be overridden by FluxCD policies.
First look take a look in the ConfigMap itself.
kubectl get cm cilium-config -n kube-system -o yaml
apiVersion: v1
data:
agent-not-ready-taint-key: node.cilium.io/agent-not-ready
arping-refresh-period: 30s
auto-direct-node-routes: "false"
bpf-lb-external-clusterip: "false"
bpf-lb-map-max: "65536"
bpf-lb-mode: snat
bpf-map-dynamic-size-ratio: "0.0025"
bpf-policy-map-max: "16384"
bpf-root: /sys/fs/bpf
cgroup-root: /run/cilium/cgroupv2
cilium-endpoint-gc-interval: 5m0s
cluster-id: "0"
cluster-name: default
debug: "false"
disable-cnp-status-updates: "true"
disable-endpoint-crd: "false"
enable-auto-protect-node-port-range: "true"
enable-bgp-control-plane: "false"
enable-bpf-clock-probe: "true"
enable-endpoint-health-checking: "false"
enable-endpoint-routes: "true"
enable-health-check-nodeport: "true"
enable-health-checking: "true"
enable-host-legacy-routing: "true"
enable-hubble: "false"
enable-ipv4: "true"
enable-ipv4-masquerade: "false"
enable-ipv6: "false"
enable-ipv6-masquerade: "false"
enable-k8s-terminating-endpoint: "true"
enable-l2-neigh-discovery: "true"
enable-l7-proxy: "false"
enable-local-node-route: "false"
enable-local-redirect-policy: "false"
enable-metrics: "true"
enable-policy: default
enable-remote-node-identity: "true"
enable-session-affinity: "true"
enable-svc-source-range-check: "true"
enable-vtep: "false"
enable-well-known-identities: "false"
enable-xt-socket-fallback: "true"
identity-allocation-mode: crd
install-iptables-rules: "true"
install-no-conntrack-iptables-rules: "false"
ipam: delegated-plugin
kube-proxy-replacement: strict
kube-proxy-replacement-healthz-bind-address: ""
local-router-ipv4: 169.254.23.0
metrics: +cilium_bpf_map_pressure
monitor-aggregation: medium
monitor-aggregation-flags: all
monitor-aggregation-interval: 5s
node-port-bind-protection: "true"
nodes-gc-interval: 5m0s
operator-api-serve-addr: 127.0.0.1:9234
operator-prometheus-serve-addr: :9963
preallocate-bpf-maps: "false"
procfs: /host/proc
prometheus-serve-addr: :9962
remove-cilium-node-taints: "true"
set-cilium-is-up-condition: "true"
sidecar-istio-proxy-image: cilium/istio_proxy
synchronize-k8s-nodes: "true"
tofqdns-dns-reject-response-code: refused
tofqdns-enable-dns-compression: "true"
tofqdns-endpoint-max-ip-per-hostname: "50"
tofqdns-idle-connection-grace-period: 0s
tofqdns-max-deferred-connection-deletes: "10000"
tofqdns-min-ttl: "3600"
tofqdns-proxy-response-max-delay: 100ms
tunnel: disabled
unmanaged-pod-watcher-interval: "15"
vtep-cidr: ""
vtep-endpoint: ""
vtep-mac: ""
vtep-mask: ""
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: cilium
meta.helm.sh/release-namespace: kube-system
creationTimestamp: "2023-06-02T18:14:49Z"
labels:
app.kubernetes.io/managed-by: Helm
helm.toolkit.fluxcd.io/name: cilium-adapter-helmrelease
helm.toolkit.fluxcd.io/namespace: 647a30a31348b50001a23260
name: cilium-config
namespace: kube-system
resourceVersion: "1317"
uid: 59d4ce5a-dd60-4cb1-8d24-156c7e00c80bNo potential settings are visible, but let’s consult the Cilium v1.12 documentation. Here you can learn how to easily setup Wireguard Encryption. Read more about this topic at doc.cilium.io.
To summarise you have to add the following settings to the ConfigMap (using edit , for example) and restart the Cilium DaemonSet.
kubectl edit cm -n kube-system cilium-config
# Add the following settings
enable-wireguard: "true"
encrypt-interface: eth0
encrypt-node: "true"Now restart the Cilium DaemonSet and monitor log activity if settings are applied. The actual cilium-config is mounted in the cilium-agent container as folder in the ‘/tmp’ directory.
kubectl rollout restart ds/cilium -n kube-system
kubectl logs ds/cilium -n kube-system -c cilium-agentIt can take around a minute before Cilium is fully restarted. Now use the Cilium CLI to look up the current Encryption status.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium encrypt status
Encryption: Wireguard
Interface: cilium_wg0
Public key: rUKEyX39jJfGOYCBb6EKVBMBRPSCF1gy7mzHl7jfmWo=
Number of peers: 0Awesome, you have successfully enabled Wireguard Transparent Encryption.
Last thing we can do is providing clear evidence using tcpdump to analyse if actual traffic is going over the cilium_wg0 interface.
For this experiment we temporary need to install tcpdump in the cilium container.
apt-get update
apt-get install tcpdump -yBelow the command how to start a tcpdump capturing session on the cilium_wg0 interface.
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- tcpdump -n -i cilium_wg0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cilium_wg0, link-type RAW (Raw IP), capture size 262144 bytes
20:55:52.690132 IP 10.241.0.56.50046 > 10.241.0.15.53: 21806+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.svc.cluster.local. (97)
20:55:52.690161 IP 10.241.0.56.59114 > 10.241.0.15.53: 29552+ A? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.svc.cluster.local. (97)
20:55:54.096583 IP 10.241.0.49.45816 > 10.241.0.17.27017: Flags [S], seq 1255489429, win 64240, options [mss 1460,sackOK,TS val 1698354767 ecr 0,nop,wscale 7], length 0
20:55:54.597229 IP 10.241.0.49.45830 > 10.241.0.17.27017: Flags [S], seq 2139424092, win 64240, options [mss 1460,sackOK,TS val 1698355268 ecr 0,nop,wscale 7], length 0
20:55:55.097694 IP 10.241.0.49.45844 > 10.241.0.17.27017: Flags [S], seq 2888121862, win 64240, options [mss 1460,sackOK,TS val 1698355768 ecr 0,nop,wscale 7], length 0
20:55:55.598497 IP 10.241.0.49.40889 > 10.241.0.15.53: 40266+ A? user-db.sock-shop.svc.cluster.local. (53)
20:55:57.693426 IP 10.241.0.56.51667 > 10.241.0.15.53: 364+ A? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.svc.cluster.local. (97)
20:56:00.599216 IP 10.241.0.49.45854 > 10.241.0.17.27017: Flags [S], seq 1254850900, win 64240, options [mss 1460,sackOK,TS val 1698361270 ecr 0,nop,wscale 7], length 0
20:56:01.099627 IP 10.241.0.49.45866 > 10.241.0.17.27017: Flags [S], seq 1996097473, win 64240, options [mss 1460,sackOK,TS val 1698361770 ecr 0,nop,wscale 7], length 0
20:56:01.600447 IP 10.241.0.49.45876 > 10.241.0.17.27017: Flags [S], seq 2185372130, win 64240, options [mss 1460,sackOK,TS val 1698362271 ecr 0,nop,wscale 7], length 0
20:56:02.101869 IP 10.241.0.49.59190 > 10.241.0.17.27017: Flags [S], seq 1238457071, win 64240, options [mss 1460,sackOK,TS val 1698362773 ecr 0,nop,wscale 7], length 0
20:56:02.602223 IP 10.241.0.49.59192 > 10.241.0.17.27017: Flags [S], seq 2759758146, win 64240, options [mss 1460,sackOK,TS val 1698363273 ecr 0,nop,wscale 7], length 0
20:56:02.698063 IP 10.241.0.56.40375 > 10.241.0.15.53: 18490+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.cluster.local. (93)
20:56:03.102694 IP 10.241.0.49.59202 > 10.241.0.17.27017: Flags [S], seq 4188840387, win 64240, options [mss 1460,sackOK,TS val 1698363773 ecr 0,nop,wscale 7], length 0
20:56:03.512378 IP 10.241.0.40.56157 > 10.241.0.15.53: 30521+ A? user.sock-shop.svc.cluster.local. (50)
20:56:03.512420 IP 10.241.0.40.56157 > 10.241.0.15.53: 30721+ AAAA? user.sock-shop.svc.cluster.local. (50)
20:56:04.103526 IP 10.241.0.49.32901 > 10.241.0.15.53: 38254+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:56:06.013019 IP 10.241.0.40.56157 > 10.241.0.15.53: 30521+ A? user.sock-shop.svc.cluster.local. (50)
20:56:06.013070 IP 10.241.0.40.56157 > 10.241.0.15.53: 30721+ AAAA? user.sock-shop.svc.cluster.local. (50)
20:56:07.702862 IP 10.241.0.56.55643 > 10.241.0.15.53: 58518+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.cluster.local. (93)
20:56:09.104031 IP 10.241.0.49.47850 > 10.241.0.15.53: 47460+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:56:12.703719 IP 10.241.0.56.53491 > 10.241.0.15.53: 23494+ A? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.rnfjgl5vvqwu1iaacmg53jyscf.bx.internal.cloudapp.net. (131)
20:56:14.383172 IP 10.241.0.46.47115 > 10.241.0.15.53: 42857+ AAAA? dc.services.visualstudio.com.svc.cluster.local. (64)
20:56:14.604312 IP 10.241.0.49.44510 > 10.241.0.15.53: 47808+ A? user-db.sock-shop.svc.cluster.local. (53)
20:56:17.709006 IP 10.241.0.56.43907 > 10.241.0.15.53: 425+ A? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com. (79)
20:56:19.383669 IP 10.241.0.46.52783 > 10.241.0.15.53: 9106+ AAAA? dc.services.visualstudio.com.svc.cluster.local. (64)
20:56:19.605304 IP 10.241.0.49.60944 > 10.241.0.17.27017: Flags [S], seq 2360359542, win 64240, options [mss 1460,sackOK,TS val 1698380276 ecr 0,nop,wscale 7], length 0
20:56:20.105683 IP 10.241.0.49.60960 > 10.241.0.17.27017: Flags [S], seq 1715323892, win 64240, options [mss 1460,sackOK,TS val 1698380776 ecr 0,nop,wscale 7], length 0
20:56:20.606054 IP 10.241.0.49.60966 > 10.241.0.17.27017: Flags [S], seq 2376095443, win 64240, options [mss 1460,sackOK,TS val 1698381277 ecr 0,nop,wscale 7], length 0
20:56:21.106509 IP 10.241.0.49.42581 > 10.241.0.15.53: 45145+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:56:22.723849 IP 10.241.0.56.45952 > 10.241.0.15.53: 62046+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.kube-system.svc.cluster.local. (109)
20:56:22.723994 IP 10.241.0.56.46490 > 10.241.0.15.53: 35704+ A? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.kube-system.svc.cluster.local. (109)
20:56:24.384958 IP 10.241.0.46.45158 > 10.241.0.15.53: 8904+ AAAA? dc.services.visualstudio.com.rnfjgl5vvqwu1iaacmg53jyscf.bx.internal.cloudapp.net. (98)
20:56:25.426843 IP 10.241.0.49.50147 > 10.241.0.15.53: 11826+ A? user-db.sock-shop.svc.cluster.local. (53)
20:56:27.725300 IP 10.241.0.56.48586 > 10.241.0.15.53: 44888+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.kube-system.svc.cluster.local. (109)
20:56:29.389302 IP 10.241.0.46.42854 > 10.241.0.15.53: 33321+ AAAA? dc.services.visualstudio.com.rnfjgl5vvqwu1iaacmg53jyscf.bx.internal.cloudapp.net. (98)
20:56:30.427749 IP 10.241.0.49.48530 > 10.241.0.17.27017: Flags [S], seq 2618632773, win 64240, options [mss 1460,sackOK,TS val 1698391098 ecr 0,nop,wscale 7], length 0
20:56:30.928372 IP 10.241.0.49.48536 > 10.241.0.17.27017: Flags [S], seq 1123770872, win 64240, options [mss 1460,sackOK,TS val 1698391599 ecr 0,nop,wscale 7], length 0
20:56:31.429362 IP 10.241.0.49.48546 > 10.241.0.17.27017: Flags [S], seq 2234493642, win 64240, options [mss 1460,sackOK,TS val 1698392100 ecr 0,nop,wscale 7], length 0
20:56:31.930619 IP 10.241.0.49.54142 > 10.241.0.15.53: 6301+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:56:32.726979 IP 10.241.0.56.50739 > 10.241.0.15.53: 22474+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.svc.cluster.local. (97)
20:56:34.389987 IP 10.241.0.46.36548 > 10.241.0.15.53: 35822+ A? dc.services.visualstudio.com. (46)
20:56:35.377166 IP 10.241.0.46.51760 > 10.241.0.15.53: 42682+ AAAA? data.policy.core.windows.net.svc.cluster.local. (64)
20:56:35.377214 IP 10.241.0.46.57761 > 10.241.0.15.53: 44273+ A? data.policy.core.windows.net.svc.cluster.local. (64)
20:56:36.930969 IP 10.241.0.49.58037 > 10.241.0.15.53: 53952+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:56:37.728990 IP 10.241.0.56.57096 > 10.241.0.15.53: 53424+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.cluster.local. (93)
20:56:37.729059 IP 10.241.0.56.54281 > 10.241.0.15.53: 62533+ A? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.cluster.local. (93)
20:56:39.393602 IP 10.241.0.46.55856 > 10.241.0.15.53: 57316+ A? dc.services.visualstudio.com. (46)
20:56:40.377423 IP 10.241.0.46.46667 > 10.241.0.15.53: 5577+ A? data.policy.core.windows.net.svc.cluster.local. (64)
20:56:42.431678 IP 10.241.0.49.58151 > 10.241.0.15.53: 36456+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:56:42.729171 IP 10.241.0.56.48232 > 10.241.0.15.53: 51854+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.cluster.local. (93)
20:56:45.378153 IP 10.241.0.46.40028 > 10.241.0.15.53: 61063+ A? data.policy.core.windows.net.cluster.local. (60)
20:56:47.432578 IP 10.241.0.49.58876 > 10.241.0.17.27017: Flags [S], seq 4138949722, win 64240, options [mss 1460,sackOK,TS val 1698408103 ecr 0,nop,wscale 7], length 0
20:56:47.732577 IP 10.241.0.56.49192 > 10.241.0.15.53: 49846+ A? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.rnfjgl5vvqwu1iaacmg53jyscf.bx.internal.cloudapp.net. (131)
20:56:47.732577 IP 10.241.0.56.45654 > 10.241.0.15.53: 13805+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.rnfjgl5vvqwu1iaacmg53jyscf.bx.internal.cloudapp.net. (131)
20:56:47.933292 IP 10.241.0.49.58882 > 10.241.0.17.27017: Flags [S], seq 4020941278, win 64240, options [mss 1460,sackOK,TS val 1698408604 ecr 0,nop,wscale 7], length 0
20:56:48.434065 IP 10.241.0.49.58884 > 10.241.0.17.27017: Flags [S], seq 1250164924, win 64240, options [mss 1460,sackOK,TS val 1698409105 ecr 0,nop,wscale 7], length 0
20:56:48.935271 IP 10.241.0.49.39560 > 10.241.0.15.53: 1072+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:56:50.382405 IP 10.241.0.46.59224 > 10.241.0.15.53: 16336+ A? data.policy.core.windows.net.cluster.local. (60)
20:56:52.733108 IP 10.241.0.56.38849 > 10.241.0.15.53: 60003+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.rnfjgl5vvqwu1iaacmg53jyscf.bx.internal.cloudapp.net. (131)
20:56:53.936302 IP 10.241.0.49.54258 > 10.241.0.17.27017: Flags [S], seq 3612814160, win 64240, options [mss 1460,sackOK,TS val 1698414607 ecr 0,nop,wscale 7], length 0
20:56:54.436866 IP 10.241.0.49.54260 > 10.241.0.17.27017: Flags [S], seq 1852220279, win 64240, options [mss 1460,sackOK,TS val 1698415108 ecr 0,nop,wscale 7], length 0
20:56:54.937368 IP 10.241.0.49.54270 > 10.241.0.17.27017: Flags [S], seq 1299576379, win 64240, options [mss 1460,sackOK,TS val 1698415608 ecr 0,nop,wscale 7], length 0
20:56:55.382974 IP 10.241.0.46.57593 > 10.241.0.15.53: 62465+ AAAA? data.policy.core.windows.net.rnfjgl5vvqwu1iaacmg53jyscf.bx.internal.cloudapp.net. (98)
20:56:55.438711 IP 10.241.0.49.42532 > 10.241.0.15.53: 23501+ A? user-db.sock-shop.svc.cluster.local. (53)
20:56:57.733303 IP 10.241.0.56.42653 > 10.241.0.15.53: 31709+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com. (79)
20:57:00.387591 IP 10.241.0.46.40143 > 10.241.0.15.53: 64139+ AAAA? data.policy.core.windows.net.rnfjgl5vvqwu1iaacmg53jyscf.bx.internal.cloudapp.net. (98)
20:57:00.439071 IP 10.241.0.49.48066 > 10.241.0.15.53: 45065+ A? user-db.sock-shop.svc.cluster.local. (53)
20:57:02.738265 IP 10.241.0.56.52469 > 10.241.0.15.53: 18224+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com. (79)
20:57:05.940035 IP 10.241.0.49.33438 > 10.241.0.17.27017: Flags [S], seq 700986103, win 64240, options [mss 1460,sackOK,TS val 1698426611 ecr 0,nop,wscale 7], length 0
20:57:06.440704 IP 10.241.0.49.33440 > 10.241.0.17.27017: Flags [S], seq 1595742128, win 64240, options [mss 1460,sackOK,TS val 1698427111 ecr 0,nop,wscale 7], length 0
20:57:06.941446 IP 10.241.0.49.33454 > 10.241.0.17.27017: Flags [S], seq 4071375737, win 64240, options [mss 1460,sackOK,TS val 1698427612 ecr 0,nop,wscale 7], length 0
20:57:07.894278 IP 10.241.0.56.41662 > 10.241.0.15.53: 30005+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.kube-system.svc.cluster.local. (109)
20:57:07.942498 IP 10.241.0.49.58839 > 10.241.0.15.53: 9365+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:57:07.942662 IP 10.241.0.49.52345 > 10.241.0.15.53: 42626+ A? user-db.sock-shop.svc.cluster.local. (53)
20:57:12.895545 IP 10.241.0.56.36025 > 10.241.0.15.53: 5919+ AAAA? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.kube-system.svc.cluster.local. (109)
20:57:12.942838 IP 10.241.0.49.40413 > 10.241.0.15.53: 10810+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
20:57:17.895835 IP 10.241.0.56.56054 > 10.241.0.15.53: 9387+ A? 3edc25e1-7d10-427d-8fcb-c8ce9fe20482.ods.opinsights.azure.com.svc.cluster.local. (97)
20:57:18.443272 IP 10.241.0.49.45314 > 10.241.0.15.53: 57101+ AAAA? user-db.sock-shop.svc.cluster.local. (53)
^C
79 packets captured
79 packets received by filter
0 packets dropped by kernelAs you may expect, you can see all cluster traffic passing by like communication with Azure and other used services.
Last great Cilium tool to mention here is debuginfo. It provides JSON output and again more troubleshooting information on Wireguard, interfaces, ports, peers and last received handshakes. We will use jq to parse the JSON output.
Try it out yourself!
kubectl exec ds/cilium -n kube-system -c cilium-agent -it -- cilium debuginfo --output json | jq .encryptionUpgrade to Isovalent Cilium Enterprise
If your team or organisation want to benefit from the mentioned features above you may want to upgrade your current “Azure CNI Powered by Cilium” towards “Isovalent Cilium Enterprise”, which includes support from Isovalent. This can be easily done through the Azure Marketplace.
Ensure you are in the Azure Portal, can open the Marketplace and just search for Cilium.
Now first click on the Isovalent Cilium Enterprise tile that is displayed and the product overview is shown as below.
Click on Plans + Pricing and select the first and currently only software plan.
Click Create. Now the actual deployment window is opened.
Ensure that you have selected the correct subscription and resource group. You can choose ‘No’, since you already have an AKS cluster you may want to upgrade. Just click Next and go to the Cluster Details.
Select your newly created cluster called ‘cilium-playground’. Notice the information message that tells us that you must already have Azure CNI Powered by Cilium deployed.
Now review the plan and take notice of additional costs. Let’s create the Isovalent Cilium Enterprise.
After approx 15 minutes the components are all installed. You can use the following command to validate the AKS extension status.
az k8s-extension show --name cilium --cluster-name cilium-playground --resource-group aks-cilium-playground --cluster-type managedClustersMore information about Isovalent Enterprise for Cilium from the Azure Marketplace can be found at isovalent.com/blog.
Conclusion
Cilium as Azure CNI for Azure IP Address Management (IPAM) is a reliable and scalable solution for Microsoft Azure customers. It accelerates both networking performance and provides better insights in cluster traffic.
Of course there are still some features that can be added to the data plane implementation which will make your life easier. Think about full support of applying Cilium network policies from L3/L4 to L7 for more granular API control. For this feature there is still an issue open at their GitHub repository.
Next to that, features like node-to-node transparent encryption and integrated Hubble would be very beneficial, both from Security and Observability standpoint. We can deploy Cilium OSS or leverage Isovalent Cilium Enterprise as AKS BYO CNI.
It would be great to have most of the Cilium features supported by Azure CNI Powered by Cilium, but I cannot tell you if these are part of the current roadmap.
Finally as quick reminder and reference when you are planning to start using the Azure CNI data plane in Production, always read the documentation, since there are still some limitations (also see Frequently Asked Questions) as mentioned above.