EKS Anywhere, jiving with Cilium OSS and BGP Load Balancer
47 min readJun 7, 2024
This article is part of the EKS Anywhere series EKS Anywhere, extending the Hybrid cloud momentum | by Ambar Hassani
Up until now, I had always used MetalLB as the LoadBalancer and NGINX Ingress Controller for my on premises EKS-Anywhere clusters. In that context, I wanted to swap out the CNI and Load-balancer to a full spectrum of capabilities provided by Cilium OSS.
This blog will focus on replacing the default Cilium installation on EKS Anywhere with Cilium OSS and deploy:
- BGP based load balancer for EKS Anywhere
- Replacement of Kube-proxy with eBPF
- Cilium Ingress with TLS
- Gateway API with TLS
- Hubble UI with Ingress and TLS
The below visual articulates the setup.
Note that EKS-Anywhere is by default bootstrapped with a minimal install of Cilium CNI. However, to achieve the below functionality, we need to switch over to Cilium OSS CNI.
Let’s begin!
Create the EKS Anywhere cluster. In the below logs, one can observe the creation of the cluster with Kubernetes version 1.29
One thing to observe is that we need to setup the cluster spec with the skip upgrade setting for default Cilium CNI
Your cluster.spec should look like this
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
annotations:
anywhere.eks.amazonaws.com/managed-by-cli: "true"
anywhere.eks.amazonaws.com/management-components-version: v0.19.5
name: eksanag23
namespace: default
spec:
clusterNetwork:
cniConfig:
cilium:
skipUpgrade: true
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12Next, we create the cluster…
Creating new bootstrap cluster
Provider specific pre-capi-install-setup on bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific post-setup
Installing EKS-A custom components on bootstrap cluster
Installing EKS-D components
Installing EKS-A custom components (CRD and controller)
Creating new workload cluster
Creating EKS-A namespace
Installing cluster-api providers on workload cluster
Installing EKS-A secrets on workload cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components on workload cluster
Installing EKS-D components
Installing EKS-A custom components (CRD and controller)
Applying cluster spec to workload cluster
Installing GitOps Toolkit on workload cluster
Enumerating objects: 620, done.
Counting objects: 100% (564/564), done.
Compressing objects: 100% (535/535), done.
Total 620 (delta 180), reused 0 (delta 0), pack-reused 56
Adding cluster configuration files to Git
Finalized commit and committed to local repository {"hash": "467c20947e16322097c94a29e20c6d632b300ea2"}
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
Installing helm chart on cluster {"chart": "eks-anywhere-packages", "version": "0.4.3-eks-a-65"}
ubuntu@eksa-admin-machine-new-1:~$
kubectl get nodes
NAME STATUS ROLES AGE VERSION
eksanag12-5jn9h Ready control-plane 32m v1.29.1-eks-61c0bbb
eksanag12-7mgxw Ready control-plane 33m v1.29.1-eks-61c0bbb
eksanag12-lvf7q Ready control-plane 30m v1.29.1-eks-61c0bbb
eksanag12-md-0-hqnhc-hf4xc Ready <none> 29m v1.29.1-eks-61c0bbb
eksanag12-md-0-hqnhc-npkhc Ready <none> 29m v1.29.1-eks-61c0bbbInstall Cilium CLI on the EKS-Anywhere admin machine
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}Install Hubble CLI on the EKS-Anywhere admin machine
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}With these CLI utilities in place, let’s observe the default installation of Cilium on EKS-Anywhere. It’s important for us to note that the default deployment of Cilium on EKS-Anywhere is a minimal install and does not provide the full range of Cilium OSS capabilities.
cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode)
\__/¯¯\__/ Hubble Relay: disabled
\__/ ClusterMesh: disabled
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
Containers: cilium Running: 5
cilium-operator Running: 2
Cluster Pods: 16/16 managed by Cilium
Helm chart version:
Image versions cilium public.ecr.aws/isovalent/cilium:v1.13.9-eksa.1: 5
cilium-operator public.ecr.aws/isovalent/operator-generic:v1.13.9-eksa.1: 2SinceSince EKS Anywhere comes with a default minimal install, we can check the cilium config of the default installation
cilium config view
agent-not-ready-taint-key node.cilium.io/agent-not-ready
arping-refresh-period 30s
auto-direct-node-routes false
bpf-lb-external-clusterip false
bpf-lb-map-max 65536
bpf-lb-sock false
bpf-map-dynamic-size-ratio 0.0025
bpf-policy-map-max 16384
bpf-root /sys/fs/bpf
cgroup-root /run/cilium/cgroupv2
cilium-endpoint-gc-interval 5m0s
cluster-id 0
cluster-name default
cni-chaining-mode portmap
cni-uninstall true
custom-cni-conf false
debug false
debug-verbose
disable-cnp-status-updates true
disable-endpoint-crd false
egress-gateway-healthcheck-timeout 2s
egress-gateway-reconciliation-trigger-interval 1s
enable-auto-protect-node-port-range true
enable-bgp-control-plane false
enable-bpf-clock-probe false
enable-cluster-aware-addressing false
enable-endpoint-health-checking true
enable-health-check-nodeport true
enable-health-checking true
enable-host-legacy-routing true
enable-hubble true
enable-inter-cluster-snat false
enable-ipv4 true
enable-ipv4-masquerade true
enable-ipv6 false
enable-ipv6-big-tcp false
enable-ipv6-masquerade true
enable-k8s-terminating-endpoint true
enable-l2-neigh-discovery true
enable-l7-proxy true
enable-local-redirect-policy false
enable-metrics true
enable-policy default
enable-remote-node-identity true
enable-sctp false
enable-svc-source-range-check true
enable-vtep false
enable-well-known-identities false
enable-xt-socket-fallback true
hubble-disable-tls false
hubble-listen-address :4244
hubble-socket-path /var/run/cilium/hubble.sock
hubble-tls-cert-file /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file /var/lib/cilium/tls/hubble/server.key
identity-allocation-mode crd
identity-gc-interval 15m0s
identity-heartbeat-timeout 30m0s
install-no-conntrack-iptables-rules false
ipam kubernetes
kube-proxy-replacement disabled
monitor-aggregation medium
monitor-aggregation-flags all
monitor-aggregation-interval 5s
node-port-bind-protection true
nodes-gc-interval 5m0s
operator-api-serve-addr 127.0.0.1:9234
operator-prometheus-serve-addr :9963
policy-cidr-match-mode
preallocate-bpf-maps false
procfs /host/proc
prometheus-serve-addr :9962
proxy-prometheus-port 9964
remove-cilium-node-taints true
set-cilium-is-up-condition true
sidecar-istio-proxy-image cilium/istio_proxy
skip-cnp-status-startup-clean false
synchronize-k8s-nodes true
tofqdns-dns-reject-response-code refused
tofqdns-enable-dns-compression true
tofqdns-endpoint-max-ip-per-hostname 50
tofqdns-idle-connection-grace-period 0s
tofqdns-max-deferred-connection-deletes 10000
tofqdns-min-ttl 3600
tofqdns-proxy-response-max-delay 100ms
tunnel geneve
unmanaged-pod-watcher-interval 15
vtep-cidr
vtep-endpoint
vtep-mac
vtep-maskAlso observe the CRDs created for the default Cilium installed. As one can see there is nothing much in there for BGP, etc.
kubectl get crd | grep cilium
ciliumendpoints.cilium.io 2024-06-12T18:47:58Z
ciliumidentities.cilium.io 2024-06-12T18:47:58Z
ciliumnodes.cilium.io 2024-06-12T18:47:58ZBefore we moving to full Cilium OSS., we will need to pause reconciliation of the cluster.
kubectl -n eksa-system annotate clusters.cluster.x-k8s.io $CLUSTER_NAME cluster.x-k8s.io/paused=true
cluster.cluster.x-k8s.io/eksanag23 annotatedTo deploy the full Cilium OSS, we need to overwrite the default Cilium deployment. This can be done via the helm, however with a caveat that there are previous resources of Service accounts, Secrets, ConfigMaps, RoleBindings that will need to be patched.
Patch the resources from the default Cilium installation
kubectl -n kube-system annotate serviceaccount cilium meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate serviceaccount cilium meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label serviceaccount cilium app.kubernetes.io/managed-by=Helm
kubectl -n kube-system annotate serviceaccount cilium-operator meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate serviceaccount cilium-operator meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label serviceaccount cilium-operator app.kubernetes.io/managed-by=Helm
kubectl -n kube-system annotate secret hubble-ca-secret meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate secret hubble-ca-secret meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label secret hubble-ca-secret app.kubernetes.io/managed-by=Helm
kubectl -n kube-system annotate secret hubble-server-certs meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate secret hubble-server-certs meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label secret hubble-server-certs app.kubernetes.io/managed-by=Helm
kubectl -n kube-system annotate configmap cilium-config meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate configmap cilium-config meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label configmap cilium-config app.kubernetes.io/managed-by=Helm
kubectl -n kube-system annotate secret cilium-ca meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate secret cilium-ca meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label secret cilium-ca app.kubernetes.io/managed-by=Helm
kubectl -n kube-system annotate service hubble-peer meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate service hubble-peer meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label service hubble-peer app.kubernetes.io/managed-by=Helm
kubectl -n kube-system annotate daemonset cilium meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate daemonset cilium meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label daemonset cilium app.kubernetes.io/managed-by=Helm
kubectl -n kube-system annotate deployment cilium-operator meta.helm.sh/release-name=cilium
kubectl -n kube-system annotate deployment cilium-operator meta.helm.sh/release-namespace=kube-system
kubectl -n kube-system label deployment cilium-operator app.kubernetes.io/managed-by=Helm
kubectl annotate clusterrole cilium meta.helm.sh/release-name=cilium
kubectl annotate clusterrole cilium meta.helm.sh/release-namespace=kube-system
kubectl label clusterrole cilium app.kubernetes.io/managed-by=Helm
kubectl annotate clusterrole cilium-operator meta.helm.sh/release-name=cilium
kubectl annotate clusterrole cilium-operator meta.helm.sh/release-namespace=kube-system
kubectl label clusterrole cilium-operator app.kubernetes.io/managed-by=Helm
kubectl annotate clusterrolebinding cilium meta.helm.sh/release-name=cilium
kubectl annotate clusterrolebinding cilium meta.helm.sh/release-namespace=kube-system
kubectl label clusterrolebinding cilium app.kubernetes.io/managed-by=Helm
kubectl annotate clusterrolebinding cilium-operator meta.helm.sh/release-name=cilium
kubectl annotate clusterrolebinding cilium-operator meta.helm.sh/release-namespace=kube-system
kubectl label clusterrolebinding cilium-operator app.kubernetes.io/managed-by=Helm
kubectl annotate role cilium-config-agent -n kube-system meta.helm.sh/release-name=cilium
kubectl annotate role cilium-config-agent -n kube-system meta.helm.sh/release-namespace=kube-system
kubectl label role cilium-config-agent -n kube-system app.kubernetes.io/managed-by=Helm
kubectl annotate rolebinding cilium-config-agent -n kube-system meta.helm.sh/release-name=cilium
kubectl annotate rolebinding cilium-config-agent -n kube-system meta.helm.sh/release-namespace=kube-system
kubectl label rolebinding cilium-config-agent -n kube-system app.kubernetes.io/managed-by=HelmNow we are ready to install Cilium with Helm. One can enable multiple capabilities of Cilium, however for now we are enabling the BGP control
helm install cilium cilium/cilium --version 1.15.5 \
--namespace kube-system \
--set eni.enabled=false \
--set ipam.mode=kubernetes \
--set tunnel=vxlan \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set bgpControlPlane.enabled=trueAs one can see and compare the default installation (seen earlier) is now replaced with Cilium OSS and per above helm settings, we have enabled the BGP control plane
cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode)
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 5
cilium-operator Running: 2
hubble-ui Running: 1
hubble-relay Running: 1
Cluster Pods: 18/18 managed by Cilium
Helm chart version:
Image versions cilium quay.io/cilium/cilium:v1.15.5@sha256:4ce1666a73815101ec9a4d360af6c5b7f1193ab00d89b7124f8505dee147ca40: 5
cilium-operator quay.io/cilium/operator-generic:v1.15.5@sha256:f5d3d19754074ca052be6aac5d1ffb1de1eb5f2d947222b5f10f6d97ad4383e8: 2
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1
hubble-ui quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
hubble-relay quay.io/cilium/hubble-relay:v1.15.5@sha256:1d24b24e3477ccf9b5ad081827db635419c136a2bd84a3e60f37b26a38dd0781: 1We need to now resume the cluster reconciliation
kubectl -n eksa-system annotate clusters.cluster.x-k8s.io $CLUSTER_NAME cluster.x-k8s.io/paused-Verify BGP control plane
cilium config view | grep -i bgp
bgp-secrets-namespace kube-system
enable-bgp-control-plane trueCreate upstream BGP connection and IPAM pool for LoadBalancer services. Note that IPAM pool should be different than the subnet used for the actual machines. There are plenty of configuration options to target specific services for the Load Balancer IPAM pool and other settings. For now, we will work with the below simplicity. In the configurations we are also providing references to perform peering with the BGP BIRD ubuntu host.
cat <<EOF | kubectl apply -f -
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
name: ippool
spec:
cidrs:
- cidr: "172.26.1.0/24" #This range is outside of the host network.
disabled: false
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
name: 01-bgp-peering-policy
spec:
nodeSelector:
matchLabels:
group: md-0
virtualRouters:
- localASN: 64512
exportPodCIDR: true
neighbors:
- peerAddress: '10.204.111.49/32' #establish iBGP with upstream Bird router
peerASN: 64513
serviceSelector:
matchExpressions:
- {key: somekey, operator: NotIn, values: ['never-used-value']}
EOFLet’s observe the BGP status on Cilium. The upstream peer (ubuntu BIRD machine) is not yet setup for BGP peering and hence the peer status is IDLE.
cilium bgp peers
Node Local AS Peer AS Peer Address Session State Uptime Family Received Advertised
eksanag12-md-0-hqnhc-hf4xc 64512 64513 10.204.111.49 idle 0s ipv4/unicast 0 0
ipv6/unicast 0 0
eksanag12-md-0-hqnhc-npkhc 64512 64513 10.204.111.49 idle 0s ipv4/unicast 0 0
ipv6/unicast 0 0Next, we will need the worker node IP addresses as they will act as the remote peers for the BGP daemon running via BIRD on the ubuntu machine
kubectl get nodes --selector='!node-role.kubernetes.io/control-plane' \
-o template \
--template='{{range.items}}{{range.status.addresses}}{{if eq .type "InternalIP"}}{{.address}}{{end}}{{end}} {{end}}'Setting up Upstream BGP BIRD on Ubuntu 20.04
Install and Configure BIRD on a ubuntu 20.04 machine to emulate an upstream L3 router. Adjust the configuration for the remote peers, etc.
echo "net.ipv4.conf.all.forwarding=1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv4.conf.default.forwarding=1" | sudo tee -a /etc/sysctl.conf
sudo add-apt-repository ppa:cz.nic-labs/bird
sudo apt-get install bird
sudo cat <<EOF > /etc/bird/bird.conf
# This is a minimal configuration file, which allows the bird daemon to start
# but will not cause anything else to happen.
#
# Please refer to the documentation in the bird-doc package or BIRD User's
# Guide on http://bird.network.cz/ for more information on configuring BIRD and
# adding routing protocols.
# Change this into your BIRD router ID. It's a world-wide unique identification
# of your router, usually one of router's IPv4 addresses.
router id 10.204.111.49;
# The Kernel protocol is not a real routing protocol. Instead of communicating
# with other routers in the network, it performs synchronization of BIRD's
# routing tables with the OS kernel.
protocol kernel {
# learn; # Learn all alien routes from the kernel
persist; # Don't remove routes on bird shutdown
scan time 20; # Scan kernel routing table every 20 seconds
# import none; # Default is import all
export all; # Default is export none
# kernel table 5; # Kernel table to synchronize with (default: main)
}
# Advertise a dummy local route via bgp
protocol static static_bgp {
route 192.168.2.0/24 via 10.204.111.49;
}
# Create eBGP session to EKS-A Worker node-1
protocol bgp link1 {
description "BIRD BGP CONFIG";
local as 64513;
neighbor xx.xx.xx.xx as 64512;
graceful restart;
import all;
export where proto = "static_bgp";
}
# Create eBGP session to EKS-A worker node-2
protocol bgp link2 {
description "BIRD BGP CONFIG";
local as 64513;
neighbor xx.xx.xx.xx as 64512;
graceful restart;
import all;
export where proto = "static_bgp";
}
# The Device protocol is not a real routing protocol. It doesn't generate any
# routes and it only serves as a module for getting information about network
# interfaces from the kernel.
protocol device {
scan time 60;
}
log syslog { debug, trace, info, remote, warning, error, auth, fatal, bug };
log stderr all;
EOFRestart and Enable BIRD service on the ubuntu host
sudo systemctl restart bird
sudo systemctl enable birdAccess bird CLI and observe BGP related information.
sudo birdc
BIRD 1.6.8 ready.
bird> show protocols
name proto table state since info
kernel1 Kernel master up 06:14:15
static_bgp Static master up 06:14:15
link1 BGP master up 06:14:16 Established
link2 BGP master up 06:14:22 Established
device1 Device master up 06:14:15Observe BGP information from Cilium end
cilium bgp peers
Node Local AS Peer AS Peer Address Session State Uptime Family Received Advertised
eksanag12-md-0-hqnhc-hf4xc 64512 64513 10.204.111.49 established 58s ipv4/unicast 1 1
ipv6/unicast 0 0
eksanag12-md-0-hqnhc-npkhc 64512 64513 10.204.111.49 established 1m4s ipv4/unicast 1 1
ipv6/unicast 0 0At this point our BGP peering is in an established state, and we are good to go.
Deploy a sample NGINX pod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: simple-pod
labels:
app: simple-pod
spec:
containers:
- name: my-app-container
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: simple-pod # Make sure this matches the label of the Pod
ports:
- protocol: TCP
port: 8080
targetPort: 80
type: LoadBalancer
EOFObserve the service for the sample pod and note that we have received 172.26.1.1 as an External IP from LB IPAM pool
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 42m
my-service LoadBalancer 10.96.84.205 172.26.1.1 8080:31733/TCP 5sAt the same token, let’s observe the route information received at BIRD BGP machine
Note that Cilium will not advertise the entire LB IPAM pool as a single prefix. Instead, it granularly advertises each of the services External IPs assigned as /32 prefixes.
bird> show route
192.168.3.0/24 via 10.204.111.9 on ens160 [link2 05:09:34] * (100) [AS64512i]
192.168.4.0/24 via 10.204.110.28 on ens160 [link1 05:09:23] * (100) [AS64512i]
172.26.1.1/32 via 10.204.110.28 on ens160 [link1 05:21:53] * (100) [AS64512i]
via 10.204.111.9 on ens160 [link2 05:21:55] (100) [AS64512i]
172.16.2.0/24 via 10.204.111.49 on ens160 [static_bgp 05:09:21] ! (200)Let’s curl up to validate application reachability to the NGINX pod from the BIRD BGP machine.
curl 172.26.1.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>The case for kube-proxy replacement
Now let’s replace the kube-proxy with Cilium and revalidate this scenario. kube-proxy is generally a default component of Kubernetes that handles routing traffic for services within the cluster. There are two backends available for Layer 3/4 load balancing in upstream kube-proxy - iptables and IPVS. Cilium’s kube-proxy replacement offers advanced configuration modes to cater to your specific needs. Features like client source IP preservation ensure that your service connections remain intact, while Maglev Consistent Hashing enhances load balancing and resiliency. With support for Direct Server Return (DSR) and Hybrid DSR/SNAT modes, you can optimize traffic routing and improve performance.
To replace the kube-proxy with Cilium, we need to remove the daemonset and configmap related to kube-proxy
Let’s delete our application and the BGP peering to a clean slate.
kubectl delete pod simple-pod
kubectl delete service my-service
kubectl delete CiliumBGPPeeringPolicy 01-bgp-peering-policy
kubectl delete CiliumLoadBalancerIPPool ippoolVerify Cilium BGP. All gone for good!
cilium bgp peers
Node Local AS Peer AS Peer Address Session State Uptime Family Received AdvertisedDelete kube-proxy references such that we can then use the Cilium attributes.
kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxyUpgrade Cilium setting the kube proxy replacement flag. Note that we will need to specify the k8s services host and port to ensure Cilium can act as a replacement for kube-proxy
helm upgrade cilium cilium/cilium --version 1.15.5 \
--namespace kube-system \
--set eni.enabled=false \
--set ipam.mode=kubernetes \
--set tunnel=vxlan \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set bgpControlPlane.enabled=true \
--set kubeProxyReplacement=true \
--set k8sServiceHost=10.204.111.57 \
--set k8sServicePort=6443 \
--set externalIPs.enabled=trueVerify Cilium status
cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode)
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Containers: hubble-ui Running: 1
cilium Running: 5
cilium-operator Running: 2
hubble-relay Running: 1
Cluster Pods: 18/18 managed by Cilium
Helm chart version:
Image versions cilium quay.io/cilium/cilium:v1.15.5@sha256:4ce1666a73815101ec9a4d360af6c5b7f1193ab00d89b7124f8505dee147ca40: 5
cilium-operator quay.io/cilium/operator-generic:v1.15.5@sha256:f5d3d19754074ca052be6aac5d1ffb1de1eb5f2d947222b5f10f6d97ad4383e8: 2
hubble-relay quay.io/cilium/hubble-relay:v1.15.5@sha256:1d24b24e3477ccf9b5ad081827db635419c136a2bd84a3e60f37b26a38dd0781: 1
hubble-ui quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1Redeploy the BGP peering policy and LB IPAM pool
cat <<EOF | kubectl apply -f -
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
name: ippool
spec:
cidrs:
- cidr: "172.26.1.0/24" #This range is outside of the host network.
disabled: false
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
name: 01-bgp-peering-policy
spec:
nodeSelector:
matchLabels:
group: md-0
virtualRouters:
- localASN: 64512
exportPodCIDR: true
neighbors:
- peerAddress: '10.204.111.49/32' #establish iBGP with upstream Bird router
peerASN: 64513
serviceSelector:
matchExpressions:
- {key: somekey, operator: NotIn, values: ['never-used-value']}
EOFVerify BGP on Cilium end
cilium bgp peers
Node Local AS Peer AS Peer Address Session State Uptime Family Received Advertised
eksanag12-md-0-hqnhc-hf4xc 64512 64513 10.204.111.49 established 27s ipv4/unicast 1 1
ipv6/unicast 0 0
eksanag12-md-0-hqnhc-npkhc 64512 64513 10.204.111.49 established 29s ipv4/unicast 1 1
ipv6/unicast 0 0Redeploy the test pod and service
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: simple-pod
labels:
app: simple-pod
spec:
containers:
- name: my-app-container
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: simple-pod # Make sure this matches the label of the Pod
ports:
- protocol: TCP
port: 8080
targetPort: 80
type: LoadBalancer
EOFVerify if the sample service received an External IP
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 70m
my-service LoadBalancer 10.107.135.37 172.26.1.1 8080:30159/TCP 6sVerify incoming prefix on the BIRD BGP ubuntu machine
BIRD 1.6.8 ready.
bird> show protocol
name proto table state since info
kernel1 Kernel master up 06:14:15
static_bgp Static master up 06:14:15
link1 BGP master up 06:41:40 Established
link2 BGP master up 06:41:42 Established
device1 Device master up 06:14:15
bird>
bird> show route
192.168.3.0/24 via 10.204.110.31 on ens160 [link2 06:41:42] * (100) [AS64512i]
192.168.4.0/24 via 10.204.110.30 on ens160 [link1 06:41:40] * (100) [AS64512i]
172.26.1.1/32 via 10.204.110.30 on ens160 [link1 06:43:02] * (100) [AS64512i]
via 10.204.110.31 on ens160 [link2 06:43:02] (100) [AS64512i]
172.16.2.0/24 via 10.204.111.49 on ens160 [static_bgp 06:14:15] ! (200)
bird>
bird>Verify application accessibility from the BIRD BGP Ubuntu machine
curl 172.26.1.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>Let’s move to building an Ingress Controller
Paint the wall clean
kubectl delete pod simple-pod
kubectl delete service my-service
kubectl delete CiliumBGPPeeringPolicy 01-bgp-peering-policy
kubectl delete CiliumLoadBalancerIPPool ippoolpod "simple-pod" deleted
service "my-service" deleted
ciliumbgppeeringpolicy.cilium.io "01-bgp-peering-policy" deleted
ciliumloadbalancerippool.cilium.io "ippool" deletedRedeploy Cilium with Ingress controller and with load balancer mode as dedicated. In addition to this, we will enable a recommended setting for ebpf based masquerading along with creating a dedicated daemonset for Envoy based L7 proxy.
Very Important: I had issues with deploying Envoy as a dedicated daemonset via the below configuration. Each time I deployed the configuration, everything came up smooth except cilium-envoy daemonset pods, which went into pending state forever. Upon trial and error, figured out that the cilium daemonset has to be deleted via the below command and then apply the helm configuration.
kubectl delete daemonset cilium --namespace kube-systemhelm upgrade cilium cilium/cilium --version 1.15.5 \
--namespace kube-system \
--set eni.enabled=false \
--set ipam.mode=kubernetes \
--set bpf.masquerade=true \
--set tunnel=vxlan \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set bgpControlPlane.enabled=true \
--set kubeProxyReplacement=true \
--set k8sServiceHost=10.204.111.57 \
--set k8sServicePort=6443 \
--set externalIPs.enabled=true \
--set bgp.announce.loadbalancerIP=true \
--set ingressController.enabled=true \
--set ingressController.loadbalancerMode=dedicated \
--set envoy.enabled=true \
--set loadBalancer.l7.backend=envoy \
--set debug.enabled=true \
--set debug.verbose=flowRelease "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Sun Jun 9 03:54:07 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 7
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.
Your release version is 1.15.5.
For any further help, visit https://docs.cilium.io/en/v1.15/gettinghelpLet’s view the cilium config
agent-not-ready-taint-key node.cilium.io/agent-not-ready
arping-refresh-period 30s
auto-direct-node-routes false
bgp-secrets-namespace kube-system
bpf-lb-acceleration disabled
bpf-lb-external-clusterip false
bpf-lb-map-max 65536
bpf-lb-sock false
bpf-map-dynamic-size-ratio 0.0025
bpf-policy-map-max 16384
bpf-root /sys/fs/bpf
cgroup-root /run/cilium/cgroupv2
cilium-endpoint-gc-interval 5m0s
cluster-id 0
cluster-name default
cni-exclusive true
cni-log-file /var/run/cilium/cilium-cni.log
custom-cni-conf false
debug true
debug-verbose flow
dnsproxy-enable-transparent-mode true
egress-gateway-reconciliation-trigger-interval 1s
enable-auto-protect-node-port-range true
enable-bgp-control-plane true
enable-bpf-clock-probe false
enable-bpf-masquerade true
enable-endpoint-health-checking true
enable-envoy-config true
enable-health-check-loadbalancer-ip false
enable-health-check-nodeport true
enable-health-checking true
enable-hubble true
enable-hubble-open-metrics false
enable-ingress-controller true
enable-ingress-proxy-protocol false
enable-ingress-secrets-sync true
enable-ipv4 true
enable-ipv4-big-tcp false
enable-ipv4-masquerade true
enable-ipv6 false
enable-ipv6-big-tcp false
enable-ipv6-masquerade true
enable-k8s-networkpolicy true
enable-k8s-terminating-endpoint true
enable-l2-neigh-discovery true
enable-l7-proxy true
enable-local-redirect-policy false
enable-masquerade-to-route-source false
enable-metrics true
enable-policy default
enable-remote-node-identity true
enable-sctp false
enable-svc-source-range-check true
enable-vtep false
enable-well-known-identities false
enable-xt-socket-fallback true
enforce-ingress-https true
external-envoy-proxy true
hubble-disable-tls false
hubble-export-file-max-backups 5
hubble-export-file-max-size-mb 10
hubble-listen-address :4244
hubble-metrics dns drop tcp flow icmp http
hubble-metrics-server :9965
hubble-socket-path /var/run/cilium/hubble.sock
hubble-tls-cert-file /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file /var/lib/cilium/tls/hubble/server.key
identity-allocation-mode crd
identity-gc-interval 15m0s
identity-heartbeat-timeout 30m0s
ingress-default-lb-mode dedicated
ingress-lb-annotation-prefixes service.beta.kubernetes.io service.kubernetes.io cloud.google.com
ingress-secrets-namespace cilium-secrets
ingress-shared-lb-service-name cilium-ingress
install-no-conntrack-iptables-rules false
ipam kubernetes
ipam-cilium-node-update-rate 15s
k8s-client-burst 20
k8s-client-qps 10
kube-proxy-replacement true
kube-proxy-replacement-healthz-bind-address
loadbalancer-l7 envoy
loadbalancer-l7-algorithm round_robin
loadbalancer-l7-ports
max-connected-clusters 255
mesh-auth-enabled true
mesh-auth-gc-interval 5m0s
mesh-auth-queue-size 1024
mesh-auth-rotated-identities-queue-size 1024
monitor-aggregation medium
monitor-aggregation-flags all
monitor-aggregation-interval 5s
node-port-bind-protection true
nodes-gc-interval 5m0s
operator-api-serve-addr 127.0.0.1:9234
operator-prometheus-serve-addr :9963
policy-cidr-match-mode
preallocate-bpf-maps false
procfs /host/proc
proxy-connect-timeout 2
proxy-idle-timeout-seconds 60
proxy-max-connection-duration-seconds 0
proxy-max-requests-per-connection 0
proxy-xff-num-trusted-hops-egress 0
proxy-xff-num-trusted-hops-ingress 0
remove-cilium-node-taints true
routing-mode tunnel
service-no-backend-response reject
set-cilium-is-up-condition true
set-cilium-node-taints true
sidecar-istio-proxy-image cilium/istio_proxy
skip-cnp-status-startup-clean false
synchronize-k8s-nodes true
tofqdns-dns-reject-response-code refused
tofqdns-enable-dns-compression true
tofqdns-endpoint-max-ip-per-hostname 50
tofqdns-idle-connection-grace-period 0s
tofqdns-max-deferred-connection-deletes 10000
tofqdns-proxy-response-max-delay 100ms
tunnel-protocol vxlan
unmanaged-pod-watcher-interval 15
vtep-cidr
vtep-endpoint
vtep-mac
vtep-mask
write-cni-conf-when-ready /host/etc/cni/net.d/05-cilium.conflistRestart the operator and Cilium daemonset
kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/ciliumdeployment.apps/cilium-operator restarted
daemonset.apps/cilium restartedVerify Cilium status
cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode)
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
Containers: cilium Running: 5
cilium-operator Running: 2
hubble-relay Running: 1
hubble-ui Running: 1
Cluster Pods: 19/19 managed by Cilium
Helm chart version:
Image versions hubble-relay quay.io/cilium/hubble-relay:v1.15.5@sha256:1d24b24e3477ccf9b5ad081827db635419c136a2bd84a3e60f37b26a38dd0781: 1
hubble-ui quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1
cilium quay.io/cilium/cilium:v1.15.5@sha256:4ce1666a73815101ec9a4d360af6c5b7f1193ab00d89b7124f8505dee147ca40: 5
cilium-operator quay.io/cilium/operator-generic:v1.15.5@sha256:f5d3d19754074ca052be6aac5d1ffb1de1eb5f2d947222b5f10f6d97ad4383e8: 2Also observe that we are running Envoy as a separate daemonset providing additional benefits.
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
cilium-45hl6 1/1 Running 0 11m
cilium-cssvt 1/1 Running 0 11m
cilium-envoy-4zlx7 1/1 Running 0 11m
cilium-envoy-jcrpg 1/1 Running 0 11m
cilium-envoy-pcnt9 1/1 Running 0 11m
cilium-envoy-rqlsg 1/1 Running 0 11m
cilium-envoy-xtgpd 1/1 Running 0 11m
cilium-gffkl 1/1 Running 0 11m
cilium-jm449 1/1 Running 0 10m
cilium-operator-748f8f9fb5-j7hcb 1/1 Running 0 11m
cilium-operator-748f8f9fb5-pgzxt 1/1 Running 0 11m
cilium-t64zf 1/1 Running 0 11m
coredns-7b5898fc49-22zrr 1/1 Running 0 13h
coredns-7b5898fc49-w5zgm 1/1 Running 0 13h
etcd-eksanag15-5667n 1/1 Running 0 13h
etcd-eksanag15-5xfrb 1/1 Running 0 13h
etcd-eksanag15-6g97k 1/1 Running 0 13h
hubble-relay-66db7678d6-qbvg6 1/1 Running 0 11m
hubble-ui-6548d56557-6zh5x 2/2 Running 0 11m
kube-apiserver-eksanag15-5667n 1/1 Running 0 13h
kube-apiserver-eksanag15-5xfrb 1/1 Running 0 13h
kube-apiserver-eksanag15-6g97k 1/1 Running 0 13h
kube-controller-manager-eksanag15-5667n 1/1 Running 0 13h
kube-controller-manager-eksanag15-5xfrb 1/1 Running 0 13h
kube-controller-manager-eksanag15-6g97k 1/1 Running 0 13h
kube-scheduler-eksanag15-5667n 1/1 Running 0 13h
kube-scheduler-eksanag15-5xfrb 1/1 Running 0 13h
kube-scheduler-eksanag15-6g97k 1/1 Running 0 13h
kube-vip-eksanag15-5667n 1/1 Running 0 13h
kube-vip-eksanag15-5xfrb 1/1 Running 0 13h
kube-vip-eksanag15-6g97k 1/1 Running 1 (13h ago) 13h
vsphere-cloud-controller-manager-hq6tv 1/1 Running 0 13h
vsphere-cloud-controller-manager-rv9q4 1/1 Running 1 (13h ago) 13h
vsphere-cloud-controller-manager-s4k5l 1/1 Running 1 (13h ago) 13h
vsphere-cloud-controller-manager-wr5k5 1/1 Running 1 (13h ago) 13h
vsphere-cloud-controller-manager-zcbdv 1/1 Running 1 (13h ago) 13hCreate the BGP policy and LB IPAM pool
cat <<EOF | kubectl apply -f -
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
name: ippool
spec:
cidrs:
- cidr: "172.26.1.0/24" #This range is outside of the host network.
disabled: false
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
name: 01-bgp-peering-policy
spec:
nodeSelector:
matchLabels:
group: md-0
virtualRouters:
- localASN: 64512
exportPodCIDR: true
neighbors:
- peerAddress: '10.204.111.49/32' #establish iBGP with upstream Bird router
peerASN: 64513
serviceSelector:
matchExpressions:
- {key: somekey, operator: NotIn, values: ['never-used-value']}
EOFValidate BGP sessions
cilium bgp peers
Node Local AS Peer AS Peer Address Session State Uptime Family Received Advertised
eksanag15-md-0-wsxnd-f9hdk 64512 64513 10.204.111.49 established 5m58s ipv4/unicast 1 3
ipv6/unicast 0 0
eksanag15-md-0-wsxnd-xj97h 64512 64513 10.204.111.49 established 5m48s ipv4/unicast 1 3
ipv6/unicast 0 0Create the NGINIX sample pod/service
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: simple-pod
labels:
app: simple-pod
spec:
containers:
- name: my-app-container
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: simple-pod # Make sure this matches the label of the Pod
ports:
- protocol: TCP
port: 8080
targetPort: 80
EOFCreate a TLS secret for our sample NGINX pod/service. I already have the ca certificate and key. You may wish to obtain it per your convenience.
kubectl create secret tls nginx-test-cert \
--key tls.key \
--cert tls.crtCreate the Ingress resource with the ingressClassName as cilium and specifying the backend along with TLS host and secret name
cat <<EOF | kubectl create -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tls-ingress
namespace: default
spec:
ingressClassName: cilium
rules:
- host: nginx-test.poc.thecloudgarage.com
http:
paths:
- backend:
service:
name: my-service
port:
number: 8080
path: /
pathType: Prefix
tls:
- hosts:
- nginx-test.poc.thecloudgarage.com
secretName: nginx-test-cert
EOFValidate if a dedicated load balancer service has been created for our Ingress resource. As one can observe, there are two services, one by the name cilium-ingress which is the default shared loadbalancer service for Ingresses, and the other one which is cilium-ingress-tls-ingress. This is a dedicated loadbalancer service created for our Ingress resource named as tls-ingress.
kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-webhook-service ClusterIP 10.99.116.161 <none> 443/TCP 13h
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-webhook-service ClusterIP 10.107.89.240 <none> 443/TCP 13h
capi-system capi-webhook-service ClusterIP 10.109.185.214 <none> 443/TCP 13h
capv-system capv-webhook-service ClusterIP 10.99.205.167 <none> 443/TCP 13h
cert-manager cert-manager ClusterIP 10.104.55.142 <none> 9402/TCP 13h
cert-manager cert-manager-webhook ClusterIP 10.105.176.229 <none> 443/TCP 13h
cilium-test cilium-ingress-other-node NodePort 10.96.76.195 <none> 80:31906/TCP,443:32032/TCP 3h55m
cilium-test cilium-ingress-same-node NodePort 10.96.4.233 <none> 80:30331/TCP,443:31265/TCP 3h55m
cilium-test echo-other-node NodePort 10.110.154.84 <none> 8080:31358/TCP 3h55m
cilium-test echo-same-node NodePort 10.105.86.35 <none> 8080:31228/TCP 3h55m
default cilium-ingress-tls-ingress LoadBalancer 10.109.70.84 172.26.1.2 80:31232/TCP,443:32300/TCP 5m36s
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 13h
default my-service ClusterIP 10.110.54.84 <none> 8080/TCP 5m56s
eksa-system eksa-webhook-service ClusterIP 10.100.92.78 <none> 443/TCP 13h
etcdadm-bootstrap-provider-system etcdadm-bootstrap-provider-webhook-service ClusterIP 10.106.23.14 <none> 443/TCP 13h
etcdadm-controller-system etcdadm-controller-webhook-service ClusterIP 10.96.105.0 <none> 443/TCP 13h
flux-system notification-controller ClusterIP 10.110.115.175 <none> 80/TCP 13h
flux-system source-controller ClusterIP 10.105.176.255 <none> 80/TCP 13h
flux-system webhook-receiver ClusterIP 10.108.157.179 <none> 80/TCP 13h
kube-system cilium-agent ClusterIP None <none> 9964/TCP 13h
kube-system cilium-envoy ClusterIP None <none> 9964/TCP 8m34s
kube-system cilium-ingress LoadBalancer 10.96.249.182 172.26.1.1 80:31393/TCP,443:32206/TCP 8m34s
kube-system cloud-controller-manager NodePort 10.109.52.141 <none> 443:32076/TCP 13h
kube-system hubble-metrics ClusterIP None <none> 9965/TCP 8m34s
kube-system hubble-peer ClusterIP 10.109.64.78 <none> 443/TCP 7h48m
kube-system hubble-relay ClusterIP 10.103.38.11 <none> 80/TCP 8m34s
kube-system hubble-ui ClusterIP 10.109.54.148 <none> 80/TCP 8m34s
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 13hValidate if our Ingress resource is built and that it received an IP Address associated with the Ingress controller
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
tls-ingress cilium nginx-test.poc.thecloudgarage.com 172.26.1.2 80, 443 7m40sAdditionally a CRD resource for CiliumEnvoyConfig has been instantiated for our this Ingress
kubectl get ciliumenvoyconfig
NAME AGE
cilium-ingress-default-tls-ingress 10m
kubectl describe ciliumenvoyconfig cilium-ingress-default-tls-ingress
Name: cilium-ingress-default-tls-ingress
Namespace: default
Labels: cilium.io/use-original-source-address=false
Annotations: <none>
API Version: cilium.io/v2
Kind: CiliumEnvoyConfig
Metadata:
Creation Timestamp: 2024-06-10T16:46:29Z
Generation: 1
Owner References:
API Version: networking.k8s.io/v1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: tls-ingress
UID: 5c4de0e3-ceda-4884-bde1-16d393e63557
Resource Version: 386548
UID: 675ae7ad-7497-400a-82ac-ca0e37d3c384
Spec:
Backend Services:
Name: my-service
Namespace: default
Number:
8080
Resources:
@type: type.googleapis.com/envoy.config.listener.v3.Listener
Filter Chains:
Filter Chain Match:
Transport Protocol: raw_buffer
Filters:
Name: envoy.filters.network.http_connection_manager
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
Common Http Protocol Options:
Max Stream Duration: 0s
Http Filters:
Name: envoy.filters.http.grpc_web
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
Name: envoy.filters.http.grpc_stats
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig
Emit Filter State: true
Enable Upstream Stats: true
Name: envoy.filters.http.router
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
Rds:
Route Config Name: listener-insecure
Stat Prefix: listener-insecure
Upgrade Configs:
Upgrade Type: websocket
Use Remote Address: true
Filter Chain Match:
Server Names:
nginx-test.poc.thecloudgarage.com
Transport Protocol: tls
Filters:
Name: envoy.filters.network.http_connection_manager
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
Common Http Protocol Options:
Max Stream Duration: 0s
Http Filters:
Name: envoy.filters.http.grpc_web
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
Name: envoy.filters.http.grpc_stats
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig
Emit Filter State: true
Enable Upstream Stats: true
Name: envoy.filters.http.router
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
Rds:
Route Config Name: listener-secure
Stat Prefix: listener-secure
Upgrade Configs:
Upgrade Type: websocket
Use Remote Address: true
Transport Socket:
Name: envoy.transport_sockets.tls
Typed Config:
@type: type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
Common Tls Context:
Tls Certificate Sds Secret Configs:
Name: cilium-secrets/default-nginx-test-cert
Listener Filters:
Name: envoy.filters.listener.tls_inspector
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
Name: listener
Socket Options:
Description: Enable TCP keep-alive (default to enabled)
Int Value: 1
Level: 1
Name: 9
Description: TCP keep-alive idle time (in seconds) (defaults to 10s)
Int Value: 10
Level: 6
Name: 4
Description: TCP keep-alive probe intervals (in seconds) (defaults to 5s)
Int Value: 5
Level: 6
Name: 5
Description: TCP keep-alive probe max failures.
Int Value: 10
Level: 6
Name: 6
@type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration
Name: listener-insecure
Virtual Hosts:
Domains:
nginx-test.poc.thecloudgarage.com
nginx-test.poc.thecloudgarage.com:*
Name: nginx-test.poc.thecloudgarage.com
Routes:
Match:
Prefix: /
Redirect:
Https Redirect: true
@type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration
Name: listener-secure
Virtual Hosts:
Domains:
nginx-test.poc.thecloudgarage.com
nginx-test.poc.thecloudgarage.com:*
Name: nginx-test.poc.thecloudgarage.com
Routes:
Match:
Prefix: /
Route:
Cluster: default:my-service:8080
@type: type.googleapis.com/envoy.config.cluster.v3.Cluster
Connect Timeout: 5s
Eds Cluster Config:
Service Name: default/my-service:8080
Name: default:my-service:8080
Outlier Detection:
Split External Local Origin Errors: true
Type: EDS
Typed Extension Protocol Options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
@type: type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
Common Http Protocol Options:
Idle Timeout: 60s
Use Downstream Protocol Config:
http2ProtocolOptions:
Services:
Listener:
Name: cilium-ingress-tls-ingress
Namespace: default
Events: <none>And then we head over to the BGP BIRD Ubuntu machine to do a quick validation
curl -v https://nginx-test.poc.thecloudgarage.com
* Trying 172.26.1.2:443...
* Connected to nginx-test.poc.thecloudgarage.com (172.26.1.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=thecloudgarage.com
* start date: May 13 05:30:07 2024 GMT
* expire date: Aug 11 05:30:06 2024 GMT
* subjectAltName: host "nginx-test.poc.thecloudgarage.com" matched cert's "*.poc.thecloudgarage.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: nginx-test.poc.thecloudgarage.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: envoy
< date: Mon, 10 Jun 2024 16:58:25 GMT
< content-type: text/html
< content-length: 615
< last-modified: Tue, 28 May 2024 13:22:30 GMT
< etag: "6655da96-267"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 1
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host nginx-test.poc.thecloudgarage.com left intactLet’s hurl in the Gateway API support.
While we are traversing this path, might as well implement the Gateway API support for Cilium. We will need to first install the pre-requisite CRDs
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io createdWe will continue with the Ingress configurations so as to have a co-existence of Ingress and Gateway API implementation. In that case, we are not recreating the LB IPAM Pool or the BGP peering policy as those already exist from the previous scenario.
Redeploy Cilium with Gateway API support
helm upgrade cilium cilium/cilium --version 1.15.5 \
--namespace kube-system \
--set eni.enabled=false \
--set ipam.mode=kubernetes \
--set bpf.masquerade=true \
--set tunnel=vxlan \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set bgpControlPlane.enabled=true \
--set kubeProxyReplacement=true \
--set k8sServiceHost=10.204.111.57 \
--set k8sServicePort=6443 \
--set externalIPs.enabled=true \
--set bgp.announce.loadbalancerIP=true \
--set ingressController.enabled=true \
--set ingressController.loadbalancerMode=dedicated \
--set gatewayAPI.enabled=true \
--set envoy.enabled=true \
--set loadBalancer.l7.backend=envoy \
--set debug.enabled=true \
--set debug.verbose=flowRestart Cilium operator and daemonset
kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/ciliumVerify Cilium
cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium-envoy Desired: 5, Ready: 5/5, Available: 5/5
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
Containers: cilium-operator Running: 2
cilium Running: 5
hubble-ui Running: 1
hubble-relay Running: 1
cilium-envoy Running: 5
Cluster Pods: 25/25 managed by Cilium
Helm chart version:
Image versions cilium quay.io/cilium/cilium:v1.15.5@sha256:4ce1666a73815101ec9a4d360af6c5b7f1193ab00d89b7124f8505dee147ca40: 5
hubble-ui quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1
hubble-relay quay.io/cilium/hubble-relay:v1.15.5@sha256:1d24b24e3477ccf9b5ad081827db635419c136a2bd84a3e60f37b26a38dd0781: 1
cilium-envoy quay.io/cilium/cilium-envoy:v1.28.3-31ec52ec5f2e4d28a8e19a0bfb872fa48cf7a515@sha256:bc8dcc3bc008e3a5aab98edb73a0985e6ef9469bda49d5bb3004c001c995c380: 5
cilium-operator quay.io/cilium/operator-generic:v1.15.5@sha256:f5d3d19754074ca052be6aac5d1ffb1de1eb5f2d947222b5f10f6d97ad4383e8: 2Verify BGP peering
cilium bgp peers
Node Local AS Peer AS Peer Address Session State Uptime Family Received Advertised
eksanag15-md-0-wsxnd-f9hdk 64512 64513 10.204.111.49 established 40s ipv4/unicast 1 4
ipv6/unicast 0 0
eksanag15-md-0-wsxnd-xj97h 64512 64513 10.204.111.49 established 42s ipv4/unicast 1 4
ipv6/unicast 0 0Build a new pod/service for NGINX sample app that will be used in combination with Gateway API
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: simple-pod-1
labels:
app: simple-pod-1
spec:
containers:
- name: my-app-container
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: my-service-1
spec:
selector:
app: simple-pod-1 # Make sure this matches the label of the Pod
ports:
- protocol: TCP
port: 8080
targetPort: 80
EOFCreate the Gateway resource and HTTP route. Note we are reusing the same TLS cert as it includes a wildcard certificate for my domain.
cat <<EOF | kubectl create -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: tls-gateway
spec:
gatewayClassName: cilium
listeners:
- name: https-1
protocol: HTTPS
port: 443
hostname: "nginx-test-gateway.poc.thecloudgarage.com"
tls:
certificateRefs:
- kind: Secret
name: nginx-test-cert
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: https-app-route-1
spec:
parentRefs:
- name: tls-gateway
hostnames:
- "nginx-test-gateway.poc.thecloudgarage.com"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: my-service-1
port: 8080
EOFVerify if a dedicated LoadBalancer service is created for the gateway resource
kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-webhook-service ClusterIP 10.99.116.161 <none> 443/TCP 14h
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-webhook-service ClusterIP 10.107.89.240 <none> 443/TCP 14h
capi-system capi-webhook-service ClusterIP 10.109.185.214 <none> 443/TCP 14h
capv-system capv-webhook-service ClusterIP 10.99.205.167 <none> 443/TCP 14h
cert-manager cert-manager ClusterIP 10.104.55.142 <none> 9402/TCP 14h
cert-manager cert-manager-webhook ClusterIP 10.105.176.229 <none> 443/TCP 14h
cilium-test cilium-ingress-other-node NodePort 10.96.76.195 <none> 80:31906/TCP,443:32032/TCP 4h37m
cilium-test cilium-ingress-same-node NodePort 10.96.4.233 <none> 80:30331/TCP,443:31265/TCP 4h37m
cilium-test echo-other-node NodePort 10.110.154.84 <none> 8080:31358/TCP 4h37m
cilium-test echo-same-node NodePort 10.105.86.35 <none> 8080:31228/TCP 4h37m
default cilium-gateway-tls-gateway LoadBalancer 10.102.173.94 172.26.1.3 443:31507/TCP 85s
default cilium-ingress-tls-ingress LoadBalancer 10.109.70.84 172.26.1.2 80:31232/TCP,443:32300/TCP 48m
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 14h
default my-service ClusterIP 10.110.54.84 <none> 8080/TCP 48m
default my-service-1 ClusterIP 10.101.176.214 <none> 8080/TCP 6m41s
eksa-system eksa-webhook-service ClusterIP 10.100.92.78 <none> 443/TCP 14h
etcdadm-bootstrap-provider-system etcdadm-bootstrap-provider-webhook-service ClusterIP 10.106.23.14 <none> 443/TCP 14h
etcdadm-controller-system etcdadm-controller-webhook-service ClusterIP 10.96.105.0 <none> 443/TCP 14h
flux-system notification-controller ClusterIP 10.110.115.175 <none> 80/TCP 14h
flux-system source-controller ClusterIP 10.105.176.255 <none> 80/TCP 14h
flux-system webhook-receiver ClusterIP 10.108.157.179 <none> 80/TCP 14h
kube-system cilium-agent ClusterIP None <none> 9964/TCP 14h
kube-system cilium-envoy ClusterIP None <none> 9964/TCP 51m
kube-system cilium-ingress LoadBalancer 10.96.249.182 172.26.1.1 80:31393/TCP,443:32206/TCP 51m
kube-system cloud-controller-manager NodePort 10.109.52.141 <none> 443:32076/TCP 14h
kube-system hubble-metrics ClusterIP None <none> 9965/TCP 51m
kube-system hubble-peer ClusterIP 10.109.64.78 <none> 443/TCP 8h
kube-system hubble-relay ClusterIP 10.103.38.11 <none> 80/TCP 51m
kube-system hubble-ui ClusterIP 10.109.54.148 <none> 80/TCP 51m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 14hLet’s observe if the gateway resource is created properly and that it bears an IP address
kubectl get gateway
NAME CLASS ADDRESS PROGRAMMED AGE
tls-gateway cilium 172.26.1.3 True 13mObserve that a new ciliumenvoyconfig resource is created for the gateway resource
kubectl get ciliumenvoyconfig
NAME AGE
cilium-gateway-tls-gateway 13m
cilium-ingress-default-tls-ingress 60m
kubectl describe ciliumenvoyconfig cilium-gateway-tls-gateway
Name: cilium-gateway-tls-gateway
Namespace: default
Labels: cilium.io/use-original-source-address=false
Annotations: <none>
API Version: cilium.io/v2
Kind: CiliumEnvoyConfig
Metadata:
Creation Timestamp: 2024-06-10T17:33:09Z
Generation: 2
Owner References:
API Version: gateway.networking.k8s.io/v1
Controller: true
Kind: Gateway
Name: tls-gateway
UID: e817fce7-99a5-41d1-8813-8fd42b322ba8
Resource Version: 408332
UID: 63129a5a-516f-4d63-ae5b-f38c3afabf41
Spec:
Backend Services:
Name: my-service-1
Namespace: default
Number:
8080
Resources:
@type: type.googleapis.com/envoy.config.listener.v3.Listener
Filter Chains:
Filter Chain Match:
Transport Protocol: raw_buffer
Filters:
Name: envoy.filters.network.http_connection_manager
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
Common Http Protocol Options:
Max Stream Duration: 0s
Http Filters:
Name: envoy.filters.http.grpc_web
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
Name: envoy.filters.http.grpc_stats
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig
Emit Filter State: true
Enable Upstream Stats: true
Name: envoy.filters.http.router
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
Rds:
Route Config Name: listener-insecure
Stat Prefix: listener-insecure
Upgrade Configs:
Upgrade Type: websocket
Use Remote Address: true
Filter Chain Match:
Server Names:
nginx-test-gateway.poc.thecloudgarage.com
Transport Protocol: tls
Filters:
Name: envoy.filters.network.http_connection_manager
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
Common Http Protocol Options:
Max Stream Duration: 0s
Http Filters:
Name: envoy.filters.http.grpc_web
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
Name: envoy.filters.http.grpc_stats
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig
Emit Filter State: true
Enable Upstream Stats: true
Name: envoy.filters.http.router
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
Rds:
Route Config Name: listener-secure
Stat Prefix: listener-secure
Upgrade Configs:
Upgrade Type: websocket
Use Remote Address: true
Transport Socket:
Name: envoy.transport_sockets.tls
Typed Config:
@type: type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
Common Tls Context:
Tls Certificate Sds Secret Configs:
Name: cilium-secrets/default-nginx-test-cert
Listener Filters:
Name: envoy.filters.listener.tls_inspector
Typed Config:
@type: type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
Name: listener
Socket Options:
Description: Enable TCP keep-alive (default to enabled)
Int Value: 1
Level: 1
Name: 9
Description: TCP keep-alive idle time (in seconds) (defaults to 10s)
Int Value: 10
Level: 6
Name: 4
Description: TCP keep-alive probe intervals (in seconds) (defaults to 5s)
Int Value: 5
Level: 6
Name: 5
Description: TCP keep-alive probe max failures.
Int Value: 10
Level: 6
Name: 6
@type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration
Name: listener-secure
Virtual Hosts:
Domains:
nginx-test-gateway.poc.thecloudgarage.com
nginx-test-gateway.poc.thecloudgarage.com:*
Name: nginx-test-gateway.poc.thecloudgarage.com
Routes:
Match:
Prefix: /
Route:
Cluster: default:my-service-1:8080
@type: type.googleapis.com/envoy.config.cluster.v3.Cluster
Connect Timeout: 5s
Eds Cluster Config:
Service Name: default/my-service-1:8080
Name: default:my-service-1:8080
Outlier Detection:
Split External Local Origin Errors: true
Type: EDS
Typed Extension Protocol Options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
@type: type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
Common Http Protocol Options:
Idle Timeout: 60s
Use Downstream Protocol Config:
http2ProtocolOptions:
Services:
Listener:
Name: cilium-gateway-tls-gateway
Namespace: default
Events: <none>Let’s verify access to our new NGINX sample app via the gateway resource by curling it from the BGP BIRD Ubuntu machine. Ensure that the hosts entry or DNS record is created accordingly.
curl -v https://nginx-test-gateway.poc.thecloudgarage.com
* Trying 172.26.1.3:443...
* Connected to nginx-test-gateway.poc.thecloudgarage.com (172.26.1.3) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=thecloudgarage.com
* start date: May 13 05:30:07 2024 GMT
* expire date: Aug 11 05:30:06 2024 GMT
* subjectAltName: host "nginx-test-gateway.poc.thecloudgarage.com" matched cert's "*.poc.thecloudgarage.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: nginx-test-gateway.poc.thecloudgarage.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: envoy
< date: Mon, 10 Jun 2024 17:50:02 GMT
< content-type: text/html
< content-length: 615
< last-modified: Tue, 28 May 2024 13:22:30 GMT
< etag: "6655da96-267"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 1
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host nginx-test-gateway.poc.thecloudgarage.com left intactWhy stop, let’s look at Hubble
In this stride, we will enable Hubble UI via Ingress with TLS.
Let’s begin by creating the TLS secret for Hubble in kube-system namespace
kubectl create secret tls hubble-cert -n kube-system --key tls.key --cert tls.crtWith the secret in place, let’s upgrade our Helm by including the Ingress configuration for Hubble-UI. I have chosen hubble1.poc.thecloudgarage.com as the preferred hostname for my UI.
helm upgrade cilium cilium/cilium --version 1.15.5 \
--namespace kube-system \
--set eni.enabled=false \
--set ipam.mode=kubernetes \
--set bpf.masquerade=true \
--set tunnel=vxlan \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set hubble.ui.ingress.enabled=true \
--set hubble.ui.ingress.className=cilium \
--set hubble.ui.ingress.hosts[0]="hubble1.poc.thecloudgarage.com" \
--set hubble.ui.ingress.tls[0].hosts[0]="hubble1.poc.thecloudgarage.com" \
--set hubble.ui.ingress.tls[0].secretName="hubble-cert" \
--set bgpControlPlane.enabled=true \
--set kubeProxyReplacement=true \
--set k8sServiceHost=10.204.111.57 \
--set k8sServicePort=6443 \
--set externalIPs.enabled=true \
--set bgp.announce.loadbalancerIP=true \
--set ingressController.enabled=true \
--set ingressController.loadbalancerMode=dedicated \
--set gatewayAPI.enabled=true \
--set envoy.enabled=true \
--set loadBalancer.l7.backend=envoy \
--set debug.enabled=true \
--set debug.verbose=flowRestart Cilium operator and daemonset
kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/ciliumLet’s observe the Cilium configuration
agent-not-ready-taint-key node.cilium.io/agent-not-ready
arping-refresh-period 30s
auto-direct-node-routes false
bgp-secrets-namespace kube-system
bpf-lb-acceleration disabled
bpf-lb-external-clusterip false
bpf-lb-map-max 65536
bpf-lb-sock false
bpf-map-dynamic-size-ratio 0.0025
bpf-policy-map-max 16384
bpf-root /sys/fs/bpf
cgroup-root /run/cilium/cgroupv2
cilium-endpoint-gc-interval 5m0s
cluster-id 0
cluster-name default
cni-exclusive true
cni-log-file /var/run/cilium/cilium-cni.log
custom-cni-conf false
debug true
debug-verbose flow
dnsproxy-enable-transparent-mode true
egress-gateway-reconciliation-trigger-interval 1s
enable-auto-protect-node-port-range true
enable-bgp-control-plane true
enable-bpf-clock-probe false
enable-bpf-masquerade true
enable-endpoint-health-checking true
enable-envoy-config true
enable-gateway-api true
enable-gateway-api-secrets-sync true
enable-health-check-loadbalancer-ip false
enable-health-check-nodeport true
enable-health-checking true
enable-hubble true
enable-hubble-open-metrics false
enable-ingress-controller true
enable-ingress-proxy-protocol false
enable-ingress-secrets-sync true
enable-ipv4 true
enable-ipv4-big-tcp false
enable-ipv4-masquerade true
enable-ipv6 false
enable-ipv6-big-tcp false
enable-ipv6-masquerade true
enable-k8s-networkpolicy true
enable-k8s-terminating-endpoint true
enable-l2-neigh-discovery true
enable-l7-proxy true
enable-local-redirect-policy false
enable-masquerade-to-route-source false
enable-metrics true
enable-policy default
enable-remote-node-identity true
enable-sctp false
enable-svc-source-range-check true
enable-vtep false
enable-well-known-identities false
enable-xt-socket-fallback true
enforce-ingress-https true
external-envoy-proxy true
gateway-api-secrets-namespace cilium-secrets
hubble-disable-tls false
hubble-export-file-max-backups 5
hubble-export-file-max-size-mb 10
hubble-listen-address :4244
hubble-metrics dns drop tcp flow icmp http
hubble-metrics-server :9965
hubble-socket-path /var/run/cilium/hubble.sock
hubble-tls-cert-file /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file /var/lib/cilium/tls/hubble/server.key
identity-allocation-mode crd
identity-gc-interval 15m0s
identity-heartbeat-timeout 30m0s
ingress-default-lb-mode dedicated
ingress-lb-annotation-prefixes service.beta.kubernetes.io service.kubernetes.io cloud.google.com
ingress-secrets-namespace cilium-secrets
ingress-shared-lb-service-name cilium-ingress
install-no-conntrack-iptables-rules false
ipam kubernetes
ipam-cilium-node-update-rate 15s
k8s-client-burst 20
k8s-client-qps 10
kube-proxy-replacement true
kube-proxy-replacement-healthz-bind-address
loadbalancer-l7 envoy
loadbalancer-l7-algorithm round_robin
loadbalancer-l7-ports
max-connected-clusters 255
mesh-auth-enabled true
mesh-auth-gc-interval 5m0s
mesh-auth-queue-size 1024
mesh-auth-rotated-identities-queue-size 1024
monitor-aggregation medium
monitor-aggregation-flags all
monitor-aggregation-interval 5s
node-port-bind-protection true
nodes-gc-interval 5m0s
operator-api-serve-addr 127.0.0.1:9234
operator-prometheus-serve-addr :9963
policy-cidr-match-mode
preallocate-bpf-maps false
procfs /host/proc
proxy-connect-timeout 2
proxy-idle-timeout-seconds 60
proxy-max-connection-duration-seconds 0
proxy-max-requests-per-connection 0
proxy-xff-num-trusted-hops-egress 0
proxy-xff-num-trusted-hops-ingress 0
remove-cilium-node-taints true
routing-mode tunnel
service-no-backend-response reject
set-cilium-is-up-condition true
set-cilium-node-taints true
sidecar-istio-proxy-image cilium/istio_proxy
skip-cnp-status-startup-clean false
synchronize-k8s-nodes true
tofqdns-dns-reject-response-code refused
tofqdns-enable-dns-compression true
tofqdns-endpoint-max-ip-per-hostname 50
tofqdns-idle-connection-grace-period 0s
tofqdns-max-deferred-connection-deletes 10000
tofqdns-proxy-response-max-delay 100ms
tunnel-protocol vxlan
unmanaged-pod-watcher-interval 15
vtep-cidr
vtep-endpoint
vtep-mac
vtep-mask
write-cni-conf-when-ready /host/etc/cni/net.d/05-cilium.conflistOnce the Cilium and BGP statuses are confirmed, observe that a new LoadBalancer service for cilium-ingress-hubble-ui has been created with an ExternalIP
kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-webhook-service ClusterIP 10.99.116.161 <none> 443/TCP 25h
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-webhook-service ClusterIP 10.107.89.240 <none> 443/TCP 25h
capi-system capi-webhook-service ClusterIP 10.109.185.214 <none> 443/TCP 25h
capv-system capv-webhook-service ClusterIP 10.99.205.167 <none> 443/TCP 25h
cert-manager cert-manager ClusterIP 10.104.55.142 <none> 9402/TCP 25h
cert-manager cert-manager-webhook ClusterIP 10.105.176.229 <none> 443/TCP 25h
cilium-test cilium-ingress-other-node NodePort 10.96.76.195 <none> 80:31906/TCP,443:32032/TCP 15h
cilium-test cilium-ingress-same-node NodePort 10.96.4.233 <none> 80:30331/TCP,443:31265/TCP 15h
cilium-test echo-other-node NodePort 10.110.154.84 <none> 8080:31358/TCP 15h
cilium-test echo-same-node NodePort 10.105.86.35 <none> 8080:31228/TCP 15h
default cilium-gateway-tls-gateway LoadBalancer 10.102.173.94 172.26.1.3 443:31507/TCP 10h
default cilium-ingress-tls-ingress LoadBalancer 10.109.70.84 172.26.1.2 80:31232/TCP,443:32300/TCP 11h
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 25h
default my-service ClusterIP 10.110.54.84 <none> 8080/TCP 11h
default my-service-1 ClusterIP 10.101.176.214 <none> 8080/TCP 10h
eksa-system eksa-webhook-service ClusterIP 10.100.92.78 <none> 443/TCP 25h
etcdadm-bootstrap-provider-system etcdadm-bootstrap-provider-webhook-service ClusterIP 10.106.23.14 <none> 443/TCP 25h
etcdadm-controller-system etcdadm-controller-webhook-service ClusterIP 10.96.105.0 <none> 443/TCP 25h
flux-system notification-controller ClusterIP 10.110.115.175 <none> 80/TCP 25h
flux-system source-controller ClusterIP 10.105.176.255 <none> 80/TCP 25h
flux-system webhook-receiver ClusterIP 10.108.157.179 <none> 80/TCP 25h
kube-system cilium-agent ClusterIP None <none> 9964/TCP 25h
kube-system cilium-envoy ClusterIP None <none> 9964/TCP 11h
kube-system cilium-ingress LoadBalancer 10.96.249.182 172.26.1.1 80:31393/TCP,443:32206/TCP 11h
kube-system cilium-ingress-hubble-ui LoadBalancer 10.107.249.246 172.26.1.4 80:31236/TCP,443:30647/TCP 59m
kube-system cloud-controller-manager NodePort 10.109.52.141 <none> 443:32076/TCP 25h
kube-system hubble-metrics ClusterIP None <none> 9965/TCP 11h
kube-system hubble-peer ClusterIP 10.109.64.78 <none> 443/TCP 19h
kube-system hubble-relay ClusterIP 10.103.38.11 <none> 80/TCP 11h
kube-system hubble-ui ClusterIP 10.109.54.148 <none> 80/TCP 11h
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 25hWe can also see an ingress resource with an IP address for the Hubble UI in the kube-system namespace
kubectl get ingress -n kube-system
NAME CLASS HOSTS ADDRESS PORTS AGE
hubble-ui cilium hubble1.poc.thecloudgarage.com 172.26.1.4 80, 443 61mLet’s head over to the BGP BIRD Ubuntu host if this prefix has been received via the BGP route
ip route
default via 10.204.110.1 dev ens160 proto static
10.204.110.0/23 dev ens160 proto kernel scope link src 10.204.111.49
172.16.2.0/24 via 10.204.111.49 dev ens160 proto bird
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-65b73db83171 proto kernel scope link src 172.18.0.1 linkdown
172.19.0.0/16 dev br-367eea0c41bc proto kernel scope link src 172.19.0.1 linkdown
172.20.0.0/16 dev br-b1dd61ad5edf proto kernel scope link src 172.20.0.1 linkdown
172.21.0.0/16 dev br-a94ebdb591eb proto kernel scope link src 172.21.0.1 linkdown
172.22.0.0/16 dev br-e91c56ab7045 proto kernel scope link src 172.22.0.1 linkdown
172.23.0.0/16 dev br-f8107365302c proto kernel scope link src 172.23.0.1 linkdown
172.24.0.0/16 dev br-4cc2a2dd5387 proto kernel scope link src 172.24.0.1 linkdown
172.26.1.1 via 10.204.110.9 dev ens160 proto bird
172.26.1.2 via 10.204.110.9 dev ens160 proto bird
172.26.1.3 via 10.204.110.9 dev ens160 proto bird
172.26.1.4 via 10.204.110.9 dev ens160 proto bird
192.168.2.0/24 via 10.204.111.49 dev ens160 proto bird
192.168.3.0/24 via 10.204.110.32 dev ens160 proto bird
192.168.4.0/24 via 10.204.110.9 dev ens160 proto birdNow that we have everything in place for the Hubble UI to be accessed via Ingress, add in the DNS record or a manual hosts entry in the BGP BIRD Ubuntu machine.
nslookup hubble1.poc.thecloudgarage.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Name: hubble1.poc.thecloudgarage.com
Address: 172.26.1.4For the sake of remote access, I will install xrdp on my Ubuntu host to gain desktop and browser access. In that way, we will be able to browse the Hubble UI from the BGP BIRD Ubuntu host itself.
sudo apt update
sudo apt install xrdp -yWith XRDP installed, we can RDP to the BGP BIRD Ubuntu host and browse the Hubble UI HTTPS
Let’s select the Kube-system namespace in the UI and there you have it., live flows from the hubble-ui
We can inspect any other pod/app in any of the given namespaces.
Oh yes., I forgot to share some more insight on BGP peering policy setting called exportPodCIDR
Generally speaking, one always needs a service to expose and communicate with pods. This is especially true when services are accessed from external entities via a service type LoadBalancer.
However, cilium BGP peering policy allows for a special setting of exposing podCIDRs directly to the peers
Let’s look at what this means in reality! If we pay attention to the BGP peering policy defined above while creating the Ingress controller, the setting exportPodCIDR was set to true
My EKS Anywhere cluster’s POD CIDR is set at the default value of 192.16.0.0/16 with two worker nodes. In that sense, the entire range is split into individual CIDR /24 blocks and assigned to each node allowing each to host a max of 256 pods. This setting can be changed but that’s a diffferent conversation.
Now that we know that the podCIDR is split into /24 blocks and assigned to each node, any new pod depending on node placement will get it’s IP from the respective /24 block.
When the setting exportPodCIDR is set as true in BGP peering policy, these individual node POD CIDR blocks are also advertised as /24 prefixes to the remote peer. These prefixes are on top of any prefixes which represent the LB IPAM pool, which are generally awarded to services with type LoadBalancer. We are talking about the podCIDR ranges being advertised across the peer. As a result, the remote peer can view the podCIDRs as routable via BGP and thus can directly access the pods directly on the container port (not the service port). This container port is also defined as the targetPort in the service YAML.
First let’s observe what Pod CIDRs are assigned to my 2 worker nodes. As you can see the two worker nodes have been assigned the Pod CIDRs as 192.168.3.0/24 and 192.168.4.0/24
kubectl describe ciliumnodes eksanag15-md-0-wsxnd-f9hdk
Name: eksanag15-md-0-wsxnd-f9hdk
Namespace:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=vsphere-vm.cpu-4.mem-8gb.os-ubuntu
beta.kubernetes.io/os=linux
group=md-0
kubernetes.io/arch=amd64
kubernetes.io/hostname=eksanag15-md-0-wsxnd-f9hdk
kubernetes.io/os=linux
node.cluster.x-k8s.io/esxi-host=mc-mg-node01.edub.csc
node.kubernetes.io/instance-type=vsphere-vm.cpu-4.mem-8gb.os-ubuntu
Annotations: <none>
API Version: cilium.io/v2
Kind: CiliumNode
Metadata:
Creation Timestamp: 2024-06-10T03:17:08Z
Generation: 37
Owner References:
API Version: v1
Kind: Node
Name: eksanag15-md-0-wsxnd-f9hdk
UID: 802d1ca5-83d4-490b-a6d7-4f1dc2660ea1
Resource Version: 1456630
UID: bc843866-82c4-4430-9fc6-a7a7f891dbc2
Spec:
Addresses:
Ip: 10.204.110.9
Type: InternalIP
Ip: 10.204.110.9
Type: ExternalIP
Ip: 192.168.4.222
Type: CiliumInternalIP
Alibaba - Cloud:
Azure:
Bootid: 4e905fdb-8512-4db2-9c69-f31ee9bfa7de
Encryption:
Eni:
Health:
ipv4: 192.168.4.233
Ingress:
ipv4: 192.168.4.87
Ipam:
Pod CID Rs:
192.168.4.0/24
kubectl describe ciliumnodes eksanag15-md-0-wsxnd-xj97h
Name: eksanag15-md-0-wsxnd-xj97h
Namespace:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=vsphere-vm.cpu-4.mem-8gb.os-ubuntu
beta.kubernetes.io/os=linux
group=md-0
kubernetes.io/arch=amd64
kubernetes.io/hostname=eksanag15-md-0-wsxnd-xj97h
kubernetes.io/os=linux
node.cluster.x-k8s.io/esxi-host=mc-mg-node05.edub.csc
node.kubernetes.io/instance-type=vsphere-vm.cpu-4.mem-8gb.os-ubuntu
Annotations: <none>
API Version: cilium.io/v2
Kind: CiliumNode
Metadata:
Creation Timestamp: 2024-06-10T03:17:08Z
Generation: 37
Owner References:
API Version: v1
Kind: Node
Name: eksanag15-md-0-wsxnd-xj97h
UID: c4c111c8-5fd7-4162-89c5-8dadda9b8aa0
Resource Version: 1456951
UID: 96a357c3-61e7-48d1-9556-884af5d6499b
Spec:
Addresses:
Ip: 10.204.110.32
Type: InternalIP
Ip: 10.204.110.32
Type: ExternalIP
Ip: 192.168.3.215
Type: CiliumInternalIP
Alibaba - Cloud:
Azure:
Bootid: 45ad0a2b-17fe-483b-a31e-86ed8c0d7f45
Encryption:
Eni:
Health:
ipv4: 192.168.3.104
Ingress:
ipv4: 192.168.3.102
Ipam:
Pod CID Rs:
192.168.3.0/24Let’s view the Cilium prefix advertisements. As one can observe 192.168.3.0/24 and 192.168.4.0/24 are being advertised as the exportPodCIDR setting is set to true.
cilium bgp routes
(Defaulting to `available ipv4 unicast` routes, please see help for more options)
Node VRouter Prefix NextHop Age Attrs
eksanag15-md-0-wsxnd-f9hdk 64512 172.16.2.0/24 10.204.111.49 30m13s [{Origin: i} {AsPath: 64513} {Nexthop: 10.204.111.49}]
64512 172.26.1.1/32 0.0.0.0 30m21s [{Origin: i} {Nexthop: 0.0.0.0}]
64512 172.26.1.3/32 0.0.0.0 30m21s [{Origin: i} {Nexthop: 0.0.0.0}]
64512 172.26.1.4/32 0.0.0.0 30m21s [{Origin: i} {Nexthop: 0.0.0.0}]
64512 192.168.4.0/24 0.0.0.0 30m21s [{Origin: i} {Nexthop: 0.0.0.0}]
eksanag15-md-0-wsxnd-xj97h 64512 172.16.2.0/24 10.204.111.49 29m55s [{Origin: i} {AsPath: 64513} {Nexthop: 10.204.111.49}]
64512 172.26.1.1/32 0.0.0.0 30m0s [{Origin: i} {Nexthop: 0.0.0.0}]
64512 172.26.1.3/32 0.0.0.0 30m0s [{Origin: i} {Nexthop: 0.0.0.0}]
64512 172.26.1.4/32 0.0.0.0 30m0s [{Origin: i} {Nexthop: 0.0.0.0}]
64512 192.168.3.0/24 0.0.0.0 30m0s [{Origin: i} {Nexthop: 0.0.0.0}]Let’s validate this on the BGP BIRD Ubuntu machine. Notice the last two entries, those are the POD CIDRs and their respective next-hops as the respective worker nodes as BGP peers
ip route
default via 10.204.110.1 dev ens160 proto static
10.204.110.0/23 dev ens160 proto kernel scope link src 10.204.111.49
172.16.2.0/24 via 10.204.111.49 dev ens160 proto bird
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-65b73db83171 proto kernel scope link src 172.18.0.1 linkdown
172.19.0.0/16 dev br-367eea0c41bc proto kernel scope link src 172.19.0.1 linkdown
172.20.0.0/16 dev br-b1dd61ad5edf proto kernel scope link src 172.20.0.1 linkdown
172.21.0.0/16 dev br-a94ebdb591eb proto kernel scope link src 172.21.0.1 linkdown
172.22.0.0/16 dev br-e91c56ab7045 proto kernel scope link src 172.22.0.1 linkdown
172.23.0.0/16 dev br-f8107365302c proto kernel scope link src 172.23.0.1 linkdown
172.24.0.0/16 dev br-4cc2a2dd5387 proto kernel scope link src 172.24.0.1 linkdown
172.26.1.1 via 10.204.110.9 dev ens160 proto bird
172.26.1.3 via 10.204.110.9 dev ens160 proto bird
172.26.1.4 via 10.204.110.9 dev ens160 proto bird
192.168.2.0/24 via 10.204.111.49 dev ens160 proto bird
192.168.3.0/24 via 10.204.110.32 dev ens160 proto bird
192.168.4.0/24 via 10.204.110.9 dev ens160 proto birdOur NGINX sample apps are reachable via Ingress (simple-pod) and Gatway API (simple-pod-1) and in addition, they are also seen via their actual POD IP address.
Shall we validate direct access to pods?
kubectl get pods
NAME READY STATUS RESTARTS AGE
simple-pod 1/1 Running 0 17m
simple-pod-1 1/1 Running 0 18mBoth these pods have an IP from the POD CIDR and the Port as 80
kubectl describe pod simple-pod | grep IP:
IP: 192.168.4.223
kubectl describe pod simple-pod-1 | grep IP:
IP: 192.168.4.160We have already verified above that the sample applications can via their exposed ExternalIPs. Now, we will try directly accessing the Pod from the BGP BIRD Ubuntu on its POD IP and container port. Notice we are not using the service port but the container port (80) itself
curl 192.168.4.223
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
curl 192.168.4.160
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>Hopefully, that explains a very important attribute that can give rise to tremendous advantage from a container networking perspective
Finally, let’s dabble with Pod CIDR IPAM
In the default install of Cilium and/or the above scenarios, the PodCIDR IPAM is governed by Kubernetes, which can be limiting in advanced solutions.
Let’s switch to cluster-pool IPAM and observe.
Note: Per official recommendation, this should not be done on an existing cluster.
So, I have created a new cluster with skipUpgrade option for the default Cilium CNI. In addition, I have performed the steps (patching, pausing reconciliation, deleting kube-proxy and cilium daemonset) and then finally configure Cilium OSS on the cluster via Helm.
Observe the settings for IPAM which are now configured with cluster-pool and the CIDR List and per node mask size. If we do not specify the CIDR list, it will default to 10.0.0.0/8
helm install cilium cilium/cilium --version 1.15.5 \
--namespace kube-system \
--set eni.enabled=false \
--set ipam.mode=cluster-pool \
--set ipam.operator.clusterPoolIPv4PodCIDRList=192.168.0.0/16 \
--set ipam.operator.clusterPoolIPv4MaskSize=24 \
--set bpf.masquerade=true \
--set tunnel=vxlan \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set bgpControlPlane.enabled=true \
--set kubeProxyReplacement=true \
--set nodePort.enabled=true \
--set k8sServiceHost=10.204.111.57 \
--set k8sServicePort=6443 \
--set externalIPs.enabled=true \
--set bgp.announce.loadbalancerIP=true \
--set ingressController.enabled=true \
--set ingressController.loadbalancerMode=dedicated \
--set gatewayAPI.enabled=true \
--set envoy.enabled=true \
--set loadBalancer.l7.backend=envoy \
--set debug.enabled=true \
--set debug.verbose=flowOnce deployed, we can verify the cilium status
cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
DaemonSet cilium-envoy Desired: 5, Ready: 5/5, Available: 5/5
DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 5
cilium-operator Running: 2
hubble-ui Running: 1
hubble-relay Running: 1
cilium-envoy Running: 5
Cluster Pods: 18/18 managed by Cilium
Helm chart version:
Image versions cilium quay.io/cilium/cilium:v1.15.5@sha256:4ce1666a73815101ec9a4d360af6c5b7f1193ab00d89b7124f8505dee147ca40: 5
cilium-operator quay.io/cilium/operator-generic:v1.15.5@sha256:f5d3d19754074ca052be6aac5d1ffb1de1eb5f2d947222b5f10f6d97ad4383e8: 2
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1
hubble-ui quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
hubble-relay quay.io/cilium/hubble-relay:v1.15.5@sha256:1d24b24e3477ccf9b5ad081827db635419c136a2bd84a3e60f37b26a38dd0781: 1
cilium-envoy quay.io/cilium/cilium-envoy:v1.28.3-31ec52ec5f2e4d28a8e19a0bfb872fa48cf7a515@sha256:bc8dcc3bc008e3a5aab98edb73a0985e6ef9469bda49d5bb3004c001c995c380: 5Resume the cluster reconciliation
kubectl -n eksa-system annotate clusters.cluster.x-k8s.io $CLUSTER_NAME cluster.x-k8s.io/paused-Let’s verify cilium config for specific settings
cilium config view | egrep 'ipam|ipv4'
cluster-pool-ipv4-cidr 192.168.0.0/16
cluster-pool-ipv4-mask-size 24
enable-ipv4 true
enable-ipv4-big-tcp false
enable-ipv4-masquerade true
ipam cluster-poolObserve the Cilium node configs
kubectl get ciliumnodes.cilium.io
NAME CILIUMINTERNALIP INTERNALIP AGE
eksanag23-2mzgt 192.168.0.224 10.204.111.25 47m
eksanag23-425c5 192.168.2.104 10.204.110.60 47m
eksanag23-md-0-wnmmn-2cm5z 192.168.4.106 10.204.110.65 46m
eksanag23-md-0-wnmmn-f9zgs 192.168.3.17 10.204.110.61 46m
eksanag23-phvvj 192.168.1.162 10.204.110.59 47m
kubectl describe ciliumnode eksanag23-md-0-wnmmn-2cm5z
Name: eksanag23-md-0-wnmmn-2cm5z
Namespace:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=vsphere-vm.cpu-4.mem-8gb.os-ubuntu
beta.kubernetes.io/os=linux
group=md-0
kubernetes.io/arch=amd64
kubernetes.io/hostname=eksanag23-md-0-wnmmn-2cm5z
kubernetes.io/os=linux
node.cluster.x-k8s.io/esxi-host=mc-mg-node04.edub.csc
node.kubernetes.io/instance-type=vsphere-vm.cpu-4.mem-8gb.os-ubuntu
Annotations: <none>
API Version: cilium.io/v2
Kind: CiliumNode
Metadata:
Creation Timestamp: 2024-06-13T03:35:26Z
Generation: 3
Owner References:
API Version: v1
Kind: Node
Name: eksanag23-md-0-wnmmn-2cm5z
UID: d5c708e5-81b6-403c-be3f-b8602b6ef28d
Resource Version: 9469
UID: 0aee23ef-37be-49e9-a3ab-82d1f9135274
Spec:
Addresses:
Ip: 10.204.110.65
Type: InternalIP
Ip: 10.204.110.65
Type: ExternalIP
Ip: 192.168.4.106
Type: CiliumInternalIP
Alibaba - Cloud:
Azure:
Bootid: a42638ef-068f-4f16-9374-cd3677e03f42
Encryption:
Eni:
Health:
ipv4: 192.168.4.228
Ingress:
ipv4: 192.168.4.177
Ipam:
Pod CIDRs:
192.168.4.0/24
Pools:
Status:
Alibaba - Cloud:
Azure:
Eni:
Ipam:
Operator - Status:
Events: <none>A Final twist, will we have problems upgrading cluster with a Custom CNI… let’s verify!!!
I have bumped up the machine specs in the cluster template for the control plane and worker nodes from 4 vCPUs to 8 vCPUs
kubectl apply -f $HOME/$CLUSTER_NAME/$CLUSTER_NAME-eks-a-cluster.yaml --kubeconfig $HOME/$CLUSTER_NAME/$CLUSTER_NAME-eks-a-cluster.kubeconfigThe new nodes have started popping in
kubectl get nodes
NAME STATUS ROLES AGE VERSION
eksanag23-2mzgt Ready control-plane 84m v1.29.1-eks-61c0bbb
eksanag23-425c5 Ready control-plane 82m v1.29.1-eks-61c0bbb
eksanag23-md-0-jjzfs-h4zcv NotReady <none> 8s v1.29.1-eks-61c0bbb
eksanag23-md-0-wnmmn-2cm5z Ready <none> 81m v1.29.1-eks-61c0bbb
eksanag23-md-0-wnmmn-f9zgs Ready <none> 81m v1.29.1-eks-61c0bbb
eksanag23-phvvj Ready control-plane 83m v1.29.1-eks-61c0bbb
eksanag23-zhn5n NotReady control-plane 4s v1.29.1-eks-61c0bbbOld nodes are being deleted
kubectl get nodes
NAME STATUS ROLES AGE VERSION
eksanag23-4lqgb Ready control-plane 3m54s v1.29.1-eks-61c0bbb
eksanag23-7gw6c Ready control-plane 6m38s v1.29.1-eks-61c0bbb
eksanag23-7mn5s Ready control-plane 10m v1.29.1-eks-61c0bbb
eksanag23-md-0-7w5j5-rmjqq Ready <none> 61s v1.29.1-eks-61c0bbb
eksanag23-md-0-jjzfs-h4zcv Ready,SchedulingDisabled <none> 12m v1.29.1-eks-61c0bbb
eksanag23-md-0-jjzfs-j8pnh Ready <none> 10m v1.29.1-eks-61c0bbbAll done!
kubectl get nodes
NAME STATUS ROLES AGE VERSION
eksanag23-4lqgb Ready control-plane 6m52s v1.29.1-eks-61c0bbb
eksanag23-7gw6c Ready control-plane 9m36s v1.29.1-eks-61c0bbb
eksanag23-7mn5s Ready control-plane 13m v1.29.1-eks-61c0bbb
eksanag23-md-0-7w5j5-gwcrl Ready <none> 113s v1.29.1-eks-61c0bbb
eksanag23-md-0-7w5j5-rmjqq Ready <none> 3m59s v1.29.1-eks-61c0bbbLet’s verify cilium status
cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
DaemonSet cilium-envoy Desired: 5, Ready: 5/5, Available: 5/5
DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Containers: cilium Running: 5
cilium-envoy Running: 5
hubble-ui Running: 1
cilium-operator Running: 2
hubble-relay Running: 1
Cluster Pods: 18/18 managed by Cilium
Helm chart version:
Image versions cilium quay.io/cilium/cilium:v1.15.5@sha256:4ce1666a73815101ec9a4d360af6c5b7f1193ab00d89b7124f8505dee147ca40: 5
cilium-envoy quay.io/cilium/cilium-envoy:v1.28.3-31ec52ec5f2e4d28a8e19a0bfb872fa48cf7a515@sha256:bc8dcc3bc008e3a5aab98edb73a0985e6ef9469bda49d5bb3004c001c995c380: 5
hubble-ui quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1
cilium-operator quay.io/cilium/operator-generic:v1.15.5@sha256:f5d3d19754074ca052be6aac5d1ffb1de1eb5f2d947222b5f10f6d97ad4383e8: 2
hubble-relay quay.io/cilium/hubble-relay:v1.15.5@sha256:1d24b24e3477ccf9b5ad081827db635419c136a2bd84a3e60f37b26a38dd0781: 1All thumbs up and green! Successfuly upgraded the cluster with full version of Cilium OSS
So, there we have it., a full Cilium experience without Kube-proxy and BGP based load-balancing along with Ingress, Gateway API with TLS and Hubble UI!!!
Hope this was a worthwhile read into replacing the default Cilium CNI with Cilium OSS on EKS-Anywhere.
cheers,
Ambar@thecloudgarage
#iwork4dell
