flannel
Flannel is the network plug-in solution adopted by DOK by default. vxlan is the default mode for DOK clusters. In the privatization scenario, if you confirm that the customer’s hosts are in the same subnet, you can use host-gw mode to improve network performance.
The installation logic of Flannel is as follows. There are two initContainers in the installed yaml file, which are specially used for the installation of CNI and Flannel configuration, so the names are also called install-cni-plugin and install-cni.
So how are these two containers mainly installed? It’s actually very simple. You can look at the args field. In fact, it is to copy the binary of flannel, as well as cni-conf.json and 10-flannel.conflist via cp to specified directory.
initContainers:
- name: install-cni-plugin
#image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply)
image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
command:
- cp
args:
- -f
- /flannel
- /opt/cni/bin/flannel
volumeMounts:
- name: cni-plugin
mountPath: /opt/cni/bin
- name: install-cni
#image: flannelcni/flannel:v0.18.1 for ppc64le and mips64le (dockerhub limitations may apply)
image: rancher/mirrored-flannelcni-flannel:v0.18.1
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
-
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
Where do these configuration files come from? They actually come from configMap.
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
These configuration files will not drop the files to the host like initContainer, but provide them to the container running the Flannel binary through volumeMount, so these files are found in the /etc/kube-flannel/ directory on the host If not, you can only see it when you enter the Flannel container.
# kiexec
Namespace: kube-system | Pod: ✔ kube-flannel-ds-82mww
/ # ls /etc/kube-flannel/
cni-conf.json net-conf.json
vxlan is the mode adopted by Flannel by default. The node routing in this mode is as follows:
# ip r
default via 172.22.0.1 dev eth0
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink
10.244.4.0/24 via 10.244.4.0 dev flannel.1 onlink
10.244.5.0/24 via 10.244.5.0 dev flannel.1 onlink
169.254.0.0/16 dev eth0 scope link metric 1002
By modifying the configuration, you can also switch Flannel to host-gw, and the node routing in this mode becomes:
# ip r
default via 172.22.0.1 dev eth0
10.4.0.0/24 dev nerdctl0 proto kernel scope link src 10.4.0.1
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 172.22.1.176 dev eth0
10.244.2.0/24 via 172.22.0.117 dev eth0
10.244.3.0/24 via 172.22.0.76 dev eth0
10.244.4.0/24 via 172.22.0.212 dev eth0
10.244.5.0/24 via 172.22.0.64 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1002
172.22.0.0/20 dev eth0 proto kernel scope link src 172.22.0.239
After switching, the log of Flannel is as follows:
I0826 03:22:37.551391 1 main.go:463] Found network config - Backend type: host-gw
I0826 03:22:37.551432 1 match.go:195] Determining IP address of default interface
I0826 03:22:37.551838 1 match.go:248] Using interface with name eth0 and address 172.22.1.176
I0826 03:22:37.551860 1 match.go:270] Defaulting external address to interface address (172.22.1.176)
I0826 03:22:37.569614 1 kube.go:351] Setting NodeNetworkUnavailable
I0826 03:22:37.579433 1 main.go:341] Setting up masking rules
I0826 03:22:37.758215 1 main.go:362] Changing default FORWARD chain policy to ACCEPT
I0826 03:22:37.758315 1 main.go:375] Wrote subnet file to /run/flannel/subnet.env
I0826 03:22:37.758326 1 main.go:379] Running backend.
I0826 03:22:37.758343 1 main.go:400] Waiting for all goroutines to exit
I0826 03:22:37.761081 1 route_network.go:55] Watching for new subnet leases
I0826 03:22:37.761153 1 route_network.go:92] Subnet added: 10.244.0.0/24 via 172.22.0.239
W0826 03:22:37.761524 1 route_network.go:151] Replacing existing route to {Ifindex: 5 Dst: 10.244.0.0/24 Src: <nil> Gw: 10.244.0.0 Flags: [onlink] Table: 254 Realm: 0} with {Ifindex: 2 Dst: 10.244.0.0/24 Src: <nil> Gw: 172.22.0.239 Flags: [] Table: 0 Realm: 0}
I0826 03:22:37.848961 1 route_network.go:92] Subnet added: 10.244.2.0/24 via 172.22.0.117
W0826 03:22:37.849059 1 route_network.go:151] Replacing existing route to {Ifindex: 5 Dst: 10.244.2.0/24 Src: <nil> Gw: 10.244.2.0 Flags: [onlink] Table: 254 Realm: 0} with {Ifindex: 2 Dst: 10.244.2.0/24 Src: <nil> Gw: 172.22.0.117 Flags: [] Table: 0 Realm: 0}
I0826 03:22:37.849360 1 route_network.go:92] Subnet added: 10.244.3.0/24 via 172.22.0.76
W0826 03:22:37.849454 1 route_network.go:151] Replacing existing route to {Ifindex: 5 Dst: 10.244.3.0/24 Src: <nil> Gw: 10.244.3.0 Flags: [onlink] Table: 254 Realm: 0} with {Ifindex: 2 Dst: 10.244.3.0/24 Src: <nil> Gw: 172.22.0.76 Flags: [] Table: 0 Realm: 0}
I0826 03:22:37.850273 1 route_network.go:92] Subnet added: 10.244.4.0/24 via 172.22.0.212
W0826 03:22:37.850377 1 route_network.go:151] Replacing existing route to {Ifindex: 5 Dst: 10.244.4.0/24 Src: <nil> Gw: 10.244.4.0 Flags: [onlink] Table: 254 Realm: 0} with {Ifindex: 2 Dst: 10.244.4.0/24 Src: <nil> Gw: 172.22.0.212 Flags: [] Table: 0 Realm: 0}
I0826 03:22:37.850675 1 route_network.go:92] Subnet added: 10.244.5.0/24 via 172.22.0.64
W0826 03:22:37.850758 1 route_network.go:151] Replacing existing route to {Ifindex: 5 Dst: 10.244.5.0/24 Src: <nil> Gw: 10.244.5.0 Flags: [onlink] Table: 254 Realm: 0} with {Ifindex: 2 Dst: 10.244.5.0/24 Src: <nil> Gw: 172.22.0.64 Flags: [] Table: 0 Realm: 0}
Among them, the log of Subnet added: 10.244.0.0/24 via 172.22.0.239 has been very clear. The route adjusted here is to use the ip of a certain node as the gateway of a certain subnet, so the network packet does not need to be encapsulated, just It can be directly routed to this node. In addition, because host-gw does not need packet unpacking, the value of MTU will be automatically changed to 1500 by Flannel.
# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=true
After modifying the configuration, do other containers need to be restarted? Normal situation is unnecessary, because the network stack of the container will only send the data packets of the container to the device cni0. As for whether to go through vxlan or host-gw, it depends entirely on the **routing Configure **, but it does not rule out that some components are sensitive to changes in routing and network solutions. Please carefully test before implementing any changes. In addition, host-gw has better performance, but it is useless Certain conditions need to be met, the most basic is that the worker nodes need to be in the same subnet, that is, the second layer can communicate.
The benchmark tool uses k8s-bench-suite, and the specific command is knb --verbose --client-node node2 --server-node node3, Tested on the same machine, the measured results show that the vxlan mode is compared with the host-gw mode, and there will be about 10% extra consumption (data depends on hardware and network quality).
=========================================================
Benchmark Results
=========================================================
Name : knb-12885
Date : 2022-08-26 07:11:41 UTC
Generator : knb
Version : 1.5.0
Server : node2
Client : node3
UDP Socket size : auto
=========================================================
Discovered CPU : Intel Xeon Processor (Skylake, IBRS)
Discovered Kernel : 5.4.127-1.el7.elrepo.x86_64
Discovered k8s version : v1.21.7
Discovered MTU : 1450
Idle :
bandwidth = 0 Mbit/s
client cpu = total 6.97% (user 2.53%, nice 0.05%, system 4.21%, iowait 0.03%, steal 0.15%)
server cpu = total 8.09% (user 2.73%, nice 0.05%, system 5.18%, iowait 0.00%, steal 0.13%)
client ram = 1233 MB
server ram = 1198 MB
Pod to pod :
TCP :
bandwidth = 845 Mbit/s
client cpu = total 5.06% (user 1.35%, nice 0.05%, system 3.49%, iowait 0.07%, steal 0.10%)
server cpu = total 10.78% (user 1.76%, nice 0.02%, system 8.98%, iowait 0.02%, steal 0.00%)
client ram = 1235 MB
server ram = 1197 MB
UDP :
bandwidth = 877 Mbit/s
client cpu = total 26.54% (user 2.83%, nice 0.05%, system 23.57%, iowait 0.07%, steal 0.02%)
server cpu = total 13.43% (user 3.74%, nice 0.03%, system 9.56%, iowait 0.00%, steal 0.10%)
client ram = 1234 MB
server ram = 1198 MB
Pod to Service :
TCP :
bandwidth = 856 Mbit/s
client cpu = total 5.25% (user 1.40%, nice 0.05%, system 3.68%, iowait 0.05%, steal 0.07%)
server cpu = total 10.31% (user 1.92%, nice 0.02%, system 8.37%, iowait 0.00%, steal 0.00%)
client ram = 1233 MB
server ram = 1199 MB
UDP :
bandwidth = 835 Mbit/s
client cpu = total 27.90% (user 2.94%, nice 0.02%, system 24.82%, iowait 0.07%, steal 0.05%)
server cpu = total 13.29% (user 3.74%, nice 0.03%, system 9.49%, iowait 0.00%, steal 0.03%)
client ram = 1236 MB
server ram = 1203 MB
=========================================================
=========================================================
Benchmark Results
=========================================================
Name : knb-8657
Date : 2022-08-26 07:08:07 UTC
Generator : knb
Version : 1.5.0
Server : node2
Client : node3
UDP Socket size : auto
=========================================================
Discovered CPU : Intel Xeon Processor (Skylake, IBRS)
Discovered Kernel : 5.4.127-1.el7.elrepo.x86_64
Discovered k8s version : v1.21.7
Discovered MTU : 1500
Idle :
bandwidth = 0 Mbit/s
client cpu = total 3.35% (user 1.56%, nice 0.02%, system 1.70%, iowait 0.07%, steal 0.00%)
server cpu = total 2.45% (user 1.14%, nice 0.09%, system 1.22%, iowait 0.00%, steal 0.00%)
client ram = 1258 MB
server ram = 1194 MB
Pod to pod :
TCP :
bandwidth = 875 Mbit/s
client cpu = total 4.53% (user 1.37%, nice 0.00%, system 3.00%, iowait 0.09%, steal 0.07%)
server cpu = total 7.61% (user 1.49%, nice 0.07%, system 5.98%, iowait 0.02%, steal 0.05%)
client ram = 1250 MB
server ram = 1197 MB
UDP :
bandwidth = 944 Mbit/s
client cpu = total 34.08% (user 4.70%, nice 0.03%, system 28.94%, iowait 0.03%, steal 0.38%)
server cpu = total 18.45% (user 4.81%, nice 0.02%, system 13.11%, iowait 0.02%, steal 0.49%)
client ram = 1245 MB
server ram = 1197 MB
Pod to Service :
TCP :
bandwidth = 931 Mbit/s
client cpu = total 4.01% (user 1.25%, nice 0.05%, system 2.62%, iowait 0.09%, steal 0.00%)
server cpu = total 8.14% (user 1.59%, nice 0.02%, system 6.48%, iowait 0.00%, steal 0.05%)
client ram = 1242 MB
server ram = 1197 MB
UDP :
bandwidth = 896 Mbit/s
client cpu = total 26.61% (user 2.79%, nice 0.02%, system 23.73%, iowait 0.07%, steal 0.00%)
server cpu = total 11.16% (user 3.18%, nice 0.03%, system 7.89%, iowait 0.00%, steal 0.06%)
client ram = 1236 MB
server ram = 1197 MB
=========================================================