kubeadm
This article mainly introduces how DOK sets up kubeadm’s configuration file, and creates a cluster through kubeadm. In addition, it should be noted that the environment variable DOK_KUBEADM_CONFIG can be defined on the execution machine to point to the custom configuration file, because /root/dok-release/bin/k8s/dok.yaml is the default value, so if there is no such environment variable, DOK will read the environment variable according to the custom address.
kubeadm init --upload-certs --config "${DOK_KUBEADM_CONFIG-/root/dok-release/bin/k8s/dok.yaml}"
The kubeadm used by DOK is not the open source kubeadm of the community. The main point of modification is about the certificate. The kubeadm certificate of the community is 1 year as default. In order to expand related application services, it is modified to 100 years with the principle of minimum, which would reduce operation and maintenance in the future. Problems that may be caused when switching and replacing certificates. In order to ensure that the cluster created by DOK uses the modified kubeadm, a mandatory check will be performed during the installation process to prevent kubeadm from using open source when the user recreates the installation package version.
# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 01, 2023 07:13 UTC 364d no
apiserver Aug 01, 2023 07:13 UTC 364d ca no
apiserver-etcd-client Aug 01, 2023 07:13 UTC 364d etcd-ca no
apiserver-kubelet-client Aug 01, 2023 07:13 UTC 364d ca no
controller-manager.conf Aug 01, 2023 07:13 UTC 364d no
etcd-healthcheck-client Aug 01, 2023 07:13 UTC 364d etcd-ca no
etcd-peer Aug 01, 2023 07:13 UTC 364d etcd-ca no
etcd-server Aug 01, 2023 07:13 UTC 364d etcd-ca no
front-proxy-client Aug 01, 2023 07:13 UTC 364d front-proxy-ca no
scheduler.conf Aug 01, 2023 07:13 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 29, 2032 07:12 UTC 9y no
etcd-ca Jul 29, 2032 07:12 UTC 9y no
front-proxy-ca Jul 29, 2032 07:12 UTC 9y no
diff -r kubernetes/cmd/kubeadm/app/constants/constants.go kubernetes-1.21.7/cmd/kubeadm/app/constants/constants.go
49,50c49
< // CertificateValidity = time.Hour * 24 * 365
< CertificateValidity = time.Hour * 24 * 365 * 100
---
> CertificateValidity = time.Hour * 24 * 365
diff -r kubernetes/staging/src/k8s.io/client-go/util/cert/cert.go kubernetes-1.21.7/staging/src/k8s.io/client-go/util/cert/cert.go
66,67c66
< // NotAfter: now.Add(duration365d * 10).UTC(),
< NotAfter: now.Add(duration365d * 100).UTC(),
---
> NotAfter: now.Add(duration365d * 10).UTC(),
diff -r kubernetes/vendor/k8s.io/client-go/util/cert/cert.go kubernetes-1.21.7/vendor/k8s.io/client-go/util/cert/cert.go
66,67c66
< // NotAfter: now.Add(duration365d * 10).UTC(),
< NotAfter: now.Add(duration365d * 100).UTC(),
---
> NotAfter: now.Add(duration365d * 10).UTC(),
After modifying the code related to the certificate, execute this command in the Kubernetes warehouse to compile and package normally build/run.sh make kubeadm KUBE_BUILD_PLATFORMS=linux/amd64, after recompiling, it is best to check the version again and confirm the kubeadm compilation time.
# _output/dockerized/bin/linux/amd64/kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7", GitCommit:"1f86634ff08f37e54e8bfcd86bc90b61c98f84d4", GitTreeState:"archive", BuildDate:"2022-07-04T05:37:38Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
After the cluster is finally created, check the certificate period. By default, kubeadm in the DOK installation package supports 100-year certificates.
# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 11, 2122 01:19 UTC 99y no
apiserver Jul 11, 2122 01:18 UTC 99y ca no
apiserver-etcd-client Jul 11, 2122 01:19 UTC 99y etcd-ca no
apiserver-kubelet-client Jul 11, 2122 01:18 UTC 99y ca no
controller-manager.conf Jul 11, 2122 01:19 UTC 99y no
etcd-healthcheck-client Jul 11, 2122 01:19 UTC 99y etcd-ca no
etcd-peer Jul 11, 2122 01:19 UTC 99y etcd-ca no
etcd-server Jul 11, 2122 01:19 UTC 99y etcd-ca no
front-proxy-client Jul 11, 2122 01:18 UTC 99y front-proxy-ca no
scheduler.conf Jul 11, 2122 01:19 UTC 99y no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 11, 2122 01:18 UTC 99y no
etcd-ca Jul 11, 2122 01:18 UTC 99y no
front-proxy-ca Jul 11, 2122 01:18 UTC 99y no
DOK is created through kubeadm. In fact, the command kubeadm init --upload-certs --config /root/dok-release/bin/k8s/dok.yaml is executed. Therefore, for kubeadm configuration problems, you can check The file /root/dok-release/bin/k8s/dok.yaml, if you have other requirements for the configuration of Kubernetes components, you can edit the above file after downloading the DOK installation package, then repackage and compress, and create an installation The md5 file of the compressed package is enough.
EphemeralContainers=true is mainly for kubectl debug.
apiServer:
timeoutForControlPlane: 4m0s
extraArgs:
feature-gates: TTLAfterFinished=true,EphemeralContainers=true
extraVolumes:
- name: localtime
hostPath: /etc/localtime
mountPath: /etc/localtime
readOnly: true
pathType: File
certSANs:
- "127.0.0.1"
- "localhost"
controllerManager:
extraArgs:
bind-address: 0.0.0.0
feature-gates: TTLAfterFinished=true,EphemeralContainers=true
experimental-cluster-signing-duration: 876000h
extraVolumes:
- hostPath: /etc/localtime
mountPath: /etc/localtime
name: localtime
readOnly: true
pathType: File
scheduler:
extraArgs:
bind-address: 0.0.0.0
feature-gates: TTLAfterFinished=true,EphemeralContainers=true
extraVolumes:
- hostPath: /etc/localtime
mountPath: /etc/localtime
name: localtime
readOnly: true
pathType: File
For ipvs mode.
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
metricsBindAddress: "0.0.0.0:10249"
metrics would be exposed by HTTP.
etcd:
local:
extraArgs:
listen-metrics-urls: "http://0.0.0.0:2381"