Here in the lab is an old Kubernetes 1.19 cluster that I haven't checked on in a while. I lost access to the kube-apiserver, and it turns out to be caused by an expired cert. Here are the steps I used to identify the issue and resolve it.
I Got a Problem
I am actively studying for the Certified Kubernetes Administrator (CKA) and one of the tasks you are required to perform during the exam is an upgrade of a cluster. Perfect, I have a Kubernetes 1.19 cluster from many moons ago. The only problem is I can't seem to connect to it.
$ kubectl get nodes
The connection to the server 10.0.1.159:6443 was refused - did you specify the right host or port?
Logging into the management node and checking for I errors I find:
$ journalctl -e
Oct 18 01:24:56 kube1 kubelet[696]: E1018 01:24:56.399808 696 kubelet.go:2183] node "kube1" not found
The kube-apiserver runs inside the kube-system namespace. Since this cluster uses Docker as the container runtime I
can simply check the logs
$ docker ps -a | grep kube-api
d5e5415186ff 607331163122 "kube-apiserver --ad…" 4 minutes ago Exited (1) 4 minutes ago k8s_kube-apiserver_kube-apiserver-kube1_kube-system_e62e9c7b79c44695c4f889ce8dccbe97_8245
$ docker logs k8s_kube-apiserver_kube-apiserver-kube1_kube-system_e62e9c7b79c44695c4f889ce8dccbe97_8245
W1018 01:31:47.390824 1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-10-18T01:31:47Z is after 2022-09-24T19:19:54Z". Reconnecting...
Well, sure enough it seems like the issue with with the TLS cert.
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:38:53Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Since this is an older version of kubeadm I need to use the alpha flag to check the certs.
$ kubeadm alpha certs check-expiration
W1018 01:38:09.425304 168347 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Sep 24, 2022 19:19 UTC <invalid> no
apiserver Sep 24, 2022 19:19 UTC <invalid> ca no
apiserver-etcd-client Sep 24, 2022 19:19 UTC <invalid> etcd-ca no
apiserver-kubelet-client Sep 24, 2022 19:19 UTC <invalid> ca no
controller-manager.conf Sep 24, 2022 19:19 UTC <invalid> no
etcd-healthcheck-client Sep 24, 2022 19:19 UTC <invalid> etcd-ca no
etcd-peer Sep 24, 2022 19:19 UTC <invalid> etcd-ca no
etcd-server Sep 24, 2022 19:19 UTC <invalid> etcd-ca no
front-proxy-client Sep 24, 2022 19:19 UTC <invalid> front-proxy-ca no
scheduler.conf Sep 24, 2022 19:19 UTC <invalid> no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Sep 21, 2030 01:01 UTC 7y no
etcd-ca Sep 21, 2030 01:01 UTC 7y no
front-proxy-ca Sep 21, 2030 01:01 UTC 7y no
You certainly can tell this is not my primary cluster from the fact that this has been non-operational for a couple weeks without me noticing. Anyway, I need to get this back into a working state before I can work on upgrading it.
Sweet New Certs
Now I was going to go into a long diatribe about creating certs and CSRs, but I decided to just take the easy way out.
$ kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
W1018 20:01:29.055667 899153 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
That's it. That's all you needed. Just remember the version of my kubeadm was old enough to require using the alpha
subcommand. Don't worry about the cluster configuration at this time, since the kube-apiserver pod is down. I rebooted
the machine, copied the admin.conf to my user's home directory, and checked to make sure I could reach the cluster nodes.
$ cp /etc/kubernetes/admin.conf ~/admin.conf
$ kubectl --kubeconfig admin.conf get nodes
NAME STATUS ROLES AGE VERSION
kube1 Ready master 2y25d v1.19.2
kube2 Ready <none> 2y25d v1.19.2
kube3 Ready <none> 2y25d v1.19.2
Time To Say Goodbye
Hope this serves you well in the future when you find yourself logging into that old cluster.