Kubernetes Cert Renewal

Posted by Drew Bednar on October 17, 2022
Tags: TLS

Here in the lab is an old Kubernetes 1.19 cluster that I haven't checked on in a while. I lost access to the kube-apiserver, and it turns out to be caused by an expired cert. Here are the steps I used to identify the issue and resolve it.

I Got a Problem

I am actively studying for the Certified Kubernetes Administrator (CKA) and one of the tasks you are required to perform during the exam is an upgrade of a cluster. Perfect, I have a Kubernetes 1.19 cluster from many moons ago. The only problem is I can't seem to connect to it.

$ kubectl get nodes 

The connection to the server 10.0.1.159:6443 was refused - did you specify the right host or port?

Logging into the management node and checking for I errors I find:

$ journalctl -e

Oct 18 01:24:56 kube1 kubelet[696]: E1018 01:24:56.399808     696 kubelet.go:2183] node "kube1" not found

The kube-apiserver runs inside the kube-system namespace. Since this cluster uses Docker as the container runtime I can simply check the logs 

$ docker ps -a | grep kube-api

d5e5415186ff        607331163122           "kube-apiserver --ad…"   4 minutes ago       Exited (1) 4 minutes ago                        k8s_kube-apiserver_kube-apiserver-kube1_kube-system_e62e9c7b79c44695c4f889ce8dccbe97_8245

$ docker logs k8s_kube-apiserver_kube-apiserver-kube1_kube-system_e62e9c7b79c44695c4f889ce8dccbe97_8245

W1018 01:31:47.390824       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-10-18T01:31:47Z is after 2022-09-24T19:19:54Z". Reconnecting...

Well, sure enough it seems like the issue with with the TLS cert.

$ kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:38:53Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Since this is an older version of kubeadm I need to use the alpha flag to check the certs.

$ kubeadm alpha certs check-expiration

W1018 01:38:09.425304  168347 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Sep 24, 2022 19:19 UTC   <invalid>                               no
apiserver                  Sep 24, 2022 19:19 UTC   <invalid>       ca                      no
apiserver-etcd-client      Sep 24, 2022 19:19 UTC   <invalid>       etcd-ca                 no
apiserver-kubelet-client   Sep 24, 2022 19:19 UTC   <invalid>       ca                      no
controller-manager.conf    Sep 24, 2022 19:19 UTC   <invalid>                               no
etcd-healthcheck-client    Sep 24, 2022 19:19 UTC   <invalid>       etcd-ca                 no
etcd-peer                  Sep 24, 2022 19:19 UTC   <invalid>       etcd-ca                 no
etcd-server                Sep 24, 2022 19:19 UTC   <invalid>       etcd-ca                 no
front-proxy-client         Sep 24, 2022 19:19 UTC   <invalid>       front-proxy-ca          no
scheduler.conf             Sep 24, 2022 19:19 UTC   <invalid>                               no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Sep 21, 2030 01:01 UTC   7y              no
etcd-ca                 Sep 21, 2030 01:01 UTC   7y              no
front-proxy-ca          Sep 21, 2030 01:01 UTC   7y              no

You certainly can tell this is not my primary cluster from the fact that this has been non-operational for a couple weeks without me noticing. Anyway, I need to get this back into a working state before I can work on upgrading it.

Sweet New Certs

Now I was going to go into a long diatribe about creating certs and CSRs, but I decided to just take the easy way out.

$ kubeadm alpha certs renew all

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

W1018 20:01:29.055667  899153 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

That's it. That's all you needed. Just remember the version of my kubeadm was old enough to require using the alpha subcommand. Don't worry about the cluster configuration at this time, since the kube-apiserver pod is down. I rebooted the machine, copied the admin.conf to my user's home directory, and checked to make sure I could reach the cluster nodes.

$ cp /etc/kubernetes/admin.conf ~/admin.conf
$ kubectl --kubeconfig admin.conf get nodes

NAME    STATUS   ROLES    AGE     VERSION
kube1   Ready    master   2y25d   v1.19.2
kube2   Ready    <none>   2y25d   v1.19.2
kube3   Ready    <none>   2y25d   v1.19.2

Time To Say Goodbye

Hope this serves you well in the future when you find yourself logging into that old cluster.

Tags: TLS