swirl
Home Software Blog Wallpapers Webtools
Renewing certificates in a Kubernetes cluster
Monday 06, June 2022   |   Post link

If your Kubernetes cluster has suddenly stopped working, it might be because of expired certificates. I spent almost the entire Sunday afternoon troubleshooting this only because of extremely poor error reporting by Kubernetes services (kubelet) and useless documentation found online.

First what symptoms

The three VMs hosting the Kubernetes cluster started up as usual and to my horror running kubectl gave me:

The connection to the server x.x.x.:6443 was refused - did you specify the right host 
or port

Nice, right?

Initial thoughts

Now this machine has not been touched for 5 days, everything was working fine till I shutdown 5 days back so what could be the problem? Looking at the problem I started troubleshooting kubectl. I moved from kubctl to api-server and finally to the Kubelet service. I noticed that the kubelet service was failing with no information hinting why. I read numerous posts each claiming to fix the problem. I noticed the version of Kubernets was quite out of date: 1.14 when I have 1.21.1. Many of the command mentioned in multiple blog posts were the same or had slight variations of each other like this command (link):

$ sudo kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname)
Command "user" is deprecated, please use the same command under "kubeadm kubeconfig"
required flag(s) "config" not set
To see the stack trace of this error execute with --v=5 or higher

What worked

What worked is a mishmash of steps from multiple blog posts. Some of the steps may even be redundant. However I am placing these steps exactly how I used them hoping they will work for you. Note: do not try this in production and it would be a great idea to take a snapshot of your master node before trying these commands.

Before jumping into copy-pasting the commands

Before jumping right in let's at least be sure the problem is expired certificates.

$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 22, 2022 19:16 UTC                                  no
apiserver                  May 22, 2022 19:16 UTC          ca                      no
apiserver-etcd-client      May 22, 2022 19:16 UTC          etcd-ca                 no
apiserver-kubelet-client   May 22, 2022 19:16 UTC          ca                      no
controller-manager.conf    May 22, 2022 19:16 UTC                                  no
etcd-healthcheck-client    May 22, 2022 19:16 UTC          etcd-ca                 no
etcd-peer                  May 22, 2022 19:16 UTC          etcd-ca                 no
etcd-server                May 22, 2022 19:16 UTC          etcd-ca                 no
front-proxy-client         May 22, 2022 19:16 UTC          front-proxy-ca          no
scheduler.conf             May 22, 2022 19:16 UTC                                  no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      May 20, 2031 19:16 UTC   8y              no
etcd-ca                 May 20, 2031 19:16 UTC   8y              no
front-proxy-ca          May 20, 2031 19:16 UTC   8y              no

As we can see all the certificates expired on May 22 so its quite likely this is what is causing all the problem.

Backing up the old certs and configs

Many of the articles had this steps so here it is:

$ mkdir -p $HOME/k8s-old-certs/pki
$ sudo /bin/cp -p /etc/kubernetes/pki/*.* $HOME/k8s-old-certs/pki
$ sudo /bin/cp -p /etc/kubernetes/*.conf $HOME/k8s-old-certs
$ mkdir -p $HOME/k8s-old-certs/.kube
$ sudo /bin/cp -p ~/.kube/config $HOME/k8s-old-certs/.kube/.

Renewing the certificates

$ sudo kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed 

The certiciate used by kubelet

You'll find four files /var/lib/kubelet/pki/. One of them is kubelet.crt. This file has also expired if we check with openssl:

$ sudo cat /var/lib/kubelet/pki/kubelet.crt | openssl x509 -noout -enddate
notAfter=May 22 18:16:16 2022 GMT

I tried to read about this file but there seems to be a lot of confusion what this file is used for, in fact there is disagreement as to whether this file is even needed at all.

Deleting old certificates

Stopping kubectl was not mentioned in any of the articles but I did it anyway:

$ sudo systemctl stop kubelet
$ sudo rm /etc/kubernetes/kubelet.conf
$ sudo ls /var/lib/kubelet/pki
-rw------- 1 root root 2830 May 22  2021 kubelet-client-2021-05-22-19-16-17.pem
lrwxrwxrwx 1 root root   59 May 22  2021 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-05-22-19-16-17.pem
-rw-r--r-- 1 root root 2266 May 22  2021 kubelet.crt
-rw------- 1 root root 1675 May 22  2021 kubelet.key

$ sudo rm /var/lib/kubelet/pki/kubelet-client-2021-05-22-19-16-17.pem
$ sudo rm /var/lib/kubelet/pki/kubelet-client-current.pem    
$ sudo rm /var/lib/kubelet/pki/kubelet.crt
$ sudo rm /var/lib/kubelet/pki/kubelet.key

One blog post mentioned that simply restarting all Kubernetes services should fix the problem after deleting the above files. Needless to say that did not work.

The silver bullet

This specific command regenerated the kube config file. The only article which mentions this step is this one.

$ sudo kubeadm init phase kubeconfig kubelet
$ sudo systemctl start kubelet

After this step we at least have kubelet service starting up. However on trying to run any kubectl commands, we see:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
error: You must be logged in to the server (the server has asked for the client to provide credentials)    

Now it looks like there is something which is preventing the client (kubectl) talking to the api server - a slight improvement.

Updating the client data configs

This part of the answer came from here. We need to copy two strings from /etc/kubernetes/admin.conf to ~/.kube/config. Replace the strings for "client-certificate-data" and "client-key-data" in ~/.kube/config using values present /etc/kubernetes/admin.conf. Let's now run kubectl for the last time:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

What about the minions?

Running get nodes will results in something like this:

$ kubectl get nodes
NAME         STATUS     ROLES                  AGE    VERSION
k8s-master   Ready      control-plane,master   378d   v1.21.1
k8s-node1    Ready      <none>           378d   v1.21.1
k8s-node2    NotReady   <none>           378d   v1.21.1

We got the master node working with renewed certificates but not the worker nodes. Again this information was not readily available and multiple links, posts and some imagination led to the solution. After trying a lot of stuff, I finally asked myself 'why not just rejoin the cluster?' and that worked. First run the following command on the master node to get the join command:

$ sudo kubeadm token create --print-join-command
kubeadm join 192.168.220.30:6443 --token scc32h.8ycsccrktmjyaruv --discovery-token-ca-cert-hash sha256:4b80f208d7ad0937f0ca3e1b2bc5f02e9b4a7e04d76847e9ce33731fa1ab1224

SSH into each worker node, run the following commands:

$ sudo kubeadm reset
$ kubeadm join 192.168.220.30:6443 --token scc32h.8ycsccrktmjyaruv --discovery-token-ca-cert-hash sha256:4b80f208d7ad0937f0ca3e1b2bc5f02e9b4a7e04d76847e9ce33731fa1ab1224

Replace the join command with the one generated for your system. Running get nodes now prints:

$ kubectl get nodes
NAME         STATUS   ROLES                  AGE    VERSION
k8s-master   Ready    control-plane,master   378d   v1.21.1
k8s-node1    Ready    <none>           378d   v1.21.1
k8s-node2    Ready    <none>           378d   v1.21.1    

Yaay, phew ... I can now start working on what I wanted to 5 hours back!



Categories: Kubernetes (1)

Comments

Posts By Year

2022 (9)
2021 (5)
2020 (12)
2019 (6)
2018 (8)
2017 (11)
2016 (6)
2015 (17)
2014 (2)
2013 (4)
2012 (2)

Posts By Category

.NET (4)
.NET Core (2)
ASP.NET MVC (4)
AWS (4)
AWS API Gateway (1)
Android (1)
Apache Camel (1)
Architecture (1)
Azure (2)
Book review (3)
Business (1)
C# (3)
C++ (2)
CloudHSM (1)
Containers (4)
Corporate culture (1)
Database (3)
Database migration (1)
Desktop (1)
Docker (1)
DotNet (2)
DotNet Core (2)
Entity Framework (3)
Git (2)
IIS (1)
JDBC (1)
Java (8)
Kubernetes (1)
Lambda (1)
Learning (1)
Life (7)
Linux (1)
Lucene (1)
Multi-threading (1)
Music (1)
OData (1)
Office (1)
PHP (1)
Photography (1)
PowerShell (2)
Programming (26)
Rants (5)
SQL (2)
SQL Server (1)
Security (2)
Software Engineering (1)
Software development (1)
Solr (1)
Sql Server (2)
Storage (1)
T-SQL (1)
TDD (1)
TSQL (5)
Tablet (1)
Technology (1)
Test Driven (1)
Unit Testing (1)
Unit Tests (1)
Utilities (3)
VC++ (1)
VMWare (1)
VSCode (1)
Visual Studio (2)
Wallpapers (1)
Web API (2)
Win32 (1)
Windows (9)
XML (2)

Posts By Tags

.NET(6) API Gateway(1) ASP.NET(4) AWS(2) Adults(1) Advertising(1) Android(1) Anti-forgery(1) Asynch(1) Authentication(2) Azure(2) Backup(1) Beliefs(1) BlockingQueue(1) Book review(2) Books(1) Busy(1) C#(4) C++(3) CLR(1) CORS(1) CSRF(1) CTE(1) Callbacks(1) Camel(1) Certificates(1) Checkbox(1) CloudHSM(1) Cmdlet(1) Company culture(1) Complexity(1) Consumer(1) Consumerism(1) Containers(3) Core(2) Custom(2) DPI(1) Data-time(1) Database(4) Debugging(1) Delegates(1) Developer(2) Dockers(2) DotNetCore(3) EF 1.0(1) Earphones(1) Elastic Search(1) Encrypted(1) Entity framework(1) Events(1) File copy(1) File history(1) Font(1) Git(2) HierarchyID(1) IIS(1) Installing(1) Intelli J(1) JDBC(1) JSON(1) JUnit(1) JWT(1) Java(2) JavaScript(1) Kubernetes(1) Life(1) LinkedIn(1) Linux(1) Localization(1) Log4J(1) Log4J2(1) Lucene(1) MVC(4) Management(2) Migration history(1) Mobile Apps(1) Modern Life(1) Money(1) Music(1) NGINX(1) NTFS(1) NUnit(2) OData(1) OPENXML(1) Objects(1) Office(1) OpenCover(1) Organization(1) PHP(1) Paths(1) PowerShell(2) Producer(1) Programming(1) Python(2) Quality(1) REDIS(2) REST(1) Runtimes(1) S3-Select(1) SD card(1) SLF4J(1) SQL(2) SQL Code-first Migration(1) SSH(2) Sattelite assemblies(1) School(1) Secrets Manager(1) Self reliance(1) Shell(1) Solr(1) Sony VAIO(1) Spirituality(1) Sql Express(1) System Image(1) TDD(1) TSQL(3) Table variables(1) Tables(1) Tablet(1) Ubuntu(1) Url rewrite(1) VMWare(1) VSCode(1) Validation(2) VeraCode(1) Wallpaper(1) Wallpapers(1) Web Development(4) Windows(2) Windows 10(2) Windows 2016(2) Windows 8.1(1) Work culture(1) XML(1) Yii(1) renew(1)