In this lab, we're going to practice
troubleshooting worker node failures.
We'll wait for the lab environment to be ready
and we'll get started.
While we wait, I'm going to set
the alias for kubectl.
The first question is to fix the broken cluster.
Let's run a kubectl get nodes,
and we see that indeed there is something broken.
Node01 is in a not ready state.
Let's see why that is.
Let's do a describe node01.
Here we see a set of events.
All of these seem to be normal
so they're starting the kubelet.
All of these are good statuses
and the node itself is ready.
There's something missing.
Let's go to that node, ssh node01
and we'll check the status.
The first thing that we want to check on a worker node is the kubelet,
because as we know, the kubelet is the controller of a node,
and the kubelet is what helps in communication
between a worker node and the master or the control plane node.
Let's do the service of the kubelet.
Service kublet status,
and we see that it's loaded, but it's in an inactive state.
We don't know why it's an inactive state.
We're going to just try and start it.
Let's do a service kubelet start
and let's check the status again.
We see that it's now in an active and running state.
Let's go back to the control plane
and let's do a kubectl get nodes.
We see that both are in a ready state.
I think that should fix that question.
Let's go to the next one.
The cluster is broken again, investigate and fix the issue.
Let's check the status again.
We see that is again in a not ready state.
Let's check the status of the node,
node01
and put a full screen.
We see that
all of these events are normal, so that's okay.
There's nothing here either.
Let's go to the node
and check the status of the kubelet again,
service kubelet status,
and we see that it's in an activating state
and it looks like there is an exit code
and it's exited.
The status is 255.
That means there is something wrong in the process,
it's not able to start.
Even if we do a service kubelet start,
it's not actually going to start the service
because there seems to be something wrong.
Let's check the status of the service again,
and it's in an activating state.
To look at the logs of a service,
do a journalctl -u kubelet
and check the status.
These are the logs and let's look for the most recent logs.
We are here.
Let's go up.
All of these seem to be an error from a previous error.
All of these seem to be an error message
and here you can see that there's an error that says
unable to load client CA file /etc/kubernetes/pki/WRONG-CA-FILE.
There seems to be a wrong file.
Let's try and find that out.
The kubelet configuration files
are one in the /etc/kubernetes/
kubelet.conf.
That's one file,
but this is basically the kube config file
used by the kubelet.
Here you have the certificate,
the context, and everything that the kubelet uses
to connect to the kube-apiserver.
This is not the place where there's an issue.
It's basically the kubelet service itself.
The kubelet service picks the options
from a file located at var/lib/kubelet.
This is something to remember.
Here you have a file called config.yaml,
so var/lib/kubelet/config.yaml.
Within this file, there are some properties
that are passed through to the kubelet.
Here you can see there is this WRONG-CA-FILE right here,
and that's the clientCAfile.
It should be the correct file.
Let's find what the correct file is.
There's this ca.crt file.
That's what it should be,
ca.crt instead of the WRONG-CA-FILE.
Let's edit that.
Let me put the ca.crt.
Save that.
Now we're going to restart the kubelet service.
Let's just check the status of the service now.
We see that it's in an active running state.
Let's go back.
Let's go back to the control plane node
and let's check the status of the nodes.
We see that they're in a ready state.
Let's go to the next one.
Now the cluster is broken again, so let's check.
We see that, yes, it is indeed broken and node01 is not ready.
Let's go to node01
and let's check the status of the kubelet.
We see that it's active and it's in a running state.
Let's check the logs,
journalctl -u kubelet.
Scroll all the way to the end
and let's work our way backwards.
Let's see if there's something useful here.
Here you can see, unable to register node.
You can see the control plane
and dial TCP 10:54:130:2:6553, connection refused.
This seems to be the error that's causing the remaining errors.
If you look at it, it says connection refused to this.
This is basically for the control plane that we have.
It looks like this port number is incorrect
because we know that the control plane port number is 6443 and not 6553.
That could be the error.
Let's take a look at the kube config file used by the kubelet.
We saw that the cube config file used by the kubelet
is etc/kubernetes/kubelet.conf.
Let's take a look at the contents of the file.
Here if you look at this, you see that right here,
the control plane port number is incorrect.
Let's get that corrected.
Let's set it to 6443.
We're going to do a service kubelet restart
and service kubelet status.
You see that it's now in an active state.
Let's check the logs as well.
We no longer see those errors,
and it seems to be good.
Let's go back
and let's check the status of the nodes.
We see that it's now ready.
Let's check our work.
That's the end of this lab.
When it comes to troubleshooting worker nodes,
always first check the status from here, from the control plane
and check whatever you can find from here,
and then go to each of the worker node that has the issue
and start with checking the kubelet service.
If the kubelet service is good, check the logs of the kubelet service
and continue your way from there.
That's all for this lab.
Thank you.