In this lecture,
we will see different ways of
troubleshooting worker node failures.
Again,
we start by checking the status
of the nodes in the cluster,
are they reported as ready or not ready?
If they are reported as not ready,
check details about the nodes
using the kubectl describe node command.
Each node has a set of conditions
that can point us in a direction
as to why a node might have failed.
Depending on the status,
they're either set to true or false or unknown.
When the node is out of disk space,
the OutOfDisk flag is set to true.
When a node is out of memory,
the MemoryPressure flag is set to true.
When the disk capacity is low,
the DiskPressure flag is set to true.
Similarly, when there are too many processes,
the PIDPressure flag is set to true.
Finally, if the node as a whole is healthy,
the Ready flag is set to true.
When a worker node stops communicating with the master,
maybe due to a crash,
these statuses are set to unknown.
This can indicate a possible loss of a node.
Check the LastHeartbeatTime field
to find out
the time when the node might have crashed.
In such cases,
proceed to checking the status of the node itself.
If the node is online at all or is crashed.
If it's crashed, bring it back up.
Check for possible CPU memory and disk space on the nodes.
Check the status of the kubelet.
Check the kubelet logs for possible issues.
Check the kubelet certificates,
ensure they're not expired
and they are part of the right group
and that the certificates are issued
by the right CA.
That's it for this lecture,
head over to the practice test,
and practice fixing broken clusters.