In this video, we're going to work on troubleshooting controlplane failure.
The first question is, the cluster is broken.
We tried to play an application, but it's not working.
Troubleshoot and fix the issue.
Before we begin, a quick tip.
If you don't have the alias set already on your prompt.
You could do a alias k=kubectl.
Also, if you don't have auto-completion, then one thing you could do
is go to the Kubernetes documentation pages.
Search for kubectl cheat sheet.
The first thing that comes here is the kubectl completion bash.
Let's copy and let's run that.
This basically gives you auto-completion.
You do a kubectl get and then it automatically gives you
the options.
The other thing is, if you're setting an alias,
you should do this as well to get the alias
for the short form for kubectl.
I'm going to set this as well.
That way I can just do kubectl get
and then autocompletion
for the short form.
We're going to look at the status of the cluster.
Let's do a kubectl get nodes.
We see that the node seems to be in a ready state.
That's good.
It says there are deployments, so let's take a look at the deployments.
We see that there is a app deployed, but it's not ready.
There is one container or one pod in the deployment,
but that pod is not ready.
Let's take a look at it.
Let's do
a describe deployment app.
We see that there's one desired, zero available.
There's one unavailable.
There isn't anything here that's really helpful.
Here, it says scaled up replica set to one.
Let's take a look at the replica set.
get rs.
We have one replica set.
Let's take a look at that.
describe replica set app.
We see that the desired is one,
but it's still waiting.
Here, you have the events, and it says creating pod.
Let's take a look at the pod.
Now, we see that there is a pod created, but it's in a pending state.
Let's take a closer look at that pod.
We see the events have not started yet, but everything else seems okay,
but it's in a pending state.
When a pod is in a pending it's most likely that that indicates
that it has not been assigned a node.
If you look at the node here, it's set to none.
That means the pod has not been assigned to a node.
The job of assigning a pod to a node is a scheduler's job.
We have to look at the status of the scheduler.
We know that a scheduler runs as a pod in the kube-system namespace.
Here, you have the scheduler and it has control plane at the end.
That means it's a static pod.
Its status is CrashLoopBackOff.
Let's take a look at the pod in more detail
in the kube-system namespace.
We're looking at the kube scheduler.
We see at the bottom here that seems to be restarting quite often,
and then it says failed to start container kube-scheduler,
error response from daemon: OCI runtime create failed.
Starting container process, kube-scheduler: executable file not found.
It says this executable is not found.
Obviously, it looks like it's not right.
Let's take a look at the options or the command that are used
to run this pod.
Here, we can see that there is an incorrect command that's run,
and that's the reason.
We know that the kube scheduler is a static pod,
so the file is in /etc/kubernetes/manifests,
and we have a kube-scheduler.yaml file here.
This is the manifest file for kube scheduler.
Within that, there is a run command.
Let's go and edit that file
and then we remove the extra characters,
and we're going to save that.
We'll see the status of the pod,
and it's in
a CreatingContainerConfigError state.
It's now in a running state,
and we'll just watch until it's ready.
Let's give it a few seconds.
Still not ready.
Let's take a look at the logs.
It's finally in a ready state.
Let's take a look at the pods, and we have the pod is now
in a ready state.
Let's take a look at the deployment, and we have one of one ready.
That's good.
Let's go to the next one.
Now, it says scale the deployment app to two pods.
Let's do a kubectl.
Let's get deploy,
and we have app with one pod.
Let's scale it up.
kubectl scale deployment app
and replicas is two.
There's two pods.
Even though the deployment was scaled to two,
the number of pods does not seem to increase.
Investigate and fix the issue.
Let's check the status of pods.
Now, we see that it's still just one.
We have the deployment here, it says one of two ready.
Let's take a look at the deployment.
kubectl describe deploy app,
and we see the image set.
We see scaled up replica set to one, but we don't see it,
the deployment scaling up the replica set to two.
However, here we see the number of replica to be two.
Two is the desired state,
and only one is available.
The job of updating a deployment, the job of updating a replica set
or scaling up a replica set is the controller managers.
It's the deployment controller and all the controllers as we know,
the deployment controller, the replica set controller,
they're all controlled by the controller manager.
Let's take a look at the status of the controller manager.
We're going to do a kube-system, and we see that the kube-controller-manager
is in a CrashLoopBackOff state.
Let's take a look at the kube-controller-manager.
Within the new kube-system,
we're going to describe this particular pod.
We see that it says Back-Off restarting failed container.
We don't see anything useful here in the events.
Let's take a look at it.
Here it says, CrashLoopBackOff, Terminated, Error.
The exit code is one,
and the command seems to be correct.
There isn't anything added there.
Let's take a look at the logs.
Let's do kubectl logs.
This one in the kube-system namespace.
Here it says, this particular file, there's no such file or directory.
This file is not found.
Let's check if that file exists.
Of course, that file does not exist because there seems
to be something wrong with it.
Let's check where this file is specified in the kube-controller-manager.
We know that the kube-controller-manager is also a static pod.
That means it must be
in the cat /etc/kubernetes/manifests directory,
and then we have controller-manager there.
Let's see.
It's just a grep for XXX because that's what we saw.
Here we have it specified.
The kubeconfig file is configured to this location,
but let's see
where the actual kube-config file is.
The actual kubeconfig file for the controller manager
is at /etc/kubernetes/controller-manager.conf.
Let's just verify the contents of that file.
Let's do a controller-manager.
We see that this is indeed the kubeconfig file.
That's the problem.
We're going to edit
the controller-manager manifest file
to fix that issue.
We're going to remove this XXXX
that is not required and save that,
and we'll check the pod system.
It's in a pending state.
We're going to watch that, and we'll give you some time
for the kube-controller-manager to be ready.
It's now in a running state and it's also already.
Let's take a look at the pods, and we see that there are now two pods indeed.
Let's take a look at the deployments, and we see that there are two deployments.
Let's check the status.
We're breaking something now.
Something is wrong with scaling again.
We just tried scaling the deployment to three replicas, but it's not happening.
Let's take a look at that.
Let's do a get pods.
It's only two.
Let's do a get deployment and it's two of three.
It looks like it's gone to three.
Let's check.
Let's look at the deployment.
We see that here we have scaled up the replica set to one, then two,
which we did earlier, but it has not been scaled up to three.
In the previous one, we've learned that if the scaling up isn't happening,
if the changes that we've applied on the deployment isn't going into effect,
then the culprit is usually the controller manager
that is responsible for this.
Let's take a look at the controller manager again.
We're going to look at the logs of this pod,
controller manager.
We're going to do a kubectl logs.
This is the controlplane, controller-manager,
and the kube-system namespace.
We see that it says unable to load client CA file.
The etc/kubernetes/pki/ca.crt is not available.
Let's try and find this file locally.
We see that we do have this file locally.
Yes, this file seems to be here, but it says it's not able to load it.
There's no such file or directory within the kube-controller-manager.
We know that the way that the kube-controller-manager
or any other controlplane component is set up is that the certificate files
are on the controlplane, on the host, and then we use volumes
to mount these directories within the same directory
within the controller manager.
That's set up in the manifest file.
Let's take a look at the manifest file.
We're going to look at kube-controller-manager.
We see that all of these that are configured here,
so etc/kubernetes and etc/kubernetes/pki.
These are all in fact on the host, but they're mapped as a volume through
to these containers.
The way that it's done is through volumeMounts.
We look at volumeMounts here.
We can see there are multiple volumeMounts.
You have /etc/ssl/certs, the CA certs, but what we want is the Kubernetes
and pki a volume.
This is what we are concerned about, because this is what it claims
that is not available.
Here, the mounting looks good, but the volume that is mounted
is called k8s certs.
Let's take a look at the k8s certs volume.
These are the volumes that are configured here.
Here we have the k8s certs volume, and this is the section for that.
Within this, we have the path which is given at the top.
Here, you can see that there's an incorrect path given.
It says WRONG-PKI-DIRECTORY.
We're going to have to get rid of that and add the right pki directory
which is just pki.
Just going to save that.
Let's take a look at the pods' status now.
It seems to be in a running state.
Let's watch it and wait
until it is back up and running.
It's now in a running state.
Let's take a look at the status of pods.
We see that we have three pods indeed.
If we take a look at the status of the deployment,
we see that we have three out of three ready.
That's the end of this lab.