In this video, we're going to look at the solution
for the lab on viewing certificate details.
The first question is to identify the certificate
file used for the kube-api server.
Let's look at the manifest files for the kube-api server.
We know that they are under
/etc/kubernetes/manifests,
and we have the kube-api server.
If you look at this file,
we know that we need to look
for the certificate file used for the kube-api server.
These are the files you connect to the etcd server.
These are the kubelet and the proxy.
The certificate for the kube-api server itself are these.
We have the certificate file here and the key here,
so /etc/kubernetes/pki/apiserver.crt.
Pki/apiserver.crt.
That's this one.
Identify the certificate file used to authenticate
kube-api server as a client to etcd server.
This is the kube-api server configuration,
and kube-api server connecting to etcd server
is connecting to etcd server as a client.
These are the relevant files.
This is the address of the etcd server and then you have the cafile,
the certificate file, and the key.
Here, we're asked to identify the certificate file
used to authenticate the kube-api server to etcd.
The cert file is this.
That's /etc/kuberntes/pki/apiserver-etcd-client.crt.
Etcd-client.crt.
Looks like it's this one.
Identify the key used to authenticate
the kube-api server to the kubelet server.
These are the files that are relevant to the kubelet
and you have the certificate file here and then the key.
That's /etc/kubernetes/pki/apiserver-kubelet-client.
Apiserver-kubelet-client.
/etc/kubernetes/pki.
We're asked to identify the key.
It's not the crt.
It's the one that has key at the end.
Just to summarize here,
these are the files used by the kube-api server itself to host itself.
That's under /etc/kubernetes/pki,
and then you have the apiserver.crt and apiserver.key.
Everything for connecting to the etcd server,
the cafile is under etcd,
but then the apiserver-client files are still under the pki directory.
Then here to the kubelet-client files,
they're still under the pki directory.
That's what we are concerned about.
If you look at it, anything to do with the kubernetes-apiserver,
they're under the pki directory.
Either the apiserver certificate and key itself or the client files,
which are used to connect to either etcd or kubelet,
they're all under /etc/kubernetes/pki under the pki directory.
Think about it that way.
Now, we're asked to identify the etcd server
certificate used to host the etcd server itself.
For this, we're going to look at the etcd server manifest file.
Here, we have the certificate file.
It is under the etcd directory.
The key is also under the etcd directory.
Then you have the cafile.
It's also under the etcd directory.
Anything to do with the etcd server itself for its own
hosting is under
the /etc/kubernetes/pki/etcd directory.
In this case, we are asked to identify the etcd server certificate.
That's this certificate.
That's /etc/kubernetes/pki/etcd/server.crt.
That's /etc/kubernetes/pki.
There's no etcd here.
/etc/kubernetes/pki/etcd/server.crt.
That's this one.
Now, identify the etcd server CA Root certificate
used to serve the etcd server.
Here, we have the CA file here, which is a trusted CA file.
This is under the kubernetes/pki/etcd/ directory.
As we discussed, in this case, the etcd server has its own CA file,
trusted CA Root file which is under the etcd directory.
The kube-api server has its own which is in a different directory.
Let's keep that in mind.
This is under /etc/kubernetes/pki/etcd/ca.crt.
It's etcd/ca.crt.
It's this one.
What is the common name configured on the kube-api server certificate?
We know that kube-api server cerificate is this one.
We run that through the openssl x509,
filename and text, noout.
That command is going to give us this information.
If you look inside here,
you have information about the certificate.
The issuer is Kubernetes.
The subject which is what this certificate is issued to
is the command name for that is kube-api server.
That's kube-api server.
That's the answer to this question.
That's kube-api server.
Now, what is the name of this CA
who issued the kube-api server certificate?
If you look at here, you have the issuer and the name is Kubernetes.
Kubernetes is the CA
who issued the kube-api server certificate.
Now, which of the below alternate names
is not configured on the kube-api server?
As we discussed, the kube-api server has multiple names.
It could be controlled name kubernetes, kubernetes.default,
kubernetes.default.svc, kubernetes.default.svc.cluster.local,
and this IP address is as well.
Kubernetes, yes.
Controlplane, yes.
Kubernetes.default.svcs, yes.
Kubemaser, no.
There is no kubemaser.
In this case, that's the answer to that question.
Now, what is the common name configured on the etcd server certificate?
Let's look at the etcd server certificate, and that is this one.
Let's run it through the same command.
We're just going to replace it with the etcd server certificate.
Here we can see the issuer is etcd-ca.
Etcd has its own CA.
That's what we need to understand from here.
If you remember, kubernetes had the CA Kubernetes,
but here you have anther CA.
The common name configured on the etcd server certificate,
if you look at this common name for the etcd server is controlplane.
This is not the controlplane as this.
This is not the kubernetes controlplane.
This is referring to the etcd controlplane, in fact.
It's like that.
Now, how long from the issued date
is the kube-api server certificate valid for?
We have the output for the apiserver here.
If you look at the validity, it's not valid before.
This is basically the starting.
This is April 17, 2022.
It's valid until April 17, 2023.
That's one year.
How long from the issued date is the Root CA certificate valid for?
The Root CA certificate what we know is here.
Let's run the same command.
This will fnd this.
If you look at the validity, it's April 17, 2022
to April 14, 2032.
That's approximately just 10 years.
Let's start that.
Going to a change window, there are some changes happening.
Kubectl suddenly stops responding to your command after a change window.
Check it out.
Someone recently modified this pretty little file.
You're asked to investigate and fix the issue.
Once you fix the issue, wait for some time for kubectl to respond.
Let's see that.
Let's do a ctl get pods,
and let's see what's happening.
This might take some time.
Okay. It says, "The connection to the server was refused.
Did you specify the right host or port?"
Apparently none of the Kubernetes commands are working.
If this happens, this means that the kubectl utility
is not able to communicate with the kube-api server.
We need to look at the status of the kube-api server,
but you can't do that with the kubectl command,
because the kubectl command cannot communicate
with the kube-api server.
We have to then use Docker,
because we know that these pods are in fact
running as containers on the host Docker container.
Let's do a docker ps and then grep for kube-api server
and we see that we have the kube-api server
and we have a kube-api server that exited at 27 seconds ago.
Then we have a new kube-api server,
so let's quickly check the logs of that container.
We see that it has this error.
Error while dialing tcp 127.0.0.1:2379.
Now we know that 2379 is a port of etcd.
This is the etcd server port.
Basically, the kube-api server is not able to connect to the etcd server.
Let's look at the etcd server then.
Let's do a docker ps -a
and let's look for etcd.
We have the etcd
about here.
If you look at this, let's see if we can find--
Okay. Right here it says,
"etcdmain cannot find
/etc/kubernetes/pki/etcd/server-certificate.crt,
it's not found."
Let's see where that it, that's
/etc/kubernetes/pki/etcd.
If you look at this directory, there is server.crt,
but there is no server-certificate.crt, so that's the error.
Something has changed in the etcd server,
so let's look at the etcd
manifest files.
That's /etc/kubernetes/manifest/etcd.yaml
and let's see
where we have this configured, which is server-certificate.crt.
Someone's put in the wrong path to the certificate here.
That's the crt file and that's the issue, so we have to fix this.
The crt file for etcd as we know
and we have seen earlier is the server.crt file.
It's not server-certificate, it's just server.crt.
We're going to modify this.
Let's go in, edit this file,
and let's remove this.
Okay, let's save that
and let's now give it some time to start up.
Let's look at
etcd again.
See that it's powered up now, a new container.
Let's check the logs of that container
and see that the error is no longer there.
Let's see if--
And it's still not working so we might have to give
it some time for the kube-api server to pick this up.
Let's check the kube-api server.
We'll check the logs.
This is the previous instance of the kube-api server,
so we just have to wait for a new instance to come up.
Okay.
Looks like there's there's a new container.
Let's check the logs of that container.
Let's do a local logs,
and we don't see arrows anymore.
Let's check.
Yes. Okay.
Now we have the kubectl utility back
and the node is now ready?
Okay, let's give that a shot.
Okay. That's success.
After a few hours.
I believe someone's going to break something again.
We have the kube-api server, and it stopped again, check it out.
Let's check that out.
Yes, it looks like that's stopped again.
Now, inspect the kube-api server logs
and identify the root cause and fix the issue.
Let's look at the logs again.
Okay. This time, it says,
"Unable to connect to the server TLS handshake timeout."
It looks like there's some issue with the TLS certificates.
Let's check that again.
Let's check the kube-api server again.
Docker, ps-a, grep kube-api server.
You see that succeeded.
Let's do docker logs.
Here it says, authentication handshake failed,
certificate signed by unknown authority.
Let's see if there's something else.
Okay, that's the error.
Also, here, you have some information
that it's failed to connect to 127.0.0.1:2379.
We know that 2379 is etcd server,
so it's got something to do with the etcd server.
Basically, the kube-api server is unable to connect to the etcd server,
because of a handshake issue.
It's because the certificate is signed by unknown authorities.
Let's check that out.
Let's first look at the--
Let's delete pod and see if that has any issues.
If you look at this, the etcd container has been running for four minutes
so let's check the logs of that container.
We see that it started--
Well, there were no errors,
but then started getting these rejected connection from,
and this is a remote and it's a bad certificate.
It's rejecting connections from somewhere.
We assume that that's the kube-api server
trying to connect to that etcd server
and that's because it is sending in a bad certificate.
Okay, let's try to look at the certificates
configured on the api server under manifests,
kube-api server.
We're basically focusing on the etcd side
of things so let's just look at those alone.
Okay.
This is what we need to look at.
Here we have the etcd-ca file and that's set to--
It is in kubernetes/pki/ca.crt and then you have the etcd cert file.
This is the client certificate for api server
so that apiserver-etcd-client.crt set correctly.
This is a key file, again,
this is the api server-etcd-client.key file which is right
and the etcd servers is set to the correct URL as well.
As we discussed earlier, the Kubernetes has its own CA file which is this
and etcd has its own CA file which is not this.
Right, so etcd has a CA file, which is under
kubernetes/pki/etcd,
and then the CA file that's here.
Right. This is the CA file for etcd.
Here you should be specifying that and not the Kubernetes CA file.
That seems to be the problem.
Lets go in and change.
You can just add this here like this.
Lets check the status of kube-api server.
It started eight seconds ago.
Lets check the logs.
We don't see those errors any more.
These are all informational.
Lets check if kube-api server is back up.
Yes, it seems to be back up.
That was the problem.
We looked at the logs on the kube-api server.
We looked at the logs on the etcd server and the continues for both.
We realize that it was a certificate issue somewhere,
so we went through and looked at the certificate files
configured on each of them and finally figured out it was this.
That was a bit tricky,
but that's just something you have to keep in mind.
Well, that's all for this lab.