Hi. Let's go through this practice test on troubleshooting
and application failure.
The first question is a simple two-tier
application is deployed in the alpha namespace,
it must display a green webpage on success.
Click on the App tab at the top of your terminal to view your application.
I can click here to open the application
and I see that it is indeed in a failed state.
If I look at the error message, here,
we can see some of the environment variables,
which we can use while troubleshooting but it says,
"Can't connect to MySQL server on MySQL service:3306,
name does not resolve."
Basically, it looks like this web application is unable to reach
the MySQL service because there's some issue with the name,
is not able to resolve it.
Let's look at this architecture diagram.
Here, so it's a two-tier application.
You have the web service here, and you have the database service here
and then you have the user accessing it.
We know that this is most likely a deployment or a pod,
the web application is on 8080
and then you have this web service, which is on 8080
and then this is the node port that the users are accessing,
which is 30081 and then you have the MySQL service,
which is a service that the web application user
tries to connect to and this is a MySQL pod.
Now looking at this state, so we know that the web application loads,
it's failed, the application is failed, but at least the interface loads.
Looking at it, this area seems to be okay because this port 30081,
this is the port that we are on, 30081.
That's good.
This patch looks okay,
this web service has loaded the application.
This also looks okay.
There's some issue in this area,
the application is unable to reach the MySQL service
at this name or at this port.
Let's first take a look at the application.
It's in the Alpha namespace.
We're going to do a kubectl, we'll get pods-namespace alpha.
We're going to run a lot of commands in the alpha namespace.
It's best if we switch our context
and set the alpha namespace as the default namespace
because we don't want to be typing in the namespace for each command.
Let's do a kubectl config and let's check the help,
find out the command.
We can use a set context.
We're going to do a set context and then let's check the help.
Now, what we want to do is, we want to set the namespace.
We're going to use the set context
and then the current context.
We want to set the namespace of the current context
to the namespace equals alpha.
Okay, so that's nullified.
Let's try because we'll just get pods now
and we'll see that we can see the pods in the current namespace.
As we suspected, the web application is a deployment.
Let's do a deploy.
The web app is a deployment and the SQL Server is a pod
and we also have services
and so the SQL Server has the MySQL service
and the web application has the web service.
There's another way to test if the web application is accessible or not,
is basically at this level of test is to do a curl
and then we're going to go to localhost
because it's accessible on the node port, and the port is 30081.
We can see this in a text format,
and we can see that database connection has failed,
and can't connect to MySQL Server on MySQL service.
We're going to look more into this area.
Let's look at the deployment.
We're going to do a kubectl describe deployment webapp
MySQL, so that's the deployment name.
Here we see that we have the pod template.
The image is simple web app MySQL, the port is 8080
and these are some of the environment variables.
The database host is MySQL service, the user is root,
and the password is this password.
The host is MySQL service.
Let's look at the service called MySQL service.
Let's do a get service and we see that the name
of the service is not MySQL service, it's instead MySQL.
That seems to be the problem here.
This and this should match
because from within this application is going to try
and reach a service by the name MySQL service,
and this doesn't look like that.
That seems to be the problem.
What we're going to do
is we're going to fix that.
Let's try and edit the service.
We can't edit the service name, but let's give it a shot.
All we want to do is change this to MySQL service.
Let's try and save that.
Now, it's not going to allow you to save that,
but the changes we made is in this temporary file.
This is what we need.
Here, you have a service and you have the MySQL service
and everything else seems to be as what we want.
The port numbers are there,
selector is there or another approach we could do
is to just delete the service and export
or expose the MySQL service again, but for now,
we could just go ahead with this.
We could do a kubectl delete SVC and delete the MySQL service.
Then we're going to create SVC--
No, we're going to create using this file.
Let's check service now.
We see that the MySQL service has now been created on port 3306.
Let's go ahead and check this.
When you check this application, it's now a success.
Check our work.
That's successful.
Let's proceed to the next one.
In this one, the same two-tier application is deployed in the beta namespace,
and it must display a green web page on success.
Click on the App.
Let's click on this app to see
the status.
As you can see, it's in a failed state.
If you look at this, you have the environment variables.
The database host is MySQL service, user is root, password.
It says, "Can't connect to MySQL server on 'MySQL-service:3306'."
It says, "Connection refused."
Let's take a look at it.
First, we're going to switch context to beta.
We're going to do a set-context,
config set-context command,
then we're going to set the current context
namespace to beta.
Now we're going to do a get pods.
We see the MySQL pod
and you have the webapp pod, so that's fine.
Then we also have services called the MySQL service
and the webapp service.
If you look at this, again, as before,
the port 30081 seems to be correct as we can access that here
and this whole application loads, so this whole thing should be good.
Again, it looks like it's a connectivity issue
between the web application
and the SQL server, so this is what we have to verify.
Now, last time the issue was with the name,
but now it looks like the name is correct.
It is MySQL service,
but we will anyway also check the deployment on what is set,
describe deploy webapp MySQL.
We see that the database host given here is MySQL service,
and this is MySQL service as well.
That's good.
Now let's just look at the service.
This to this is good.
Now let's just focus on this.
We're going to do a describe service MySQL-service,
and we see the selector is right.
The endpoints, it has detected the endpoint, so that's 10.42.0.12.
Let's make sure that this endpoint is the pod, which is the MySQL port.
Let's do a kubectl get pods-o wide to see the IP of the pod.
We see that the MySQL pod has an IP address, 10.42.0.12
and this is 10.42.0.12.
That's right.
If you look at this right here, it says port 8080
and so the IP is okay, so then we check the port.
It's port 8080, and 8080 is not the port for the database.
If you look at it here, the port here is 3306 and not 8080.
That seems to be the problem.
Let's do an edit as we see, MySQL service.
You can see the port here is 3306, which is right.
The target port given is 8080, and that is not right.
That should be 3306 as well.
Let's edit that.
Let's make sure the changes took place.
Now is 3306, and let's check the endpoint
and we see that the endpoint is now 10.42.0.12:3306.
Let's now check our application and yes, you can see that it's a success.
Let's check our work.
Okay.
We're two down.
The next question is the same two-tier application
is deployed in the gama namespace,
it must display green webpage
and click on the App tab at the top to view.
It looks like it's the same question.
Let's go and open the application.
Just keeps spinning
and meanwhile, we'll go here and clear our work and
we'll switch to the gama namespace.
Let's check status of pods and let's check services.
We have both the pods are running and the services are there,
but this doesn't look like it's loading.
Let's try and find out why.
It looks like the issue is somewhere here, the page itself isn't loading,
the service itself isn't loading.
Let's start looking at it right from here.
The first thing to look at is this port 30081,
because that is the port that we are trying to access, 30081
and it's now timed out, so 30081.
If you look at the service called web service,
you see that it's set to node port and the node port is 30081,
so that's fine.
That looks good.
Next thing is the service itself.
Let's check if the service has a endpoint.
We're going to do describe service web-service.
We see the name, web service, namespace is gama,
has the right selectors and has the endpoints 10.42.0.14
and the ports are 8080 and 8080.
That looks right.
Let's make sure
the IP address, the endpoints are picked up correctly.
This endpoint is 10.42.0.14
and the web app is on 10.42.0.14
as well, so that checks out.
The next step is--
The web service is good.
Let's check the deployment itself.
We do our kubectl, we get pod.
We do
a describe deployment webapp MySQL,
and we see that it has one desired and one available, so that's fine.
The image looks correct.
The database host is MySQL service, user is root, password.
All of that seems to be okay.
It's still, we're unable to use the deployment.
Let's check out the logs because it looks like the webserver isn't responding.
Let's do a logs
on the pod
and this seems to be okay.
This application itself, it says it started, it's available on this port.
That seems to be okay.
However, we are still not able to access the application.
Let's anyway, go through and complete our checks because we assume
that it's going to be somewhere here, but anyway,
let's go down and complete our checks.
MySQL service is supposed to be on 3306.
Let's check that.
We have the MySQL service and it's on 3306.
Let's check the endpoint,
describe MySQL service and here you have the endpoints
but it looks like there are no endpoints.
Apparently, the service hasn't detected MySQL pod as an endpoint.
Let's take a look at why that is.
If you look at the selectors right here,
it says name equals SQL00001 and for the pod,
describe pod MySQL.
We know that the labels for this is name equals MySQL.
That should be the selector not name equals SQL00001
and that's why doesn't have any endpoints.
Let's go ahead and edit that.
Not the pod, we're going to edit the service, MySQL service,
and we're going to change the selector
to MySQL, then save that.
Okay, now we're going to do describe service MySQL
and we see that we have the service protected.
Let's go and check out our application.
Okay, so that's connected successfully.
Apparently, even though we were not able to access the web application,
that's because the web application is trying to access the MySQL service
and that service was not responding,
which is why the web application was not responding either.
Even though we thought that the issue is somewhere here,
we went through and we made sure that everything's fine
and we just went through the remaining, just to check,
and just to be sure,
and that's how we identified that the issue is somewhere here.
That's another approach or technique
that you should use if you're not really sure.
You start with an intuition on where the issue
could be but if you can really figure out,
you just keep looking at those things just
to make sure that they're working as expected.
Let's check our work,
and let's wait for the next one.
Okay.
Here, the same two-tier application is deployed in the delta namespace
and it looks like it's the same setup.
Here we have, and it's in a failed state.
Let's first change our context to
delta
and then let's check the pods
and let's check our services.
Let's check the error message.
The DB host is MySQL service, users, SQL user, password is password.
It says,"Can't connect to MySQL server
on MySQL service: 3306.
Connection refused."
Okay, I'm going to try that out again.
I'm just refresh it, and it says,
"Access denied for our user SQL user at 10.42.0.16.
Using password."
It says, "Access denied for the user."
Here, you can see the user details.
The DB host has to be MySQL service,
and the DB user is root and the password is password.
Let's check the settings for that.
The web application must access MySQL service,
and this is where the credentials are stored.
Let's do a describe on webapp MySQL, describe deployment,
on webapp MySQL.
Let's look at the settings.
Here you have the environment variables, and you have the host,
which is MySQL service.
The user it says SQL user,
but it's actually root and the password is password.
This seems to be the problem.
Let's edit that
and update the user to root.
Okay, and let's wait for the pod to be running.
It's running already.
Let's refresh the page and that's a success.
That was the issue here.
Let's check that out,
and while the new one loads, let's close this one,
let's clear our screen.
The next one is the same application is deployed in the epsilon namespace.
It must display a green web page on success.
Let's open the web page and we have a similar error message.
It says, "The environment variables, DB hosts, MySQL service database.
It's not set.
User is SQL user and the password access denied for user SQL user."
That's kind of the similar error.
Let's take a look at it but first let's change the context to
episilon
and select the pods.
We know that it's a similar error message where it says,
"Access denied for SQL user."
That's what we just solved.
The user is not a SQL user.
It's a root user, so let's go ahead and fix that.
Edit deployment webapp MySQL.
Here we see the user is SQL user.
Let's change that to root.
Let's check the pods, it's running.
Let's rephrase the page.
Now the error message has changed and it says,
"Access denied for user root."
Again, so the access is still denied,
but the user has now been updated to root.
The user credentials are either set on the application
that's trying to connect to the database or the database itself.
We've already fixed it on the deployment here,
but now the issue could be here.
Let's check out the pod because that's the only other place
where anything could be set.
Here we see additional details, and here we have the environment variable
and the MySQL root password is set to something other than password,
which should be this.
That needs to be changed.
Let's do a edit pod MySQL
and let's change this password to paswrd.
Save that.
It's not going to allow us because it's a pod.
We're going to do kubectl replace,
replace force and we'll give the file.
It's deleted.
It's back up.
It's running.
Let's try to refresh the page.
Now it say, "Connection refused."
No, that was because we tried too early.
The pod was just being initiated.
Now it's success.
That was the issue.
Now, another thing to note here is in this case, it's very simple,
straightforward examples of setting environment variables,
but this could also be part of config map.
There could be another config map associated with the MySQL service,
which is how it is usually configured.
If that is the case,
then you should inspect the config maps as well.
Let's get ready for the final one.
The same two-tier application is deployed in the zeta name space.
Let's first set the context to zeta
and let's check out the application.
It says, "Bad gateway,"
right upfront.
Let's start from here.
We're assuming because it says
we're not even able to access the web service,
we're going to start right from the top.
The node pod is 30081,
because that's what we are trying to access here.
Let's see the status of the services.
We see that the web service has a node port, but here you see,
it's not on 30081, it's on 30088.
That's what we're going to fix.
Edit service web service.
The node port has to be 30081.
We'll save that and let's make sure that it's updated.
It's 30081.
Okay.
That's done.
We're now able to access the application at least.
Now it says, "Access denied for user SQL user using password,"
so that's something we have seen already so let's check out.
Okay. Let's look at the deployment.
Describe deploy webapp
MySQL,
and we see that their environment variables.
MySQL services, SQL server, SQL user, and password.
Let's edit that
to change it to root.
Okay.
The new pod has been deployed.
Now the username is updated, but it still says,
"Access denied for our user root."
We have to go down and check the password
configured on the MySQL service.
Let's do a describe pod MySQL.
We see that we have the password issue here, the environment variable.
Let's edit that
and it should be password.
Let's save that.
That's not going to go through, so let's do a replace with this.
Let's wait for the pod to run.
Okay, yes.
Now that's successful, check our work.
Okay. That's about it.
I know that there's a little bit of repetition
on the type of questions
and that's really in case you didn't get it the first time
for you to just get that practice again
and get it the next time and build some confidence.
I hope that was a good troubleshooting session
and we'll have more coming up in the upcoming labs.