Let us now look at the backup and restore methods.
We have a working kubernetes cluster with a set of applications running.
Let's first explore the setup.
Let's see how many deployments exist.
Currently, we see that there are two applications.
There is a red and there is a blue application running.
There are two deployments.
Now, what is the version of ETCD running on the cluster?
Check the ETCD pod or process.
Let's first see get pods so we know
that ETCD runs in the kube-system namespace.
We see that we have the etcd-controlplane and this is the pod.
From this, we can also guess that it's a static pod.
That's what we learn in the static pod section.
Let's explore this in more detail.
Let's do a describe pod kube-system.
We see that we have the single container which is etcd,
and it has the image which is etcd: 3.5.1,
so that's basically the answer to this question.
The version is 3.5.1.
Let's select that.
At what address can you reach
the ETCD cluster from the controlplane node?
While most of those details are given here, if you look at this,
this container uses the ETCD command and all of these options.
From this, we see that the URL
or the address that ETCD listens to is 127.0.0.1 and 2379.
It's 127.0.0.1 and 2379.
2379 is the pod that ETCD listens on.
There are many other options as well.
Let's just select this for now.
Where is the ETCD server certificate file located?
If you look at this, we have the ETCD cert-file,
so this is the ETCD server certificate file.
It's etc/kubernetes/pki/etcd/server.crt, so that's this option right here.
Where is the ETCD CA certificate file located?
The CA certificate is the option that has the CA in it,
so the trusted CA file.
That's the etc/kubernetes/pki/etcd/ca.crt.
That's this one right here.
The next one is a task.
The master node in our cluster is planned for a regular maintenance reboot tonight.
Now, while we do not anticipate anything to go wrong,
we are required to take the necessary backups.
Take a snapshot of the ETCD database using the built-in snapshot functionality.
We're going to have to take a snapshot.
We've learned that we used the etcdctl utility.
Let's understand this a little bit more.
Here, we know that we have the cert file.
We have the data directory which is set to var/lib/etcd.
We have the key file, so this is the private key file.
We have the CA file.
We also have the URL that it's listening to.
We'll need all of that information for us to take a backup.
Let me first take a look at
the definition file that's used to create this pod.
We know that it's a static pod.
We've learned that the static pods are created
from the etc/kubernetes/manifests directory.
Here are the files that are used for creating this pod.
Here, the ETCD is the one that's used, so let's look into that file.
If you look at this, this is a pod.
Here are all the configurations for it.
This is just the stuff that we just saw.
I also want to take your attention to some of the relevant information here.
If you look at this,
we have volumes configured which is set to hostPath.
This means that this particular pod has these volumes,
and these could be external volumes, but in this case, it says hostPath.
hostPath just refers to a path on the node that this particular pod is running on.
In this case, it's just a directory on the host that this pod is running on,
which happens to be the control plane node in this case.
There's a volume called etcd-certs
which is created from this path on the host.
That's /etc/kubernetes/pki/etcd,
and this basically is used as a volumeMount here.
What this says is that this particular path on the host
or on the controlplane is mounted
to this particular path within the container.
This is /etc/kubernetes/pki/etcd.
It's the same kind of path, but this is on the controlplane host,
and this is within the etcd container.
Similarly, we have the data path to etcd-data.
This is where it actually stores all the data.
On the host, it's at /var/lib/etcd
and the same is mount to /var/lib/etcd within the container.
If you look at the options that are passed here,
you have the data directory and the data directory is
/var/lib/etcd within the container.
That's how it maps.
Basically, what I'm trying to say is if you look at these characters,
if you look at this directory on the host which is /var/lib/etcd,
and that's what's given here.
If you look at this directory on the host,
you see these files or this directory and anything that are under it.
These are the files that are created by etcd to store information.
Similarly, if you look at this particular path on the local host,
these are the certificate files that are used by ETCD,
and these are automatically mapped into the respective folders.
In this case, they happen to be the same kind of path.
They have the same name set.
They have the same path set between this and this,
so they are identical, but they don't really have to be.
Knowing that information,
let's try and perform the task of taking a backup.
We have the etcdctl snapshot command.
If you just run it directly as like this,
it's going to tell that there's no help topic for snapshot.
Basically, etcdctl can't recognize this command.
You might come across this
when you're working on a Kubernetes cluster,
or when you're working with etcdctl, or maybe in the exam.
You should know that this happens not because your command is wrong.
It happens because you've not set the ETCDCTL_API version to three.
This environment variable needs to be set before you run this command.
If you don't want to specify this on each command,
one thing you could do is just export
in that environment that you're working on.
You could do an export ETCDCTL and API was three.
Now, you could just do etcdctrl snapshot
either you pass it in at the beginning of each command or you export it
as an environment variable.
We have the etcdctrl snapshot command,
and we need to pass some information to it,
so we're going to do a save.
We have to take a snapshot.
Let's do a save and then we need some of these parameters,
so you have to pass
in the CA certificate files for it to communicate with the endpoint.
We definitely need the endpoint, so we're going to have that first.
The endpoint as we have seen here is this.
Copy that.
You don't really need this.
That's the first thing.
Next, we need the cacert.
If you go to the output of the file,
we have this
and then we need the certificate
which happens to be this.
Then, we need the key,
which happens to be this.
Just remember that there are three certificate files
that you should pass through.
We've done the etcdctl snapshot save,
and then we've specified the endpoint,
and the certificate files that are required.
The one thing there you should specify is the location of the backup.
Store the backup file at location this, so we're going to pass that.
Let's give it a shot.
Snapshot saved at this locations.
Make sure that file is there.
The snapshot has been taken.
Great.
Let us now wait for the maintenance window to finish.
Go get some sleep and let's click OK to continue.
It's about 2:00 AM at midnight and you get a call.
Wake up.
We have a conference call.
After the reboot, the master nodes came back online,
but none of our applications are accessible.
Check the status of the applications on the cluster.
What's wrong?
Let's do a kubectl get deploy.
See that the applications are not there.
Check the pods.
Pods are not there.
Let's check services.
We only have the built-in kubernetes service,
but the service for the applications are not there.
The deployments are not present.
The services are not present.
The pods are not present.
Nothing seems to be there.
That's all of the above, not above actually,
all of the options basically.
Luckily, we took a backup.
Restore the original state of the cluster using the backup file.
Let's do that next.
The restore option, what we're going to do is,
so we have a file with the backup.
Now, we could use the etcdctl restore command to restore
a given file and restore the data to a directory on the local host.
What we're going to do is we're going to restore the data
to another directory on the local host.
Then, we're going to go into this etcd manifests file,
and we're going to change.
We're going to reconfigure this to use the data not from /var/lib/etcd,
but from that new path that we have restored the data to.
That's the strategy that we're going to follow.
To restore etcd, again, remember
if you've not set the export etcdctl API version,
remember to pass that in, but in this case, since I've said it,
I'm not going to put that in the beginning of this command,
so etcdctl snapshot restore.
Now, remember that when you're doing a snapshot,
say when you're taking a snapshot,
you're actually taking a snapshot by contacting the etcd server directly.
For that, you need all the certificates.
Now, our goal is that we have the file already,
and all we're going to do is we're going to restore
this backup file to a folder.
That's what we're going to do.
We're going to specify a data directly
where we want the backup to be restored to.
We'll create a new directory called /var/lib/etcd-from-backup, for instance.
This is where the data is going to be restored to.
We just specify the location of the backup.
Remember that we're not going to specify the endpoints or the certificate details
because this particular operation of restoring data from an etcd backup
is not done by communicating with the etcd server.
It's just a local activity that we're running to restore it
from this path and extracting the data into a new directory that will be created.
The restore is complete.
Let's check the data is in /var/lib/etcd-from-backup.
That's the data.
Now, what all we need to do is go back
into kubernetes/manifests file into the etcd.yaml.
All we need to do here is go here
and change the hostPath and add etcd-from-backup here.
Basically what's going to happen is,
as we know that any change on this file is going to recreate this pod,
and when this pod recreate is recreated,
it's going to consider this as the data
on the local host that has the data for the etcd server,
and then is going to mount this to the /var/lib/etcd within the pod.
It doesn't matter if these are different.
If this has changed, that's fine.
This volume called etcd-data is then going to be monitored
to /var/lib/etcd within this.
Either you can just leave it like this
or you can change this and also change this.
If you change this,
then you have to make sure that you go up to the data directory here
and change this.
This line and this line should match, not all the time,
but this and this may not match only
because they are connected using this name here called etcd-data.
This is what connects them.
If these values are different, that should be fine.
I'm going to save this.
When I save this what's going to happen
is I might lose access for a bit because now,
the etcd server is going to go down for a short period of time.
Because I've changed the etcd manifest file,
the kubernetes is now going to restart etcd.
It's going to shut down the running etcd pod.
Then, it's going to create a new pod-based on the new information.
Let's just give it a few minutes.
This can take a few minutes time.
Let's just wait for it to restore and then come back.
I just got an output,
but this is an old output
because as you can see still says etcd is 37 minutes old which is not true.
I expect it to be just a few seconds.
Right now, as you can see,
it's just up and it's just been up for 14 seconds.
This is the new etcd-controlplane.
Let's do a watch and wait for
this etcd-controlplane to go from a pending state to a running state.
As you can see here,
when the etcd-controlplane at etcd pod is restarted,
it also restarts
the kube-controller-manager and the scheduler.
Let's take a look at the logs of the etcd-controlplane.
logs -n kube-system etcd-controlplane.
It's waiting.
It's in a ContainerCreating state.
Let's take a look at it.
As you can see, there are Liveness and Startup probes that are failing.
Let's do one thing.
Let's take the status again.
If it does not automatically go to a running state,
it might need a restart.
Let's just delete it.
It gets restarted.
Check it again.
This time, it's back, and it's now in a running state.
Let's see status of deployments now.
We can see that the deployments are back.
Pods are back and the services as well are back.
That's the end of this lab.