[No audio]
Filesystem corrupted or if there
is a filesystem corruption.
Well, what is a filesystem?
Filesystem consists of the directories or folders
that's you create in your operating system.
So there are different type of filesystems.
The filesystems are like ext3, Linux
ext4, xfs, NTFS or Fat and Windows.
These are all different type of file
systems that we have in operating system.
A filesystem layout and partitions
are like /var, /etc, /root, /home.
So whenever you carve a disk and you give that disk
to your operating system, the first thing you have to define
is what type of filesystem do you want for it?
Then you pick a filesystem, for example, ext4.
Once that is assigned to your filesystem, the
next step is to partition that disk or that
filesystem in the way you want.
If you leave it a default file
system, then everything goes under slash.
But if you wanted to partition with advanced partitioning
option while you're doing the installation, then you could
use /var, /etc, or whichever the partition or
mount point that you want to specify.
So what happened is when the filesystem is corrupted,
it usually gets corrupted in one of these partitions.
Maybe the /var is bad, /etc, or /root is bad.
That's why sometimes the system administrators, the
way they design the operating system is
they slice each partition differently.
So this way if there is a problem
with var, it only stays with var.
It doesn't impact the whole slash filesystem.
So that is one of the main reason people
pick the partitioning of your disk instead of just
picking one big large partition which is slash.
Now, how do we check the filesystem?
Well, the first command that you
need to run is df command.
We already talked about it.
df command tells you what type of file
system you have and where that filesystem
is being partitioned and where is it mounted.
Also you could run the command fdisk with minus
l option to see all the disks that you have
in your system and also each partition of that disk.
If your system is running on lvm, then
you could run many lvm commands like
lvdisplay, pvdisplay, the lvblock.
So there are many commands that you
can run to see your filesystem.
Now, let's get into troubleshooting steps.
So you know your system is running slow
or you have identified that you have a
filesystem corruption in one of your partitions.
So what do you do?
You first check the /var/log/messages to
confirm that it is actually the filesystem.
And you will see a lot of error messages that will
come up in your /var/log/messages that will say that bad
log on this sector, on this disk and you need to
fix it or you need to run fsck,
you will see that in your /var/log/messages.
So /var/log/messages or any type of log
like /var/log/syslog are very important in a
system structure to identify where the problem is.
Then once you know that you do have
a filesystem problem or a corruption and
you identified which partition that filesystem is
corrupted, then you run the command fsck.
This is the command fsck,
which stands for filesystem check.
This command has been used with many different Linux
as well as Solaris or Unix flavors as well.
It's still there, it works perfectly.
There are some of the new commands that
are in there for different filesystem.
Maybe XFS filesystem will run xfs_repair,
which we'll talk about in a little bit.
So anyway, we run the fsck on
block device which is /dev/sda.
We will not run on the mount point, meaning
these are the mount points to /var, we cannot run fsck
on /var. We have to run on the actual mount
where it's coming from, the source.
That would be the block device.
Then make sure we are unmounting the
filesystem before we are running fsck.
Because if you run fsck on a mounted filesystem or
mounted partition that is already mounted on /var, /etc, it
will give you an error message and it will say you
cannot mount it because that filesystem is busy.
That is why one of the things that a lot
of people get into trouble saying when they are trying
to do the corruption cleanup of the corruption filesystem
and they run fsck, and it doesn't run.
And then they're sitting in trying to find out why it
is not running, read the error message, what it says and
it will tell you why it's not running because it cannot
run that command on the mounted filesystem.
So let's get into our Linux machine.
I have already opened a terminal to my Linux machine
and the first thing we will see is, we'll see
the layout of our filesystem, and for that we'll
run the command df -h. h is for human readable option.
You could pick any option you want.
If you pick, I suppose t, that will also
tell you what type of filesystem it is.
It is xfs, tmp filesystem, or exe filesystem.
So let's just run df -h, and you will see there
are different filesystems are also there as tmpfs.
These are just tmporarily, this is
not the actual filesystem.
So what you could do is you could do df
-h and grep -v to exclude tmpfs.
So now you know exactly how many partitions you have.
You have two partitions.
One is mounted on slash, which is just slash
on the root level and the second one is
/dev/sda1, which is mounted for boot.
boot is usually mapped on a regular block device.
It does not get mapped on the lvm.
So up here, you'll see this is the lvm.
It always starts with /dev/mapper.
So we identified it.
Now, in your system, if you are working in a
corporate environment or any of your system,
you're probably going to see,
more mounted points.
You probably have /var, /etc, /opt,
/sbin, or /tmp, whatever partition you have,
it's ok. I'm not going to go into everyone of them
But I'm just saying, you're probably going
to see a lot more mount points on your system.
If you don't, that is fine.
But let's go, come in here,
and to cover this, I have actually
attached a disk through my virtual machine.
And I am going to mount that disk first
just to show you how we could run fsdk.
So that disk I will do df -k, fdisk -k.
Oh sorry, fdisk -l,
and here, if you notice, I have a
new disk, which I have already partitioned.
I just wanted to add that I
have already partitioned that new disk.
If you notice right here, I have
a disk, /dev/sdb, which is five gig.
This is what I got from my virtual
machine, and I have already partitioned that.
Now I have to mount that somewhere.
So I will mount it in one of
the directory that I have created earlier.
So that directory is in my slash directory,
which is I named it sparedata.
So I'm going to mount that mount
/dev/sdb1 to /sparedata/.
Okay, now it is mounted.
Now, when I run df -h, of course
I will grep everything, all the temporary filesystem.
And now you will see I have basically two filesystems.
One is sda, and one is sdb.
They are the same filesystem,
XFS. I wanted to know, I have discovered that this
filesystem or this partition point is corrupted.
Now, I need to fix that or clean
any bad sectors that is on this partition.
So I will run the command fsck space /sparedata.
Now, the reason I'm putting that here, because I want you
to know that it will not work, because I am trying
to run fsck on the mount point, which does not work.
And it will tell you it's a
directory while trying to open this.
So we don't do on the directory, we do
on the block device, as I mentioned before.
So I will do fsck /dev/sdb1.
Now I'm running this, it will still fail. Why?
Because as I mentioned in my last part, it
says unmount filesystem and then run fsck.
Unmounting means it is already
mounted on this directory.
So that's why it's not going to work.
But I will go ahead and run
it to see the error message.
Okay, so when I ran it, it says fsck from utility-linux so
and so, and the version and the date, and it says
/dev/sdb1 is mounted. e2fsck: Cannot continue.
So aborting.
Basically, this e2fsck is a newer command.
But for fsck, I'm sure fsck probably have a link to this.
This is the new command that you could run. Anyway,
we could talk about new commands later.
But right now our focus is to find out
why we cannot run fsck on /dev/sdb1.
And we know we cannot, because it is mounted.
So what do I have to do?
I have to unmount it.
[No audio]
umount, the command to unmount is umount /sparedata.
That's the partition or that's the mount point I have.
So I'll run that command and now it is unmounted.
I will do df -h again
and you'll see it is not there.
Now I could run fsck /dev/sdb1 by hitting up arrow keys
which will repeat all the commands that we ran earlier,
I hit Enter. As soon as I hit enter,
it says fscl from utility so and so, and version, date,
it says /dev/sdb1, and clean.
And it has some calculations and the files, it's telling you
how many files it has, how many blocks it has, since
this is only five gig and it went pretty fast and
there is no files or data on that spare disc right
now, that is why it went pretty fast and you see
you got the result right away.
If you have that, let's say 100 gig of disk and has it so
much data on it, then it could take a long time to run.
It will go into it and it will check everything that it has
to check on for the bad blocks and it will fix it.
So it's also possible while it is fixing it, it
will ask you, hey, I found the fix for this,
do you want me to fix that bad block?
And you will have to enter y,
to say yes, go ahead.
So one thing is you could keep saying y, y,
and keep entering in on the prompt, or you
could just simply say fsck -y.
It means whatever that you're going to ask me,
my answer is yes.
So then you hit Enter and it will go through
all the filesystem again and it will clean up.
Now, next thing I wanted to talk about, which
is one of the biggest thing a lot of
people experience is they say, okay, my filesystem
is corrupted, but it is root filesystem.
Now how do I clean that when it is mounted?
If I unmount it, it's not going to let me unmount
because that's where I am running my entire operating system.
So how can I do that?
Well, that's a very good question if
that is coming in your mind.
Many times we have that issues we cannot clean up.
So the only way we could do that is we have to
boot our machine through the CD ROM, we have to mount our
CentOS or Red Hat cd to our Linux machine right here and
we have to boot from that CD ROM and then once it
boots, it will give us the rescue option.
We'll pick the rescue option and then we can
fix this issue if there are any bad sectors
on the root or any partition that cannot be
mounted while the operating system is running.
Now, how can I do that?
So for that, you cannot use a PuTTY session for that.
You have to actually console into the machine
because you will be rebooting the machine and
you will be bringing down all the network
connections and all the multi user environment.
So I will go into my Linux machine through console.
Okay. I am logged into console.
Now I have to attach the device,
sorry, the ISO image of my CentOS.
So go to Device, Optical Drives, and choose a disk image.
Once I go in here, it will tell me where the file is.
I'll say Desktop, it is on my Desktop,
and the file is CentOS, click Open.
If you want to confirm it has attached,
you could go back to Devices, Optical Drives, and
you see the check mark is right here.
So it means it is mounted.
So now all I have to do is do a reboot and hit Enter.
[No audio]
When it reboots, you see in the back I have the
PuTTY session and it says it lost the server unexpectedly.
That's fine.
Now I am at this section
where it says Install CentOS 7.
You probably have seen that if
you have recently done the installation.
So Install CentOS is perfect.
You don't want us to do that because this
is if you are doing the install from scratch,
Test this media & install CentOS.
It's the same thing, but it will test it again.
Our option is Troubleshooting.
I'll go in and hit Enter in Troubleshooting.
Now the next option is Install CentOS in
basic graphical mode.
Again, we are not installing, we are doing Rescue.
So come down to the selection to the
option where it says Rescue ascent OS system.
If you are using Redhat it
would say rescue a Redhat system.
So hit Enter.
[No audio]
Now it's basically starting up the Operating System.
Not actually the operating system, but the system
that is actually on the disk that is
used to go through the installer.
See, it says starting installer.
Well, in fact it's not doing the
installer, so don't be alarmed by that.
Now here we are in the rescue mode.
Read it very carefully, says the environment.
The rescue environment will now attempt
to find your Linux installation and
mounted under the directory /mount/sysimage.
You can then make any changes required to your system.
Choose 1 to proceed
With this step you can choose to
mount your filesystem read-only.
Instead of read, write by choosing 2.
If for some reason this process does not work,
choose 3 to skip directly to a show.
So if I mounted, if I pick one, then it will
mount my filesystem which is root to /mnt/sysimage.
[No audio]
So suddenly I lost the screen.
But anyway, so I don't want to mount that.
If I mounted then I cannot run fsck
which I will be in the same situation.
I don't want to mount it as read-only.
So I will just skip straight to the shell.
So I will pick option three and hit Enter.
Now I have the shell, so I run df -h command and
you will see none of the filesystem will show up here.
That actually the partitions that we have when the actual
operating system runs, the one that you see right now,
it is actually coming straight from the cd.
So you know that already.
Your filesystem that you need to
clean actually lives on /dev/sda.
So you will run now you run fsck minus
y, if you want to/dev/sda1,
because you have to specify the entire partition for
every partition that you have on the disk.
Now you hit Enter and it says fsck from
utility-linux 2.23.2. /sbin/fsck.xfs : XFS file system.
It means I cannot run fsck for
the partition that is on that list on
or that has assignment of XFL filesystem.
So I have to run the command fsck.xfs.
Now run /dev/sda1.
Now if you wish to check the consistency of
XFL system or repair damaged filesystem, see this.
Now it's saying now go to this command.
This is the one I was talking about.
You will have to run this command instead of fsck.
So now you run a /dev/sda1, hit Enter and
it's going through all the phases, phase one to six.
And it's actually resetting the content, resetting checking the
blocks and it's verifying and it says verify and
correct link counts and at the end don't.
So basically it went through the whole lot.
If you had a big disk space with a lot of data
on it, then it will take a long time as I mentioned.
So now since it's mounted, it's cleaned, your
corruption is cleared on your filesystem.
Now you could simply do reboot, but before you
do the reboot, just unmount your optical drive so
it won't start off the ISO image.
Force Unmount and hit Reboot.
It will reboot and it will come
back as clean disk and no corruption.
You will also check in the /var/log/messages.
Anyway, so this is about
this lecture filesystem corruption.
Then you have to follow these steps one by one again.
If you have any questions, please send me
a message and I'll be happy to answer
anything that you haven't in mind.
[No audio]