[No audio]
System is running slow.
Okay, this is the lecture where I'm
going to cover a lot of commands.
So sit tight and make sure you are ready to
go over all the commands because many many many times
you have this issue and people say system is running
slow and you are being reported this problem by someone
else either in your group or somewhere else or actually
you find out your own system is running slow and
what are the things that you have to do?
Well, first of all, you need to understand the
problem. As I discussed that earlier as well,
understanding the problem, where exactly
is it coming from?
This is the more important thing before
you actually start troubleshooting any issue.
So you need to find out whether it's process
related, your processor is the one that's running slow
or what exactly the problem is with the disk,
is that the disk is causing the issue.
Maybe you have external disk where your system is writing
or you even have your internal disk that probably one
of the disk has issues or bad blocks or degraded.
You need to find that out.
After that you have to look into
the networking tool like file transfer.
Like if someone is transferring files from your system
to another system or anywhere within the network, you
have to find out whether that transfer is causing
by processing, disk issue, or networking itself, or memory
that is actually in use.
So memory will be the part of the processing as well.
Then at the end we have to
know whether it is hardware related.
So these are the few things you have
to keep in mind whether it's processing or
memory, disk issues, networking issue or hardware issue.
So we have different tools for every one of
them when we are troubleshooting the system slowness issues.
So we'll talk about the troubleshooting steps
now. The troubleshooting step, the first thing is
always check if the right system is reported
or you are on the right system.
Yes, when someone reports there's a problem with system
ABC and in fact it's not ABC, it's XYZ.
So you need to find out what
system they are reporting the problem on.
Whether it's a database server, it's
a system server, a web server.
Once you find that out, when you log into that
system, of course always do hostname to check you
are on the right system before you start troubleshooting and
start digging into it because I don't want you to
spend like hours, and at the end finding out you are
not on the right system.
Then we will check the disk space.
The first thing you'll check is if the disk
is full because most of the time when the
disk is full it will cause problems for everything
that is processing, memory, networking or disk related issues.
Everything will actually end up at the disk.
If your disk is 99% full, your memory, your
processor cannot do anything at that point because they
cannot write to the disk because it's full.
So the commands that we'll learn is about
df and du, which we already covered, but
I'll quickly cover in this lecture as well.
Then about the check the processing and
memory of course, and the commands that
we'll cover is top, free, lsmem,
we'll check the /proc/meminfo
that is currently in use.
We'll check them vmstat, pmap which will
tell me which process individual process is using how much memory,
not just the entire system, dmidecode which
will tell me some information about the hardware.
What type of hardware I have installed
or added to my system, lscpu.
What the process is listed.
How much it is used.
How much it is used by other processes.
Also the /proc/cpuinf you can use the same
thing that will give you the same output as list
CPU. Then we'll cover about the check on the disk issues.
And the main command that I always use is
iostat and I have every five second interval.
So while you're troubleshooting issue, you could run the iostat
and check the disk status, meaning what is going out,
what is coming in to your system and what's the
status on that traffic going in and out.
Then we'll also cover the lsof which stands for list
open file, any files that are up and open at
this point, that is taking up the memory or CPU,
what are those files that are open?
We'll find those out that. That lsof command is also used
sometimes when you're trying to unmount a file system and you
cannot because it's telling you it's busy, but you don't know
how it is busy, maybe there's a script in it, maybe
there's a file in it, in the mounted file system
that is used by another process or another user.
So you will use lsof to find that
out and you could kill that process.
Then comes to the check networking port.
So the networking commands that we will be using is
the main one is the tcpdump with -i
option I'll use on the interface, i stands for interface.
Also at the same time we'll use lsof to see what
is listening, what the port is open and we'll also grep
for the listen what the files are open, what protocol, what
sockets are open and what is it listening to.
We'll use netstat -plnt, and by the way, netstat has
been replaced in the new version of Linux system in CentOS
and Redhat is replaced with ss as in Socket and
same options can be used that were used in Netstat.
Then there's another command iftop, which is very
useful when you are troubleshooting network related issue.
iftop does not come preinstalled with your operating
system, you would have to actually install couple of
packages like yum install epel-release, and then you
will install yum install iftop. Then this command
will be available to you.
It's like an add on command, but it is very useful.
That will tell you exactly what is coming
in, what is going out, what is being
exported, transferred, what is the transfer rate?
If it's high, it's low.
Then we'll also check the system uptime because the
system is running slow, sometimes what happens system crashed
and you don't know if it's crashed or not.
So you should check whether the system
just been booted up or not.
So update will give you the time when it has
been up and running and it will also give you
the system load every 5, 10, and 15 minutes.
That is pretty much the same way the
load that you see on the top command.
We will also check the logs.
Log is one of the main resource that you
have where you could actually go in and check
if there is something wrong with your operating system,
Kernel, memory, CPU, hardware, any type of errors that
are reported will be reported in your logs.
Then there are other things that you
could check on the hardware side that
are available to your operating system.
And if you cannot get to the hardware
issue, you need to actually go into the
system console to actually find that information out.
Then there are other tools that you could install.
Again, these tools are not added
when you install the operating system.
You would have to install this using your yum command.
And some of these tools are htop, iotop, IP Traffic
PS account, it's related to the user account status.
So these are the few commands if you want to
use it, keep them handy, keep them in your mind.
So these are the few commands that you need to know
for being a good system administrator, how you will use them,
how you will run them in the system, and when you
get the output, how are you going to read them.
So let's log into my Linux machine and I will
show you each one of these commands one by one
and hopefully we'll cover pretty much all of them.
So, as I said, first check you are
on the right system, or the system that
is reported to you is the right system.
So for that you always run the hostname which will
tell you oh yes, this is the right host,
because many times in a bigger organization you have servers,
10-20 servers using the same hostname, but they have
01, 02 or incremental numbers at the end.
So just make sure you are logged into the right system.
Then check the disk space.
That's the first thing you should always do.
And to run that you do df -h.
So df -h, h for human readable.
So this is going to tell you the file system on
the left side, the size, what is used, what is available
in percentage and what is it mounted on. All the tmpfs
can be ignored because these are the memory files system.
You could run df -h and you could grep minus
v to exclude tmpfs to exactly see the status.
So if you look at my system status, I
have a pretty healthy system in terms of disk.
The root is at 59%, sd1 is
at one and the boot is 17.
So the system at the disk level,
the usage of the disk looks okay.
If you see anything here that says 99% or 100% or
any threshold that you think has exceeded the normal threshold, then
you have to check why it is using that much and
for that you will have to use the command du.
When you do du -a, which is all and specify
which file system that you're trying to look for.
You'll see all the file system that's come all the
files in the system that will show up here and
it will tell you every file and its size.
You could do the -h for human
readable as well and it will tell you
in Kilobyte, Megabyte or Gigabyte as well.
If you want to sort this out, you
could do sort in the reverse order.
So you'll see that the highest usage of file at the top
you will do this and it will sort everything for you.
But of course it's going to go so fast
so you will have to actually more it.
So pipe it to more.
So it is actually going to the
proc which is just the memory one.
So no such file while you're running it. Don't worry about it.
You could ignore any of that that
comes up and says cannot access.
But if you are getting an error message
that saying hey, Permission denied, then it definitely
means that you are not root.
So make sure you are root before you run
the du -a command or -h command on
any file system, so it could go into every
file without getting their Permission denied error.
Let's clear the screen then.
The next step is about check processing.
So let me make this window a little smaller so
we could go over the commands one by one.
The first one we have top which
we have already covered previously as well.
I'm just going to go quickly again.
The first line tells you about the system uptime,
how many users are logged in, and the load
average in every 5, 10, and 15 minutes.
This one tells you total tasks
that are running, how many sleeping.
This line tells you about the CPU status.
This line tells you about the memory status.
This line tells you about the swap status. Then
coming down, this is the list of every process
that you are running in your system.
Mostly the high utilized process you
will see at the top.
The first column is the process ID who is using
it, the virtualization, reservation, shared CPU, memory, time, command, and
it has a lot of information as well.
Then we'll go into the next command which is free.
free is specifically for memory utilization.
You can also do free -m
if you wanted to see in Megabytes.
So this will tell you how much memory you
have total, how much used, how much free, shared,
how much is it buffer, and how much is
available to be used by the other processes.
Yes, other scripts or other programs.
Okay, now next one we have list memory, lsmem.
This is telling you exactly about the
memory block size, total online memory that
we have available to the system.
It is one gig total offline memory that is
sitting that we are not using, which is zero.
We don't have any.
So this is going to tell you about your memory as well.
Same thing pretty much when you do /proc/meminfo
cat sorry, /proc/meminfo, this is going to tell you exactly
what is going on in the system right now.
Total memory, free memory, memory available.
It has a bunch of information about your
memory, and how memory is allocated to every
process, every page in your system.
So you have to read through it, every one of them.
If you really wanted to know exactly what
each of them means, if I start explaining
them, it's going to take me forever.
So I will recommend you,
you could just go to Google and type
/proc/meminfo and it will give you
a lot of information about memory info.
Then vmstat, which is the virtual memory status.
This is telling you about your swap space.
Swap is pretty much the same as a virtual memory.
How much it has been assigned, how much
is it free, how much is it buffer.
This will also tell you if you have memory.
So if you are running free -m, and you
see the total is 991, and used is 991, nothing
is used, then the next thing you have to find
out is whether you have the same issue with your
swap or virtual machine, sorry, virtual memory as well.
By the way, free can also give you both as
well the regular memory and swap memory as well.
So check the memory status, make
sure the memory is not used.
If it is used, then you actually have
to assign more memory to this machine because
it is using up a lot of processes.
If you run top again and if you see a lot of
processes here and that are using this column right here, this column
has used up a lot of memory, by a lot of different
processes, then you actually have to add more memory.
Or you could also use a new command that
I'm going to show you which is pmap.
So right here if you notice, the
top command has a process ID 6610.
So if I do open up another session,
[No audio]
and I type pmap,
and put in the process ID, then it is telling me
this is a total process that is being used by the top
command that are the top process that I'm running.
So just that specific information about the pmap.
You could also find out about the
other process if you want to know.
Like, let's say process ID one.
This one actually gives you the process information
sorry, the memory information of one particular process.
Instead of going around and finding out every
process, you could check it over here. Okay.
So the next one is dmidecode.
dmidecode. This is going to give you a lot of information
about your hardware, what is inside of your computer.
So when you run this, and if you notice you
have an issue with the CPU or memory and you
have to actually call the vendor and the vendor asks
you some information about give me the information about the
CPU serial number or memory serial number or what's the
size or what's the release or what's the version information,
you will find all the information right here.
As you can see, the manufacturer
information, the product name, the chassis
information says manufacturer is Oracle Corporation.
So you know who to call when you are trying
to open up a case and with who, so if you
do not know which server that you are on,
this is the dmidecode, will tell you
whether it's HP, Oracle machine, Spark machine,
sorry, not sparks,
they have different ones like any other
hardware, like Dell or any other hardware
that run 64 bit operating systems.
Okay, then lscpu, which is kind of similar to lsmem,
it gives you the information about your CPU architecture.
CPU architecture is x86 64 bit.
It actually supports 32-bit and 64-bit models.
The byte size CPU is one any
socket, any core per socket, the vendor
CPU family, model name, Intel CPU, Megahertz.
Everything that you need to know about
CPU can also be found in lscpu.
Okay, next one is /proc/cpuinfo, which is again similar to
[No audio]
lscpu, /proc/cpuinfo.
If you notice, pretty much the same as the other
one, but probably it is rearranged, but it gives you
similar information to what you would get with lscpu.
Okay, let's clear the screen.
So you have find out so many things about CPU.
You know, your memory
information, your disk information.
Now moving on to the disk status.
You know the disk, when you run df -h, you see that
the disk usage is good and it's actually 59%.
So there isn't a disk in terms of disk
capacity issue, but maybe the disk input output disk
itself is running in a degraded mode.
So for that to find out, you have to run
iostat, meaning input output statistics, -y, with five.
I always use that because I am kind of used to it,
I'll say that this way, there are many other options
that you could use with iostat as well.
I would say there are many options
with every command that I'm showing you.
So the best way is that you run man on every
command and find out every option that is available to you.
Anyway, let's run iostat -y with every 5 seconds.
Now this is actually going to the disk
and it's going to ask disk give me your
statistics every five minutes, sorry, every 5 seconds.
And here it's telling you your disks.
You have two disks, SDA,
SDB and it's kilobyte read, kilobyte written, how much
it is reading, how much it is writing it.
So by looking at my system, my system is
running very efficient because it's pretty much 0.0.
Nothing is, I don't run any production type
of application on my system.
So that's why you see that everything is zero.
But if you are running, you will see how it is
going through, how much it is KB reading, how much it
is writing, how much it is transferring per second.
So all that information will show up here.
If you see the number very high, it means either you
are of course writing so much or reading so much or
it's just your disk is simply having some issues reading or
writing or actually queuing up the entire transactions.
So this is about iostat.
Now let's get into networking part.
Now you confirmed nothing is wrong with the disk.
Everything looks good at the processor, everything
looks good on the memory side.
Now let's look at the networking. If you are actually,
if your programs are actually involved
into transferring or actually receiving, you
would have to check the networking.
So first command that we will run is tcpdump.
TCP, every protocol that goes in and out is mostly on TCP.
So with -i option it's interface.
First you need to find out what interface
you're going to run tcpdump on.
So for that you have to run if config.
And you notice my interface is enp0s3.
So I'm going to run on my tcpdump -i enps03.
And when I run that enp, wrong interface, spelled wrong.
Okay, now you see it is going so fast
because my PC, where I'm running my machine or
where I'm actually showing you this whole lecture, it
is communicating to this virtual machine.
That is why it's going back and
forth, back and forth, back and forth.
So if I just close this or CTRL C
out of it and open up another window.
By the way, you have to be
root to run most of these commands.
So always log in root.
Now, if I try to do ping,
let's say www.google.com, it is pinging,
but now as it's pinging, I could actually
check this traffic on my tcpdump.
So let's just grab for, let's say google.
[No audio]
So as it's going down no, actually it's actually not
going to Google, it's actually going to the IP address.
So tcpdump actually is not going to give you Google.
It's actually going to give you the IP address.
So let me grep for 162.243.10.151
and there you go.
See, it's telling me my machine, which name
is myfirstlinuxvm is going to
the Google with the protocol ICMP.
Anytime you ping a machine, that
protocol is known as ICMP.
And so this machine, my machine is using ICMP protocol and
pinging this machine which is Google,
the protocol is listed echo request,
and again this machine which is Google, coming
back to my machine and acknowledging the traffic.
It's getting reply.
So I am look at this very carefully.
This is my machine going to Google with this
protocol and requesting to send the sequence the length
of 64 bit 64 bytes right here.
Sequence time ICMP.
That's what it's telling you.
So then when it goes there, the Google is
coming back to me and say hey, I got
it, you are the one who's pinging me.
And I'm replying back to you and
telling you yes, I'm alive, I'm available.
So it's just a simple way of doing tcpdump.
If you are trying to troubleshoot an issue where
your machine is talking to a database server, then
I would say put that database server IP address
or use the protocol in the tcpdump.
There are different options to use the protocol
specifically for that port, sorry, not the protocol,
I meant to say the port.
If you are going to, let's say 192.168.1.13,
and that is your database server and that is
using a protocol, let's say 2049 no, sorry, the
database protocol is sorry, the port number that is
used on database is anyway, I forgot.
But I hope you understand what I'm trying to say.
So you could actually specify that port number to listen,
whether this is listing on the port number or not.
So, tcpdump is a very long kind of discussion
that I could go into, but I want you to
try it on and this will give you a lot
of information when you're troubleshooting traffic issues.
Next one is lsof, list open files and
-i, -p, and -n.
This will give you what is open, what is listening.
So, let's say if my machine, this is the one
is actually running a service, let's say SSH and I
cannot connect from this machine to this machine using SSH,
then I will come into this machine, I'll run that
command and I will grep for 22.
And as you could see right here, this will
list all established connection and also listening connection.
Right here it says SSH is listening on port 22.
So I could tell, hey, my machine is listening.
I have no problem, if something is your
machine that's not listening or maybe you haven't
listening, maybe my machine is listening and probably
I have a firewall that is blocking any
traffic coming from this machine to my machine.
So I have to check firewall as well.
Okay, then the next one I have is netstat
[No audio]
-plnt.
This will pretty much give me the same information
that I just typed in a different way.
So this is another command.
Another thing that I was going to talk to you
about is Netstat is now deprecated and the newer command
that is used is ss which stands for Socket.
You could use the same options to list
all your ports that are open or listening.
You can also do man on ss and you will
find another utility to investigate sockets that are open.
Socket is something that runs on your computer which
actually attaches the port number and opens up every
time a traffic comes in on that port or
anything that comes in and trying to knock on
the door that has 22 label on it.
The last one is about check networking
and I have this command iftop.
And as I said earlier, iftop is
a lead that does not come with your
Linux operating system, you have to install that.
And to install that you have to run the command
yum install epel-release and then yum install iftop.
I have already installed this command in my system, so I'll
run it iftop, and you will see right here, this
is telling me about the transfer that is going from to
this machine and from this machine to any other place.
So if I'm using FTP to transfer some files
from one machine to another, or I'm using protocols
like NFS or SCP protocols, every time you use
them, every time you run those commands,
and if you wanted to know how much bandwidth
is used and how much megabytes or kilobytes per
second is being transferred, this command is amazing.
It will give you all that information from to
the rate and in kilobyte, how much it has
transferred, how much is remaining, and whole bunch of
information if you want to learn more about it.
I think when you install it, it actually
also installed the man pages, I believe.
Yes it did. And it says display bandwidth usage
on an interface by host.
So use that command.
If you do not have it, I highly recommend you
to install it with the instruction I have given you.
Next one is check system uptime, see
if your system has been rebooted recently.
The time is there with the system load
average. Moving on the check the logs,
all the logs are in /var/logs, and when you
do ls -l, ltr whichever option you prefer and
the best log which has all the information or log
that I always look for is messages. I could always
do tail -100 show me last 100 lines of messages,
and I could easily grep for error there's no error.
So I could just take out entire tail,
just cut it and grep for errors.
And here I will see if there are any errors that
have been reported that might help me find the issue.
I would also say look for, always put -i
by the way, I would also say put warning.
I will also say put error.
I will say put fail as in failure.
You'll find so many information that
could help you troubleshoot the issue.
Check the hardware status by
logging into the system console.
You could do that by logging in,
if it's HP, go into ILO, open up a browser, put in
the ILO IP and you will see the logs in there.
If it's Dell, put in the
Idrag and if it's other system.
Every system does have console access.
Find that out and log in and
find that hardware information or hardware message.
The last one are other tools there are
other tools that are available to you.
You have to download them as
well, just like I downloaded.
iftop these are the tools are also helpful.
Some of them like htop, iotop. iotop is helpful, which
works the same way for disks as iostat.
So these are some of the tools.
I want you to actually go through
them one by one, read about them.
They will definitely really help you
if you are pursuing system administration type of job or
Linux related job or any type of computer related job,
you will be using these type of commands quite often.
[No audio]