- In modern Linux, we have this important feature, cgroups.
Cgroups is all about resource allocation,
and reservation, and limitation.
And systemd is taking care of it.
So cgroups or control groups plays resources in controllers
that represent the type of resource.
The most significant default controllers are cpu,
memory, and blkio,
and they allow you to work with well CPU restrictions,
memory restrictions, as well as block IO.
And these controllers are subdivided in a tree structure
where different weights or limits
are applied to each branch.
Each of these branches is a cgroups,
and one or more processes are assigned to a cgroup.
Cgroups can be applied from the command line
or from systemd.
In the past, you could manually create cgroups
using the cgconfig service and the cgred process.
But then we are talking about the early 2000s
before systemd became a relevant thing.
In all cases, cgroup settings are written
to /sys/fs/cgroups.
So that's a sys file system that we talked about before.
The pseudo file system that is used
for managing hardware properties.
And that is where you can go
if you wanna know what is happening under the hood.
In a cgroups environment, we have slices.
And slices are the primary division
of your operating system.
They apply to CPU, to blkio, to memory,
and that's three of them.
The system slice, which is for system processes and daemons.
The machine slice, which is for virtual machines,
as well as containers.
And the user slice, which is for user settings.
Every user gets its own slice by default,
which perfectly isolates the resources allocated to one user
from the resources allocated to another user,
and which guarantees that every user has the same claim
to system resources.
Apart from these default slices,
you can also create custom slices.
All right, let me make a drawing
to explain how the systemd slices relate to one another.
So we have the system slice, we have the user slice,
and there is the machine slice.
And these are pairs to one another.
And when we talk about cgroups,
you might be working with CPU shares.
And the CPU shares are assigned to the different slices.
So if all of the slices have CPU shares of 1024,
that is the relative weight between the different slices.
That means that at the slice level,
if you have full activity in all of the slices,
and each slice will get 1/3 of the available CPU resources.
Now the interesting thing is that within a slice,
and that goes for each of these slices,
you can work with scopes.
And on these scopes you can set CPU shares as well.
So let's say that we have 1024 here and 1024 here,
but in a machine slice, we also have scopes
and maybe in a machine slice we have four of them
with 1024 each.
Then this 1024 relates to this one.
So the scopes decide within the slice
how much CPU shares they get.
Likewise for here.
But even if this machine slice
has twice the amount of scopes,
at the slice level, it's still 1024.
So if you have four processes running here in the scopes
and two processes running here,
this will get half of the CPU resources.
And all these processes in the machine slice
get help of the CPU resources as well.
Within a scope or directly within a slice,
you can have the different services or units.
And each system, the unit,
well, typically that will be a service,
will get its CPU share as well.
So 1024, and 512, and 2048,
where the numbers explain relative weight
between these different services within the same level,
but still within the context of the limitation
that applies to the slice,
which is within the context of the limitation
that applies to the system unit.
As a result, some surprising things might be happening.
Let's say we have user Bob,
and user Bob is starting a process.
Now, what do we get?
We get a bob slice at that moment.
And this bob slice, if user Bob is the only user around,
then user Bob is starting a very active job.
He will get all the CPU shares within the user slice.
And that means that one user
is capable of getting an equal amount of CPU shares
as all of your units within the system scope.
And that's definitely something to be aware of.
So let me show you how to use cgroups
in a systemd environment.
All right, in order to run this demo,
you need access to the course Git repository.
In case you have not yet installed it,
git clone https://github.com/sandervanvugt/lfcs.
And in that course Git repository,
you will find stress1.service, stress2.service,
which are custom systemd unit files.
So what is in there?
Well, stress1.service is a very simple service.
The type is simple and it is running a dd process.
And this dd process is going to cost a 100% system load.
And CPUShares,
that's a parameter that this demo is all about.
CPUShares is set to 1024.
If you have a look at the contents of stress2.service,
you can see it's very similar,
with the only difference that CPUShares is set to 2048.
The meaning is that the CPUShares 2048
is getting twice the amount of CPU cycles as the other one.
Now, in order to run these custom services,
we should copy them to the appropriate location,
which will be /etc/systemd/system.
You remember there's /usr/lib/systemd/system,
that's where unit files that come from packages.
So you shouldn't touch them yourself.
There is /etc/systemd system, that's for your own stuff.
And this is typically my own stuff.
Now, let's run it, systemctl start on stress1,
as well as stress2.
And then let's observe on top what is going on.
And what do we see?
Well, we see something that might surprise you.
And that is that the dd processes
are about consuming almost 100%.
So we don't see the difference in CPUShares.
And that's a very good reason for that,
and that is because this is a multi-CPU system.
Let me press 1 from the top interface.
Let's have a look at the third line
where you can now see Cpu0 and Cpu1.
We have multiple CPUs,
and that is why this demo seems to be failing.
But fortunately, thanks to the sys pseudo file system,
there's something that we can do about it.
I am going to use echo 0 > /sys/bus/.
And then I need /cpu/devices/cpu1/online.
Now what is this?
Well, the /sys/bus/cpu/devices/cpu1/online file
is what you use to enable or disable Cpu1.
So if I echo a zero to that,
then suddenly we have a one CPU system
instead of a two CPU system.
And if you get back to top,
I press 1 again from the top interface.
Now, we can see that indeed, there is only one CPU.
And you can also see the cgroups in action.
So we have one dd process process
getting twice the amount of CPU cycles as the other one.
But, hey, there is one thing that you need to be aware of.
Let me open a new window.
And in this new window,
let me move it to the lower-right corner
so that we can still see top in the background.
As an ordinary user,
I'm going to use while true; do true; done.
And now we need to observe what is happening.
You see what's happening?
Best process getting about 50%,
and both dd processes getting about 50% as well.
How come?
Well, that is for the simple reason
that we are in the user slice
for this while true; do true; thing
and we are in the system slice for the dd processes.
And by default, the user slice and the system slice
have an equal weight.
And that is why one ordinary user process
is capable of of pushing away these other processes,
and that's not good.
Let me use Control + C, and let me quit these these things
so that we can move forward.
Oh, one thing by the way.
I want to use systemd-cg for cgroup.
And there we have a cgroup top and a cgroup ls.
So here is a systemd-cgtop
where you can see where the activity is.
And you can see clearly indicated,
stress2.service and stress1.service
are the most active processes right now.
An alternative view is cgls.
Cgls is showing a list of everything that is happening
in all of the cgroups.
This is not convenient
for monitoring what's going on right now
and which process is busiest,
but it is convenient to get an overview
of all these different slices and services.
Now, let me finish this by using killall on $ pidof dd.
And that should make make them go away.
Back to top.
And, oh boy, the killall $ pid dd didn't work
so let me kill it manually.
Oh, there is number one, and K to kill,
and there is number two.
And now they are gone.
There's only one thing remaining,
and that's a few commands ago.
I offlined my Cpu1.
Now, by echoing a 1 into /sys/bus/cpu/devices/cpu1/online,
I am getting it back online again.
And you can still hear probably my computer making noise
because it has been so busy.
That will calm down in not too much time
because the really occupying processes
are now no longer active.
As you can see, the load average is already decreasing.
It's going slowly,
but it will reach value below one in not too much time.
Now, one more thing about these cgroups.
You remember the modification
that we made earlier to sshd.service?
Let me show it again.
There we go.
MemoryMax is four megabytes.
That's also cgroup functionality.
In Linux, on a process level,
you can set the maximum amount of memory.
And you know where else cgroups are being used?
In containers.
We'll later talk about containers as isolated processes.
And cgroups are one of the important pillars
in the working of containers, together with namespaces,
but we will talk about it in more detail later.