1
00:00:06,277 --> 00:00:07,200
- In this video

2
00:00:07,200 --> 00:00:10,053
we are going to explore
callback and timer plugins.

3
00:00:11,000 --> 00:00:12,890
The use of callback and timer plugins

4
00:00:12,890 --> 00:00:14,270
is completely different.

5
00:00:14,270 --> 00:00:16,293
It's more about performance measuring.

6
00:00:17,210 --> 00:00:19,320
So callback plugins extend Ansible

7
00:00:19,320 --> 00:00:21,860
by modifying how it respond to events.

8
00:00:21,860 --> 00:00:23,010
And to do their work,

9
00:00:23,010 --> 00:00:26,193
callback plugins modify the
output of the ansible commands.

10
00:00:27,170 --> 00:00:30,810
You can use ansible-doc -t callback -l

11
00:00:30,810 --> 00:00:32,230
for a complete list,

12
00:00:32,230 --> 00:00:34,420
and to get an impression of what exactly

13
00:00:34,420 --> 00:00:36,950
you can do with these callback plugins.

14
00:00:36,950 --> 00:00:39,515
And specify the name
of the specific plugin

15
00:00:39,515 --> 00:00:42,600
as an argument to ansible-doc -t callback

16
00:00:42,600 --> 00:00:45,280
if you want to get help, use it.

17
00:00:45,280 --> 00:00:47,500
If you are going to use callback plugins

18
00:00:47,500 --> 00:00:48,420
you need to enable them

19
00:00:48,420 --> 00:00:51,190
in ansible.cfg and in order to do so

20
00:00:51,190 --> 00:00:54,900
you use the callback_whitelist setting

21
00:00:54,900 --> 00:00:56,380
that looks like callback_whitelist

22
00:00:56,380 --> 00:01:00,660
is timer, profile_tests, cgroup_perf_recap

23
00:01:00,660 --> 00:01:04,480
if you want to include all
of these callback plugins,

24
00:01:04,480 --> 00:01:08,363
really it depends on which
callback plugin you want to use.

25
00:01:09,690 --> 00:01:12,880
Let's talk about cgroup_perf_recap.

26
00:01:12,880 --> 00:01:14,690
What is cgroup_perf_recap?

27
00:01:14,690 --> 00:01:16,900
Well, that is a callback
plugin that allows you to

28
00:01:16,900 --> 00:01:20,370
monitor detailed resource
usage when running a playbook.

29
00:01:20,370 --> 00:01:22,260
And you can use it to measure impacts

30
00:01:22,260 --> 00:01:25,520
after changing the forks
parameter for instance.

31
00:01:25,520 --> 00:01:29,300
This is all about seeing
how much resources

32
00:01:29,300 --> 00:01:31,243
are spent where exactly.

33
00:01:32,780 --> 00:01:34,810
Now the challenge with cgroup_perf_recap

34
00:01:34,810 --> 00:01:37,600
is that it is based on
Cgroups functionality.

35
00:01:37,600 --> 00:01:40,020
And Cgroups allow for
system performance tuning

36
00:01:40,020 --> 00:01:41,290
and monitoring.

37
00:01:41,290 --> 00:01:43,570
Cgroups is a generic Linux setting.

38
00:01:43,570 --> 00:01:46,080
And that means that
your Linux distribution

39
00:01:46,080 --> 00:01:48,550
must offer support for Cgroups.

40
00:01:48,550 --> 00:01:52,030
And that might to be configured

41
00:01:52,030 --> 00:01:54,963
before you can use cgroup_perf_recap.

42
00:01:55,840 --> 00:01:58,920
At this moment cgroup_perf_recap needs

43
00:01:58,920 --> 00:02:01,240
Cgroups version one support.

44
00:02:01,240 --> 00:02:02,940
The additional challenge is that

45
00:02:02,940 --> 00:02:05,950
on recent Linux distributions like Fedora

46
00:02:05,950 --> 00:02:08,450
Cgroups version two has been introduced.

47
00:02:08,450 --> 00:02:11,540
And if you are on rail eight for instance,

48
00:02:11,540 --> 00:02:14,893
there Cgroups version
one is still the default.

49
00:02:16,150 --> 00:02:18,240
Now, if you want to make sure

50
00:02:18,240 --> 00:02:21,390
that you are using Cgroup version one

51
00:02:21,390 --> 00:02:22,950
this is what you can do.

52
00:02:22,950 --> 00:02:25,320
You can edit the etc default grub

53
00:02:25,320 --> 00:02:28,900
on the control host that is
using Cgroup version two.

54
00:02:28,900 --> 00:02:33,330
And you use the parameter
systemd_unified_cgroup_hierarchy=0

55
00:02:33,330 --> 00:02:36,050
to the end of the line
that loads the kernel.

56
00:02:36,050 --> 00:02:38,120
And that will give you a presentation

57
00:02:38,120 --> 00:02:41,793
of Cgroup output that is
compatible with cgroup_perf_recap.

58
00:02:42,680 --> 00:02:47,680
Do not forget if you make
changes to the Grub configuration

59
00:02:47,710 --> 00:02:50,750
you will need to learn sudo grub2-mkconfig

60
00:02:50,750 --> 00:02:52,520
or whatever you need to do

61
00:02:52,520 --> 00:02:56,190
on your distribution to
rewrite Grub configuration.

62
00:02:56,190 --> 00:02:58,310
This command works on Fedora.

63
00:02:58,310 --> 00:03:01,633
Other distributions may
need a different approach.

64
00:03:02,730 --> 00:03:04,550
Now let me walk you through a demo

65
00:03:04,550 --> 00:03:09,550
and in order to do so this
time I have prepared a slide

66
00:03:09,600 --> 00:03:12,780
because this slide contains
a lot of information

67
00:03:12,780 --> 00:03:14,530
and I wanted to make it easy for you

68
00:03:14,530 --> 00:03:18,750
to run all of these commands.

69
00:03:18,750 --> 00:03:19,863
Let me demonstrate.

70
00:03:20,840 --> 00:03:24,090
So I need to start in the ansible.cfg

71
00:03:25,790 --> 00:03:30,210
and in the ansible cfg
in the defaults setting

72
00:03:32,130 --> 00:03:36,175
I'm going to include callback,

73
00:03:36,175 --> 00:03:41,175
callback_whitelist
equals cgroup_perf_recap.

74
00:03:45,160 --> 00:03:46,720
We need a corresponding setting.

75
00:03:46,720 --> 00:03:49,890
And that setting is in
an additional section

76
00:03:49,890 --> 00:03:51,500
of the configuration file

77
00:03:51,500 --> 00:03:53,353
which will be callback_cgroup_perf_recap.

78
00:03:59,800 --> 00:04:04,800
And in there we need control_group
equals ansible_profile.

79
00:04:07,450 --> 00:04:09,330
Now thing is if you work with Cgroups

80
00:04:10,220 --> 00:04:12,660
you need to create a so-called controller.

81
00:04:12,660 --> 00:04:14,020
And in this controller

82
00:04:14,020 --> 00:04:16,670
you are going to put
whatever you want to measure.

83
00:04:16,670 --> 00:04:19,980
And this defines the name
of the cgroup controller.

84
00:04:19,980 --> 00:04:22,990
You need Linux commands
to create these cgroups.

85
00:04:22,990 --> 00:04:26,940
And the command is cgcreates.yaml

86
00:04:31,150 --> 00:04:33,610
Just to make sure that
we do this Ansible style

87
00:04:33,610 --> 00:04:35,710
here we can see how it works.

88
00:04:35,710 --> 00:04:40,200
So we need to install
Cgroups and in order to do so

89
00:04:40,200 --> 00:04:43,610
we need to use become
parameters on localhost

90
00:04:43,610 --> 00:04:45,260
which is the control note.

91
00:04:45,260 --> 00:04:48,750
And first we need to
make sure that we install

92
00:04:48,750 --> 00:04:52,650
the packets that contains
the cgcreate binary

93
00:04:52,650 --> 00:04:55,410
which happens to be libcgroup-tools

94
00:04:56,360 --> 00:05:00,203
without libcgroup-tools insulation
you cannot use cgcreate.

95
00:05:01,793 --> 00:05:04,970
Then we create the Cgroups
and I'm using command

96
00:05:04,970 --> 00:05:07,840
and I know command is
not a prettiest option,

97
00:05:07,840 --> 00:05:09,330
but it's the only option if there is

98
00:05:09,330 --> 00:05:10,843
no specific Ansible model.

99
00:05:11,930 --> 00:05:13,680
So in this command cgcreate

100
00:05:14,620 --> 00:05:16,797
I'm using -a ansible:ansible
-t ansible:ansible

101
00:05:19,290 --> 00:05:24,290
which is about the permissions
I'm using ansible_profile

102
00:05:25,310 --> 00:05:27,410
which is a Cgroup I want to create

103
00:05:27,410 --> 00:05:30,300
and I want to put in three types of data.

104
00:05:30,300 --> 00:05:31,820
These are the common types of data,

105
00:05:31,820 --> 00:05:33,220
so don't worry about this.

106
00:05:33,220 --> 00:05:35,810
Just make sure that you
include CPU accounting,

107
00:05:35,810 --> 00:05:39,960
memory, and PIDs so that you can work

108
00:05:39,960 --> 00:05:41,720
with all of these,

109
00:05:41,720 --> 00:05:45,140
and also make sure that
you replace ansible:ansible

110
00:05:45,140 --> 00:05:47,168
with the actual name of the user

111
00:05:47,168 --> 00:05:50,310
that is going to use Ansible.

112
00:05:50,310 --> 00:05:55,310
So ansible-playbook
cgcreate.yaml is doing what?

113
00:05:57,390 --> 00:05:58,890
It's generating an error.

114
00:05:58,890 --> 00:06:01,330
That's an interesting one
cannot open, blah, blah.

115
00:06:01,330 --> 00:06:03,280
Why is it generating this error?

116
00:06:03,280 --> 00:06:05,760
Well, in this error, you can already see

117
00:06:05,760 --> 00:06:09,900
that it is trying to refer
to the cgroups information

118
00:06:12,700 --> 00:06:17,030
but the cgroups information
does not yet exist.

119
00:06:17,030 --> 00:06:19,550
Now the playbook is creating it.

120
00:06:19,550 --> 00:06:23,120
So let's run it again
and see what it is doing.

121
00:06:23,120 --> 00:06:27,690
And there, we can see
disappointing information.

122
00:06:27,690 --> 00:06:31,280
It does show information,
but not in the right way.

123
00:06:31,280 --> 00:06:35,190
And that is because we
need to run the playbook

124
00:06:35,190 --> 00:06:36,407
using cgexec.

125
00:06:38,586 --> 00:06:41,419
So cgexec -g cpuacct, memory, pids

126
00:06:46,455 --> 00:06:49,122
ansible_profile ansible-playbook

127
00:06:53,270 --> 00:06:56,100
on let's use cgcreate.yaml.

128
00:06:59,490 --> 00:07:01,170
So what is this command doing?

129
00:07:01,170 --> 00:07:03,533
All in this command we are telling cgexec

130
00:07:05,950 --> 00:07:08,427
that it needs to run
this specific command.

131
00:07:08,427 --> 00:07:11,190
And the command in this
case is ansible-playbook.

132
00:07:11,190 --> 00:07:13,600
We need to run the ansible-playbook

133
00:07:13,600 --> 00:07:17,380
within the Cgroup and the
Cgroup is ansible_profile

134
00:07:17,380 --> 00:07:18,910
and we are going to record data

135
00:07:18,910 --> 00:07:22,590
about CPU accounting, memories and PIDs.

136
00:07:22,590 --> 00:07:24,760
So here we go, we run it again.

137
00:07:24,760 --> 00:07:28,010
And this time we do see information.

138
00:07:28,010 --> 00:07:30,560
We do see information regarding memory,

139
00:07:30,560 --> 00:07:34,673
regarding CPU usage,
regarding the number of PIDs

140
00:07:34,673 --> 00:07:39,230
for all of the different
steps in the playbook.

141
00:07:39,230 --> 00:07:41,910
So it's measuring this
for every single task,

142
00:07:41,910 --> 00:07:44,723
and this makes this a very precise tool

143
00:07:44,723 --> 00:07:49,723
to measure exact performance
utilization of your playbooks,

144
00:07:50,200 --> 00:07:52,110
but do keep in mind, it only works

145
00:07:52,110 --> 00:07:55,180
if you run your playbooks through cgexec,

146
00:07:55,180 --> 00:07:58,293
without cgexec you won't see any result.

147
00:07:59,480 --> 00:08:02,610
Now Cgroup functionality is pretty cool

148
00:08:02,610 --> 00:08:04,380
but it involves a lot of work.

149
00:08:04,380 --> 00:08:06,730
And sometimes you might
want to measure performance

150
00:08:06,730 --> 00:08:09,270
using something a little
bit more lightweight.

151
00:08:09,270 --> 00:08:13,280
And that is where timer
can be interesting.

152
00:08:13,280 --> 00:08:16,540
Timer just shows the duration
of playbook execution

153
00:08:16,540 --> 00:08:20,370
or profile_tasks which shows
start time of each task

154
00:08:20,370 --> 00:08:22,800
as well as how much time is spent

155
00:08:22,800 --> 00:08:26,070
or profile_roles which is
doing the same for roles.

156
00:08:26,070 --> 00:08:29,540
And I'm going to demonstrate that as well.

157
00:08:29,540 --> 00:08:32,510
In this demo we are going to start

158
00:08:32,510 --> 00:08:35,660
by enabling these plugins.

159
00:08:35,660 --> 00:08:39,290
And once they are enabled, they
don't require anything else.

160
00:08:39,290 --> 00:08:40,580
We can just run on them.

161
00:08:40,580 --> 00:08:44,193
And that is what you are probably
going to like about these.

162
00:08:45,990 --> 00:08:50,990
So let me use the ansible
cfg, that's where we start.

163
00:08:52,250 --> 00:08:56,450
And I am just going to comment
out the Cgroup information

164
00:08:59,822 --> 00:09:04,822
and I am going to use a
new callback_whitelist

165
00:09:06,460 --> 00:09:10,840
and set that to timer comma profile_tasks

166
00:09:13,390 --> 00:09:14,943
and profile_roles.

167
00:09:19,282 --> 00:09:22,800
And then I am going to
run my playbook again

168
00:09:23,950 --> 00:09:26,023
without a cgexec this time.

169
00:09:26,980 --> 00:09:29,990
And we can see information.

170
00:09:29,990 --> 00:09:32,758
When did it start, how
much time was created,

171
00:09:32,758 --> 00:09:34,640
and we can see a nice summary

172
00:09:34,640 --> 00:09:37,700
of how much time was spent everywhere.

173
00:09:37,700 --> 00:09:42,700
Now, the thing is that these
plugins are not as detailed.

174
00:09:42,743 --> 00:09:46,679
They just show you current execution time

175
00:09:46,679 --> 00:09:49,490
but in some cases, that might be enough.

176
00:09:49,490 --> 00:09:51,220
So if you just want information

177
00:09:51,220 --> 00:09:54,540
about time that was spent
handling a specific task

178
00:09:54,540 --> 00:09:57,400
have a look at timer,
profile_task, profile_roles.

179
00:09:57,400 --> 00:09:59,830
If you want detailed
information about memory

180
00:09:59,830 --> 00:10:02,600
use usage in every single
part of the playbook

181
00:10:02,600 --> 00:10:04,590
definitely go for Cgroups.

182
00:10:04,590 --> 00:10:06,300
Both of them are very interesting

183
00:10:06,300 --> 00:10:08,603
but it really depends
on what you want to do.