1
00:00:06,790 --> 00:00:08,280
- Now let's talk about log collections

2
00:00:08,280 --> 00:00:10,973
with Amazon CloudWatch Logs.

3
00:00:12,000 --> 00:00:15,850
So, before we get into the nature

4
00:00:15,850 --> 00:00:17,850
and details of CloudWatch logs,

5
00:00:17,850 --> 00:00:20,163
let's first look at where logs come from.

6
00:00:21,056 --> 00:00:23,090
And we have applications
running in different places.

7
00:00:23,090 --> 00:00:27,030
And we have other resources
that are generating logs.

8
00:00:27,030 --> 00:00:30,160
And so, a number of places,

9
00:00:30,160 --> 00:00:32,380
including the Elastic Load Balancer ,

10
00:00:32,380 --> 00:00:35,490
will generate access logs.

11
00:00:35,490 --> 00:00:38,000
We could have applications running on EC2

12
00:00:38,000 --> 00:00:39,210
that can be running

13
00:00:39,210 --> 00:00:42,210
within the elastic container service,

14
00:00:42,210 --> 00:00:43,453
within containers.

15
00:00:44,310 --> 00:00:46,460
We might have applications running

16
00:00:46,460 --> 00:00:47,710
within Lambda.

17
00:00:47,710 --> 00:00:50,544
Of course, Lambda will
generate its own logs,

18
00:00:50,544 --> 00:00:53,360
whether or not your application

19
00:00:53,360 --> 00:00:55,150
is writing any logs or not.

20
00:00:55,150 --> 00:00:58,050
Lambda will include its own logs.

21
00:00:58,050 --> 00:01:01,050
And then of course we have S3 access logs.

22
00:01:01,050 --> 00:01:04,060
So, again, if we're storing

23
00:01:04,060 --> 00:01:06,129
static content, for example;

24
00:01:06,129 --> 00:01:10,850
images, CSS pdf files and and so on in S3,

25
00:01:10,850 --> 00:01:12,720
and those are publicly available

26
00:01:13,715 --> 00:01:15,040
as far as web assets

27
00:01:15,040 --> 00:01:18,630
for some kind of a website application,

28
00:01:18,630 --> 00:01:21,190
then as those files are accessed,

29
00:01:21,190 --> 00:01:23,913
S3 will generate its own access logs.

30
00:01:24,770 --> 00:01:29,040
And then of course, we can
also collect VPC flow logs.

31
00:01:29,040 --> 00:01:33,000
So, an important component
of network security

32
00:01:33,000 --> 00:01:35,270
is understanding the nature of the traffic

33
00:01:35,270 --> 00:01:38,750
whether traffic is being
accepted or rejected.

34
00:01:38,750 --> 00:01:42,020
And so, you can look at at least

35
00:01:42,020 --> 00:01:44,650
metadata about network traffic

36
00:01:44,650 --> 00:01:46,860
with VPC flow logs.

37
00:01:46,860 --> 00:01:49,000
We also have CloudFront access logs.

38
00:01:49,000 --> 00:01:53,520
If our Load Balancer or
our S3 buckets are behind

39
00:01:53,520 --> 00:01:55,010
a CloudFront distribution,

40
00:01:55,010 --> 00:01:56,280
and our users are going through

41
00:01:56,280 --> 00:01:59,030
those edge locations
to retrieve that date,

42
00:01:59,030 --> 00:02:01,860
then CloudFront will generate access logs.

43
00:02:01,860 --> 00:02:05,030
We also have CloudTrail.
CloudTrail is a service

44
00:02:05,030 --> 00:02:08,910
that collects and records API calls.

45
00:02:08,910 --> 00:02:11,959
So anything you do through
through the AWS API,

46
00:02:11,959 --> 00:02:14,270
for the purpose of creating

47
00:02:14,270 --> 00:02:18,150
or managing some Amazon resource,

48
00:02:18,150 --> 00:02:21,330
then those calls to the API are recorded.

49
00:02:21,330 --> 00:02:23,340
The credentials that are used

50
00:02:23,340 --> 00:02:25,530
and metadada about that request

51
00:02:25,530 --> 00:02:28,620
are recorded through CloudTrail.

52
00:02:28,620 --> 00:02:30,903
And of course, there are
plenty of more places

53
00:02:30,903 --> 00:02:34,170
where logs could be generated from.

54
00:02:34,170 --> 00:02:36,730
So again, in terms of capturing logs,

55
00:02:36,730 --> 00:02:38,840
let's take a look at this diagram.

56
00:02:38,840 --> 00:02:40,100
And you'll see here, ofcourse,

57
00:02:40,100 --> 00:02:41,637
we have our users out there

58
00:02:41,637 --> 00:02:46,570
and our users are accessing
our dynamic content

59
00:02:46,570 --> 00:02:50,710
and our static content,
through those edge locations.

60
00:02:50,710 --> 00:02:55,170
Of course, those edge
locations, those CloudFront,

61
00:02:55,170 --> 00:03:00,170
is sending CloudFront access logs to S3.

62
00:03:01,910 --> 00:03:02,810
And then of course,

63
00:03:02,810 --> 00:03:07,260
we have our Load Balancer here,

64
00:03:07,260 --> 00:03:09,429
and our Load Balancer is sending

65
00:03:09,429 --> 00:03:13,513
access logs directly to S3.

66
00:03:14,610 --> 00:03:17,413
We have our VPC flow logs,

67
00:03:17,413 --> 00:03:19,250
which are being collected

68
00:03:19,250 --> 00:03:21,020
directly by CloudWatch logs.

69
00:03:21,020 --> 00:03:23,680
Our Lambda functions are being collected

70
00:03:23,680 --> 00:03:25,015
by CloudWatch logs.

71
00:03:25,015 --> 00:03:27,670
And then, of course, we could,

72
00:03:27,670 --> 00:03:31,470
for our applications running on EC2,

73
00:03:31,470 --> 00:03:33,220
we have a couple of options.

74
00:03:33,220 --> 00:03:36,570
We could use the CloudWatch logs API

75
00:03:36,570 --> 00:03:39,540
to have our application logged directly

76
00:03:39,540 --> 00:03:41,040
to CloudWatch logs.

77
00:03:41,040 --> 00:03:43,980
Or we can have our applications log

78
00:03:43,980 --> 00:03:45,940
to local files,

79
00:03:45,940 --> 00:03:48,160
and then have the CloudWatch agent

80
00:03:48,160 --> 00:03:50,520
essentially stream those files

81
00:03:50,520 --> 00:03:54,150
in near realtime to the
CloudWatch log service.

82
00:03:54,150 --> 00:03:56,620
So either way, we have our choice there

83
00:03:56,620 --> 00:03:57,680
in terms of how we get

84
00:03:57,680 --> 00:04:00,970
our application logs to CloudWatch logs.

85
00:04:00,970 --> 00:04:03,730
But, the point here is that, either way,

86
00:04:03,730 --> 00:04:07,880
we can collect those
logs to CloudWatch logs.

87
00:04:07,880 --> 00:04:12,880
So the benefit here is that those days

88
00:04:13,030 --> 00:04:15,750
of shelling in to the individual instances

89
00:04:15,750 --> 00:04:18,167
and trying to find the log
files, "Where are they?

90
00:04:18,167 --> 00:04:19,596
"Are they in (speaker mumbles),

91
00:04:19,596 --> 00:04:22,407
or are they up somewhere?"

92
00:04:23,840 --> 00:04:25,370
Trying to find them to begin with

93
00:04:25,370 --> 00:04:27,050
and then getting to that directory

94
00:04:27,050 --> 00:04:29,860
only to find out that they've been rotated

95
00:04:29,860 --> 00:04:32,090
or maybe they haven't.

96
00:04:32,090 --> 00:04:33,430
Maybe they're so large

97
00:04:33,430 --> 00:04:36,430
that you can't find
what you're looking for.

98
00:04:36,430 --> 00:04:37,500
And so I've been there.

99
00:04:37,500 --> 00:04:38,333
I've done that.

100
00:04:38,333 --> 00:04:40,090
It's a painful process.

101
00:04:40,090 --> 00:04:43,639
And so the idea here with CloudWatch logs

102
00:04:43,639 --> 00:04:46,210
is that one of the
patterns of Cloud computing

103
00:04:47,460 --> 00:04:51,183
is to treat logs as streams.

104
00:04:51,183 --> 00:04:54,840
You could have within EC2,

105
00:04:54,840 --> 00:04:57,240
remember that we could have

106
00:04:57,240 --> 00:05:00,140
just about any number of instances here.

107
00:05:00,140 --> 00:05:02,260
We could have two, we
could have two hundred,

108
00:05:02,260 --> 00:05:03,810
we can have two thousand.

109
00:05:03,810 --> 00:05:06,260
And, like I've mentioned before,

110
00:05:06,260 --> 00:05:10,090
that these instances are
essentially disposable.

111
00:05:10,090 --> 00:05:11,780
They could be an issue

112
00:05:11,780 --> 00:05:13,770
where they just go away on their own,

113
00:05:13,770 --> 00:05:16,420
or auto scaling could terminate them,

114
00:05:16,420 --> 00:05:18,800
or we may decide to terminate them.

115
00:05:18,800 --> 00:05:21,480
Or they could be terminated by accident.

116
00:05:21,480 --> 00:05:25,310
We can't tell on the EC2
instance being there.

117
00:05:25,310 --> 00:05:26,690
Therefore, we can't count

118
00:05:26,690 --> 00:05:30,510
on the logs being there
for any length of time.

119
00:05:30,510 --> 00:05:34,110
So, when you have such a dynamic system

120
00:05:34,110 --> 00:05:36,780
where EC2 instances are coming and going,

121
00:05:36,780 --> 00:05:37,750
it's really important

122
00:05:37,750 --> 00:05:40,030
that we have one central place

123
00:05:40,030 --> 00:05:42,540
to which we can record our logs.

124
00:05:42,540 --> 00:05:44,130
That way we don't have to worry about

125
00:05:44,130 --> 00:05:45,600
the machine being gone.

126
00:05:45,600 --> 00:05:47,870
We know that we've already collected

127
00:05:47,870 --> 00:05:50,520
as much logs as we can from that instance

128
00:05:50,520 --> 00:05:51,723
before it went away.

129
00:05:52,620 --> 00:05:54,980
Not only do we have one
place to collect it,

130
00:05:54,980 --> 00:05:56,240
but we have one place

131
00:05:56,240 --> 00:05:58,700
from which we can pull those logs

132
00:05:58,700 --> 00:06:00,360
instead of having to hunt them down

133
00:06:00,360 --> 00:06:03,803
across all of these
various disparate places.

134
00:06:05,220 --> 00:06:06,840
Now of course, like I mentioned earlier,

135
00:06:06,840 --> 00:06:09,160
we also have CloudTrail.

136
00:06:09,160 --> 00:06:14,060
Now, CloudTrial could go directly to S3.

137
00:06:14,060 --> 00:06:18,150
We could have CloudTrail send files to S3,

138
00:06:18,150 --> 00:06:19,640
but we also have the option

139
00:06:19,640 --> 00:06:24,610
of having CloudTrail send
the various API events

140
00:06:24,610 --> 00:06:26,400
directly to CloudWatch logs,

141
00:06:26,400 --> 00:06:30,160
so that we get more of a realtime insight

142
00:06:30,160 --> 00:06:32,040
into what's happening with CloudTrail

143
00:06:32,040 --> 00:06:33,590
and what's happening to our credentials

144
00:06:33,590 --> 00:06:35,580
and AWS environment.

145
00:06:35,580 --> 00:06:40,250
We also have Route 53 DNS queries.

146
00:06:40,250 --> 00:06:42,460
So as our users are out there

147
00:06:42,460 --> 00:06:44,860
and they are making DNS queries

148
00:06:44,860 --> 00:06:46,710
to various domain names,

149
00:06:46,710 --> 00:06:49,650
and maybe we use different subdomains

150
00:06:49,650 --> 00:06:50,730
for different purposes.

151
00:06:50,730 --> 00:06:52,500
Maybe we use different subdomains

152
00:06:52,500 --> 00:06:55,070
for different marketing campaigns.

153
00:06:55,070 --> 00:06:59,690
So perhaps there is some
kind of data to be found

154
00:06:59,690 --> 00:07:02,310
or some type of value to be found

155
00:07:02,310 --> 00:07:04,340
in analyzing DNS queries.

156
00:07:04,340 --> 00:07:06,540
And so we can capture those as well.

157
00:07:06,540 --> 00:07:10,600
Route 53 can send its DNS query data

158
00:07:10,600 --> 00:07:12,150
to CloudWatch logs.

159
00:07:12,150 --> 00:07:16,620
And so, we could also have CouldWatch logs

160
00:07:16,620 --> 00:07:20,201
archive those logs off to S3

161
00:07:20,201 --> 00:07:22,900
and then perhaps from S3,

162
00:07:22,900 --> 00:07:26,560
we can have EMR or Athena,

163
00:07:26,560 --> 00:07:28,290
pulls those logs and perform

164
00:07:28,290 --> 00:07:29,620
some kind of an analysis

165
00:07:29,620 --> 00:07:33,660
or a sequel query on the data from S3.

166
00:07:33,660 --> 00:07:37,400
And maybe that kind of thing is performed.

167
00:07:37,400 --> 00:07:39,240
Some kind of processing is performed,

168
00:07:39,240 --> 00:07:40,900
in the first week,

169
00:07:40,900 --> 00:07:43,310
or the first 30 days of those logs.

170
00:07:43,310 --> 00:07:44,661
And then after that,

171
00:07:44,661 --> 00:07:46,800
perhaps we need to keep those logs

172
00:07:46,800 --> 00:07:48,970
for some extended period of time

173
00:07:48,970 --> 00:07:52,441
because of legal or regulatory compliance.

174
00:07:52,441 --> 00:07:54,730
We also have the ability

175
00:07:54,730 --> 00:07:57,750
to configure those life cycle rules,

176
00:07:57,750 --> 00:08:01,290
and have those logs archived to Glacier.

177
00:08:01,290 --> 00:08:03,640
So we could lower our long term cost

178
00:08:03,640 --> 00:08:06,360
of archiving data that we have to keep.

179
00:08:06,360 --> 00:08:09,320
So, again, when you think
about your applications

180
00:08:09,320 --> 00:08:12,321
running in these various places within AWS

181
00:08:12,321 --> 00:08:16,280
whether they're running
on EC2 or in containers,

182
00:08:16,280 --> 00:08:17,960
whether they're running in Lambda,

183
00:08:17,960 --> 00:08:22,300
and you want a central
place to collect those logs,

184
00:08:22,300 --> 00:08:24,773
take a look at Amazon CloudWatch logs.