1
00:00:06,855 --> 00:00:08,723
- Now let's talk about log collection

2
00:00:08,723 --> 00:00:11,690
with Amazon CloudWatch Logs.

3
00:00:11,690 --> 00:00:14,824
With Amazon CloudWatch
Logs, we have the ability

4
00:00:14,824 --> 00:00:17,407
to stream our logs in real time

5
00:00:18,848 --> 00:00:21,594
to the Amazon CloudWatch Logs service.

6
00:00:21,594 --> 00:00:23,586
So, the way that this works is that

7
00:00:23,586 --> 00:00:27,118
we install the Amazon
CloudWatch Logs agent

8
00:00:27,118 --> 00:00:28,868
on our EC2 instances.

9
00:00:30,041 --> 00:00:33,838
We configure it to
watch whatever log files

10
00:00:33,838 --> 00:00:35,672
are being written to disk.

11
00:00:35,672 --> 00:00:38,535
It could be Apache or NGINX log files,

12
00:00:38,535 --> 00:00:40,549
it could be our application,

13
00:00:40,549 --> 00:00:44,292
whatever application is writing log files,

14
00:00:44,292 --> 00:00:46,337
the Amazon CloudWatch Log agent

15
00:00:46,337 --> 00:00:49,504
will essentially follow that log file.

16
00:00:50,660 --> 00:00:54,621
Whatever goes into it will
be captured and streamed

17
00:00:54,621 --> 00:00:59,249
in real time to the Amazon
CloudWatch Logs service.

18
00:00:59,249 --> 00:01:02,449
By default, Amazon CloudWatch logs

19
00:01:02,449 --> 00:01:05,496
will retain those log files indefinitely.

20
00:01:05,496 --> 00:01:08,219
We can shorten that if we wanted to,

21
00:01:08,219 --> 00:01:11,385
perhaps we only needed to store
certain things for a week.

22
00:01:11,385 --> 00:01:14,568
We could also have Amazon CloudWatch Logs

23
00:01:14,568 --> 00:01:17,925
archive those off to S3, so that we can

24
00:01:17,925 --> 00:01:21,149
very easily, as we've seen, pull those

25
00:01:21,149 --> 00:01:24,483
into Elastic MapReduce,
we can pull them into

26
00:01:24,483 --> 00:01:28,867
a data pipeline and transform
and do other things with them.

27
00:01:28,867 --> 00:01:31,867
We can also very easily stream those

28
00:01:33,264 --> 00:01:38,207
into Amazon Elasticsearch,
which is, very much like RDS,

29
00:01:38,207 --> 00:01:42,110
is a fully managed elastic search service.

30
00:01:42,110 --> 00:01:45,867
And it's also worth noting
that Amazon Elasticsearch

31
00:01:45,867 --> 00:01:48,713
includes, for those of
us who might be familiar,

32
00:01:48,713 --> 00:01:51,917
it includes Kibana, which is a very nice

33
00:01:51,917 --> 00:01:54,711
and very powerful UI tool

34
00:01:54,711 --> 00:01:57,357
for searching and performing analytics

35
00:01:57,357 --> 00:02:01,340
against records stored in Elasticsearch.

36
00:02:01,340 --> 00:02:05,507
So, this particular service is
incredibly powerful and rich.

37
00:02:07,088 --> 00:02:09,573
It enables us to capture these log files

38
00:02:09,573 --> 00:02:12,266
and very easily get them into a place

39
00:02:12,266 --> 00:02:14,832
where we can do more with them.

40
00:02:14,832 --> 00:02:16,377
It's very important that,

41
00:02:16,377 --> 00:02:18,554
especially in a distributed environment,

42
00:02:18,554 --> 00:02:22,304
where you have potentially
dozens to hundreds

43
00:02:23,578 --> 00:02:27,745
of EC2 instances, instead of
all of them being responsible

44
00:02:29,604 --> 00:02:32,497
for packaging their log files,

45
00:02:32,497 --> 00:02:36,192
writing scripts to get their
log files to various places,

46
00:02:36,192 --> 00:02:38,336
it's a whole lot easier and simpler

47
00:02:38,336 --> 00:02:40,993
just to install the CloudWatch Logs agent

48
00:02:40,993 --> 00:02:43,753
and allow that to handle the collection

49
00:02:43,753 --> 00:02:46,872
and aggregation of logs for you.

50
00:02:46,872 --> 00:02:49,785
Now, another thing that we
can do within CloudWatch Logs

51
00:02:49,785 --> 00:02:54,003
is we can search and filter that log data.

52
00:02:54,003 --> 00:02:56,727
So we do have a very specific syntax

53
00:02:56,727 --> 00:02:58,780
and I would encourage you to, again,

54
00:02:58,780 --> 00:03:01,225
make the documentation your best friend.

55
00:03:01,225 --> 00:03:05,141
I've been using Amazon web
services for seven years,

56
00:03:05,141 --> 00:03:07,523
just about every day over
the last seven years,

57
00:03:07,523 --> 00:03:09,393
and I still go to the documentation,

58
00:03:09,393 --> 00:03:13,122
just because it's so much
information to remember.

59
00:03:13,122 --> 00:03:17,712
So, you can search JSON,
a lot of the applications

60
00:03:17,712 --> 00:03:20,425
that I work with being written in node js

61
00:03:20,425 --> 00:03:24,541
leverage JSON and store our logs in JSON,

62
00:03:24,541 --> 00:03:27,633
and so yes, you can
search on those fields.

63
00:03:27,633 --> 00:03:31,776
We can create what we
call subscription filters,

64
00:03:31,776 --> 00:03:34,783
so not only can we go in and just type

65
00:03:34,783 --> 00:03:37,523
something like this, error exception,

66
00:03:37,523 --> 00:03:41,690
in the search field, which
would match messages like these,

67
00:03:45,223 --> 00:03:46,945
we could also turn those searches

68
00:03:46,945 --> 00:03:49,623
into what we call subscription filters,

69
00:03:49,623 --> 00:03:53,803
so that as these logs
come in in real time,

70
00:03:53,803 --> 00:03:56,136
each log message is compared

71
00:03:57,445 --> 00:03:59,342
against that subscription filter,

72
00:03:59,342 --> 00:04:01,969
and messages that pass through,

73
00:04:01,969 --> 00:04:03,798
messages that match that filter,

74
00:04:03,798 --> 00:04:07,078
could then be streamed into Kinesis,

75
00:04:07,078 --> 00:04:10,408
we could stream them into AWS Lambda,

76
00:04:10,408 --> 00:04:12,951
and do further processing from there.

77
00:04:12,951 --> 00:04:16,258
We could also create metric filters

78
00:04:16,258 --> 00:04:19,909
so that, for example, here,
if we created a metric filter

79
00:04:19,909 --> 00:04:24,076
based on this here, every
time we see the word error

80
00:04:25,536 --> 00:04:28,351
or the word exception,
we could count that,

81
00:04:28,351 --> 00:04:30,793
and say plus one, plus one,

82
00:04:30,793 --> 00:04:32,616
and then this would end up counting

83
00:04:32,616 --> 00:04:36,374
as a custom metric
within Amazon CloudWatch.

84
00:04:36,374 --> 00:04:39,596
From that, we could create alarms.

85
00:04:39,596 --> 00:04:44,436
So we might want to count,
you know, the number of 404s.

86
00:04:44,436 --> 00:04:48,603
In the case of a web server,
we might want to count

87
00:04:49,517 --> 00:04:51,519
the number of bytes that were transferred

88
00:04:51,519 --> 00:04:55,686
by that web server per request
or over a period of time.

89
00:04:56,773 --> 00:05:00,429
As our applications write their log files,

90
00:05:00,429 --> 00:05:03,158
we might want to count
the customer conversions.

91
00:05:03,158 --> 00:05:06,100
Every time a customer
closes a shopping cart

92
00:05:06,100 --> 00:05:09,218
due to a sale, we add that log message,

93
00:05:09,218 --> 00:05:12,991
we can configure our filter
patterns to search for that,

94
00:05:12,991 --> 00:05:16,379
and then count the number
of sales that are happening.

95
00:05:16,379 --> 00:05:18,805
So you can see between Amazon CloudWatch,

96
00:05:18,805 --> 00:05:23,659
Amazon CloudWatch Alarms,
and Amazon CloudWatch Logs,

97
00:05:23,659 --> 00:05:25,359
we have a very rich and powerful system

98
00:05:25,359 --> 00:05:28,428
for collecting metrics
and for collecting logs

99
00:05:28,428 --> 00:05:31,345
and that is Amazon CloudWatch Logs.