- Now let's talk about log collection
with Amazon CloudWatch Logs.
With Amazon CloudWatch Logs, we have the ability
to stream our logs in real time
to the Amazon CloudWatch Logs service.
So, the way that this works is that
we install the Amazon CloudWatch Logs agent
on our EC2 instances.
We configure it to watch whatever log files
are being written to disk.
It could be Apache or NGINX log files,
it could be our application,
whatever application is writing log files,
the Amazon CloudWatch Log agent
will essentially follow that log file.
Whatever goes into it will be captured and streamed
in real time to the Amazon CloudWatch Logs service.
By default, Amazon CloudWatch logs
will retain those log files indefinitely.
We can shorten that if we wanted to,
perhaps we only needed to store certain things for a week.
We could also have Amazon CloudWatch Logs
archive those off to S3, so that we can
very easily, as we've seen, pull those
into Elastic MapReduce, we can pull them into
a data pipeline and transform and do other things with them.
We can also very easily stream those
into Amazon Elasticsearch, which is, very much like RDS,
is a fully managed elastic search service.
And it's also worth noting that Amazon Elasticsearch
includes, for those of us who might be familiar,
it includes Kibana, which is a very nice
and very powerful UI tool
for searching and performing analytics
against records stored in Elasticsearch.
So, this particular service is incredibly powerful and rich.
It enables us to capture these log files
and very easily get them into a place
where we can do more with them.
It's very important that,
especially in a distributed environment,
where you have potentially dozens to hundreds
of EC2 instances, instead of all of them being responsible
for packaging their log files,
writing scripts to get their log files to various places,
it's a whole lot easier and simpler
just to install the CloudWatch Logs agent
and allow that to handle the collection
and aggregation of logs for you.
Now, another thing that we can do within CloudWatch Logs
is we can search and filter that log data.
So we do have a very specific syntax
and I would encourage you to, again,
make the documentation your best friend.
I've been using Amazon web services for seven years,
just about every day over the last seven years,
and I still go to the documentation,
just because it's so much information to remember.
So, you can search JSON, a lot of the applications
that I work with being written in node js
leverage JSON and store our logs in JSON,
and so yes, you can search on those fields.
We can create what we call subscription filters,
so not only can we go in and just type
something like this, error exception,
in the search field, which would match messages like these,
we could also turn those searches
into what we call subscription filters,
so that as these logs come in in real time,
each log message is compared
against that subscription filter,
and messages that pass through,
messages that match that filter,
could then be streamed into Kinesis,
we could stream them into AWS Lambda,
and do further processing from there.
We could also create metric filters
so that, for example, here, if we created a metric filter
based on this here, every time we see the word error
or the word exception, we could count that,
and say plus one, plus one,
and then this would end up counting
as a custom metric within Amazon CloudWatch.
From that, we could create alarms.
So we might want to count, you know, the number of 404s.
In the case of a web server, we might want to count
the number of bytes that were transferred
by that web server per request or over a period of time.
As our applications write their log files,
we might want to count the customer conversions.
Every time a customer closes a shopping cart
due to a sale, we add that log message,
we can configure our filter patterns to search for that,
and then count the number of sales that are happening.
So you can see between Amazon CloudWatch,
Amazon CloudWatch Alarms, and Amazon CloudWatch Logs,
we have a very rich and powerful system
for collecting metrics and for collecting logs
and that is Amazon CloudWatch Logs.