- Now let's talk about log collections
with Amazon CloudWatch Logs.
So, before we get into the nature
and details of CloudWatch logs,
let's first look at where logs come from.
And we have applications running in different places.
And we have other resources that are generating logs.
And so, a number of places,
including the Elastic Load Balancer ,
will generate access logs.
We could have applications running on EC2
that can be running
within the elastic container service,
within containers.
We might have applications running
within Lambda.
Of course, Lambda will generate its own logs,
whether or not your application
is writing any logs or not.
Lambda will include its own logs.
And then of course we have S3 access logs.
So, again, if we're storing
static content, for example;
images, CSS pdf files and and so on in S3,
and those are publicly available
as far as web assets
for some kind of a website application,
then as those files are accessed,
S3 will generate its own access logs.
And then of course, we can also collect VPC flow logs.
So, an important component of network security
is understanding the nature of the traffic
whether traffic is being accepted or rejected.
And so, you can look at at least
metadata about network traffic
with VPC flow logs.
We also have CloudFront access logs.
If our Load Balancer or our S3 buckets are behind
a CloudFront distribution,
and our users are going through
those edge locations to retrieve that date,
then CloudFront will generate access logs.
We also have CloudTrail. CloudTrail is a service
that collects and records API calls.
So anything you do through through the AWS API,
for the purpose of creating
or managing some Amazon resource,
then those calls to the API are recorded.
The credentials that are used
and metadada about that request
are recorded through CloudTrail.
And of course, there are plenty of more places
where logs could be generated from.
So again, in terms of capturing logs,
let's take a look at this diagram.
And you'll see here, ofcourse,
we have our users out there
and our users are accessing our dynamic content
and our static content, through those edge locations.
Of course, those edge locations, those CloudFront,
is sending CloudFront access logs to S3.
And then of course,
we have our Load Balancer here,
and our Load Balancer is sending
access logs directly to S3.
We have our VPC flow logs,
which are being collected
directly by CloudWatch logs.
Our Lambda functions are being collected
by CloudWatch logs.
And then, of course, we could,
for our applications running on EC2,
we have a couple of options.
We could use the CloudWatch logs API
to have our application logged directly
to CloudWatch logs.
Or we can have our applications log
to local files,
and then have the CloudWatch agent
essentially stream those files
in near realtime to the CloudWatch log service.
So either way, we have our choice there
in terms of how we get
our application logs to CloudWatch logs.
But, the point here is that, either way,
we can collect those logs to CloudWatch logs.
So the benefit here is that those days
of shelling in to the individual instances
and trying to find the log files, "Where are they?
"Are they in (speaker mumbles),
or are they up somewhere?"
Trying to find them to begin with
and then getting to that directory
only to find out that they've been rotated
or maybe they haven't.
Maybe they're so large
that you can't find what you're looking for.
And so I've been there.
I've done that.
It's a painful process.
And so the idea here with CloudWatch logs
is that one of the patterns of Cloud computing
is to treat logs as streams.
You could have within EC2,
remember that we could have
just about any number of instances here.
We could have two, we could have two hundred,
we can have two thousand.
And, like I've mentioned before,
that these instances are essentially disposable.
They could be an issue
where they just go away on their own,
or auto scaling could terminate them,
or we may decide to terminate them.
Or they could be terminated by accident.
We can't tell on the EC2 instance being there.
Therefore, we can't count
on the logs being there for any length of time.
So, when you have such a dynamic system
where EC2 instances are coming and going,
it's really important
that we have one central place
to which we can record our logs.
That way we don't have to worry about
the machine being gone.
We know that we've already collected
as much logs as we can from that instance
before it went away.
Not only do we have one place to collect it,
but we have one place
from which we can pull those logs
instead of having to hunt them down
across all of these various disparate places.
Now of course, like I mentioned earlier,
we also have CloudTrail.
Now, CloudTrial could go directly to S3.
We could have CloudTrail send files to S3,
but we also have the option
of having CloudTrail send the various API events
directly to CloudWatch logs,
so that we get more of a realtime insight
into what's happening with CloudTrail
and what's happening to our credentials
and AWS environment.
We also have Route 53 DNS queries.
So as our users are out there
and they are making DNS queries
to various domain names,
and maybe we use different subdomains
for different purposes.
Maybe we use different subdomains
for different marketing campaigns.
So perhaps there is some kind of data to be found
or some type of value to be found
in analyzing DNS queries.
And so we can capture those as well.
Route 53 can send its DNS query data
to CloudWatch logs.
And so, we could also have CouldWatch logs
archive those logs off to S3
and then perhaps from S3,
we can have EMR or Athena,
pulls those logs and perform
some kind of an analysis
or a sequel query on the data from S3.
And maybe that kind of thing is performed.
Some kind of processing is performed,
in the first week,
or the first 30 days of those logs.
And then after that,
perhaps we need to keep those logs
for some extended period of time
because of legal or regulatory compliance.
We also have the ability
to configure those life cycle rules,
and have those logs archived to Glacier.
So we could lower our long term cost
of archiving data that we have to keep.
So, again, when you think about your applications
running in these various places within AWS
whether they're running on EC2 or in containers,
whether they're running in Lambda,
and you want a central place to collect those logs,
take a look at Amazon CloudWatch logs.