- Now let's review monitoring, with Amazon CloudWatch.
Now, with Amazon CloudWatch we can gain instrumentation
for our infrastructure by one, collecting metrics,
key performance metrics from our EC2 instances and
EOB volumes, RDS instances, and so on.
We can also collect logs from those systems
and we'll talk more about logs later on,
but for now let's focus on collecting metrics.
So again, this is one of the key things that
Amazon CloudWatch does, is to collect performance metrics.
And it's important to remember that these metrics
are only stored for up to 2 weeks and so
if you needed the ability to see much more historical data,
if for example, if you wanted to compare this week,
this year, with this week last year, then you would need
to pull those metrics into either your own system,
perhaps on EC2 or on premises
or rely on a third party service, such as New Relic
or Data Dog or something else.
And of course, like all Amazon services, CloudWacth allows
us to pull these metrics out by accessing the API.
And we can give third parties access to that API, so that
a third party like New Relic or Data Dog or
Noggios, running on premises can connect to CloudWatch
and pull those metrics in.
Now, every service, or most services anyway,
will generate their own unique set of metrics.
So EC2 has a particular set of metrics, RDS, redshift,
elasticash, and so on, they all generate their own unique
set of metrics. We also have the ability to publish
custom metrics into CloudWatch. So if our own applications,
if we wanted to publish something like JVM heap size,
or the number of processes, number of concurrent threads,
render times, those kinds of things, if there are metrics
that are very easy for our application to readily grab,
then it's very fairly trivial to publish those to CloudWatch
so that we can monitor, not only things reported by,
say the hypervisor,
but also things reported by our applications.
And then of course we can correlate things
that are happening within our application to things
that are happening within our infrastructure.
And so now for example, if we take a look at EC2,
with EC2 the default interval collection is every 5 minutes.
Right? And so for things like auto scaling,
that's going to be too long. We may need to scale within
a minute or two. And so we can pay an extra fee per instance
to gain access to a detailed one minute interval.
Now a thing to keep in mind about EC2, is that these metrics
that EC2 collects and reports are reported by the hypervisor
and now the hypervisor will have a very accurate,
perhaps the most accurate metric on something like CPU usage
and network IO, disk IO, and a few other metrics.
But one thing to keep in mind is that the hypervisor has
no idea how memory is being used.
And so while the hypervisor may know how much memory an
EC2 instance actually has,
it doesn't know how it is being used.
So, if you need more detailed memory usage
that kind of metric needs to be reported to CloudWatch,
from inside the instance.
And of course, CloudWatch does provide a very powerful agent
that can just readily out of the box report very detailed
memory usage and of course those are reported as
custom metrics and we do pay an additional fee
per custom metric.
Now for the elastic load bouncer, as an example, the default
is 1 minute, and that's so that
that service can support scaling.
For the relational database service, we do get access to
memory. Because the database engine is already running,
and the database, typically, the database engines are aware
of memory usage and connections, and disk IO,
and things like that, then we are relying on the software
that's already apart of
that instance to report those things.
With DynamoDB, we do get such things like read and write
throughput and again, this is not an exhaust of it,
but just an example of the type of metrics that we might see
with different services.
Another powerful feature that we have with CloudWatch
is the ability to create alarms, where we either want to be
notified when something is happening, or we want to leverage
a, you know the breach of a threshold trigger some other
process. And so with CloudWatch alarms,
we define a threshold.
We say, well I want something to happen, I want to create
an alarm when some number, some metric is either too high
or too low.
And so even though we use the word alarm, it does not
necessarily signal an emergency.
It simply means when the alarm is triggered,
it simply means that a number is too high or too low,
for too long of a period of time.
And whether or not there's an emergency, that's up to you
to determine based on the nature of your application on that
particular infrastructure. But,
we can use those alarms trigger things like auto scaling.
We can also use that alarm to simply terminate an instance.
We can also use the alarm to reboot an instance.
There could be a case where,
maybe you are aware of a memory leak, as an example,
and you're waiting on developers to fix it in code,
but in the meantime, you're trying to mitigate the issue
within infrastructure.
So perhaps, as a stop gap measure you have a,
you're collecting memory as a custom metric and then
when memory becomes too high you simply reboot the machine.
Of course that's not a longterm solution,
just one example of something that you could do with alarms.
And it's also important to remember,
that with most Amazon services there are both hard limits
and soft limits.
Hard limits are just a nature of the technology,
but soft limits can be overridden by submitting a ticket
to Amazon support.
And so with CloudWatch there is an initial limit of 5000
alarms per account.
So that's 5000 alarms across all the regions.
Some limits within AWS are specific to a region
and some are more broad,
some are specific to an account across all regions.
So, let's take a look here at a diagram.
In this diagram, we are collecting.
You can see here that we have a load balancer
and this load balancer is sending metrics to CloudWatch,
such as requests per minute, um the number of 500s,
the number of 400s, backend arrows and those kind of things.
Our EC2 instances individually are also reporting
those metrics, such as CPU usage, disk IO, network IO.
And then our auto scaling group is also reporting
metrics about that group in aggregate, so average CPU,
average disk usage and network usage and so on.
Our RDS instance is also reporting metrics into CloudWatch.
So, CloudWatch is collecting these metrics from various
different types of places.
And so in the least, one really helpful thing is,
if you are seeing some performance degradation,
you could look back on those CloudWatch metrics in the
console and create a graph and overlay different metrics
and then correlate, you know what you're seeing in
the load balancer, maybe request per minute
how that effects CPU or disk IO within your EC2 instances
and/or how that is effecting your database. Right?
And then of course here, we have an alarm.
We've, you can see here that we've defined an alarm
that says when CPU utilization is greater than 80% for
two periods of one minute. So essentially, when CPU is
greater than 80% for two minutes, then we want an alarm
to go off. Now again, it's not necessarily an emergency.
And what happens is totally up to us.
We get to configure what happens when that alarm goes off.
It's very possible that nothing happens. We can do that.
We can have an alarm that just goes off
and then nothing happens. But here, in this example,
we may want that alarm, you can see
that we could use that alarm to trigger auto scaling.
So that perhaps that alarm is signifies that their is more
work to be done than what can be done with
our current set of instances. And so this could signal
the need to grow and expand our fleet of EC2 instances,
in order to meet that demand.
We can also send that alarm out
through the simple notifications service.
And so here we have a simple notification topic
and then from there we can do a number of things.
We can have lambda respond to that, right?
So we can write lambda function, that would respond
to that alarm in some kind of intelligent way,
doing some kind of automated process in response
to that alarm.
Another thing that we could do is to perhaps, you know,
if that alarm is for a particular application,
then perhaps we send that alarm to a slack channel
so that our developers are basically,
in mini teams that I've worked with developers have 2 things
in front of them all the time,
their IDE and their slack channel, and so instead of putting
notifications and alarms off in some place that requires
them to go look for them, then it's better in my experience,
to have that alarm go right to where they're already at,
such as slack. And so,
and also instead of that alarm or notification
going to one person and waiting for that one person,
we can send that notification to an entire team of people
so that we have a greater chance of multiple people
being aware of an issue. And I've seen that be
a really powerful pattern for helping development teams
jump on issues much faster.
We can also have those alarms be sent to some other kind of
ticketing issue or bug tracking software like Gira,
or some kind of sim management system, or like a mension,
maybe you're not using slack,
but maybe you're using some other kind of chat system.
The point is to automate the collection of metrics,
automate the triggering of alarms and then get those
notifications to the appropriate people in an efficient way.
So you can see that CloudWatch plays a very key role
within our EWS infrastructure.
And we will take a closer look at a CloudWatch
and CloudWatch logs as the course progresses.