- Now let's review a use case of analytics
for a subscription video on demand service.
Well, as you can see when it comes to analytics,
we have a lot of options.
So, we'll just start from the top and move our way down here
and kind of explain what's going on.
So, let's just say that we have an API,
and that API is powered by perhaps, you know,
containers running within ECS or ECS for Kubernetes.
It could be running on EC2 directly,
but in this example, this comes from personal experience
of running, you know, a video subscription service
very similar to Netflix, where it was all of the
microservices were powered by ECS.
And they were all available through a load balancer.
And so, all of those microservices, you know,
a couple of dozen different applications
were all sending logs to CloudWatch Logs,
and the benefit of that is by streaming those logs
to CloudWatch Logs, we had the benefit of, you know,
if a container went away, which it often did,
then we still had those logs.
We didn't have to worry about them, you know,
being unavailable.
And then, it was very easy to connect CloudWatch Logs
to Elasticsearch, the Amazon Elasticsearch service,
and have those logs stream essentially in real-time,
into Elasticsearch, so that developers could very easily
access those logs within Elasticsearch.
Now technically, developers could access logs in
CloudWatch Logs, but Elasticsearch gave us much more power
and flexibility in how we could not only view those logs,
but run analysis on those logs as well.
So, even from an application performance standpoint,
is really the biggest benefit to our developers,
is really understanding how our applications are performing
by looking at those kinds of logs.
And so, we also had several different databases
within the relational databases service.
We had MySQL data storage.
We also had Postgre data stores, or databases,
within Aurora- and non-Aurora-based RDS instances.
Certain data stored here, like user account information,
video metadata,
and a number of other things,
and then we also had a lot of financial information
stored here in a SQL server database.
Primarily because the finance and accounting departments,
their tools were all Windows-based,
and we stored information
as it related to not just billing, but also as it related to
the cost of producing videos, as it related to the cost of
marketing those videos.
And so, by storing all of that there,
then finance and accounting could, they could use their
Windows-based tools to work with that data,
but we could also, as we'll talk more about later,
we could pull that in to Redshift.
And so, we also stored a number of other types
of information in DynamoDB.
We actually duplicated a lot of the user information,
the video information was duplicated from MySQL
into DynamoDB.
And that's because we had CMS tools,
content management systems,
that allowed, you know, editors and content creators
to edit that content here.
But then for running and production for our users,
and, you know, coming from their mobile applications
and from the website, they got a much better performance by
reading it from DynamoDB.
And so we had other services in the background that could,
you know, replicate that data from MySQL over to DynamoDB
doing an ETL operation.
We had other data coming in, as, you know, users were
performing different things.
As they, you know, clicked on a webpage,
as they searched for something,
as they, you know, pressed play, as they paused,
as they fast-forward,
and as they continued to watch a video.
There were so many different events
that users were generating
that our API was collecting.
A lot of those events were sent into Amazon Kinesis.
And Amazon Kinesis gave us that, you know, reliable way of
ingesting a very high volume of data coming in
at a high velocity.
And so, we had, we split that in a couple of different ways.
We had what we would call Kinesis-enabled applications,
or consumers, Kinesis consumers, that were reading from
this Kinesis stream and performing a number of different
kinds of analysis on that stream.
So we were doing real-time information, such as, you know,
how many people are viewing the website right now,
how many people are viewing the videos right now.
And then other kinds of longer-term trend analysis,
such as, you know, what videos are more popular based on
the length that they were watched.
You know, are people watching to a certain point
and then skipping ahead?
That kind of information.
So, we could use Kinesis-enabled applications, consumers,
to read from that stream, perform analysis on real-time
kinds of things, and store that in DynamoDB.
And then, of course, other users
could potentially read that.
Like if we wanted to also display the popularity of a
particular video, then these Kinesis applications could,
you know, determine that popularity, and then other users
could read that from DynamoDB.
We also had... these Kinesis streams could connect
to Kinesis data Firehose, and Kinesis data Firehose is
sort of an out-of-the-box solution for writing Kinesis data
either directly to S3 or directly to Redshift
among a couple of other places.
And so, here, we could leverage Kinesis Firehose,
without writing our own code, just leverage Kinesis Firehose
to get that data into flat files in S3,
where they would remain for some time.
And from there, we had access to large datasets
that we could run analysis on using Amazon Athena;
so our analytics team, our data team, could use their own,
you know, SQL-based tools connecting to Athena using,
you know, ODBC, JDBC drivers, and running
whatever ad-hoc queries
they could think of against that large dataset
within S3.
And then, of course, we had - like I mentioned earlier -
we had user data, we had video and a number of different
types of data here in MySQL, and we had
financial information here as it related to those videos.
And then we had real-time popularity information within
DynamoDB, and so we could use Data Pipeline to pull
all of that data in from all of these various sources,
performing an extract and transformation and then load
that into Redshift, so that our analytics team
could perform, you know, regular queries around, well,
how did the popularity of a video, how does the popularity
of a video, relate to, you know, the money that we put in
to produce it?
We spend a certain amount of money to, you know, make this
particular video or this series of videos, and we spend
a certain amount of money to market it.
Are we seeing, you know, a level of engagement
to justify that expense?
Right, and so the only way to know that
is to be able to join, you know, user data, account data,
with other, you know, financial data with popularity data.
And again, we can do that by pulling all of that data into
Redshift and allowing our analytics team to,
very much like Athena, they could connect to Amazon Redshift
and run any ad-hoc query they can think of
across a very large dataset, and determine not only
popularity as it relates to cost, but also how does our
marketing efforts relate to users
continuing their memberships and their subscriptions?
You know, are users canceling their subscriptions?
And is there a relationship between activity in the video
and, you know, users either canceling or renewing their
memberships?
Is there a relationship between, you know, activity on
a new series and whether or not we were getting new people
subscribed, you know, new users signing up?
And then, how did, you know, is the money that we're making
on that, is it allowing us to be profitable considering
the money that we spent to create and market that?
Right, so there was a lot of information here that required
some very complex SQL statements that could only really
be run in one place.
And Redshift served that really well.
So, as you can see, we have, when it comes to analytics
within AWS, we have a lot of options, a lot of very powerful
options, and it's very common for, you know, applications to
use a number of different data stores, because each one
of these is targeted for a particular use case, and it
serves that kind of data in that kind of scenario,
that kind of access pattern, really well.
Right, and of course there are others.
There's Amazon EMR, which was not really used
in this particular scenario, but that is also an option.
So again, as you continue to move forward in your
exploration of AWS, I would highly encourage you to explore
some of these tools, such as Kinesis, DynamoDB,
Athena, Redshift, Elasticsearch, and of course
Amazon EMR as well.