- Let's talk about real-time stream processing
with Amazon Kinesis.
Before we get into the details
of the Amazon Kinesis service,
let's first review a use case.
I think it helps to understand a real-world problem,
when it comes to real-time stream processing.
So, let's take the example of a very popular pizza chain,
around the world.
So, currently, this particular pizza chain
would have somewhere, around 11,000 locations,
nearly 40,000 delivery drivers.
Now, with these 11,000 locations,
each having, perhaps, multiple devices,
maybe laptops, tablets, at different stations,
at the dough station, at the oven, so on and so forth,
we have delivery drivers that each have mobile devices.
And then, of course, we have millions of customers,
to whom we serve, more than 1.5 million pizzas per day.
So, we have a lot of people, and a lot of data to track.
Now, what we have, is an antiquated ordering system built,
using soap and old-school technologies out of the 90's,
and we want to update that.
We want to bring it into the modern era,
hosted in the cloud,
and really make this much more
of a modern, real-time system.
Currently, we're leveraging daily, weekly,
and monthly reports, but we want to get near real-time.
We wanna be able to see what's going on now.
We don't wanna be responding to what happened yesterday,
or the week before.
We want to be able to respond to situations,
much more quickly.
Now, currently, customers have no insight
into what's happening with their pizza.
You know, where is my pizza?
What stage is it in?
Why am I waiting so long?
So, we wanna be able to give our customers
some insight into, hey, your pizza just went into the oven.
Your pizza just went into the box.
Your pizza is now in the car, and on the road.
So, currently, we have absolutely no way of doing that.
But, we're going to architect a solution,
to add that ability.
So, our goals, over the next couple of years,
will be to add an additional 5,000 locations,
an additional 10,000 drivers, and we want to, by then,
be serving more than two million pizzas per day.
So, you can see here,
we will be increasing the amount of data
that we're tracking, significantly.
We also want to enable our customers to order
with their mobile device.
We want to allow them to pay with their mobile device.
We want to allow our drivers to track where they are,
when they've delivered it, with their mobile device.
We want to enable different stations
in the pizza-making process, the ability to check in,
via, some type of a mobile device, like a tablet.
And then, we want the ability to do real-time reporting.
Right, we want not just across the board, but per location.
We wanna be able to zoom into a particular store,
and see what's happening with that particular pizza shop,
in real time.
We wanna be able to see, kind of aggregate,
pizza shops by region, by country,
and, of course, as I mentioned,
we wanna be able to see what's happening,
per pizza, in the step along the way.
And we want to enable mobile push notifications,
so that we can alert our customers,
hey, your pizza is on its way.
Or, hey, your pizza was delivered.
First, let's look at our API.
Again, we have pizza locations.
We have customers, and employees, and drivers,
sending data with their mobile devices,
with their desktops and laptops.
So, we have millions of events coming in, to our API,
which is built in a highly-available, fault-tolerant way,
using the elastic load balancer,
leveraging multiple EC2 instances,
and multiple availability zones.
So, we get high availability.
We get the ability to handle requests at a high scale.
But, what we're lacking, here,
with a normal, traditional API,
is the ability to process real-time data.
Now, yes, these EC2 instances, being a part of this API,
can ingest all of these events.
It can write it to a database.
It can write it to RDS, to DynamoDB.
It could write it to Cassandra, to MongoDB.
But, still, we're lacking the ability to actually do
any kind of meaningful analytics, on this data,
as it comes in, in real time.
And, the problem is that as events are coming in,
they are being distributed among our EC2 instances.
So, perhaps, an event on Joe's pizza, first comes in here.
And then, a little bit later,
Joe's pizza goes into the oven, so another event comes in,
and it goes to this machine.
Later, Joe's pizza goes into the car, to be delivered,
and that event goes to another machine.
And so, the problem here, with these instances,
trying to do any kind of real-time data processing,
is that each of these instances are receiving
a subset of the data.
They are not receiving all of the data,
for that particular pizza.
No one machine is receiving all of the data,
for that particular pizza location,
or that particular region.
And so, that's where Kinesis comes in.
That's how Kinesis can help.
So, we're going to augment our solution,
by adding a Kinesis Stream.
Again, we have this incoming stream of raw event data,
coming from our employees, from management,
from our customers and delivery drivers.
These millions and millions of events, coming in,
in a live stream.
And, we're going to accept those requests,
via, our highly-available application running on EC2.
But instead of just simply recording those to a database,
we're going to also send those records, those events,
we're going to put them to a Kinesis Stream.
Now, what Kinesis does, it will divide that stream
into what we call shards.
When we do this, put action, and we put that record
into a Kinesis Stream.
We specify a partition key,
something unique that sets Joe's pizza apart
from Sally's pizza.
We set store number one apart from store number three.
And so, Kinesis will automatically divide that data
into these shards,
based on the partition key.
Then, we can have another application, called consumers,
that each consumer will pull from one shard.
That way, this particular EC2 instance
can now get all of the data,
associated with a particular store.
And now, it received the whole of the data,
the full data set of a particular store.
Or, an EC2 instance would receive in the case
where we're tracking events and data based on one pizza,
and we want this particular machine
to receive the full data set about that one pizza.
So, that's how Kinesis helps,
by dividing data into shards,
and insuring that our application
that is performing this real-time analytics,
can receive the full data set of our groupings,
however it is that we group things,
whether it's by pizza, or by store, or by region,
and so on and so forth.
So, that's part one of real-time stream processing
with Amazon Kinesis.