1
00:00:06,516 --> 00:00:09,079
- Let's talk about
real-time stream processing

2
00:00:09,079 --> 00:00:10,998
with Amazon Kinesis.

3
00:00:10,998 --> 00:00:12,803
Before we get into the details

4
00:00:12,803 --> 00:00:15,215
of the Amazon Kinesis service,

5
00:00:15,215 --> 00:00:18,167
let's first review a use case.

6
00:00:18,167 --> 00:00:22,175
I think it helps to understand
a real-world problem,

7
00:00:22,175 --> 00:00:25,905
when it comes to real-time
stream processing.

8
00:00:25,905 --> 00:00:30,072
So, let's take the example of
a very popular pizza chain,

9
00:00:32,073 --> 00:00:33,100
around the world.

10
00:00:33,100 --> 00:00:36,078
So, currently, this particular pizza chain

11
00:00:36,078 --> 00:00:39,489
would have somewhere,
around 11,000 locations,

12
00:00:39,489 --> 00:00:42,072
nearly 40,000 delivery drivers.

13
00:00:43,001 --> 00:00:45,900
Now, with these 11,000 locations,

14
00:00:45,900 --> 00:00:48,968
each having, perhaps, multiple devices,

15
00:00:48,968 --> 00:00:52,292
maybe laptops, tablets,
at different stations,

16
00:00:52,292 --> 00:00:55,992
at the dough station, at the
oven, so on and so forth,

17
00:00:55,992 --> 00:00:58,906
we have delivery drivers that
each have mobile devices.

18
00:00:58,906 --> 00:01:01,594
And then, of course, we
have millions of customers,

19
00:01:01,594 --> 00:01:05,761
to whom we serve, more than
1.5 million pizzas per day.

20
00:01:07,254 --> 00:01:11,421
So, we have a lot of people,
and a lot of data to track.

21
00:01:12,624 --> 00:01:17,375
Now, what we have, is an
antiquated ordering system built,

22
00:01:17,375 --> 00:01:21,784
using soap and old-school
technologies out of the 90's,

23
00:01:21,784 --> 00:01:23,865
and we want to update that.

24
00:01:23,865 --> 00:01:26,155
We want to bring it into the modern era,

25
00:01:26,155 --> 00:01:27,267
hosted in the cloud,

26
00:01:27,267 --> 00:01:29,382
and really make this much more

27
00:01:29,382 --> 00:01:32,314
of a modern, real-time system.

28
00:01:32,314 --> 00:01:35,245
Currently, we're leveraging daily, weekly,

29
00:01:35,245 --> 00:01:39,155
and monthly reports, but we
want to get near real-time.

30
00:01:39,155 --> 00:01:41,111
We wanna be able to see
what's going on now.

31
00:01:41,111 --> 00:01:45,024
We don't wanna be responding
to what happened yesterday,

32
00:01:45,024 --> 00:01:45,976
or the week before.

33
00:01:45,976 --> 00:01:49,047
We want to be able to
respond to situations,

34
00:01:49,047 --> 00:01:51,195
much more quickly.

35
00:01:51,195 --> 00:01:54,286
Now, currently, customers have no insight

36
00:01:54,286 --> 00:01:56,898
into what's happening with their pizza.

37
00:01:56,898 --> 00:01:58,175
You know, where is my pizza?

38
00:01:58,175 --> 00:01:59,397
What stage is it in?

39
00:01:59,397 --> 00:02:01,345
Why am I waiting so long?

40
00:02:01,345 --> 00:02:03,536
So, we wanna be able to give our customers

41
00:02:03,536 --> 00:02:07,814
some insight into, hey, your
pizza just went into the oven.

42
00:02:07,814 --> 00:02:10,186
Your pizza just went into the box.

43
00:02:10,186 --> 00:02:13,767
Your pizza is now in the
car, and on the road.

44
00:02:13,767 --> 00:02:16,496
So, currently, we have
absolutely no way of doing that.

45
00:02:16,496 --> 00:02:19,145
But, we're going to architect a solution,

46
00:02:19,145 --> 00:02:20,797
to add that ability.

47
00:02:20,797 --> 00:02:25,293
So, our goals, over the
next couple of years,

48
00:02:25,293 --> 00:02:28,785
will be to add an
additional 5,000 locations,

49
00:02:28,785 --> 00:02:32,956
an additional 10,000 drivers,
and we want to, by then,

50
00:02:32,956 --> 00:02:36,195
be serving more than two
million pizzas per day.

51
00:02:36,195 --> 00:02:37,585
So, you can see here,

52
00:02:37,585 --> 00:02:40,726
we will be increasing the amount of data

53
00:02:40,726 --> 00:02:43,233
that we're tracking, significantly.

54
00:02:43,233 --> 00:02:47,793
We also want to enable
our customers to order

55
00:02:47,793 --> 00:02:49,114
with their mobile device.

56
00:02:49,114 --> 00:02:51,871
We want to allow them to pay
with their mobile device.

57
00:02:51,871 --> 00:02:55,952
We want to allow our drivers
to track where they are,

58
00:02:55,952 --> 00:02:58,393
when they've delivered it,
with their mobile device.

59
00:02:58,393 --> 00:03:00,942
We want to enable different stations

60
00:03:00,942 --> 00:03:05,109
in the pizza-making process,
the ability to check in,

61
00:03:06,201 --> 00:03:10,462
via, some type of a mobile
device, like a tablet.

62
00:03:10,462 --> 00:03:13,951
And then, we want the ability
to do real-time reporting.

63
00:03:13,951 --> 00:03:18,128
Right, we want not just across
the board, but per location.

64
00:03:18,128 --> 00:03:21,413
We wanna be able to zoom
into a particular store,

65
00:03:21,413 --> 00:03:25,883
and see what's happening with
that particular pizza shop,

66
00:03:25,883 --> 00:03:26,751
in real time.

67
00:03:26,751 --> 00:03:29,819
We wanna be able to
see, kind of aggregate,

68
00:03:29,819 --> 00:03:32,652
pizza shops by region, by country,

69
00:03:33,914 --> 00:03:35,365
and, of course, as I mentioned,

70
00:03:35,365 --> 00:03:37,737
we wanna be able to see what's happening,

71
00:03:37,737 --> 00:03:40,735
per pizza, in the step along the way.

72
00:03:40,735 --> 00:03:43,524
And we want to enable
mobile push notifications,

73
00:03:43,524 --> 00:03:45,643
so that we can alert our customers,

74
00:03:45,643 --> 00:03:47,965
hey, your pizza is on its way.

75
00:03:47,965 --> 00:03:50,615
Or, hey, your pizza was delivered.

76
00:03:50,615 --> 00:03:53,384
First, let's look at our API.

77
00:03:53,384 --> 00:03:56,234
Again, we have pizza locations.

78
00:03:56,234 --> 00:04:00,814
We have customers, and
employees, and drivers,

79
00:04:00,814 --> 00:04:03,160
sending data with their mobile devices,

80
00:04:03,160 --> 00:04:05,461
with their desktops and laptops.

81
00:04:05,461 --> 00:04:09,628
So, we have millions of
events coming in, to our API,

82
00:04:10,630 --> 00:04:14,078
which is built in a
highly-available, fault-tolerant way,

83
00:04:14,078 --> 00:04:16,310
using the elastic load balancer,

84
00:04:16,310 --> 00:04:18,679
leveraging multiple EC2 instances,

85
00:04:18,679 --> 00:04:21,256
and multiple availability zones.

86
00:04:21,256 --> 00:04:23,673
So, we get high availability.

87
00:04:25,067 --> 00:04:29,234
We get the ability to handle
requests at a high scale.

88
00:04:30,771 --> 00:04:33,049
But, what we're lacking, here,

89
00:04:33,049 --> 00:04:35,989
with a normal, traditional API,

90
00:04:35,989 --> 00:04:39,789
is the ability to process real-time data.

91
00:04:39,789 --> 00:04:44,317
Now, yes, these EC2 instances,
being a part of this API,

92
00:04:44,317 --> 00:04:46,287
can ingest all of these events.

93
00:04:46,287 --> 00:04:48,107
It can write it to a database.

94
00:04:48,107 --> 00:04:51,918
It can write it to RDS, to DynamoDB.

95
00:04:51,918 --> 00:04:54,765
It could write it to
Cassandra, to MongoDB.

96
00:04:54,765 --> 00:04:58,475
But, still, we're lacking
the ability to actually do

97
00:04:58,475 --> 00:05:02,498
any kind of meaningful
analytics, on this data,

98
00:05:02,498 --> 00:05:05,060
as it comes in, in real time.

99
00:05:05,060 --> 00:05:09,143
And, the problem is that
as events are coming in,

100
00:05:10,102 --> 00:05:14,903
they are being distributed
among our EC2 instances.

101
00:05:14,903 --> 00:05:19,070
So, perhaps, an event on Joe's
pizza, first comes in here.

102
00:05:20,946 --> 00:05:22,449
And then, a little bit later,

103
00:05:22,449 --> 00:05:26,144
Joe's pizza goes into the oven,
so another event comes in,

104
00:05:26,144 --> 00:05:28,286
and it goes to this machine.

105
00:05:28,286 --> 00:05:32,207
Later, Joe's pizza goes into
the car, to be delivered,

106
00:05:32,207 --> 00:05:34,947
and that event goes to another machine.

107
00:05:34,947 --> 00:05:39,086
And so, the problem here,
with these instances,

108
00:05:39,086 --> 00:05:41,977
trying to do any kind of
real-time data processing,

109
00:05:41,977 --> 00:05:44,787
is that each of these
instances are receiving

110
00:05:44,787 --> 00:05:47,364
a subset of the data.

111
00:05:47,364 --> 00:05:50,014
They are not receiving all of the data,

112
00:05:50,014 --> 00:05:51,657
for that particular pizza.

113
00:05:51,657 --> 00:05:54,859
No one machine is
receiving all of the data,

114
00:05:54,859 --> 00:05:57,291
for that particular pizza location,

115
00:05:57,291 --> 00:05:59,381
or that particular region.

116
00:05:59,381 --> 00:06:02,710
And so, that's where Kinesis comes in.

117
00:06:02,710 --> 00:06:05,508
That's how Kinesis can help.

118
00:06:05,508 --> 00:06:08,841
So, we're going to augment our solution,

119
00:06:10,828 --> 00:06:13,078
by adding a Kinesis Stream.

120
00:06:14,456 --> 00:06:18,258
Again, we have this incoming
stream of raw event data,

121
00:06:18,258 --> 00:06:21,727
coming from our employees,
from management,

122
00:06:21,727 --> 00:06:25,361
from our customers and delivery drivers.

123
00:06:25,361 --> 00:06:27,950
These millions and millions
of events, coming in,

124
00:06:27,950 --> 00:06:30,031
in a live stream.

125
00:06:30,031 --> 00:06:33,798
And, we're going to accept those requests,

126
00:06:33,798 --> 00:06:38,019
via, our highly-available
application running on EC2.

127
00:06:38,019 --> 00:06:42,186
But instead of just simply
recording those to a database,

128
00:06:43,339 --> 00:06:48,201
we're going to also send
those records, those events,

129
00:06:48,201 --> 00:06:52,470
we're going to put them
to a Kinesis Stream.

130
00:06:52,470 --> 00:06:56,542
Now, what Kinesis does,
it will divide that stream

131
00:06:56,542 --> 00:06:58,625
into what we call shards.

132
00:07:00,536 --> 00:07:04,086
When we do this, put action,
and we put that record

133
00:07:04,086 --> 00:07:05,741
into a Kinesis Stream.

134
00:07:05,741 --> 00:07:07,991
We specify a partition key,

135
00:07:08,855 --> 00:07:12,486
something unique that
sets Joe's pizza apart

136
00:07:12,486 --> 00:07:14,224
from Sally's pizza.

137
00:07:14,224 --> 00:07:18,073
We set store number one apart
from store number three.

138
00:07:18,073 --> 00:07:22,270
And so, Kinesis will
automatically divide that data

139
00:07:22,270 --> 00:07:23,770
into these shards,

140
00:07:27,074 --> 00:07:29,324
based on the partition key.

141
00:07:30,423 --> 00:07:34,877
Then, we can have another
application, called consumers,

142
00:07:34,877 --> 00:07:38,483
that each consumer will
pull from one shard.

143
00:07:38,483 --> 00:07:41,194
That way, this particular EC2 instance

144
00:07:41,194 --> 00:07:44,068
can now get all of the data,

145
00:07:44,068 --> 00:07:46,979
associated with a particular store.

146
00:07:46,979 --> 00:07:49,228
And now, it received
the whole of the data,

147
00:07:49,228 --> 00:07:53,219
the full data set of a particular store.

148
00:07:53,219 --> 00:07:56,647
Or, an EC2 instance
would receive in the case

149
00:07:56,647 --> 00:08:00,597
where we're tracking events
and data based on one pizza,

150
00:08:00,597 --> 00:08:03,038
and we want this particular machine

151
00:08:03,038 --> 00:08:06,701
to receive the full data
set about that one pizza.

152
00:08:06,701 --> 00:08:08,556
So, that's how Kinesis helps,

153
00:08:08,556 --> 00:08:11,635
by dividing data into shards,

154
00:08:11,635 --> 00:08:14,808
and insuring that our application

155
00:08:14,808 --> 00:08:18,509
that is performing this
real-time analytics,

156
00:08:18,509 --> 00:08:21,817
can receive the full data
set of our groupings,

157
00:08:21,817 --> 00:08:23,658
however it is that we group things,

158
00:08:23,658 --> 00:08:27,138
whether it's by pizza, or
by store, or by region,

159
00:08:27,138 --> 00:08:28,989
and so on and so forth.

160
00:08:28,989 --> 00:08:32,249
So, that's part one of
real-time stream processing

161
00:08:32,249 --> 00:08:33,916
with Amazon Kinesis.