1
00:00:07,725 --> 00:00:10,795
- Now let's talk about
Amazon Simple Queue Service.

2
00:00:10,795 --> 00:00:14,369
Amazon Simple Queue Service
very much like other Amazon

3
00:00:14,369 --> 00:00:19,258
services is built to be
inherently highly available

4
00:00:19,258 --> 00:00:20,974
and fault tolerant.

5
00:00:20,974 --> 00:00:23,772
So with a queue service
we can put messages in,

6
00:00:23,772 --> 00:00:27,837
buffer those messages up for
some period of time and then

7
00:00:27,837 --> 00:00:29,452
pull them back out.

8
00:00:29,452 --> 00:00:33,179
The Simple Queue Service or
queues in general are a really

9
00:00:33,179 --> 00:00:37,346
great way to introduce loose
coupling into our applications.

10
00:00:38,508 --> 00:00:42,176
So loose coupling or that
is the ability to separate

11
00:00:42,176 --> 00:00:46,945
different applications so that
they can scale independently,

12
00:00:46,945 --> 00:00:50,176
they don't have any inherent
knowledge of each other,

13
00:00:50,176 --> 00:00:54,018
that's a really great practice
in order to be able to scale

14
00:00:54,018 --> 00:00:59,011
out and handle large amounts
of traffic and large amounts

15
00:00:59,011 --> 00:01:00,207
of data.

16
00:01:00,207 --> 00:01:03,852
So with the Simple Queue
Service we can handle millions

17
00:01:03,852 --> 00:01:06,975
even billions of messages
without having to worry about

18
00:01:06,975 --> 00:01:11,057
the underlying infrastructure
of that service.

19
00:01:11,057 --> 00:01:14,208
Amazon just handles it for us.

20
00:01:14,208 --> 00:01:17,505
Now the messages that we
put in here will be stored

21
00:01:17,505 --> 00:01:20,002
redundantly across multiple
servers and multiple

22
00:01:20,002 --> 00:01:24,205
availability zones very much
like Simple Storage Service

23
00:01:24,205 --> 00:01:25,725
in DynamoDB.

24
00:01:25,725 --> 00:01:29,892
With the Simple Queue Service
messages will be pushed in

25
00:01:31,502 --> 00:01:34,537
and then pulled out, we
actually have to go in and ask

26
00:01:34,537 --> 00:01:37,452
for the messages, they're
not published like with other

27
00:01:37,452 --> 00:01:41,469
services, we actually have to
ask for those messages by way

28
00:01:41,469 --> 00:01:42,386
of pulling.

29
00:01:43,325 --> 00:01:45,726
It's important to understand
that with the Simple Queue

30
00:01:45,726 --> 00:01:50,431
Service the message order
is not guaranteed, but we do

31
00:01:50,431 --> 00:01:53,914
get a guarantee of at-least-once delivery.

32
00:01:53,914 --> 00:01:56,607
Now that means that a message
could be delivered two

33
00:01:56,607 --> 00:02:01,449
or three times but it will
be delivered at least once.

34
00:02:01,449 --> 00:02:05,118
We can leverage resource based
policies and IAM policies

35
00:02:05,118 --> 00:02:09,285
to control who's allowed to
use this particular queue,

36
00:02:10,319 --> 00:02:12,578
even sharing it with other accounts.

37
00:02:12,578 --> 00:02:16,030
We can also allow this queue
to be written to and read from

38
00:02:16,030 --> 00:02:18,363
anonymously if we wanted to.

39
00:02:19,677 --> 00:02:23,135
Another important feature
of the Simple Queue Service

40
00:02:23,135 --> 00:02:27,003
is what we call the visibility timeout.

41
00:02:27,003 --> 00:02:30,718
Now the way that this works
is that let's say that we have

42
00:02:30,718 --> 00:02:34,351
an application down here on
EC2 that's pulling messages

43
00:02:34,351 --> 00:02:38,125
out of the queue, and we've
pulled message A out of

44
00:02:38,125 --> 00:02:41,790
the queue and whatever that
message is, perhaps that message

45
00:02:41,790 --> 00:02:46,043
is a notification about
a video that is ready

46
00:02:46,043 --> 00:02:49,805
to be processed or some
other type of data that needs

47
00:02:49,805 --> 00:02:53,972
something to be done to it in
the background asynchronously.

48
00:02:55,840 --> 00:03:00,112
So we pull message A out
of the queue and once we

49
00:03:00,112 --> 00:03:04,573
acknowledge receipt of that
message the queue will now mark

50
00:03:04,573 --> 00:03:08,740
that message A across all
of the partitions is now

51
00:03:09,867 --> 00:03:14,128
considered invisible or it's
locked so that no other system,

52
00:03:14,128 --> 00:03:18,295
if we have our application
distributed among multiple EC2

53
00:03:19,744 --> 00:03:23,880
Instances, we don't want that
message to be processed twice,

54
00:03:23,880 --> 00:03:26,929
so now once we've retrieved
it by one, it's now invisible

55
00:03:26,929 --> 00:03:27,762
to others.

56
00:03:29,022 --> 00:03:33,583
The default visibility timeout
is 30 seconds, we can change

57
00:03:33,583 --> 00:03:38,474
that if we wanted to and
that gives us time to process

58
00:03:38,474 --> 00:03:40,250
that message.

59
00:03:40,250 --> 00:03:44,140
If processing fails and that
message is not deleted within

60
00:03:44,140 --> 00:03:48,574
that timeout window, then the
message will become visible

61
00:03:48,574 --> 00:03:52,494
again so that it can be picked
up by another EC2 Instance

62
00:03:52,494 --> 00:03:55,761
and then processing can resume.

63
00:03:55,761 --> 00:03:59,836
So this is a really great way
to prevent multiple components

64
00:03:59,836 --> 00:04:03,180
from processing the same
message at the same time,

65
00:04:03,180 --> 00:04:06,256
but again we do need to keep
in mind that because of that

66
00:04:06,256 --> 00:04:10,423
at-least-once delivery, it does
mean that if let's just say

67
00:04:13,054 --> 00:04:15,661
that this machine deletes the message,

68
00:04:15,661 --> 00:04:19,664
and then 10 milliseconds later
another EC2 Instance pulls

69
00:04:19,664 --> 00:04:23,857
messages, it's still possible
that this EC2 Instance

70
00:04:23,857 --> 00:04:27,155
will still get a copy of that same message

71
00:04:27,155 --> 00:04:31,357
so our application needs
to be what we would call

72
00:04:31,357 --> 00:04:35,904
idempotent, meaning that if
we were to process the message

73
00:04:35,904 --> 00:04:40,076
twice in a row we're still
going to get the same result.

74
00:04:40,076 --> 00:04:43,292
Another approach to that
would be to record the fact

75
00:04:43,292 --> 00:04:47,888
somewhere perhaps in DynamoDB
we might record the fact that

76
00:04:47,888 --> 00:04:51,311
we've already processed this
message so that we don't have

77
00:04:51,311 --> 00:04:53,277
to do it twice.

78
00:04:53,277 --> 00:04:58,014
So that is Amazon Simple Queue
Service, a really powerful

79
00:04:58,014 --> 00:05:02,181
service for buffering events,
buffering messages to enable

80
00:05:03,250 --> 00:05:07,627
loose coupling and asynchronous
processing between layers

81
00:05:07,627 --> 00:05:09,210
of our application.