- Now let's talk about Amazon Simple Queue Service.
Amazon Simple Queue Service very much like other Amazon
services is built to be inherently highly available
and fault tolerant.
So with a queue service we can put messages in,
buffer those messages up for some period of time and then
pull them back out.
The Simple Queue Service or queues in general are a really
great way to introduce loose coupling into our applications.
So loose coupling or that is the ability to separate
different applications so that they can scale independently,
they don't have any inherent knowledge of each other,
that's a really great practice in order to be able to scale
out and handle large amounts of traffic and large amounts
of data.
So with the Simple Queue Service we can handle millions
even billions of messages without having to worry about
the underlying infrastructure of that service.
Amazon just handles it for us.
Now the messages that we put in here will be stored
redundantly across multiple servers and multiple
availability zones very much like Simple Storage Service
in DynamoDB.
With the Simple Queue Service messages will be pushed in
and then pulled out, we actually have to go in and ask
for the messages, they're not published like with other
services, we actually have to ask for those messages by way
of pulling.
It's important to understand that with the Simple Queue
Service the message order is not guaranteed, but we do
get a guarantee of at-least-once delivery.
Now that means that a message could be delivered two
or three times but it will be delivered at least once.
We can leverage resource based policies and IAM policies
to control who's allowed to use this particular queue,
even sharing it with other accounts.
We can also allow this queue to be written to and read from
anonymously if we wanted to.
Another important feature of the Simple Queue Service
is what we call the visibility timeout.
Now the way that this works is that let's say that we have
an application down here on EC2 that's pulling messages
out of the queue, and we've pulled message A out of
the queue and whatever that message is, perhaps that message
is a notification about a video that is ready
to be processed or some other type of data that needs
something to be done to it in the background asynchronously.
So we pull message A out of the queue and once we
acknowledge receipt of that message the queue will now mark
that message A across all of the partitions is now
considered invisible or it's locked so that no other system,
if we have our application distributed among multiple EC2
Instances, we don't want that message to be processed twice,
so now once we've retrieved it by one, it's now invisible
to others.
The default visibility timeout is 30 seconds, we can change
that if we wanted to and that gives us time to process
that message.
If processing fails and that message is not deleted within
that timeout window, then the message will become visible
again so that it can be picked up by another EC2 Instance
and then processing can resume.
So this is a really great way to prevent multiple components
from processing the same message at the same time,
but again we do need to keep in mind that because of that
at-least-once delivery, it does mean that if let's just say
that this machine deletes the message,
and then 10 milliseconds later another EC2 Instance pulls
messages, it's still possible that this EC2 Instance
will still get a copy of that same message
so our application needs to be what we would call
idempotent, meaning that if we were to process the message
twice in a row we're still going to get the same result.
Another approach to that would be to record the fact
somewhere perhaps in DynamoDB we might record the fact that
we've already processed this message so that we don't have
to do it twice.
So that is Amazon Simple Queue Service, a really powerful
service for buffering events, buffering messages to enable
loose coupling and asynchronous processing between layers
of our application.