- Now let's review Amazon's Simple Queue Service, or SQS.
With Amazon Simple Queue Service,
we gain a fully managed queue service
and so there's no instance of any kind of server
that we need to provision,
nothing is sitting there idle that we're paying for,
there's no software to install.
We simply go to the Simple Queue Service and create a queue.
And then, all of the underlying infrastructure
necessary to maintain that queue,
including high availability, fault tolerance,
durability of messages, is all handled by AWS.
The system itself scales automatically.
With the standard queue, we can scale almost limitlessly
and we can control authorization
using the identity and access management
specifically for users groups or roles.
We can also use a resource policy,
so we can control usage of the queue at the queue itself
and by doing that, we can use a resource policy
to share that queue with another account,
being it an account that we own
or an account owned by a third party.
We can also share the queue to anonymous users,
if we needed to.
SQS also supports server side encryption,
so if you are dealing with messages
and data, such as personally identifiable information
or protected health information
then we can leverage server side encryption
using AES256 to ensure that those messages
are encrypted at rest.
Now, with the Simple Queue Service,
we have two different types of queues.
We have a standard queue and a FIFO queue.
With the standard queue,
keep in mind that the order is not guaranteed
and it does make a best effort to deliver it in order,
but it's not guaranteed.
It's also an at-least-once delivery model,
meaning that it's possible for a message
to be delivered more than once.
So, your application would be responsible
for any kind of either D duplication
or ensuring that your operations are item potent,
meaning that if you end up performing the same process
on the same message twice,
then you end up with the same result.
Now, with the standard queue,
we get nearly unlimited throughput.
There's really nothing for us to do
to ensure that the standard queue can scale
to just about any throughput level.
We can send millions and millions of messages
through the standard queue,
and again, AWS is responsible for ensuring
that it scales.
AWS handles all of the infrastructure
necessary to make that happen.
Now we also have First In First Out queues.
With FIFO queues, the order is guaranteed
and it also follows an exactly-once delivery
so there are plenty of applications
where you rely on the order absolutely
and many applications where you can't afford
to do things twice
so you need an exactly-once delivery.
So if you need order, if you need exactly once delivery,
then take a look at the FIFO queue.
And so the difference in throughput between these two
is fairly significant.
Again, with standard queue,
we get nearly unlimited throughput,
but with the FIFO queue,
we do have an upper limit of about 3000 messages per second
if we are using a batch type of operation.
So let's take a look at a diagram here.
The idea with the queue,
I'm sure many of us are familiar with queues,
some of us may not be,
but in terms of queues
and how they can benefit our applications,
with this diagram, you can see here that
we have some type of an application running on EC2 instances
and those are behind a load balancer.
So, it could be that users
are asking for some kind of work to be done.
They could be uploading documents
and our application on EC2 is responsible
for ingesting those documents
or it could be that they are asking for some kind of report
and then willing to wait for that report
to be delivered to them later.
Either way, what we're looking at here
is the first stage of an asynchronous process.
The idea here is that we don't really want to rely
on our backend instances here to perform everything.
We want to decouple, for example,
if we're ingesting images,
we want to decouple image ingestion from image processing.
We want to decouple a user requesting a report
from the report actually being generated.
And so, the way that we could use this
is that within Amazon SQS, within the service,
we simply create a queue and that's all we have to do.
We create a queue, we give it a name
and we get a particular DNS endpoint for that queue
and then our instances sends messages to the queue
and those messages will buffer,
they will stack up in the queue
and then our back end instances,
here we have back end instances that are responsible
for doing the work, whatever that work is.
It could be doing some type of an image processing,
it could be video transcoding,
it could be executing some type of report
on a large data set that's going to take some time.
Our user has requested it,
now they're off doing something else
while the report is being generated.
These systems back here, whatever this is,
all it has to do now
is pull the queue.
By pulling the queue, it can then receive these messages.
Once it receives the message, it can then do the work.
Once that work is completed,
it then deletes the message from the queue.
Now, these back end instances here could be anything.
Doesn't have to be EC2.
Those could be Lambda Functions.
They could be containers
running within ECS or EKS.
In fact, it doesn't even have to be anything on
or within AWS.
It could very well be servers,
traditional servers running on premises.
But the idea here is again, that we are looking to decouple
whatever work is being done back here
from whatever work is being done up here
so that if either one is down,
if our back end is down then messages
can simply continue to buffer
and stack up in the queue
and the back end being down should not bring down
our file ingestion.
We should still be able to ingest
or do the front end work
until we're able to get the back end back online.
And the converse is true as well.
If image ingestion is offline temporarily,
then our back end can continue to work through the messages
that are already in the queue.
If you have a need for a queue,
if you need the ability to send messages
between components like this,
if you are looking to decouple
different stages of your larger application,
then definitely take a look at Amazon Simple Queue Service.