Simple Queue Service (SQS)

Fully managed queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications

What is a Messaging System?

Used to provide asynchronous communication and decouple processes via messages / events from a sender and receiver (producer and consumer)

Queuing

Generally will delete messages once they are consumed. Simple communication. Not Real-time. Have to pull. Not reactive.

SQS, Sidekip, RabbitMQ

VS

Streaming

Multiple consumers can _ react_ to events (messages), Event live in the stream for long periods of time, so complex operations can be applied. Real-time

Kinesis, Kafka, NATS

Introduction to SQS

SQS is for Application Integration

AWS SQS is a solution for the distributed queuing of messages generated by your application. It connects isolate applications together by passing along messages to one another.

A queue is a temporary repository for messages that are awaiting processing.

Using the AWS SDK you write code which publishes messages onto the queue or you pull the queue for messages

SQS - Use case

  1. App publishes messages to the Queue
  2. Other app pulls the queue and finds the message and does something
  3. Other app reports that they completed their task and marks the message for completion
  4. Original app pulls the queue and sees the message is no longer in the Queue

SQS Limits - Message Size

Message Size

The size of the message can be between 1 byte and 256KB

Amazon SQS Extended Client Library for Java lets you send messages upto 2GB in size. The message will be stores in S3 and library will reference the S3 object.

Message Retention

how long SQS will hold onto a message in the queue before dropping it from the queue

Message retention by default is 4 days.

Message retention can be adjusted from a minimum of 60 seconds to a max of 14 days

SQS - Standard Queue

AWS SQS Standard Queues allow you a nearly-unlimited number of transactions per second.

Guarantees that message will be delivered AT LEAST once

More than one copy of a message could be potentially delivered out of order.

Provides best-effort ordering that helps ensure a message is generally delivered in the same order that it was sent.

FIFO (First in First Out)

AWS SQS FIFO queues support multiple ordered message groups within a single queue.

Limited to 300 transactions per second. SQS FIFO queues have all the same capabilities of a Standard queue

SQS - Visibility Timeout

How do we prevent another app from reading a message while another one is busy with that message?(Avoid someone doing the same task)

A visibility time-out is the period of time that a messages are invisible in the SQS queue3, after a reader picks up that message.

Messages will be deleted from the queue after a job has processed. (before the visibility timeout expires)

If a job is NOT processed before the visibility timeout period, the message will become visible again and another reader will process it.

This can result in the same message being delivered twice!

  • 30 seconds (default)
  • 0 seconds minimum
  • 12 hours maximum

SQS - Short vs Long Polling

Poling is the method in which we retrieve messages from the queues.

Short polling (default) returns messages immediately, even if the message queue being is empty

When you need a message right away. Short polling is what you want to use.

Long polling waits until message arrives in the queue or the long poll timeout expires

Long polling makes it inexpensive to retrieve messages from your queue as soon as the messages are available.

using long polling will reduce the cost because you can reduce the number of empty receives

Most use-cases you want to use Long Polling