Hello my friendly undead!
Today we talk about the AWS service Simple Queue Services, aka SQS. It’s a queue service that AWS provides for enhancing your system design.
So, um, what is a queue and why do I want to use one, zombie?
Life Before Queues
So, if you’ve been around the software block for a while, you may remember that programs used to be one big monolith of code. Sure, sometimes you might be smart enough to move code into smaller functions and such, but often the “Application” consisted of all pieces of code that would do everything from A to Z.
The problem with this approach is that it made for monstrous code collections that are hard to maintain and very fragile. When something breaks, and it will, trying to identify where root problem is can be a real headache. And even if you wrote the code and know every nook and cranny, all that buys you is the inability to hand off the code to others. Having done a lot of maintenance coding in my time, you should always consider the person who will have to take care of your code next.
So as we evolved as programmers, we recognized the value of splitting code into smaller, easier to maintain chunks. One way to do that is to use a queue to hand off work from one component to another. That’s what SQS does for us.
Advantages of Queues
There are a number of other advantages to using a queue as part of your application architecture.
- Scalability – you can scale the needs of the different parts of the application independently by using queues. You might need more emphasis on the ordering part of a system, and maybe less on the distribution part.
- Asynchronous processing – there may be parts of your application that need to respond quickly, and other parts that can take place at a more leisurely pace.
- Batching – some parts make sense to process as soon as something is received, and others that should be done as a batch process.
- Fault tolerance – if one part of your application goes down, you don’t want it to necessarily bring your entire application down. Using a queue will allow the parts to work independently of each other.
- Monitoring – you can more easily monitor a small part of an application than a large part. You can create specific alarms to alert you to specific areas of the application that may be having trouble.
Let’s take an example. You operate an online store that provides the best zombie fashion for the year to your undead clientele. After watching The Walking Dead premiere, you realize your current wardrobe is so last year, and you need the newest outfits for your shuffle-down-the-street-as-a-mob needs.
We have designed an ordering system that can handle web traffic coming into us, but we don’t want to create a bottleneck at the database as we update our inventory. Zombies all over the world want to buy the latest green sweater with the red blood splatter across the left arm. We get a sudden intake of orders all in the hour after the show. Crazy internet traffic could slow down my site, except that I used a queue to split the “take an order” from the “process an order”.
Setting It Up
Go ahead and login to the AWS console. Then choose Simple Queue Service
AWS always introduces you to the new service if you’ve not yet visited. Welcome to the SQS greeting page.
Exciting. Let’s get started by clicking on the “Get Started Now” button.
We’ll call our queue ZombieOrderingQueue.
Next, we have a decision to make. Do we want a “Standard Queue” or a “FIFO Queue”? What’s the diff?
Let’s start with explaining a FIFO Queue. FIFO stands for First In-First Out, which means that the queue messages will be processed exactly in the order they are received. They’re also only processed once, and should be used when the order in which needs to occur is important. However, they are limited in how many messages they can handle at a time. A maximum of 300 messages per second, whether it is a send, receive, or delete. This is a nice quantity, but you may need more.
In that case, you’re left with a Standard Queue. Standard Queues try to ensure the order in which messages are processed, but “no promises”. They can handle nearly unlimited throughput, and messages are guaranteed to be processed “at least once”, which means sometimes more. You may need to write some logic to prevent duplicate messages being processed.
For our case, let’s create a Standard Queue. Select it (selected by default), and choose “Configure Queue” to see what options we have.
There’s a lot of options here.
Let’s look at a couple of them in the Queue Attributes.
- Default Visibility Timeout: When you pull an item from the queue to process, this is how long it will hold that item exclusively for work before returning it to the queue. You may have multiple applications consuming the queue, but while it has access to the item exclusively, only that one application will process it.
- Maximum Message Size: This goes up to a maximum size of 256KB, so if you are working with large objects, you may want to store them somewhere else first (maybe in S3 or a database) and just pass the key to that item.
We also have two other areas of note:
- Dead Letter Queue (DLQ) settings: If a message can’t be processed for some reason, after a maximum number of receives it will be dumped into a DLQ. Maybe there’s an error with the data, and you can investigate what goes into the DLQ to see what happened.
- Server-Side Encryption: This area allows you to make sure the items that enter your queue are encrypted.
For now, we’ll keep the defaults. Click on “Create Queue”.
Cool, we got a nifty queue.
Using Our New Queue
Looking at the details tab, we can see the queue has a few interesting tidbits. A URL, and more importantly, an ARN. ARN is an AWS Resource Name. ARNs are important because that’s how we uniquely identify a service we create in AWS. For now, let’s go ahead and copy that ARN to a clipboard. We’ll use it later.
Also, our queue line itself has a context sensitive menu. Go ahead and right click on the queue in the queue list.
We can test our queue, see what’s in it, and more from this menu. It also looks like SQS queues play well with SNS Topics, and as you’ll recall, we’ve worked with those before.
Using SNS To Send Messages to SQS Queue
Hey, I got an idea, let’s create an SNS topic to send messages to our queue.
We can subscribe to an existing topic, but let’s go through the process of creating an SNS Topic again, and then we’ll connect it to our queue.
Go back to the services menu, and select SNS.
Click on “Create Topic”.
Let’s call our topic “WalkingDeadPremiereOrders” and give it a display name of “WalkDead1” (because of the annoying 10 character limit on this field, ya know).
Cool, we got a new topic.
We could now navigate back to SQS to subscribe our SQS topic to SNS, but we can also do it here, so let’s click on “Create subscription”.
Choose “Amazon SQS” for the protocol.
Now take the ARN you copied earlier and paste it in the endpoint.
You’ll get a brief pop-up that explains that the queue needs to agree to subscribe.
So let’s go back to SQS.
Right click on the SQS Topic and choose “Subscribe Queue to SNS Topic”. Then choose the topic we just created. It will automatically fill in our Topic ARN.
So, let’s go back to SNS, and let’s send a test message to our SNS topic.
Go ahead and fill in the fields.
Note I added a couple message attributes (Price, OrderID) so we can see what those look like. I also changed the format from JSON to Raw. Click on “Publish Message”.
We expect to see that message pass along to our SQS queue. Let’s go look.
Awesome, we have one message available in the queue now. Right click on the queue and select View/Delete Messages.
You’ll get a message about the View/Delete Messages removing the messages from being visible in the queue until the console stops polling. That’s ok. Click “Start Polling for Messages”.
You’ll see a preview pane with the messages that were polled.
Awesome, let’s click on the “More Details” link.
Delicious JSON-y goodness. Click on Message Attributes.
Empty as a grave, hmm.
Let’s go back and look more at the Message Body tab. Scroll down some.
Oh, um, here they are. Weird that SNS message attributes get added here, rather than received in SQS’s Message Attributes tab, but c’est la vie. That seems to be the way it works with SNS to SQS.
Using SQS Itself To Send Messages to SQS Queue
Let’s try putting a message into the SQS queue directly from SQS
Click on “Queue Actions” and select “Send a Message”. You can also right-click on the topic.
Add text for some blood stained, stone washed jeans. Very stylish this year I hear.
Click on Message Attributes and add a Price as a number for 26.95.
Don’t forget to click “Add Attribute”.
Click “Send Message”.
Cool, you got a confirmation. Click “Close”.
Cool, now I have two messages. Let’s look at them again. Right click and choose “View/Delete Messages”. Click “Start Polling for Messages”.
It looks like the message was stored as plain text, not JSON. Click “More Details” for our new message.
There’s our message, let’s look at attributes.
Cool, we got attributes.
Summary
So we’ve played around a bit with SQS and queues. They come in real handy when doing programming tasks, which we will look at in future posts. They chain together nicely with other AWS services, which gives them power and lets us start to explore the world of AWS service components as building blocks for applications. We’ll use them a lot.