Processing Notification and Queue data in AWS

Data ingestion in Batch (typically large datasets) and Real time (trickle feeds) are the most common forms of ingesting data. In this blog let us look processing the data which has been ingested via Notifications and Queues.

Notifications are typically used to inform users of events, while queues are typically used to store and process messages in a specific order.

Examples of notifications:

• An email notification that you have received a new message.
• A push notification on your phone that a new app update is available.
• A pop-up notification on your computer that a new file has been downloaded.

Examples of queues:

• A queue of print jobs waiting to be printed.
• A queue of emails waiting to be sent.
• A queue of tasks waiting to be executed by a background worker process.

The table below summarizes some key concepts of Notifications and queues:

Feature	Notification	Queue
Purpose	To inform a recipient of an event.	To store and process messages in a specific order.
Delivery	Notifications are typically delivered immediately to the recipient.	Messages are typically processed from a queue one at a time.
Persistence	Notifications are typically not stored persistently.	Messages in a queue can be stored for long, so that they can be retrieved and processed even if the system crashes or restarts. In Amazon SQS, you can use the MessageRetentionPeriod parameter to set the message retention period from 60 seconds (1 minute) to 1,209,600 seconds (14 days).
Audience	Notifications can be sent to one or more recipients.	Messages in a queue are typically processed by a single consumer.

Architecture flow:

1)    Amazon SNS notifications from a different AWS account is received into Amazon SQS.

2)    A Source application is sending messages to the Amazon SQS.

3)    Amazon SQS is used to collect the Amazon SNS notifications and Source messages.

4)    AWS Lambda is triggered for every message that comes to the Amazon SQS. AWS Lambda will take the message from Amazon SQS and combines it with relevant information from Amazon Aurora to create the final output.

5)    AWS Lambda output is sent to the Consumer application.

6)    The entire Request + Response + Message sent Status is captured in Amazon Aurora for auditing and other uses.

7)    If there was a failure response from the consumer application, then Lambda writes the complete constructed message to a different Error SQS.

8)    A different Lambda flow will be triggered which will check the messages on the Error SQS and try to send it to the Consumer Application. This SQS records will be deleted only if the message was a success. Else, that record will become visible once after the visibility timeout. Also, if the response from the Consumer Application is success, then the Request + Response + Message sent Status is captured in Amazon Aurora for auditing and other uses.

Note – If you a continuous feed of Notifications and/or Queue messages, you can use Amazon ECS Fargate in place of AWS Lambda.

Conclusion:

Notifications and queues are both useful tools for communicating between different components of a software system. If you need to inform a user of an event, then a notification is the best choice. If you need to store and process messages in a specific order, then a queue is the best choice.

~Narendra V Joshi