Message Queue and Message Broker explained in easy terms + Terminology
This article will discuss Message Queues in detail about their main components, how they work, and their usage. After reading this post, you are expected to know the basics of a message queue and terminologies like Message Queues, Broker, Queue, and Message.
For the sake of simplicity, software, server, service, sensor, or any particular software or hardware that can receive or produce a message will be called a Node.
Message Queue (MQ)
In simple terms, a message queue is a software component that sits between two or more nodes to collect and exchange messages. Software Systems can use message queues in any interprocess communication (IPC) or for inter-thread communication within the same process.
For example, consider a garden with many sensors that generate constant logs for temperature, moisture, etc. As the garden gets more prominent and the number of sensors rises, handling a large amount of data can become complicated and potentially leads to a bottleneck at receiving the messages and losing important data. This situation gets worse if you want to add other things to the system, like scheduling watering of the plants.
On the other hand, MQ provides you decoupling between system components and makes asynchronous messaging easy. This way, all the input messages stores in the queue(s), and whenever the receiver node(s) are ready, they consume the messages. MQ enables the system to handle many messages easily!
The joint RabbitMQ on Google Compute Engine performance test demonstrates how one of the world’s most widely adopted, open source message brokers can sustain a combined ingress/egress of over two million messages per second — a volume comparatively greater than the combined set of all U.S. text, Apple iMessages, and WhatsApp messages per day.
Usage
Backpressure
A situation where there are many requests or data simultaneously to handle. Backpressure can happen while reading and writing files, node communications, render, etc.
Read more about Backpressure.
Chat Applications
One of the main features of a good chat application is scaling up and supporting a heavy load of messages with increasing users. While developing and maintaining such a reliable system is challenging and time-consuming, a message queue can be used instead, enabling us to do so quickly.
Decoupled Communication
In modern cloud architecture, applications are decoupled into smaller, independent building blocks that are easier to develop, deploy and maintain. Message queues provide communication and coordination for these distributed applications. Message queues can significantly simplify the coding of decoupled applications while improving performance, reliability, and scalability.
Dynamic Range of Request
During peak hours, the number of requests can rise to a high level. The system may not be able to respond to all of them. Using MQ make it possible to add consumers on the fly or discard current consumers as they become idle.
More Details
now let’s go into more details and talk about different parts of a Message Queue.
Message Broker
A message broker is an architectural pattern for message validation, transformation, and routing. It mediates communication among applications, minimizing the mutual awareness that applications should have to exchange messages, effectively implementing decoupling.
Message Queue
A queue is a sequential FIFO (First In First Out) data structure with two primary operations. An item can be enqueued (published) at the tail and dequeued (consumed) from the head.
Message
A message is data transferred between sender and receiver(s) with extra information as properties or headers. The message can include information for an operation, a response to a request, or just plain text.
The message can act differently based on the properties it gets. Some of the properties are listed below:
- Scheduling: a message by default is sent only once. However, if you schedule the message, you can set the number of times you want to send it, the starting delay, and the waiting time between each sending.
- TTL (Time to Live): Expiry time each message has before transferring to DLQ (Dead Letter Queue).
- Priority: a number in a range indicating the importance of the message.
Dead Letter Queue (DLQ)
It is generally a special queue for messages which cannot be processed successfully. If any message has one or more of the following criteria, it will be sent to DLQ.
- The message is sent to an invalid queue.
- The destination of the message is full.
- The message length has exceeded the limit.
- message reading count has reached the limit but has never been acknowledged.
- The message expires due to per-message TTL.
Types of Exchanging
Direct Exchange
A direct exchange delivers messages to queues based on the
message routing key. A direct exchange is ideal for the unicast routing of messages (although they can be used for multicast routing). Here is how it works:
- A queue binds to the exchange with a routing key K
- When a new message with routing key R arrives at the direct exchange, the exchange routes it to the queue if K = R
Direct exchanges are often used to distribute tasks between multiple workers (instances of the same application) in a round-robin manner.
Fanout Exchange
A fanout exchange routes messages to all of the queues bound to it, and the routing key is ignored. If N queues are bound to a fanout exchange, when a new message is published to that exchange, a copy of the message is delivered to all N queues. Fanout exchanges are ideal for the broadcast routing of messages.
Because a fanout exchange delivers a copy of a message to every queue bound to it, its use cases are quite similar:
- Massively multiplayer online (MMO) games can use it for leaderboard updates or other global events.
- Sport news sites can use fanout exchanges for distributing score updates to mobile clients in near real-time.
- Distributed systems can broadcast various state and configuration updates
- Group chats can distribute messages between participants using a fanout exchange
Topic Exchange
Topic exchanges route messages to one or many queues based on matching between a message routing key and the pattern used to bind a queue to an exchange. The topic exchange type is often used to implement various publish/subscribe pattern variations. Topic exchanges are commonly used for the multicast routing of messages.
Topic exchanges have an extensive set of use cases. Whenever a problem involves multiple consumers/applications that selectively choose which type of messages they want to receive, topic exchanges should be considered.
Example uses:
- Distributing data relevant to specific geographic location, for example, points of sale
- Background task processing is done by multiple workers, each capable of handling a specific set of tasks.
- Stocks price updates (and updates on other kinds of financial data)
- News updates that involve categorization or tagging (for example, only for a particular sport or team)
- Orchestration of services of different kinds in the cloud
- Distributed architecture/OS-specific software builds or packaging where each builder can handle only one architecture or OS
Header Exchange
A headers exchange is designed to route multiple attributes that are more easily expressed as message headers than a routing key. Headers exchanges ignore the routing key attribute. Instead, the attributes used for routing are taken from the headers attribute. A message is considered matching if the value of the header equals the value specified upon binding.
Conclusion
Day to day, the systems get more complicated, and handling requests and data flow becomes harder. Having a decoupled system with secure and reliable middleware for moving data as messages between applications is vital. We talked about what a Message Queue is and demonstrates its usage by exampling a garden management system.
{Happy Coding}🍻
References
- Wikipedia: https://en.wikipedia.org/wiki/Message_queue
- Amazon: https://aws.amazon.com/message-queue/#:~:text=A%20message%20queue%20provides%20a,messages%2C%20or%20just%20plain%20information.
- Backpressure: https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7
- Queues: https://www.rabbitmq.com/queues.html
- Message Broker: https://en.wikipedia.org/wiki/Message_broker
- RABBITMQ: https://tanzu.vmware.com/content/blog/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine
- AMQP CONCEPTS: https://www.rabbitmq.com/tutorials/amqp-concepts.html
- DLQ: https://en.wikipedia.org/wiki/Dead_letter_queue
