What on earth is a message broker? Why are some dumb, and others smart? And what’s their job?
According to Wikipedia, a message or event broker is “an intermediary computer program module that translates a message from the formal messaging protocol of the sender (producer) to the formal messaging protocol of the receiver (consumer)”.
In other words, a message broker is a piece of software which enables your applications, systems, and services to be decoupled and still communicate with each other to exchange information. It does this by storing and translating messages between messaging protocols. So even if messages are written in different languages or implemented on different platforms, a message broker allows your interdependent services to “talk” to one another across languages.
So your broker really is a go-between! It stores, routes, validates, and/or transforms protocols between producers and consumers. Which would make you think they’re all pretty smart, right?
But there’s more to it than that.
Why are some brokers smarter than others? And does it matter?
Yes, it does matter. But first, here’s how to spot the difference.
A broker is “smart” when:
- It has built-in logic for the message ‘retry’ process.
- It keeps track of which messages have been ‘consumed’.
- It can route and/or filter messages based on certain conditions across distributed environments (cloud / on premise).
- It has advance features like transactional messages processing, dead-letter queue, detect duplicate messages, message ordering, etc.
- It can reliably keep track of multiple steps, enabling service choreography.
Whereas a broker is regarded as “dumb” if:
- It keeps messages in queues for a fixed duration of time and leaves the responsibility of reading the message from the queue to the consumer.
- It has no idea of which message has been consumed.
- Smart consumers can retrieve historical event stream on-demand to replay messages.
- It doesn’t have the advanced features found in a “smart” broker, e.g. detecting duplicate messages.
- ‘Enterprise features’ need to be built on top of the broker.
So where would you use one over the other?
A ‘smart’ broker handles the messages at pretty much the pace they are created, and removes the message from the data queue once ‘sent’. It has a broader range of intelligent capabilities and handles more complex requirements – lessening the load on the business’ technology resources.
Examples of smart broker technologies are RabbitMQ, Azure Service Bus, NServiceBus, ActiveMQ and Solace. Companies including Reddit, Trivago, T-Mobile, AT&T, UK Power Reserve, Fujitsu, Honeywell, Bank of America, eBay, Caterpillar Inc., Health First, Barclays Capital, and London Stock Exchange use smart brokers. Smart brokers are often used where enterprise features are required out of the box.
By comparison, a ‘dumb’ broker is used in a business which has a significant and consistent stream of low value real-time data to manage and needs the job done quickly. It can support many consumers and retain large amounts of data for a fixed duration of time with very little overhead. Examples of dumb broker technologies include Apache Kafka, Azure Event Hub and Kestrel. And to give you some context, dumb brokers are used by Netflix, Spotify, Uber, LinkedIn, Twitter, Rabobank, Goldman Sachs, The New York Times, eBay, PayPal, Pinterest, T-Mobile, BMW, and Schneider Electric.
Cost, capability, and use-case dictate where you use one broker over another. Message brokers can affect the performance of your data processing, though, so choosing the right one is essential.
Some organisations even use smart and dumb brokers side by side, depending on the use case (you will have noted that eBay and T-Mobile have both!). They deploy smart brokers to handle the more complex and critical message exchanges, while the dumb broker is their workhorse, processing high volume, low value log style messages.
Selecting a smart or dumb message broker can be a key decision when planning your enterprise integration strategy. Adaptiv’s consultants have successfully delivered multiple projects using both styles of broker, and can work with you to determine which solution best meets your requirements.
If you’d like to find out more about how you can use message brokers in your business, or to receive a whitepaper on A Guide to Smart and Dumb Message Brokers, contact phil.cullen@adaptiv.nz
