- Message Retention: Unlike regular message queues, messages stay in Kafka after clients consume them. A message can be consumed over and over again by the same or a different client until topic retention (by max data size or oldest message timestamp) kicks in and the oldest messages get deleted. This can be very handy in many scenarios: handling bugs in software, testing code, simple distribution of message processing, and routing messages to many different consumers simultaneously.
- Horizontal Scalability: To add more capacity, both in terms of storage and performance to a Kafka cluster, you just need to add more servers. Regular message queues usually work in a master-slave configuration and do not scale very well horizontally.
- Simplicity in operations.
Improvements to My Organization:
It has become dead simple to connect different application and services, saving a lot of development hours.
Room for Improvement:
The standard Kafka Java library, which is shipped with the product, is too complex for inexperienced users. At my company, engineering teams ended up writing wrapper libraries to solve complex issues. Kafka client libraries in general are complex, regardless of language. This is the price Kafka users have to pay for having simple, yet robust, server-side code.
What could be improved is the hard dependency on ZooKeeper. The work in this direction has already been started, though. Overall, the project is moving forward at a very good pace
Use of Solution:
I have used Kafka for three years.
Sometimes we have stability issues, but not often.
We have not had any scalability issues.
There is no official technical support as the product is 100% open source.
We used RabbitMQ before. It does not scale well.
The setup was pretty straightforward.
Cost and Licensing Advice:
There is no pricing and licensing.
Other Solutions Considered:
We didn't evaluate any other options.
Go ahead. It's a great product.
Disclosure: I am a real user, and this review is based on my own experience and opinions.