The most valuable features to me are replication, partitioning and easy integration with Apache Spark, which we use quite a bit for distributed processing.
Replication is good for high availability. It provides additional safety for data in case of node failure or data center outage. Partitioning is a really useful feature for parallelizing processing. We use Apache Spark to process data from a Kafka queue, and Spark is able to assign one executor to each Kafka partition. The more partitions we have, the more threads we can use to process data in parallel. This helps us achieve really good throughput.
Improvements to My Organization:
It will help us build a scalable platform. This will allow the company to provide better customer service.
Room for Improvement:
It’s pretty easy to use for now. I haven’t had any difficulty or problems that I can complain about. Maybe they can add a UI to the configure queues and to display statistics about data stores.
Use of Solution:
I have used Kafka for about a year.
So far, we have not encountered any stability issues.
We have not had any scalability issues. The product is horizontally scalable, so adding extra hardware is all that is needed.
We haven’t needed technical support with the product yet.
I think performance-wise, the product is very good and fits in our use case. We used other distributed message queues, but all products have their own use case
Initial setup wasn’t really complex. We use Kafka through Hortonworks Suite, which comes with many other big data tools. Ambari makes it easy to setup
Cost and Licensing Advice:
Licensing and pricing was handled by my management, so I don’t have much knowledge there.
Give it a try. It’s a valuable, high-performance, distributed processing tool.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Dec 29 2016