The ability to partition data on Kafka is valuable. But Kafka needs support and management. It is better to have it fully managed on the cloud.
The only reason I give Kafka as product a low rating is because there are far superior and cheaper alternatives in cloud-based solutions, where we save money on manpower, electricity, servers, datacenters, networking, etc.
In fact, this is the view I have for pretty much all open source software compared to cloud based services. They just make things cheaper, faster, scalable and manageable. Kafka is good, but Kafka as a cloud service is awesome!!
This is a relative rating (compared to cloud services), not that something is wrong with Kafka. I hope that is clear.
Improvements to My Organization:
We have used Kafka for streaming customer web clicks from live sessions to understand customer behavioral patterns.
Room for Improvement:
The product is good, but it needs implementation and on-going support. The whole cloud engagement model has made the adoption of Kafka better due to PaaS (Amazon Kinesis, a fully managed service by AWS).
Use of Solution:
I have used Kafka for a very short period; like six months. Then I moved to Amazon Kinesis, the PaaS offering for Kafka.
No issues here with stability.
Ah, scalability!!! We need to set up multiple servers again for handling the load, which makes Kafka not scalable, unless you subscribe to cloud services.
It’s an Apache-community based support, so it is not really prioritized if you have a business issue. This is why most enterprise customers pay for cloud services.
We didn’t have a previous solution. We started with Kafka and then switched to Amazon Kinesis (PaaS for Kafka). I think Microsoft Azure also released a competing service.
The setup was straightforward.
Cost and Licensing Advice:
Licensing issues are not applicable. Apache licensing makes it simple with almost zero cost for the software itself.
Other Solutions Considered:
We unsuccessfully, and kind of foolishly, tried Apache Camel. They were not similar in services, so we moved to Kafka rightfully, and then to AWS cloud ultimately.
If you have a dedicated Kafka resource to implement and manage the services, then go for Apache Kafka. Otherwise, do consider cloud-based services from AWS or Azure.
Disclosure: I am a real user, and this review is based on my own experience and opinions.