Apache Kafka Review

I am a solution architect and I used Apache Kafka in this role.

The processing power of Apache Kafka is good when you have requirements for high throughput and a large number of consumers. 

They need to have a proper portal to do everything because, at this moment, Kafka is lagging in this regard. It could be used to do the preprocessing or the configurations, instead of directly doing it on the queues or the topics. If you look at Solace, for example, they have come up with a portal where you don't need to touch these activities. You don't need to access the platform beyond the portal.

I have used Apache Kafka for between one and one and a half years.

Apache Kafka is stable.

This is certainly a scalable product. There are currently 30 or more people using it but we expect to scale beyond this. It is going to be an enterprise tool within the company.

I am not directly interacting with the service people at this moment. It is limited for now because we are still exploring and effecting our architecture and design, and deciding how to align it with our existing strategy. There is not much progress in this regard and it will take more time.

Prior to working with Apache Kafka, there was no messaging queue system. For many projects, they were using the Azure Event Hub, but it was not serving the purpose. So, we started moving towards Kafka, and that's why we have procured Confluent Kafka.

Several months ago, I stopped working on Apache Kafka. I am now working on Confluent Kafka. It was not my decision to switch solutions.

My current organization has chosen Confluent Kafka for various reasons. One is that we have a large number of streaming requirements, and Confluent Kafka has one more layer on top of Apache Kafka to do this transformation and connecting with other multiple lane systems.

There are out-of-the-box features along with the KSQL features. For example, things like fetching the events are kind of query-based. So, that seems to be a good feature for our requirements. That is why we ultimately procured Confluent Kafka.

For some time, I have also worked with Solace and it has an advantage. Given that my core strength is integration, I work with integration platforms such as MuleSoft, Azure functions, then TIBCO. Based on our requirements, I found that the event-driven APA implementation with Solace was easier.

Solace also has a top-notch solution for portal management and you register your producers, consumers, and preprocessing logic. All of these things are pretty easy to do. This is an area where Kafka could use some enhancement.

I don't think that the initial setup was a complex process.

MQ messaging systems are not my core strength but for any integration platform where we have a large number of APIs and events, to integrate with an IoT platform, for example, I found Kafka is better than ActiveMQ.

I'm not getting into in MQTT or other things but comparatively, when you compare ActiveMQ and Kafka, Kafka has done better.

I think that many people are using Apache Kafka just as a publishing and subscription model, but I feel that Kafka is better than that. Furthermore, Confluent Kafka is even more than that.

Confluent Kafka is offering features that are equal to those of a data lake. You can do lots with data, and huge data can be persisted. However, many people are not using that feature. Rather than make use of persistence logic, they are pushing the messages and consuming them. Maybe if people were using it for persistence, they would see the impact or real power of Kafka.

I would rate this solution a seven out of ten.