Apache Kafka Review
Does real-time streaming and persistence into distributed nodes. It provides a mechanism to create, publish, and subscribe.

Valuable Features

Real-time streaming and persistence into distributed nodes. It provides a simple mechanism to create, publish, and subscribe.

Improvements to My Organization

We are using Kafka as part of our product. It is one of the messaging layers used to interact between various layers of software modules. This provides a clear separation of modules and leverages it for development and testing of different modules.

Room for Improvement

The management tools are getting mature. When we have thousands of topics, it is hard to visualize.

Use of Solution

I’ve been using Kafka for two years.

Stability Issues

We have not encountered any stability issues.

Scalability Issues

We have to balance the nodes when topics partition across cluster nodes. As it assumes they are of equal sizes, sometimes some nodes may not be allocated similar resources. Reassignment moves all the partitions of specified topics which may be an issue when not planned for.

Customer Service and Technical Support

We have the source code to make changes if necessary.

Previous Solutions

Kafka rendered itself suitable for our product offering. It supports all the necessary requirements for a real-time pipeline.

Initial Setup

Setting up was easy with ZooKeeper.

Pricing, Setup Cost and Licensing

With paid support from Confluent, you get the additional benefit of Kafka Connect.

Other Solutions Considered

We used Akka Streams for faster communication, but it would require additional configuration and setup for persistence. Kafka provides those by default.

Other Advice

Kafka provides distributed persistence and streaming layers. The user has flexibility in managing as a consumer on how to consume messages if they have to handle resilience in their code. It requires ZooKeeper.

Disclosure: I am a real user, and this review is based on my own experience and opinions.

Add a Comment

Anonymous avatar x30
Why do you like it?

Sign Up with Email