Apache Kafka Room for Improvement

294bf53a 7050 4633 bd52 249f8c4d01d7 avatar
Michael Silvi
Senior Software Engineering Consultant at a tech services company with 51-200 employees
Kafka requires non-trivial expertise with DevOps to deploy in production at scale. The organization needs to understand ZooKeeper and Kafka and should consider using additional tools, such as MirrorMaker, so that the organization can survive an availability zone or a region going down. Shifting availability concerns to Kafka means that it cannot go down. It's important to understand the partitioning model and replication needs before relying on it for critical business functions. I'd suggest using it with a feature toggle for a non-critical path in production and learning from failure before relying on it. While Kafka is built to scale, that does not mean that applications can start as many consumers or producers without consideration for how Kafka brokers will perform. Considerations about scaling out brokers need to occur before publishing millions of messages. View full review »
B94acfff c187 47f3 a11a 2f35bba4f243 avatar
Sean Hickey
Solutions Architect at a consultancy with 1,001-5,000 employees
The GUI tools for monitoring and support are still very basic and not very rich. There is no help in determining a shard key for performance. View full review »
868fac92 ebf8 4953 a45c b94dc0d0ab3f avatar
Kevin Quon
Technical Architect at a tech vendor with 51-200 employees
As an open-source project, Kafka is still fairly young and has not yet built out the stability and features that other open-source projects have acquired over the many years. If done correctly, Kafka can also take over the stream-processing space that technologies such as Apache Storm cover. Currently, as it is in the big/fast data integration world, you need to piece together many different open-source technologies. For example, to create a reliable, fault-tolerant streaming processing system that ingests data, you need: * a producer service * an event/message buffer such as Kafka or a message queue * a stream processing consumer such as Spark, Flink, Storm, etc. * something to help facilitate the ingestion into target datasources such as Flume or some customized concoction. This is simply to ingest the data and does not necessarily account for the analytical pieces, which may consist of Spark ML, SystemML, ElasticSearch, Mahout, etc. What I'm getting at is basically the need for a Spring framework of big data. View full review »
Anonymous avatar x60
Founder, CEO at a tech vendor with 1-10 employees
The product is good, but it needs implementation and on-going support. The whole cloud engagement model has made the adoption of Kafka better due to PaaS (Amazon Kinesis, a fully managed service by AWS). View full review »
Anonymous avatar x60
Head of Engineering
Stability of the API and the technical support could be improved. The Kafka API is changing quite radically with the different releases. There are many new improvements and that's good. But the inherent cost of adapting to a new version of the platform was worrying me at the time. The documentation was sometimes misleading, since it was describing some feature in the new version of the API rather than the one we were using. View full review »
96d1f61b 88f4 4be7 bf9e e3ce6b7d6f17 avatar
Ivan Dyachkov
Team Lead at a financial services firm with 1,001-5,000 employees
The standard Kafka Java library, which is shipped with the product, is too complex for inexperienced users. At my company, engineering teams ended up writing wrapper libraries to solve complex issues. Kafka client libraries in general are complex, regardless of language. This is the price Kafka users have to pay for having simple, yet robust, server-side code. What could be improved is the hard dependency on ZooKeeper. The work in this direction has already been started, though. Overall, the project is moving forward at a very good pace View full review »
Anonymous avatar x60
Java Developer at a media company with 1,001-5,000 employees
It’s pretty easy to use for now. I haven’t had any difficulty or problems that I can complain about. Maybe they can add a UI to the configure queues and to display statistics about data stores. View full review »
Anonymous avatar x60
Lead Engineer at a retailer with 1,001-5,000 employees
This product guarantees at-least-once delivery. We have asked JIRA to provide features such as at-most-once delivery to remove duplicate message consumption. View full review »
98daa8e2 c94b 4886 a904 8eece49fd0de avatar
Technical Lead/Project Manager(Consulting Apple Inc) at a tech services company with 1,001-5,000 employees
I would like to see a more user-friendly GUI. View full review »
D8cacc50 c74a 4062 b407 94a51a8ca547 avatar
Chandra Keerthy
Principal Software Architect at a tech services company with 11-50 employees
The management tools are getting mature. When we have thousands of topics, it is hard to visualize. View full review »
D94612f3 2860 4f98 8b91 b334119e0435 avatar
Dori Waldman
Big Data Lead at a marketing services firm with 51-200 employees
* Maintenance: Sometimes brokers disconnect and there are repartitions issues. * Built-in monitoring application for Kafka infrastructure. * UI for Kafka would also be great. View full review »
Anonymous avatar x60
Senior Java Consultant at a tech services company with 501-1,000 employees
It’s perfect for our requirements. View full review »
Fbc6d212 93a9 4393 a7b1 de1e33e98003 avatar
Jyothish Kalavoor Parambil
Hadoop Technical Lead (Assistant Consultant) at a tech vendor with 10,001+ employees
* It needs a separate cluster and a separate administrator to manage the Kafka cluster, adding an extra cost. * It is challenging when data is moved to a mirror cluster, in the case of disaster recovery. It doesn't keep the offset. View full review »
9bc22c4d 7c47 4b6c bc4a 8aa01a3c4e05 avatar
Piyush Ranjan
SDET II at a tech services company with 5,001-10,000 employees
One improvement is in regards to the OS memory management. In case there are too many partitions, it runs into memory issues. Although this is a very rare scenario, it can happen. View full review »
50918253 dfac 4b79 a431 4fe711a85fff avatar
Mehul Jani
Deputy General Manager, DevOps Manager at a tech services company
* GUI for Kafka infrastructure monitoring and deployment View full review »
Anonymous avatar x60
Enterprise Architect at a logistics company with 1,001-5,000 employees
A good free monitor tool would be great for Apache Kafka (from Apache foundation). View full review »
A6947a15 28e5 4b37 adc2 c1655b92dd46 avatar
Sendil S
Java Architect at a tech vendor with 51-200 employees
Too much dependency on the zookeeper and leader selection is still the bottleneck for Kafka implementation. View full review »

Sign Up with Email