Apache Kafka Review

We use it as an MQ. From it, we have several consumers like Secor that upload raw data to S3.

What is most valuable?

We are using Kafka consumer and producer.

How has it helped my organization?

We are using Kafka as MQ; our servers generate events which are being sent to Kafka. From Kafka, we have several consumers like Secor (https://github.com/pinterest/secor) that upload raw data to S3; Spark stream that is doing aggregations and saving the result in Cassandra; and Druid for OLAP.

What needs improvement?

  • Maintenance: Sometimes brokers disconnect and there are repartitions issues.
  • Built-in monitoring application for Kafka infrastructure.
  • UI for Kafka would also be great (similar to http://www.kafkatool.com/).

For how long have I used the solution?

I have used this product for two years.

What do I think about the stability of the solution?

We used to have problems in Kafka every three weeks and our dev ops team fixed a few issues. For the last six months, there have been no production problems, but during the time Kafka was not stable, it was not easy to understand what was wrong and how to fix it.

What do I think about the scalability of the solution?

We have not encountered any scalability issues yet. We are growing and currently, we manage 1M events per second in Kafka.

How is customer service and technical support?

We need more documentation regarding maintenance issues.

Which solutions did we use previously?

I used RabbitMQ and ActiveMQ. Kafka is the standard, so there is no question what to use (unless you need better performance, like in ZeroMQ).

Which other solutions did I evaluate?

We did not evaluate other options as Apache Kafka is the standard.

What other advice do I have?

Read the documentation and understand the offset issues (where to save them, read from start to end).

Disclosure: I am a real user, and this review is based on my own experience and opinions.
Add a Comment
Sign Up with Email