Apache Kafka Review
This is the base streaming component of our IoT platform. It needs a separate cluster and a separate administrator.

Valuable Features

  • Distributed
  • Persistence
  • Offset management by consumer

Improvements to My Organization

This is the base streaming component of our IoT platform.

In case of disaster recovery, we mirror the data in the cluster by maintaining the offsets and store the data within Hadoop 2.8 HDFS.

Room for Improvement

  • It needs a separate cluster and a separate administrator to manage the Kafka cluster, adding an extra cost.
  • It is challenging when data is moved to a mirror cluster, in the case of disaster recovery. It doesn't keep the offset.

Use of Solution

I have used this solution for one year.

Customer Service and Technical Support

The open source community is very strong. Also, distributors like Cloudera and Hortonworks provide paid support.

Previous Solutions

For big data, we did not have a previous solution. I have used Microsoft MQ for building traditional systems.

Initial Setup

The setup was straightforward.

Pricing, Setup Cost and Licensing

This is open source with the cost of a cluster administrator.

Other Solutions Considered

We did not look at anything else. At that time, this was already accepted by the industry for streaming data processing.

Other Advice

If the Hadoop distribution is MapR, then consider MapR Streaming. MapR Streaming has overcome these fundamental issues. It stores data within the MapR-FS itself. So there is extra overhead, but with a licensing cost.

Disclosure: I am a real user, and this review is based on my own experience and opinions.

Add a Comment

Anonymous avatar x30
Why do you like it?

Sign Up with Email