- Offset management by consumer
This is the base streaming component of our IoT platform.
In case of disaster recovery, we mirror the data in the cluster by maintaining the offsets and store the data within Hadoop 2.8 HDFS.
I have used this solution for one year.
The open source community is very strong. Also, distributors like Cloudera and Hortonworks provide paid support.
For big data, we did not have a previous solution. I have used Microsoft MQ for building traditional systems.
The setup was straightforward.
This is open source with the cost of a cluster administrator.
We did not look at anything else. At that time, this was already accepted by the industry for streaming data processing.
If the Hadoop distribution is MapR, then consider MapR Streaming. MapR Streaming has overcome these fundamental issues. It stores data within the MapR-FS itself. So there is extra overhead, but with a licensing cost.