Apache Kafka Reviews

The most valuable features to me are replication, partitioning and easy integration with Apache Spark, which we use quite a bit for distributed processing.

Replication is good for high availability. It provides additional safety for data in case of node failure or data center outage. Partitioning is a really useful feature for parallelizing processing. We use Apache Spark to process data from a Kafka queue, and Spark is able to assign one executor to each Kafka partition. The more partitions we have, the more threads we can use to process data in parallel. This helps us achieve really good throughput.

It will help us build a scalable platform. This will allow the company to provide better customer service.

It’s pretty easy to use for now. I haven’t had any difficulty or problems that I can complain about. Maybe they can add a UI to the configure queues and to display statistics about data stores.

I have used Kafka for about a year.

So far, we have not encountered any stability issues.

We have not had any scalability issues. The product is horizontally scalable, so adding extra hardware is all that is needed.

We haven’t needed technical support with the product yet.

I think performance-wise, the product is very good and fits in our use case. We used other distributed message queues, but all products have their own use case

Initial setup wasn’t really complex. We use Kafka through Hortonworks Suite, which comes with many other big data tools. Ambari makes it easy to setup

Licensing and pricing was handled by my management, so I don’t have much knowledge there.

Give it a try. It’s a valuable, high-performance, distributed processing tool.

In my previous company, we had a proprietary implementation and we changed it with Kafka. We changed it because we had many different connectors available and it also allowed us to create a window to our products for the client. It was an on-premise product and it allowed the outline to take the data out, without us developing anything.

You can connect in any language and there are a lot of connectors available, it helps a lot. And it creates visibility into the data and stability. There are several alternatives but this is one of the best options for this.

It's very easy to install and it's pretty stable.

The possibility to have connectors is very helpful. Another valuable aspect is that it's mature and open-source. 

From a scalability point of view, you just add servers and it's scalable. The whole architecture is very scalable.

There is a feature that we're currently using called MirrorMaker. We use it to combine the information from different Kafka servers into another server. It's very wide and it gives a very generic scenario. I think it would be great if the possibility would exist out of the box and not as a third party. The third party is not very stable and sometimes you have problems with this component. There are some developments in newer versions and we're about to try them out, but I'm not sure if it closes the gap.

I have been using this solution for six months. I also worked with it additionally in my previous company but not so intensively. 

I have never needed to use technical support. I know it's available but we haven't needed it because there's a lot of information on the internet that has helped us to solve our issues. 

I would definitely recommend Kafka. In our current position, we use it to move a lot of data and I think it's definitely working well. I would definitely recommend it.

I would rate it an eight out of ten. 

Through its publisher-subscriber pattern, Kafka has allowed our applications to access and consume data at a real time pace.

I like the performance and reliability of Kafka. I needed a data streaming buffer that could handle thousands of messages per second with at least one processing point for an analytics pipeline. Kafka fits this requirement very well, as it is a fast, distributed message broker. It definitely does exactly what it is designed to do.

As an open-source project, Kafka is still fairly young and has not yet built out the stability and features that other open-source projects have acquired over the many years. If done correctly, Kafka can also take over the stream-processing space that technologies such as Apache Storm cover.

Currently, as it is in the big/fast data integration world, you need to piece together many different open-source technologies. For example, to create a reliable, fault-tolerant streaming processing system that ingests data, you need:

• a producer service
• an event/message buffer such as Kafka or a message queue
• a stream processing consumer such as Spark, Flink, Storm, etc.
• something to help facilitate the ingestion into target datasources such as Flume or some customized concoction.

This is simply to ingest the data and does not necessarily account for the analytical pieces, which may consist of Spark ML, SystemML, ElasticSearch, Mahout, etc.

What I'm getting at is basically the need for a Spring framework of big data.

The only stability issues we had were mostly a result of the evolving APIs and existing bugs.

Kafka is designed to be very easily scalable so I did not have any trouble here.

We used the open-source version and did not buy support from Confluent.

We did not have any other previous solutions. Our project was green field and a new type of project development.

Initial setup was straightforward. We simply hosted multiple Kafka brokers and ZooKeeper servers on AWS EC2 instances.

We implemented it in-house and then went with the Hortonworks Data Platform distribution.

We evaluated AWS Kinesis as well.

Kafka is open source and requires an administrator to maintain the servers.

This is a system for email and other small devices. There has been a relay of transactions continuously over the last two years it has been in production.

Besides better stability and scalability, there are no additional functionalities I'd like to see. Kafka is good at what it does.

The management overhead is more compared to the messaging system. There are challenges here and there. Like for long usage, it requires restarts and nodes from time to time.

We started using this solution two years ago.

There are issues with stability. It's not 100% stable like ActiveMQ, but it's maybe 98% stable.

With the containerized version we have used, we have faced challenges with the scalability.

Initial setup was not easy. It requires intermediate skills.

This is an open-source version.

I would rate this solution 8 out of 10.

We use Apache Kafka for our messaging. 

We publish a message and ask the subscriber to listen to it. We use it to save events generated by integration with external systems. There are external events, that are first published to our Kafka queue, and then to a topic, and then we save it to our own data storage system.

I believe that the speed, and especially the performance, are very good features.

Also included is a cluster with built-in failover. 

It is the performance that is really meaningful.

In terms of features, we are satisfied. I don't require any additional features. I don't believe we require any additional features at this time.

More Windows support, I believe, is one area where it can improve. We need to wrap it as a service, but there isn't one built into Windows. So that's something they could improve.

I believe Windows Server is primarily aimed at the Windows shop or those who use Windows.

I don't recall the specific version that we are using, it may be Kafka 2.11, but it is not the latest one.

It's stable. However, the Windows Service is not very stable because it is a wrapper.

We are a small team with a few people.

We might increase our usage in the future.

We don't get in touch with technical support. We rely on open-source software. We haven't used the help of technical support. We did not seek assistance. As a result, I have no opinion on the subject.

This is the first product we have used. We didn't have anything prior to that.

The initial setup is straightforward. It's easy to set up.

It took a few days to get it up and running.

We only need one or two engineers to keep this solution running. We basically let it run and monitor what's going on. We usually don't touch it unless something goes wrong.

We deployed it ourselves.

It's free. We use the free version.

VMware RabbitMQ and ActiveMQ are products that are not being used by us. I wanted to look into it. But we use different things.

We compared our findings to those of other researchers. We are primarily concerned with performance. Kafka is unquestionably the performance leader.

I would recommend trying this solution, but you should probably run it on Linux.

I like this product, I would rate Apache Kafka a nine out of ten.

I'm a software architect. The use case will depend my customers. They usually use it for data transfer from static files to a legacy system.

In the next release, I would like for there to be some authorization features and HTL security. 

We also need bigger software and better monitoring. 

I have been using Apache Kafka for the last ten years. 

The stability is good. We've never had any issues. 

Scalability is very good. 

I have never needed to contact technical support. My colleagues get support from here, in Morrocco. 

The setup is not a big deal for us. We can handle it. After the system is set up, the person who administers it has to do so with Apache Kafka.

Depending on the setup, it will usually take two weeks.

I would rate it a nine out of ten. Not a ten because of the monitoring and admin improvement I'd like for them to make. 

We are using Kafka consumer and producer.

We are using Kafka as MQ; our servers generate events which are being sent to Kafka. From Kafka, we have several consumers like Secor (https://github.com/pinterest/secor) that upload raw data to S3; Spark stream that is doing aggregations and saving the result in Cassandra; and Druid for OLAP.

• Maintenance: Sometimes brokers disconnect and there are repartitions issues.
• Built-in monitoring application for Kafka infrastructure.
• UI for Kafka would also be great (similar to http://www.kafkatool.com/).

I have used this product for two years.

We used to have problems in Kafka every three weeks and our dev ops team fixed a few issues. For the last six months, there have been no production problems, but during the time Kafka was not stable, it was not easy to understand what was wrong and how to fix it.

We have not encountered any scalability issues yet. We are growing and currently, we manage 1M events per second in Kafka.

We need more documentation regarding maintenance issues.

I used RabbitMQ and ActiveMQ. Kafka is the standard, so there is no question what to use (unless you need better performance, like in ZeroMQ).

We did not evaluate other options as Apache Kafka is the standard.

Read the documentation and understand the offset issues (where to save them, read from start to end).

I used Kafka with a client to decouple applications with different availability profiles. Before using a messaging-based architecture with Kafka as the messaging system, the client used a coordinator application to fire off various posts to as many as eight other applications. With an application that's impacting at least a customer a second in airports, where the customers demand that the system always works, there were issues with ensuring high availability.

A typical way to calculate system availability is: Availability = Uptime/(Uptime + Downtime). Hence, where there are two applications involved with a 99% availability, the total system availability degrades quickly: 99% * 99% = 98.01%.

With eight applications, total availability caused issues. However, only two systems needed to provide real-time responses, while other systems were for payment processing, CRM, promotions, etc. It was OK if those systems were not up to date in real time.

Kafka allowed the client to have temporal decoupling for writes, i.e., the flaky third-party CRM system did not need to be available at the moment for us to respond to a user with a successful response. The availability concerns shifted to Kafka, which is a better trade off because it's built for this.

Another benefit, though not required, was the addition of logical decoupling between applications. Additional consumers could be built to overlay concerns of analytics, but the systems responsible for creating the entities on a given topic did not need to be aware of the analytics applications. This simplifies the interaction between applications and concerns of an organization.

Another benefit of this architecture is that testing is simplified. A given application needs to be tested to obey a contract of reading a message and producing another message. A Kafka topic acts as the boundary for an integration test.

Kafka, as compared with other messaging system options, is great for large scale message processing applications. It offers high throughput with built-in fault-tolerance and replication.

Messaging systems in general allow for logical and temporal decoupling between applications. Given Kafka's high availability, it's a great option to use if applications require availability, but not real-time processing.

If a downstream system is offline, messages can queue up and process when possible, but the user may not necessarily need to be aware of any issues.

A messaging-based architecture becomes important as a set of micro-services need to scale with high availability. Kafka is a great choice for messaging with such architecture.

Kafka requires non-trivial expertise with DevOps to deploy in production at scale. The organization needs to understand ZooKeeper and Kafka and should consider using additional tools, such as MirrorMaker, so that the organization can survive an availability zone or a region going down.

Shifting availability concerns to Kafka means that it cannot go down. It's important to understand the partitioning model and replication needs before relying on it for critical business functions. I'd suggest using it with a feature toggle for a non-critical path in production and learning from failure before relying on it.

While Kafka is built to scale, that does not mean that applications can start as many consumers or producers without consideration for how Kafka brokers will perform. Considerations about scaling out brokers need to occur before publishing millions of messages.

Generally, there were no stability issues. However, there was one scare in production when a consumer rebalance took 30 minutes and messages were not being processed during that time.

We have not yet had scalability issues!

There are specialized consulting companies in this space and there are online resources to read. That may help companies get past hurdles.

No, we did you use a previous messaging system.

The setup was complex. One must consider setting up ZooKeeper, Kafka, multi-zone/region availability, as well as typical associated functions for running it all in production. This includes monitoring, message schema changes (consider Avro), encrypting messages if it's a concern, potentially authorization for different topics depending up on the sensitivity of data.

If an organization uses Kafka as the first messaging system, then the approach for application design must also shift significantly.

It is open source software.

The client evaluated alternatives before I arrived, but I was not there during the evaluation so I cannot comment.

Consider using a managed Kafka service, such as from Heroku.

If messaging is not a central component of the business and vendor lock-in is less of a concern, consider using something like Amazon's Kinesis. This can more rapidly provide the benefits of a messaging service without the pain of understanding it deeply, setting it up, and managing it.

It's important to use a lean approach to understand how it will break in production.

Implement a non-critical transaction with it.

Perhaps use a feature toggle within a facade and implement the behavior with the old approach and with Kafka to reduce risk.

Add it to one or two applications and monitor how it goes.

Figure out security, monitoring, scaling, schema migration, etc., before using it as a critical component in an application.