Apache Kafka Other Advice
AT
reviewer2075460
Group Manager at a media company with 201-500 employees
From an architecture and solution design perspective, I would say that before going for streaming solutions, we should analyze the data, which might be old, and decide if it's a streaming use case or not. Often, people think it's a streaming use case, but when they perform analytics on top of it, they realize they can't do a month-to-date or year-to-date analysis. So, it's essential to think again from the data basics perspective before going to Kafka.
Overall, from the product and solution perspective, I would rate it a nine based on my personal use of data.
View full review »Apache Kafka as a broker tool is a very stable and good product. When you need to create a consumer in any programming language, including Java, Golang or any other programming language, the team involved in the process of the creation of a consumer should have very strong knowledge and expertise in the use of Apache Kafka since it is not at all easy to create a consumer for the product. A highly qualified person with a good amount of experience should also know the internals of the solution, which may not seem too straightforward. Anyone cannot use Apache Kafka easily without proper knowledge or experience. When you use Apache Kafka in your actual application, you need to create some producers and some consumers. To create a consumer, you need to have a very strong understanding of the solution since it is not a process that anyone can manage easily. A company needs to have a very strong team with good technical knowledge to be able to use the product.
I rate the overall solution a nine out of ten.
The number of people required for maintenance depends on the team. They need a centralized team to offer Apache Kafka and services. Each team does have knowledge of Kafka.
This solution has a lot of features and there is no other solution on the market that has similar advanced features. It is a very good solution.
I rate Apache Kafka an eight out of ten.
View full review »Buyer's Guide
Apache Kafka
March 2024
Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: March 2024.
765,234 professionals have used our research since 2012.
Apache Kafka is a good choice, so I would recommend people not have a real-time application if they do not have to. It is better to have a very fast batch operation than a real-time operation. I would rate Apache Kafka a nine on a scale of one to ten.
View full review »I rate Apache Kafka seven out of 10. It's a good solution. They're constantly fixing bugs and adding new features.
View full review »I believe that when working with Kafka Apache, it's essential to have a specialist who thoroughly understands and can optimize all the available variables within the solution to achieve the desired behavior.
I would rate it an eight out of ten.
View full review »
I give the solution an eight out of ten.
Maintaining Kafka, the open source, can be difficult without the proper version purchased or the right infrastructure in place. However, once the initial setup is complete, it is relatively simple to maintain. The open-source version of Kafka is not a complete package, so additional maintenance may be required.
I strongly recommend reading the documentation for any issues because it is likely to contain the answer we are looking for. There is a lot of information provided that may not be immediately obvious, so take the time to explore thoroughly.
View full review »MB
Mohamed BENTAHAR
Architect at Agence Française de Développement
To be able to recommend Kafka to others, especially considering every context, we will have to set a benchmark and compare Kafka with other tools.
I rate the overall solution a seven out of ten.
We had a good experience with the solutions, the maintainability and scalability are good. I would recommend the solution to others.
I rate Apache Kafka a nine out of ten.
View full review »SP
reviewer1142973
CEO at a comms service provider with 11-50 employees
Kafka is a really good product. To be able to keep it running in the long term, you need to know very well how it works. You should have good knowledge about it. It isn't about just knowing how to install it because it is quite simple to install it. It is important to have the right knowledge and experience to do a good installation and let it run for a long period. You can also go for someone who has the right experience and knowledge.
We are very satisfied with Kafka. I would rate it an eight out of 10. It is not perfect, but it is a really good product.
View full review »A non-enterprise business with a low message load can use an open-source solution like Apache Kafka.
I would recommend the solution to enterprise businesses depending on their use cases. Suppose an enterprise business doesn't have any integration or a middleware platform and wants to do a greenfield implementation. I'll evaluate the use cases and refer Apache Kafka to them if messaging is needed only for exception handling or transferring the messages.
I have recommended Apache Kafka to some customers who wanted asynchronous messaging for logging purposes. Those messages were not business-critical messages as such.
I would recommend Apache Kafka to other users. Apache Kafka is more relevant when we use open-source integrations and when customers want to reduce the TCO. As an architect, I recommend the solution to customers based on their messaging needs. Apache Kafka and Anypoint MQ are the only two messaging products available today. The open-source Apache Kafka is always recommended if the customer really doesn't want to get into any of the license models.
Overall, I rate Apache Kafka an eight out of ten.
GT
George Thomas
Lead Architect at a financial services firm with 1,001-5,000 employees
When it comes to Apache Kafka, they must understand how it works and what its internals are. There could be numerous challenges associated with the product and its entire life cycle. You will have to have a good understanding and knowledge of the configuration. You will need a technical person who is knowledgeable in Kafka which will be an advantage and on an ongoing life partner.
It's a very good solution, I would rate Apache Kafka a nine out of ten.
View full review »I rate this solution an eight out of ten. It is good, but the documentation could be improved.
View full review »The documentation can be a challenge. There are quite advanced capabilities of Kafka, like the transformations that you can build to modify the data as needed. We found that the biggest challenge was documentation and being able to gain the knowledge of exactly how to do stuff. We also struggled on the transformation, but other components were fine, so some parts are good, and some parts are bad.
I would rate this solution as an eight out of ten.
View full review »I would recommend that other businesses do the deployment themselves, but manage the tool with the aid of a service provider, rather than in-house.
I would rate this product seven out of ten.
View full review »I don't see any major issues with using Apache Kafka. Many companies use it and it's a good solution. My advice would be to use it as a software-as-a-service rather than setting up your own cluster. This way, you can benefit from a preconfigured and maintained platform. It's better to opt for a software-as-a-service solution.
I rate Apache Kafka an eight out of ten.
View full review »AX
reviewer2116086
Senior Developer at a financial services firm with 10,001+ employees
I would give Kafka a rating of seven out of ten.
View full review »Since it has become so popular, large enterprises especially want to do it. For smaller enterprises, Kafka would probably be too expensive because they would have to hire people to maintain it.
I would rate the Apache Kafka solution a seven out of ten.
TD
reviewer1218324
Head of Technology - Money Movement Platform at a financial services firm with 10,001+ employees
I rate Apache Kafka a nine out of ten.
View full review »AV
Andre Visser
Technical Director at Metrofibre Networx
Apache Kafka is an out-of-the-box, reliable solution. For people in the fiber business, we need a reliable solution, and this solution is hundred percent reliable. If it is set up correctly, it hardly has any issues due to the more extensive user base; even if there are issues, it is sorted by the community. I rate it nine out of ten.
View full review »RG
Reza Gholami
Senior Technology Architect at a tech services company with 10,001+ employees
On a scale from one to ten, I would give Apache Kafka a rating of eight.
View full review »I would advise others to start with non-SSL implementations and try to do PoCs. Afterwards, they should move towards more secure features.
View full review »I give the solution a nine out of ten.
We have 80 people using the solution and five people are required to maintain it.
I suggest using cloud services because the solution is expensive if you are using it on-premises.
View full review »NK
Nitin Kamble
Director at Tibco
I give the solution an eight out of ten.
We test all the supported versions of the solution based on our customers' use.
We support our integration product. So we need to do dev and QA with Apache Kafka or any other messaging applications. But we do not provide support. The solution can be supported by someone else.
We don't need to have any specific staff for deployment. All the developers in QA can install and configure the solution. We don't have a separate person for maintenance.
Our team and our product dev and QAs all use the solution.
I think Apache Kafka is a good solution and I recommend it to others.
View full review »JA
Jack Angoe
Technical Lead at Interface Fintech Ltd
We do not use customer support, but there is a lot of documentation available.
I would definitely recommend this solution to other people. I would rate it as an eight out of ten.
View full review »SK
SunilKalva
Barista Brewing Espresso at Linkedln
There is room for improvement with this solution so I rate it eight out of 10.
View full review »I would rate this solution 7 out of 10.
I would recommend this solution because the queue manager is very fast and stable.
View full review »ME
Mario Estrada
CTO at Estrada & Consultores
New users should understand the product capabilities. Often, people will start putting their hands in new products without knowing the capabilities and the disadvantages in specific scenarios. In our case for example, We haven't used Kafka for financial transaction processing, for which we still use IBM MQ, but It really depends upon your knowledge and experience with the product. My advice is to understand the product very well, its pros and cons and work from there.
Finally I'd rate the solution at a nine out of ten.
DZ
Dimitrios Zigkos
Enterprise Architect at Smals vzw
The maintenance of Apache Kafka is crucial due to the complexity of the system with numerous microservices and systems communicating through Apache Kafka, requiring proper integration and configuration to prevent overloading and ensure a healthy cluster. The task is not easy and requires knowledge of the various adjustable parameters, as misadjusting even one of them can greatly slow down the cluster. For example, if the consumer group changes frequently, the messages must be regrouped and reassigned, causing significant delays. Therefore, configuring Apache Kafka correctly is essential to avoid high latency issues.
I would strongly suggest others give Apache Kafka a chance and explore the various advantages that it can offer, especially since it should not be perceived as a message bus or broker but rather an enterprise bus designed for data manipulation. It has the ability to transform data, store and reject it, and even maintain different versions of the same data simultaneously. Moreover, it operates on a pull mechanism rather than a push mechanism, which takes away the risk of losing data and places the responsibility for data loss on the consumer. On the other hand, it also ensures that the data is always available within the specified window and allows for easy replication of the past, which is extremely helpful in situations such as those involving a hacked bank database. With Apache Kafka, you can efficiently go back in time, obtain the required status and events, and make changes accordingly, without the need to go through each transaction separately. Thus, using this solution can make data management much more efficient and convenient.
I rate Apache Kafka an eight out of ten.
In order to improve its user-friendliness, engineer-friendliness, and DevOps-friendliness, the system must undertake various tasks, such as enhancing the overall operation and configuration, ensuring seamless integration with other systems, and adapting to security layers in a more comprehensive and generic manner. This will require significant efforts to make the system more functional, secure, and efficient.
Apache Kafka is one of the best open-source solutions that are available today.
I would recommend this solution to others.
I rate Apache Kafka an eight out of ten.
View full review »KS
reviewer1421481
Solution Architect at a manufacturing company with 10,001+ employees
I think that many people are using Apache Kafka just as a publishing and subscription model, but I feel that Kafka is better than that. Furthermore, Confluent Kafka is even more than that.
Confluent Kafka is offering features that are equal to those of a data lake. You can do lots with data, and huge data can be persisted. However, many people are not using that feature. Rather than make use of persistence logic, they are pushing the messages and consuming them. Maybe if people were using it for persistence, they would see the impact or real power of Kafka.
I would rate this solution a seven out of ten.
View full review »I would rate this solution six out of ten.
View full review »I rate Apache Kafka a nine out of ten.
View full review »RP
Ravi-Patel
Assistant Professor at CHAROTAR UNIVERSITY OF SCIENCE AND TECHNOLOGY
There is a lot of configuration involved in this solution. We have found many configurations that have helped us but it would be beneficial if there was automation.
I rate Apache Kafka a five out of ten.
View full review »JB
reviewer2000091
Software Support & Development Engineer at a computer software company with 501-1,000 employees
We have not used the solution in production. We do not have a lot of data at the moment.
I would recommend this solution to others.
I rate Apache Kafka an eight out of ten.
View full review »I would rate this solution 8 out of 10.
View full review »DP
DustyPressley
Sr Technical Consultant at a tech services company with 1,001-5,000 employees
My advice to others wanting to implement this solution is to start with data streaming projects, not simple messaging projects because while it is very good at general-purpose messaging, it is more suited and geared for when you are using it as a streaming solution.
I rate Apache Kafka an eight out of ten.
View full review »LP
Lakshmanan Panneerselvam
Owner at Binarylogicworks.com.au
This is currently the product that I am recommending to customers. Some customers want an open-source solution.
There are some newer products that are coming on to the market that are even faster than Kafka but this solution is very resilient.
In the long run, I think that open-source will dominate the pace.
I would rate this solution a seven out of ten.
View full review »TM
reviewer1398480
Building Event-centric Data processing Architectures at a tech services company with 51-200 employees
Overall, I rate the product a six out of ten.
View full review »TM
reviewer971988
Assistant Student at a retailer with 5,001-10,000 employees
I would rate this solution 9 out of 10.
My advice is to take some time in investigating how to implement the solution.
We used to require about half a year to implement in our organization. Someone who needs to implement Kafka has to be prepared for a quite lengthy process. Don't expect implementation to be completed in a week. It's a little bit longer because it's complex.
View full review »Be sure to define the use cases as best as possible at first.
Kafka is very good, but it is complex to support. It can handle any message size, whereas native cloud options have size limitations.
Be sure to understand what messages will be sent and how many discrete topics will be needed.
Be aware that you must code both producers and consumers.
The bulk of the work is with the consumer.
The Apache stack for Kafka is very open source. There are essentially no tools other than command line options to monitor brokers and topic health. So there are 3rd party tools that will help with that, some free, some paid – but it requires that you install agents on the servers hosting Kafka and open up ports for netbeans on the scripts that start up the Kafka services. Additionally, you also have to monitor zookeeper – which is very memory intensive. Cloud offerings that provide the whole modern data architecture stack – like AWS EMR and Azure HDInsight as well as Hortonworks and Cloudera provide a console GUI as part of each of their offerings. Also Confluent, a company founded by the Linked-In engineers that designed Kafka, also have a paid enterprise offering that has much better tools for maintain the kafka cluster. But apache Kafka with the community – you are on your own.
View full review »I rate Apache Kafka eight out of 10. I would recommend it to others.
View full review »I rate this solution a nine out of ten for streaming. I recommend it to other people. The solution is good, but its performance can be improved.
View full review »DS
reviewer1052868
Principal Technology Architect at a computer software company with 5,001-10,000 employees
I rate Apache Kafka eight out of 10. There are so many products on the market, so my advice is to consider if Kafka suits your business requirements first. If it's suitable, the next step is to check whether all the technical requirements are met. If everything checks out, I would say that Kafka is a relatively stable, sound, and scalable product, so they can try it out.
View full review »MN
MoulaliNaguri
Project Engineer at Wipro Limited
I'd rate the solution eight out of ten. It's good at scaling, and, performance-wise, it's excellent. If they could add upon the UI and allow for easier configuration, I'd rate them higher.
View full review »It's a high-performance distributed system. If you want to track the user activities or any stream processing, then this is perfect. We have used Docker Kafka for our implementation. It's very easy for setup and testing. You could also try the same.
View full review »Give it a try. It’s a valuable, high-performance, distributed processing tool.
View full review »SG
SergeyGoncharov
Developer Infrastructure at Outbrain
I would definitely recommend Kafka. In our current position, we use it to move a lot of data and I think it's definitely working well. I would definitely recommend it.
I would rate it an eight out of ten.
KQ
Kevin Quon
Senior Technical Architect at a computer software company with 51-200 employees
Kafka is open source and requires an administrator to maintain the servers.
View full review »YL
reviewer1128858
Vice President at a consultancy with 51-200 employees
I would recommend trying this solution, but you should probably run it on Linux.
I like this product, I would rate Apache Kafka a nine out of ten.
View full review »I would rate it a nine out of ten. Not a ten because of the monitoring and admin improvement I'd like for them to make.
View full review »Read the documentation and understand the offset issues (where to save them, read from start to end).
View full review »Consider using a managed Kafka service, such as from Heroku.
If messaging is not a central component of the business and vendor lock-in is less of a concern, consider using something like Amazon's Kinesis. This can more rapidly provide the benefits of a messaging service without the pain of understanding it deeply, setting it up, and managing it.
It's important to use a lean approach to understand how it will break in production.
Implement a non-critical transaction with it.
Perhaps use a feature toggle within a facade and implement the behavior with the old approach and with Kafka to reduce risk.
Add it to one or two applications and monitor how it goes.
Figure out security, monitoring, scaling, schema migration, etc., before using it as a critical component in an application.
View full review »Kafka provides distributed persistence and streaming layers. The user has flexibility in managing as a consumer on how to consume messages if they have to handle resilience in their code. It requires ZooKeeper.
View full review »I rate this solution nine out of 10.
View full review »NC
Narendra Chauhan
Chief Technology Officer at a tech services company with 1-10 employees
I rate this solution an eight out of 10.
View full review »JJ
reviewer1247268
Technology Lead at a tech services company with 10,001+ employees
In this type of solution, you need to be able to accept a high volume of messages, but not lose any, and not have any duplicates. Because we are unable to control the queue in Kafka, I cannot say that this works 100%.
The suitability of this solution depends on the use cases. There are two or three things that we are worried about, and we will be very careful in choosing solutions. In cases where the messages are well organized, or there is no worry that there will be duplicate or dropped messages, then I recommend using Kafka. Also, I recommend this solution for those looking to get involved with open-source applications.
Other than the problems with having no control over the queue, Apache Kafka is wonderful.
I would rate this solution an eight out of ten.
View full review »Go ahead. It's a great product.
View full review »The product is easy to use. However, to leverage its power, there is a need for good knowledge of event based processing. I suggest using the massive amount of material shared by the Confluent team, or what is available online.
View full review »I recommend this solution, we're probably going to use it again in another project.
I rate this solution eight out of 10.
View full review »AM
AhmadMasamreh
IBMi/MIMIX Administrator at Arab Bank
Apache Kafka is a good solution with many good features but for large deployments, I would choose IBM MQ over Kafka.
I would rate this solution a seven out of ten.
View full review »My advice would be to go through the documents and understand the topics. Learn what its effects are and take care of partitioning.
Based on my experience, I would rate it an eight out of ten. It's quite complicated and the configuration requires a lot of effort. As a developer it is quite hard to go into all these things.
This is the best tool I have ever used for asynchronous, event-based solutions.
View full review »Kafka processes asynchronous exchanges, so there are no transactional interactions.
View full review »OT
OnurTokat
Senior Big Data Developer | Cloudera at Dilisim
We're using the 2.1.30 version of the solution for our cloud-based clusters. We use the on-premises deployment model. Most customers use the on-premise solution for cloud-based clusters.
Kafka is a very good solution for log management. If you need anything done related to log management, Kafka can do it. Kafka can also store the data in the brokers. This prevents data loss as well as the duplication of data. It's quite comprehensive.
I'd rate the solution seven out of ten. If the solution could provide a user interface I'd rate it higher. This is important for managing Kafka's clusters on the administration side. It would also be helpful if two to three files could be minimized to one configuration file.
View full review »If the Hadoop distribution is MapR, then consider MapR Streaming. MapR Streaming has overcome these fundamental issues. It stores data within the MapR-FS itself. So there is extra overhead, but with a licensing cost.
View full review »DR
FounderC32bc
Founder, CEO at a tech vendor with 1-10 employees
If you have a dedicated Kafka resource to implement and manage the services, then go for Apache Kafka. Otherwise, do consider cloud-based services from AWS or Azure.
View full review »MS
reviewer1304505
Senior Consultant at a tech services company with 51-200 employees
What happens in our company is a little different. We basically provide services to other companies through Kafka, like our management services. It doesn't necessarily mean we're using the solution ourselves, however, we will be going and deploying Kafka for companies, like a systems integrator.
The version of the solution is normally 2.4, however, it depends on the requirements. Our cloud providers are always different due to the fact that the countries that we work with are all different. For example, in the US it could Amazon, Azure, or Google. It varies.
I'd advise other organizations considering using the solution to make sure they understand what the use case is. They need to know what their services will be and if they will be directed to Apache Kafka.
From a customer perspective, potential companies need to make sure they have an idea of how big it's going to be due to the fact that it's a cluster environment. It needs to be taken care of. Customers will need to know things like what is the message rate is which is coming into Kafka and how they will connect all those different microservices or any services together to Kafka.
From an infrastructure perspective, it's more of how big of a cluster a company needs. Who would be the producers to produce it, and who's the consumer who's consuming the data are a few questions that need to be asked.
I'd rate the solution eight out of ten.
View full review »RH
reviewer1289778
freelance at a tech services company with 11-50 employees
I would recommend trying this solution.
Take the time to understand it because it is a different style when it comes to working with data.
I would rate this solution a nine out of ten.
View full review »RH
reviewer1289778
freelance at a tech services company with 11-50 employees
I rate Apache Kafka nine out of 10. I think it's one of the best tools on the internet.
View full review »Although we are deployed on-premises at the moment, we are looking to have a cloud-based deployment in a year or two.
This is a solution that I can recommend but it will take a lot of time to develop the adapters.
I would rate this solution a seven out of ten.
View full review »WG
reviewer1388343
Solutions Architect at a tech services company with 201-500 employees
This is a solution that I may recommend, but its suitability depends on the needs and requirements.
I would rate this solution an eight out of ten.
View full review »Buyer's Guide
Apache Kafka
March 2024
Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: March 2024.
765,234 professionals have used our research since 2012.