Apache Kafka Other Advice

AT
Group Manager at a media company with 201-500 employees

From an architecture and solution design perspective, I would say that before going for streaming solutions, we should analyze the data, which might be old, and decide if it's a streaming use case or not. Often, people think it's a streaming use case, but when they perform analytics on top of it, they realize they can't do a month-to-date or year-to-date analysis. So, it's essential to think again from the data basics perspective before going to Kafka.

Overall, from the product and solution perspective, I would rate it a nine based on my personal use of data.

View full review »
Amit Laddha - PeerSpot reviewer
Vice President (Information and Product Management) at Tradebulls Securities (P) Limited

Apache Kafka as a broker tool is a very stable and good product. When you need to create a consumer in any programming language, including Java, Golang or any other programming language, the team involved in the process of the creation of a consumer should have very strong knowledge and expertise in the use of Apache Kafka since it is not at all easy to create a consumer for the product. A highly qualified person with a good amount of experience should also know the internals of the solution, which may not seem too straightforward. Anyone cannot use Apache Kafka easily without proper knowledge or experience. When you use Apache Kafka in your actual application, you need to create some producers and some consumers. To create a consumer, you need to have a very strong understanding of the solution since it is not a process that anyone can manage easily. A company needs to have a very strong team with good technical knowledge to be able to use the product.

I rate the overall solution a nine out of ten.

View full review »
AbhishekGupta - PeerSpot reviewer
Engineering Leader at Walmart

The number of people required for maintenance depends on the team. They need a centralized team to offer Apache Kafka and services. Each team does have knowledge of Kafka.

This solution has a lot of features and there is no other solution on the market that has similar advanced features. It is a very good solution.

I rate Apache Kafka an eight out of ten.

View full review »
Buyer's Guide
Apache Kafka
March 2024
Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: March 2024.
765,234 professionals have used our research since 2012.
Harsha Ravnikar - PeerSpot reviewer
Senior Solutions Architect at Sysmex America, Inc.

Apache Kafka is a good choice, so I would recommend people not have a real-time application if they do not have to. It is better to have a very fast batch operation than a real-time operation. I would rate Apache Kafka a nine on a scale of one to ten.

View full review »
Reza Sadeghi - PeerSpot reviewer
Software Development Team Lead at asa com

I rate Apache Kafka seven out of 10. It's a good solution. They're constantly fixing bugs and adding new features. 

View full review »
Jhon Rico - PeerSpot reviewer
Senior Solutions Architect at BVC

I believe that when working with Kafka Apache, it's essential to have a specialist who thoroughly understands and can optimize all the available variables within the solution to achieve the desired behavior.

I would rate it an eight out of ten.

View full review »
Pratul Shukla - PeerSpot reviewer
Software Engineer at a financial services firm with 10,001+ employees

I give the solution an eight out of ten.

Maintaining Kafka, the open source, can be difficult without the proper version purchased or the right infrastructure in place. However, once the initial setup is complete, it is relatively simple to maintain. The open-source version of Kafka is not a complete package, so additional maintenance may be required.

I strongly recommend reading the documentation for any issues because it is likely to contain the answer we are looking for. There is a lot of information provided that may not be immediately obvious, so take the time to explore thoroughly.

View full review »
MB
Architect at Agence Française de Développement

To be able to recommend Kafka to others, especially considering every context, we will have to set a benchmark and compare Kafka with other tools.

I rate the overall solution a seven out of ten.


View full review »
Nor EL MALKI - PeerSpot reviewer
Project Manager at Leyton & Associés, SAS

We had a good experience with the solutions, the maintainability and scalability are good. I would recommend the solution to others.

I rate Apache Kafka a nine out of ten.

View full review »
SP
CEO at a comms service provider with 11-50 employees

Kafka is a really good product. To be able to keep it running in the long term, you need to know very well how it works. You should have good knowledge about it. It isn't about just knowing how to install it because it is quite simple to install it. It is important to have the right knowledge and experience to do a good installation and let it run for a long period. You can also go for someone who has the right experience and knowledge.

We are very satisfied with Kafka. I would rate it an eight out of 10. It is not perfect, but it is a really good product.

View full review »
Bharath-Reddy - PeerSpot reviewer
Architect at Tekgeminus

A non-enterprise business with a low message load can use an open-source solution like Apache Kafka.

I would recommend the solution to enterprise businesses depending on their use cases. Suppose an enterprise business doesn't have any integration or a middleware platform and wants to do a greenfield implementation. I'll evaluate the use cases and refer Apache Kafka to them if messaging is needed only for exception handling or transferring the messages.

I have recommended Apache Kafka to some customers who wanted asynchronous messaging for logging purposes. Those messages were not business-critical messages as such.

I would recommend Apache Kafka to other users. Apache Kafka is more relevant when we use open-source integrations and when customers want to reduce the TCO. As an architect, I recommend the solution to customers based on their messaging needs. Apache Kafka and Anypoint MQ are the only two messaging products available today. The open-source Apache Kafka is always recommended if the customer really doesn't want to get into any of the license models.

Overall, I rate Apache Kafka an eight out of ten.

View full review »
GT
Lead Architect at a financial services firm with 1,001-5,000 employees

When it comes to Apache Kafka, they must understand how it works and what its internals are. There could be numerous challenges associated with the product and its entire life cycle. You will have to have a good understanding and knowledge of the configuration. You will need a technical person who is knowledgeable in Kafka which will be an advantage and on an ongoing life partner.

It's a very good solution, I would rate Apache Kafka a nine out of ten.

View full review »
Mukulit Bhati - PeerSpot reviewer
CTO at InsightGeeks Solutions Pvt.

I rate this solution an eight out of ten. It is good, but the documentation could be improved.

View full review »
Stuart-Cook - PeerSpot reviewer
CEO & Founder at a tech consulting company with 11-50 employees

The documentation can be a challenge. There are quite advanced capabilities of Kafka, like the transformations that you can build to modify the data as needed. We found that the biggest challenge was documentation and being able to gain the knowledge of exactly how to do stuff. We also struggled on the transformation, but other components were fine, so some parts are good, and some parts are bad.

I would rate this solution as an eight out of ten. 

View full review »
Rémy NOLLET - PeerSpot reviewer
Data Exchange Architect MQSeries at Decathlon International

I would recommend that other businesses do the deployment themselves, but manage the tool with the aid of a service provider, rather than in-house.

I would rate this product seven out of ten.

View full review »
Silvio Lucas Pereira Filho - PeerSpot reviewer
Senior Tech Lead at RecargaPay

I don't see any major issues with using Apache Kafka. Many companies use it and it's a good solution. My advice would be to use it as a software-as-a-service rather than setting up your own cluster. This way, you can benefit from a preconfigured and maintained platform. It's better to opt for a software-as-a-service solution.

I rate Apache Kafka an eight out of ten.

View full review »
AX
Senior Developer at a financial services firm with 10,001+ employees

I would give Kafka a rating of seven out of ten.

View full review »
Joaquin Marques - PeerSpot reviewer
CEO - Founder / Principal Data Scientist / Principal AI Architect at Kanayma LLC

Since it has become so popular, large enterprises especially want to do it. For smaller enterprises, Kafka would probably be too expensive because they would have to hire people to maintain it.

I would rate the Apache Kafka solution a seven out of ten.

View full review »
TD
Head of Technology - Money Movement Platform at a financial services firm with 10,001+ employees

I rate Apache Kafka a nine out of ten.

View full review »
AV
Technical Director at Metrofibre Networx

Apache Kafka is an out-of-the-box, reliable solution. For people in the fiber business, we need a reliable solution, and this solution is hundred percent reliable. If it is set up correctly, it hardly has any issues due to the more extensive user base; even if there are issues, it is sorted by the community. I rate it nine out of ten.

View full review »
RG
Senior Technology Architect at a tech services company with 10,001+ employees

On a scale from one to ten, I would give Apache Kafka a rating of eight.

View full review »
it_user590451 - PeerSpot reviewer
Lead Engineer at a retailer with 10,001+ employees

I would advise others to start with non-SSL implementations and try to do PoCs. Afterwards, they should move towards more secure features.

View full review »
Felipe Lopes - PeerSpot reviewer
Engineering Manager at Alice

I give the solution a nine out of ten.

We have 80 people using the solution and five people are required to maintain it.

I suggest using cloud services because the solution is expensive if you are using it on-premises.

View full review »
NK
Director at Tibco

I give the solution an eight out of ten.

We test all the supported versions of the solution based on our customers' use.

We support our integration product. So we need to do dev and QA with Apache Kafka or any other messaging applications. But we do not provide support. The solution can be supported by someone else.

We don't need to have any specific staff for deployment. All the developers in QA can install and configure the solution. We don't have a separate person for maintenance.

Our team and our product dev and QAs all use the solution.

I think Apache Kafka is a good solution and I recommend it to others.

View full review »
JA
Technical Lead at Interface Fintech Ltd

We do not use customer support, but there is a lot of documentation available.

I would definitely recommend this solution to other people. I would rate it as an eight out of ten. 

View full review »
SK
Barista Brewing Espresso at Linkedln

There is room for improvement with this solution so I rate it eight out of 10. 

View full review »
Salvatore Campana - PeerSpot reviewer
CEO & Founder at XAUTOMATA TECHNOLOGY GmbH

I would rate this solution 7 out of 10.

I would recommend this solution because the queue manager is very fast and stable.

View full review »
ME
CTO at Estrada & Consultores

New users should understand the product capabilities. Often, people will start putting their hands in new products without knowing the capabilities and the disadvantages in specific scenarios. In our case for example, We haven't used Kafka for financial transaction processing, for which we still use IBM MQ, but It really depends upon your knowledge and experience with the product. My advice is to understand the product very well, its pros and cons and work from there.

Finally I'd rate the solution at a nine out of ten.

View full review »
DZ
Enterprise Architect at Smals vzw

The maintenance of Apache Kafka is crucial due to the complexity of the system with numerous microservices and systems communicating through Apache Kafka, requiring proper integration and configuration to prevent overloading and ensure a healthy cluster. The task is not easy and requires knowledge of the various adjustable parameters, as misadjusting even one of them can greatly slow down the cluster. For example, if the consumer group changes frequently, the messages must be regrouped and reassigned, causing significant delays. Therefore, configuring Apache Kafka correctly is essential to avoid high latency issues.

I would strongly suggest others give Apache Kafka a chance and explore the various advantages that it can offer, especially since it should not be perceived as a message bus or broker but rather an enterprise bus designed for data manipulation. It has the ability to transform data, store and reject it, and even maintain different versions of the same data simultaneously. Moreover, it operates on a pull mechanism rather than a push mechanism, which takes away the risk of losing data and places the responsibility for data loss on the consumer. On the other hand, it also ensures that the data is always available within the specified window and allows for easy replication of the past, which is extremely helpful in situations such as those involving a hacked bank database. With Apache Kafka, you can efficiently go back in time, obtain the required status and events, and make changes accordingly, without the need to go through each transaction separately. Thus, using this solution can make data management much more efficient and convenient.

I rate Apache Kafka an eight out of ten.

In order to improve its user-friendliness, engineer-friendliness, and DevOps-friendliness, the system must undertake various tasks, such as enhancing the overall operation and configuration, ensuring seamless integration with other systems, and adapting to security layers in a more comprehensive and generic manner. This will require significant efforts to make the system more functional, secure, and efficient.

View full review »
Ravi Kuppusamy - PeerSpot reviewer
CEO and Founder at BAssure Solutions

Apache Kafka is one of the best open-source solutions that are available today.

I would recommend this solution to others.

I rate Apache Kafka an eight out of ten.

View full review »
KS
Solution Architect at a manufacturing company with 10,001+ employees

I think that many people are using Apache Kafka just as a publishing and subscription model, but I feel that Kafka is better than that. Furthermore, Confluent Kafka is even more than that.

Confluent Kafka is offering features that are equal to those of a data lake. You can do lots with data, and huge data can be persisted. However, many people are not using that feature. Rather than make use of persistence logic, they are pushing the messages and consuming them. Maybe if people were using it for persistence, they would see the impact or real power of Kafka.

I would rate this solution a seven out of ten.

View full review »
Moussa Chikhi - PeerSpot reviewer
Architecte Technique Senior at a computer software company with 10,001+ employees

I would rate this solution six out of ten.

View full review »
Paul Adams - PeerSpot reviewer
Consultant Solution Architect at a tech services company with 51-200 employees

I rate Apache Kafka a nine out of ten.

View full review »
RP
Assistant Professor at CHAROTAR UNIVERSITY OF SCIENCE AND TECHNOLOGY

There is a lot of configuration involved in this solution. We have found many configurations that have helped us but it would be beneficial if there was automation. 

I rate Apache Kafka a five out of ten.

View full review »
JB
Software Support & Development Engineer at a computer software company with 501-1,000 employees

We have not used the solution in production. We do not have a lot of data at the moment.

I would recommend this solution to others.

I rate Apache Kafka an eight out of ten.

View full review »
ShoaibKhan - PeerSpot reviewer
Technical Specialist at APIZone

I would rate this solution 8 out of 10.

View full review »
DP
Sr Technical Consultant at a tech services company with 1,001-5,000 employees

My advice to others wanting to implement this solution is to start with data streaming projects, not simple messaging projects because while it is very good at general-purpose messaging, it is more suited and geared for when you are using it as a streaming solution.

I rate Apache Kafka an eight out of ten.

View full review »
LP
Owner at Binarylogicworks.com.au

This is currently the product that I am recommending to customers. Some customers want an open-source solution.

There are some newer products that are coming on to the market that are even faster than Kafka but this solution is very resilient.

In the long run, I think that open-source will dominate the pace.

I would rate this solution a seven out of ten.

View full review »
TM
Building Event-centric Data processing Architectures at a tech services company with 51-200 employees

Overall, I rate the product a six out of ten.

View full review »
TM
Assistant Student at a retailer with 5,001-10,000 employees

I would rate this solution 9 out of 10.

My advice is to take some time in investigating how to implement the solution.

We used to require about half a year to implement in our organization. Someone who needs to implement Kafka has to be prepared for a quite lengthy process. Don't expect implementation to be completed in a week. It's a little bit longer because it's complex.

View full review »
it_user653562 - PeerSpot reviewer
Solutions Architect at a consultancy with 1,001-5,000 employees

Be sure to define the use cases as best as possible at first.

Kafka is very good, but it is complex to support. It can handle any message size, whereas native cloud options have size limitations.

Be sure to understand what messages will be sent and how many discrete topics will be needed.

Be aware that you must code both producers and consumers.

The bulk of the work is with the consumer.

The Apache stack for Kafka is very open source. There are essentially no tools other than command line options to monitor brokers and topic health. So there are 3rd party tools that will help with that, some free, some paid – but it requires that you install agents on the servers hosting Kafka and open up ports for netbeans on the scripts that start up the Kafka services. Additionally, you also have to monitor zookeeper – which is very memory intensive. Cloud offerings that provide the whole modern data architecture stack – like AWS EMR and Azure HDInsight as well as Hortonworks and Cloudera provide a console GUI as part of each of their offerings. Also Confluent, a company founded by the Linked-In engineers that designed Kafka, also have a paid enterprise offering that has much better tools for maintain the kafka cluster. But apache Kafka with the community – you are on your own.

View full review »
Abdul-Samad - PeerSpot reviewer
Software Engineer at a tech services company with 201-500 employees

I rate Apache Kafka eight out of 10. I would recommend it to others. 

View full review »
Guirino Ciliberti - PeerSpot reviewer
Data Governance & Lineage Product Manager at Primeur

I rate this solution a nine out of ten for streaming. I recommend it to other people. The solution is good, but its performance can be improved.

View full review »
DS
Principal Technology Architect at a computer software company with 5,001-10,000 employees

I rate Apache Kafka eight out of 10. There are so many products on the market, so my advice is to consider if Kafka suits your business requirements first. If it's suitable, the next step is to check whether all the technical requirements are met. If everything checks out, I would say that Kafka is a relatively stable, sound, and scalable product, so they can try it out. 

View full review »
MN
Project Engineer at Wipro Limited

 I'd rate the solution eight out of ten. It's good at scaling, and, performance-wise, it's excellent. If they could add upon the UI and allow for easier configuration, I'd rate them higher.

View full review »
it_user660591 - PeerSpot reviewer
Senior Java Consultant at a tech services company with 501-1,000 employees

It's a high-performance distributed system. If you want to track the user activities or any stream processing, then this is perfect. We have used Docker Kafka for our implementation. It's very easy for setup and testing. You could also try the same.

View full review »
it_user578787 - PeerSpot reviewer
Java Developer at a media company with 10,001+ employees

Give it a try. It’s a valuable, high-performance, distributed processing tool.

View full review »
SG
Developer Infrastructure at Outbrain

I would definitely recommend Kafka. In our current position, we use it to move a lot of data and I think it's definitely working well. I would definitely recommend it.

I would rate it an eight out of ten. 

View full review »
KQ
Senior Technical Architect at a computer software company with 51-200 employees

Kafka is open source and requires an administrator to maintain the servers.

View full review »
YL
Vice President at a consultancy with 51-200 employees

I would recommend trying this solution, but you should probably run it on Linux.

I like this product, I would rate Apache Kafka a nine out of ten.

View full review »
RABBAHMahmoud - PeerSpot reviewer
Senior Technical Architect at RABBAH SOFT

I would rate it a nine out of ten. Not a ten because of the monitoring and admin improvement I'd like for them to make. 

View full review »
it_user642168 - PeerSpot reviewer
Big Data Lead at a marketing services firm with 51-200 employees

Read the documentation and understand the offset issues (where to save them, read from start to end).

View full review »
it_user660627 - PeerSpot reviewer
Senior Software Engineering Consultant at a tech services company with 51-200 employees

Consider using a managed Kafka service, such as from Heroku.

If messaging is not a central component of the business and vendor lock-in is less of a concern, consider using something like Amazon's Kinesis. This can more rapidly provide the benefits of a messaging service without the pain of understanding it deeply, setting it up, and managing it.

It's important to use a lean approach to understand how it will break in production.

Implement a non-critical transaction with it.

Perhaps use a feature toggle within a facade and implement the behavior with the old approach and with Kafka to reduce risk.

Add it to one or two applications and monitor how it goes.

Figure out security, monitoring, scaling, schema migration, etc., before using it as a critical component in an application.

View full review »
it_user650223 - PeerSpot reviewer
Principal Software Architect at a tech services company with 11-50 employees

Kafka provides distributed persistence and streaming layers. The user has flexibility in managing as a consumer on how to consume messages if they have to handle resilience in their code. It requires ZooKeeper.

View full review »
Sreekar Nethagani - PeerSpot reviewer
Co-Founder at Attaika

I rate this solution nine out of 10. 

View full review »
NC
Chief Technology Officer at a tech services company with 1-10 employees

I rate this solution an eight out of 10. 

View full review »
JJ
Technology Lead at a tech services company with 10,001+ employees

In this type of solution, you need to be able to accept a high volume of messages, but not lose any, and not have any duplicates. Because we are unable to control the queue in Kafka, I cannot say that this works 100%.

The suitability of this solution depends on the use cases. There are two or three things that we are worried about, and we will be very careful in choosing solutions. In cases where the messages are well organized, or there is no worry that there will be duplicate or dropped messages, then I recommend using Kafka. Also, I recommend this solution for those looking to get involved with open-source applications.

Other than the problems with having no control over the queue, Apache Kafka is wonderful.

I would rate this solution an eight out of ten.

View full review »
it_user650004 - PeerSpot reviewer
Team Lead at a financial services firm with 1,001-5,000 employees

Go ahead. It's a great product.

View full review »
it_user647457 - PeerSpot reviewer
Head of Engineering

The product is easy to use. However, to leverage its power, there is a need for good knowledge of event based processing. I suggest using the massive amount of material shared by the Confluent team, or what is available online.

View full review »
Andrea Castorino - PeerSpot reviewer
Program Manager at SirfinPA

I recommend this solution, we're probably going to use it again in another project.

I rate this solution eight out of 10. 

View full review »
AM
IBMi/MIMIX Administrator at Arab Bank

Apache Kafka is a good solution with many good features but for large deployments, I would choose IBM MQ over Kafka.

I would rate this solution a seven out of ten.

View full review »
Mukulit Bhati - PeerSpot reviewer
CTO at InsightGeeks Solutions Pvt.

My advice would be to go through the documents and understand the topics. Learn what its effects are and take care of partitioning. 

Based on my experience, I would rate it an eight out of ten. It's quite complicated and the configuration requires a lot of effort. As a developer it is quite hard to go into all these things.

View full review »
it_user592356 - PeerSpot reviewer
Technical Lead/Project Manager(Consulting Apple Inc) at a tech services company with 1,001-5,000 employees

This is the best tool I have ever used for asynchronous, event-based solutions.

View full review »
it_user592338 - PeerSpot reviewer
Enterprise Architect at a logistics company with 1,001-5,000 employees

Kafka processes asynchronous exchanges, so there are no transactional interactions.

View full review »
OT
Senior Big Data Developer | Cloudera at Dilisim

We're using the 2.1.30 version of the solution for our cloud-based clusters. We use the on-premises deployment model. Most customers use the on-premise solution for cloud-based clusters.

Kafka is a very good solution for log management. If you need anything done related to log management, Kafka can do it. Kafka can also store the data in the brokers. This prevents data loss as well as the duplication of data. It's quite comprehensive.

I'd rate the solution seven out of ten. If the solution could provide a user interface I'd rate it higher. This is important for managing Kafka's clusters on the administration side. It would also be helpful if two to three files could be minimized to one configuration file.

View full review »
it_user642942 - PeerSpot reviewer
Hadoop Technical Lead (Assistant Consultant) at a tech services company with 10,001+ employees

If the Hadoop distribution is MapR, then consider MapR Streaming. MapR Streaming has overcome these fundamental issues. It stores data within the MapR-FS itself. So there is extra overhead, but with a licensing cost.

View full review »
DR
Founder, CEO at a tech vendor with 1-10 employees

If you have a dedicated Kafka resource to implement and manage the services, then go for Apache Kafka. Otherwise, do consider cloud-based services from AWS or Azure.

View full review »
MS
Senior Consultant at a tech services company with 51-200 employees

What happens in our company is a little different. We basically provide services to other companies through Kafka, like our management services. It doesn't necessarily mean we're using the solution ourselves, however, we will be going and deploying Kafka for companies, like a systems integrator.

The version of the solution is normally 2.4, however, it depends on the requirements. Our cloud providers are always different due to the fact that the countries that we work with are all different. For example, in the US it could Amazon, Azure, or Google. It varies.

I'd advise other organizations considering using the solution to make sure they understand what the use case is. They need to know what their services will be and if they will be directed to Apache Kafka.

From a customer perspective, potential companies need to make sure they have an idea of how big it's going to be due to the fact that it's a cluster environment. It needs to be taken care of. Customers will need to know things like what is the message rate is which is coming into Kafka and how they will connect all those different microservices or any services together to Kafka.

From an infrastructure perspective, it's more of how big of a cluster a company needs. Who would be the producers to produce it, and who's the consumer who's consuming the data are a few questions that need to be asked.

I'd rate the solution eight out of ten.

View full review »
RH
freelance at a tech services company with 11-50 employees

I would recommend trying this solution. 

Take the time to understand it because it is a different style when it comes to working with data.

I would rate this solution a nine out of ten.

View full review »
RH
freelance at a tech services company with 11-50 employees

I rate Apache Kafka nine out of 10. I think it's one of the best tools on the internet.

View full review »
it_user998961 - PeerSpot reviewer
Enterprice Architect

Although we are deployed on-premises at the moment, we are looking to have a cloud-based deployment in a year or two.

This is a solution that I can recommend but it will take a lot of time to develop the adapters.

I would rate this solution a seven out of ten.

View full review »
WG
Solutions Architect at a tech services company with 201-500 employees

This is a solution that I may recommend, but its suitability depends on the needs and requirements.

I would rate this solution an eight out of ten.

View full review »
Buyer's Guide
Apache Kafka
March 2024
Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: March 2024.
765,234 professionals have used our research since 2012.