Apache Kafka Initial Setup

AT
Group Manager at a media company with 201-500 employees

The initial two months were for capacity estimation, where we worked with the client's different business teams to understand the data volumes and use cases. Then, the next four to five months went into procurement, where we had to work with infrastructure teams and vendors to understand the servers and networks required for the cluster. 

The actual cluster setup took us two months, and it was a little longer due to a shortage of expertise on the client's networking team. We had to handle everything ourselves since it was an on-premise setup with physical servers and network connections. Currently, we are in the security review phase, and once it completes, we will start implementing various use cases like task and batch processing, archival, etc.

If you see my experience from the Apache Kafka implementation and clusters as a perspective, I will rate the setup somewhere between seven to eight out of ten.

View full review »
Amit Laddha - PeerSpot reviewer
Vice President (Information and Product Management) at Tradebulls Securities (P) Limited

If you are using the latest version of Apache Kafka in which the use of Zookeeper is not required, then it uses the KRaft protocol, which is built into Kafka broker. Since the use of Zookeeper is no longer required, I think the setup phase of dissolution is better than its previous versions. I rate the initial setup of the product somewhere between seven to eight out of ten.

Apache Kafka's initial setup is very straightforward.

The solution is deployed on an on-premises model.

Apache Kafka was deployed in our company within a couple of days.

Three people were involved in the deployment process of Apache Kafka.

View full review »
AbhishekGupta - PeerSpot reviewer
Engineering Leader at Walmart

The initial setup of Apache Kafka was complex. We were able to simplify it by doing registry-based integration of the services.

View full review »
Buyer's Guide
Apache Kafka
April 2024
Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
768,415 professionals have used our research since 2012.
Reza Sadeghi - PeerSpot reviewer
Software Development Team Lead at asa com

I rate Kafka 10 out of 10 for ease of setup. It's easy for us because we use Docker, but if you want to use another system like Linux it may be a little challenging

View full review »
Pratul Shukla - PeerSpot reviewer
Software Engineer at a financial services firm with 10,001+ employees

The initial setup was relatively easy for us since we already had Zookeeper and the necessary setup in place. We also had good knowledge of Kafka. Therefore, it was not a difficult challenge. In general, I believe that it is manageable. There are benefits and the setup is not overly complex.

Our company has implemented Ship, making our lives easier when it comes to changes or version updates. We can package everything in one place and deploy it with Ship, then implement the virtual number with a minimum of 50 changes.

Deployment time depends on our location and the task at hand. Initially, there is a lot of setup and configuration that must be done, but this can become easier with experience. Nowadays, the process is not too difficult, as all the version numbers and conflict files are already in place. However, if this is a new task for us, it may take some time to figure out all the configurations.

One person was dedicated to deploying Kafka. This person got help from our release team, who had already set up Zookeeper and other necessary components.

View full review »
MB
Architect at Agence Française de Développement

The solution's initial setup process was complex. The deployment process took three or four years.

Right now, I can't deliver the planning process required for deployment.

For deployment and maintenance, we have a manager and an operational person. However, I can't give an exact count of the people required for deployment and maintenance.

View full review »
Nor EL MALKI - PeerSpot reviewer
Project Manager at Leyton & Associés, SAS

The initial setup of Apache Kafka was easy because it is Dockerized. However, if you were to install it yourself it would be difficult. Having it Dockerized makes it worth it. 

The first deployment took approximately two hours. The updates of the solution can be done in a matter of minutes.

View full review »
SP
CEO at a comms service provider with 11-50 employees

I am into pre-sales and project management. I don't usually install Apache Kafka, but its basic installation seems quite simple.

Its deployment is usually quite short. Usually, we are able to deploy it in a few days, but data management and application development can take a few months.

View full review »
Bharath-Reddy - PeerSpot reviewer
Architect at Tekgeminus

The solution's initial setup is quite straightforward. You just have to upgrade a couple of configuration files.

View full review »
GT
Lead Architect at a financial services firm with 1,001-5,000 employees

It is not deployed on-premises. 

We use Kafka as part of the OCI Oracle Cloud platform and the Oracle Big Data platform because Kafka is included.

The Apache Kafka setup will take some time because it is not simple, and we have a lot of other components to install. It's fine because we needed all the plugins and other things for the simple implementations, but the containers' implementation is simple. The only difference is that when it comes to Zookeeper, there are a lot of supporting applications running on top of it, such as Zookeeper. As part of their area, Apache Kafka is running on top of Zookeeper. What do they think? As part of their... manageability, the Kafka area, and Apache Zookeeper. As a result, everything must be removed. And it will be preferable if the implementation is simple.  I believe Confluent is doing this, but we have not yet begun.

The deployment, and configuration, will take one hour to complete. However, it is also dependent on the fact that you require a large number of configurations, which we have.

View full review »
Mukulit Bhati - PeerSpot reviewer
CTO at InsightGeeks Solutions Pvt.

The initial setup is dependent on the individual. For example, it would be straightforward if a person practices these things a lot and understands the documentation correctly. However, since most people prefer examples instead of reviewing documentation, it would be easy to set up if they find steps on the internet but difficult if they do not have examples.

View full review »
Rémy NOLLET - PeerSpot reviewer
Data Exchange Architect MQSeries at Decathlon International

The initial setup was complex for us because we built it internally. This meant that full deployment took around a month.

View full review »
Silvio Lucas Pereira Filho - PeerSpot reviewer
Senior Tech Lead at RecargaPay

The initial setup is relatively easy as I am using Docker and the files provided by Confluent. However, setting up Apache Kafka in a production environment is not as straightforward. I prefer to use solutions, such as Confluence that already have everything preconfigured. As a developer, creating an environment for it is not a problem for me, but I think it can be challenging for those responsible for the production environment. There have been issues with data loss and other problems in the past. Configuring it for production is not easy.

My deployment was very quick because I am using it locally. We have someone else that does the cloud deployment.

View full review »
Joaquin Marques - PeerSpot reviewer
CEO - Founder / Principal Data Scientist / Principal AI Architect at Kanayma LLC

The initial setup was extremely complex. In our case, it took a team of 12 two months to deploy.

View full review »
TD
Head of Technology - Money Movement Platform at a financial services firm with 10,001+ employees

Apache Kafka is straightforward to implement.

View full review »
AV
Technical Director at Metrofibre Networx

The initial setup was straightforward because I've got a lot of experience in this field. But even for a junior person, it would be fine. There are so many resources, and it's very well documented as they are a premium service provider. So it makes the setup just easier.

The deployment takes a few days.

We set up a free cluster for this service because we use a lot of data. We use ZooKeeper to secure different products for instruction with the cluster. But, it was easy as it is a popular product, and much information is available. It can download data, like fifty gigs per day. We can effectively handle it all as well. I never developed any issues.

View full review »
it_user590451 - PeerSpot reviewer
Lead Engineer at a retailer with 10,001+ employees

The initial setup was a bit complex in terms of how to manage it across data centers. But once it was setup, we never faced issues.

View full review »
Felipe Lopes - PeerSpot reviewer
Engineering Manager at Alice

The initial setup was straightforward.

View full review »
NK
Director at Tibco

The open-source version is relatively straightforward to set up and only takes a few minutes.

View full review »
JA
Technical Lead at Interface Fintech Ltd

There was a little learning curve, but we managed it. I think it took us around six weeks to complete the deployment. 

View full review »
SK
Barista Brewing Espresso at Linkedln

The initial setup was simple although we had a couple of hiccups. It took around a week but that was several years ago and we haven't had any problems since. Our team carried out the deployment and we currently have a few engineers who deal with maintenance. 

View full review »
Salvatore Campana - PeerSpot reviewer
CEO & Founder at XAUTOMATA TECHNOLOGY GmbH

It's very complex, like Spark. 

Deployment took 50 minutes for all the Kubernetes ports, Spark, Kafka, and other components based on Sherlogic. In 30 minutes, we created an environment using this program to make installation easier.

View full review »
ME
CTO at Estrada & Consultores

The initial setup is easy. The majority of the tools today are really very easy to configure and setup. Docker Containers and Kubernetes, actually, have made life easier for architects as well as developers.

Nowadays, you just install the container, and then you don't have to really manage the internals at libraries, OS levels, et cetera. You just run the container. Everything is containerized.

View full review »
DZ
Enterprise Architect at Smals vzw

Setting up Apache Kafka is, is not an easy task, especially when trying to containerize it and make it controllable. This is because Apache Kafka has its own distributed mechanism for staying alive, checking readiness, replicating, and scaling. Ensuring that it complies with Kubernetes or OpenShift Orchestrator requires careful attention, as there is a risk of two masters attempting to perform the same task and ultimately undoing each other's work.

In comparison to Kubernetes, OpenShift is a highly skilled and advanced implementation infrastructure that automatically manages and orchestrates all the steps required for an application setup. It operates at a higher level of abstraction and eliminates the need for manual operations that are required with Kubernetes. While Kubernetes can run an application with some pipeline and configuration, OpenShift takes care of everything from finding the required images to creating ports and connecting databases. Although manual changes can be made, it's not necessary as OpenShift offers a much more course-grained management approach.

View full review »
Ravi Kuppusamy - PeerSpot reviewer
CEO and Founder at BAssure Solutions

Apache Kafka was simple to install. If you have a complicated clustered production, it takes time. However, for the development, it doesn't take more than one or two hours.

View full review »
KS
Solution Architect at a manufacturing company with 10,001+ employees

I don't think that the initial setup was a complex process.

View full review »
Moussa Chikhi - PeerSpot reviewer
Architecte Technique Senior at a computer software company with 10,001+ employees

The initial setup was simple.

View full review »
Paul Adams - PeerSpot reviewer
Consultant Solution Architect at a tech services company with 51-200 employees

The initial setup of Apache Kafka is easy to set up a cluster.  I did the initial setup on my laptop and it is straightforward. I used the Confluent version, but even if you want to run native capabilities it's straightforward to do the implementation.

View full review »
RP
Assistant Professor at CHAROTAR UNIVERSITY OF SCIENCE AND TECHNOLOGY

The installation is fairly easy. Additionally, there is a cloud-based version available if a use case requires it.

View full review »
JB
Software Support & Development Engineer at a computer software company with 501-1,000 employees

The initial setup of Apache Kafka took some time but after it was easy.

I rate the initial setup of Apache Kafka a three out of five.

View full review »
DP
Sr Technical Consultant at a tech services company with 1,001-5,000 employees

Apache Kafka on AWS is a bit complex. There is a third-party company called Confluent and they have the support that makes their installation much easier, especially for the on-premise deployment. You install Apache Kafka alone it can be a little complex compared to other queuing messaging solutions.

The on-premise deployment takes approximately a few days. The cloud or hybrid deployments including all the permissions, typologies, firewalls, and networking configuration can take weeks for all the accessibility issues to be resolved. However, the delay could have been client-related and not necessarily the solution.

View full review »
LP
Owner at Binarylogicworks.com.au

The initial setup was straightforward for me. However, it is not straightforward for everyone because there are some tricky things to implement. In single-mode it is a little bit easier, but when it is set up as a distributed system then it is more complex because there are a lot of things to be considered.

View full review »
TM
Building Event-centric Data processing Architectures at a tech services company with 51-200 employees

It is super easy to set up. I rate the ease of setup a ten out of ten. However, building and administration get quite difficult. It takes three months to make things production-ready.

View full review »
TM
Assistant Student at a retailer with 5,001-10,000 employees

Implementations took half a year for everyone to learn the solution. It was quite lengthy.

View full review »
it_user653562 - PeerSpot reviewer
Solutions Architect at a consultancy with 1,001-5,000 employees

Since it is open source, you are on your own for setup. However, the tutorials from the Apache foundation and online sources have been an immense help.

Getting started is very easy. The complexity of very large volumes of data and appropriate sharding, however, is difficult. There are fewer resources for tuning and best practices.

View full review »
Abdul-Samad - PeerSpot reviewer
Software Engineer at a tech services company with 201-500 employees

I initially found Kafka difficult to set up, so I would rate it about five out of 10 for ease of setup. After I learned more about the platform, I would rate it eight out of 10. It is deployed on-premises over a cluster of three or four PCs. You can deploy Kafka in a few hours with one person. 

View full review »
DS
Principal Technology Architect at a computer software company with 5,001-10,000 employees

We were able to deploy and use it with no problems for our use case. We didn't find it so complex. We work with so many applications, databases, Postgres, and so many other things, so we could manage it easily. We deployed Kafka in a few hours. We have an infrastructure team and DevOps. Those teams are pretty capable, and they've completely automated the whole deployment. It always takes time the first time you upgrade any application, not just Kafka. We might discover some issues, such as configuration, parameters, compatibility, etc. Once that becomes standard, it is stable, and then they only need to replicate it to the different environments or different developers groups. We have a sophisticated process.

View full review »
MN
Project Engineer at Wipro Limited

We're continuously deploying the product. We're still in the process of deployment.

View full review »
it_user660591 - PeerSpot reviewer
Senior Java Consultant at a tech services company with 501-1,000 employees

The initial setup was straightforward. We faced some issues during the development in areas such as message producer and consumer. We rectified those with the tweaking the producer and consumer configurations. The documentation is very good.

View full review »
it_user578787 - PeerSpot reviewer
Java Developer at a media company with 10,001+ employees

Initial setup wasn’t really complex. We use Kafka through Hortonworks Suite, which comes with many other big data tools. Ambari makes it easy to setup

View full review »
KQ
Senior Technical Architect at a computer software company with 51-200 employees

Initial setup was straightforward. We simply hosted multiple Kafka brokers and ZooKeeper servers on AWS EC2 instances.

View full review »
ShoaibKhan - PeerSpot reviewer
Technical Specialist at APIZone

Initial setup was not easy. It requires intermediate skills.

View full review »
YL
Vice President at a consultancy with 51-200 employees

The initial setup is straightforward. It's easy to set up.

It took a few days to get it up and running.

We only need one or two engineers to keep this solution running. We basically let it run and monitor what's going on. We usually don't touch it unless something goes wrong.

View full review »
RABBAHMahmoud - PeerSpot reviewer
Senior Technical Architect at RABBAH SOFT

The setup is not a big deal for us. We can handle it. After the system is set up, the person who administers it has to do so with Apache Kafka.

Depending on the setup, it will usually take two weeks.

View full review »
it_user660627 - PeerSpot reviewer
Senior Software Engineering Consultant at a tech services company with 51-200 employees

The setup was complex. One must consider setting up ZooKeeper, Kafka, multi-zone/region availability, as well as typical associated functions for running it all in production. This includes monitoring, message schema changes (consider Avro), encrypting messages if it's a concern, potentially authorization for different topics depending up on the sensitivity of data.

If an organization uses Kafka as the first messaging system, then the approach for application design must also shift significantly.

View full review »
it_user650223 - PeerSpot reviewer
Principal Software Architect at a tech services company with 11-50 employees

Setting up was easy with ZooKeeper.

View full review »
Sreekar Nethagani - PeerSpot reviewer
Co-Founder at Attaika

The initial setup is straightforward, it's not complicated.

View full review »
JJ
Technology Lead at a tech services company with 10,001+ employees

We had problems when we were setting up Kafka ourselves to conduct our PoC internally. Kafka would not start and it was related to parameters or property settings in Java. We were able to work around it, but we had problems like adding certificates.

View full review »
it_user650004 - PeerSpot reviewer
Team Lead at a financial services firm with 1,001-5,000 employees

The setup was pretty straightforward.

View full review »
it_user647457 - PeerSpot reviewer
Head of Engineering

The initial setup was straightforward.

View full review »
it_user644286 - PeerSpot reviewer
Deputy General Manager, DevOps Manager at a comms service provider with 10,001+ employees

Initial setup was mostly intuitive (based on rsync).

View full review »
Andrea Castorino - PeerSpot reviewer
Program Manager at SirfinPA

The initial setup was a little complex. We carried out the development ourselves. 

View full review »
AM
IBMi/MIMIX Administrator at Arab Bank

The initial setup was straightforward.

View full review »
Mukulit Bhati - PeerSpot reviewer
CTO at InsightGeeks Solutions Pvt.

It's wasn't so easy for me to deploy. 

View full review »
it_user592356 - PeerSpot reviewer
Technical Lead/Project Manager(Consulting Apple Inc) at a tech services company with 1,001-5,000 employees

The installation was somewhat straightforward.

View full review »
it_user592338 - PeerSpot reviewer
Enterprise Architect at a logistics company with 1,001-5,000 employees

The basic configuration is quite good. We have built a Hadoop cluster and the Kafka service was included.

View full review »
OT
Senior Big Data Developer | Cloudera at Dilisim

The initial setup isn't too complex. I know Kafka very well and don't find it to be overly difficult. There's also very good documentation which users can take advantage of.

Deployment, including security integration, only took about one day.

Two people handled the deployment. One person created the authentification group and after creating groups and users, another handled topic authentification and user definition for the customer.

View full review »
it_user642942 - PeerSpot reviewer
Hadoop Technical Lead (Assistant Consultant) at a tech services company with 10,001+ employees

The setup was straightforward.

View full review »
DR
Founder, CEO at a tech vendor with 1-10 employees

The setup was straightforward.

View full review »
MS
Senior Consultant at a tech services company with 51-200 employees

The difficulty or simplicity of the initial setup varies. It really depends on the organization and its requirements and infrastructure.

Deployment times vary. It can be up to a week in production, however, with some products online, some services can be deployed within minutes.

When you have already deployed the solution, and it's installed, it doesn't require very much maintenance. If it needs any, my company handles it for our clients. We have an entire team that can work on it.

View full review »
RH
freelance at a tech services company with 11-50 employees

For the most part, the initial setup is easy, but if you need a more sophisticated infrastructure or if you have to set up the topics, then you have to be careful and you have to be more knowledgable in Kafka. You will have to know the parameters for the rotations, the size of the message, and the timeouts, as an example.

For a developer it is easy, but for an administrator and production, it requires more.

View full review »
RH
freelance at a tech services company with 11-50 employees

The first time we tried to deploy Kafka, it seemed a little complicated, but the second try went better. Sometimes it isn't easy to set up the necessary communication or estimate how many partitions we need. Some applications have a vast amount of data, so we have to consider how to improve the performance and not increase the transfer times.

View full review »
it_user998961 - PeerSpot reviewer
Enterprice Architect

The initial setup is easy.

View full review »
it_user660630 - PeerSpot reviewer
SDET II at a tech services company with 5,001-10,000 employees

The initial setup was straightforward. Just go through the Kafka documentation and it will be up and running in no time.

View full review »
it_user613983 - PeerSpot reviewer
Java Architect at a tech vendor with 51-200 employees

Setup is very much straightforward for development, and cluster setup is also easy. I am not aware of the production setup yet.

View full review »
WG
Solutions Architect at a tech services company with 201-500 employees

Deploying Kafka is more complex than IBM MQ.

View full review »
Buyer's Guide
Apache Kafka
April 2024
Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
768,415 professionals have used our research since 2012.