Apache Kafka Initial Setup
AT
reviewer2075460
Group Manager at a media company with 201-500 employees
The initial two months were for capacity estimation, where we worked with the client's different business teams to understand the data volumes and use cases. Then, the next four to five months went into procurement, where we had to work with infrastructure teams and vendors to understand the servers and networks required for the cluster.
The actual cluster setup took us two months, and it was a little longer due to a shortage of expertise on the client's networking team. We had to handle everything ourselves since it was an on-premise setup with physical servers and network connections. Currently, we are in the security review phase, and once it completes, we will start implementing various use cases like task and batch processing, archival, etc.
If you see my experience from the Apache Kafka implementation and clusters as a perspective, I will rate the setup somewhere between seven to eight out of ten.
View full review »If you are using the latest version of Apache Kafka in which the use of Zookeeper is not required, then it uses the KRaft protocol, which is built into Kafka broker. Since the use of Zookeeper is no longer required, I think the setup phase of dissolution is better than its previous versions. I rate the initial setup of the product somewhere between seven to eight out of ten.
Apache Kafka's initial setup is very straightforward.
The solution is deployed on an on-premises model.
Apache Kafka was deployed in our company within a couple of days.
Three people were involved in the deployment process of Apache Kafka.
The initial setup of Apache Kafka was complex. We were able to simplify it by doing registry-based integration of the services.
View full review »Buyer's Guide
Apache Kafka
April 2024
Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
768,415 professionals have used our research since 2012.
I rate Kafka 10 out of 10 for ease of setup. It's easy for us because we use Docker, but if you want to use another system like Linux it may be a little challenging
The initial setup was relatively easy for us since we already had Zookeeper and the necessary setup in place. We also had good knowledge of Kafka. Therefore, it was not a difficult challenge. In general, I believe that it is manageable. There are benefits and the setup is not overly complex.
Our company has implemented Ship, making our lives easier when it comes to changes or version updates. We can package everything in one place and deploy it with Ship, then implement the virtual number with a minimum of 50 changes.
Deployment time depends on our location and the task at hand. Initially, there is a lot of setup and configuration that must be done, but this can become easier with experience. Nowadays, the process is not too difficult, as all the version numbers and conflict files are already in place. However, if this is a new task for us, it may take some time to figure out all the configurations.
One person was dedicated to deploying Kafka. This person got help from our release team, who had already set up Zookeeper and other necessary components.
View full review »MB
Mohamed BENTAHAR
Architect at Agence Française de Développement
The solution's initial setup process was complex. The deployment process took three or four years.
Right now, I can't deliver the planning process required for deployment.
For deployment and maintenance, we have a manager and an operational person. However, I can't give an exact count of the people required for deployment and maintenance.
The initial setup of Apache Kafka was easy because it is Dockerized. However, if you were to install it yourself it would be difficult. Having it Dockerized makes it worth it.
The first deployment took approximately two hours. The updates of the solution can be done in a matter of minutes.
View full review »SP
reviewer1142973
CEO at a comms service provider with 11-50 employees
I am into pre-sales and project management. I don't usually install Apache Kafka, but its basic installation seems quite simple.
Its deployment is usually quite short. Usually, we are able to deploy it in a few days, but data management and application development can take a few months.
The solution's initial setup is quite straightforward. You just have to upgrade a couple of configuration files.
View full review »GT
George Thomas
Lead Architect at a financial services firm with 1,001-5,000 employees
It is not deployed on-premises.
We use Kafka as part of the OCI Oracle Cloud platform and the Oracle Big Data platform because Kafka is included.
The Apache Kafka setup will take some time because it is not simple, and we have a lot of other components to install. It's fine because we needed all the plugins and other things for the simple implementations, but the containers' implementation is simple. The only difference is that when it comes to Zookeeper, there are a lot of supporting applications running on top of it, such as Zookeeper. As part of their area, Apache Kafka is running on top of Zookeeper. What do they think? As part of their... manageability, the Kafka area, and Apache Zookeeper. As a result, everything must be removed. And it will be preferable if the implementation is simple. I believe Confluent is doing this, but we have not yet begun.
The deployment, and configuration, will take one hour to complete. However, it is also dependent on the fact that you require a large number of configurations, which we have.
View full review »The initial setup is dependent on the individual. For example, it would be straightforward if a person practices these things a lot and understands the documentation correctly. However, since most people prefer examples instead of reviewing documentation, it would be easy to set up if they find steps on the internet but difficult if they do not have examples.
View full review »The initial setup was complex for us because we built it internally. This meant that full deployment took around a month.
View full review »The initial setup is relatively easy as I am using Docker and the files provided by Confluent. However, setting up Apache Kafka in a production environment is not as straightforward. I prefer to use solutions, such as Confluence that already have everything preconfigured. As a developer, creating an environment for it is not a problem for me, but I think it can be challenging for those responsible for the production environment. There have been issues with data loss and other problems in the past. Configuring it for production is not easy.
My deployment was very quick because I am using it locally. We have someone else that does the cloud deployment.
View full review »The initial setup was extremely complex. In our case, it took a team of 12 two months to deploy.
View full review »TD
reviewer1218324
Head of Technology - Money Movement Platform at a financial services firm with 10,001+ employees
Apache Kafka is straightforward to implement.
View full review »AV
Andre Visser
Technical Director at Metrofibre Networx
The initial setup was straightforward because I've got a lot of experience in this field. But even for a junior person, it would be fine. There are so many resources, and it's very well documented as they are a premium service provider. So it makes the setup just easier.
The deployment takes a few days.
We set up a free cluster for this service because we use a lot of data. We use ZooKeeper to secure different products for instruction with the cluster. But, it was easy as it is a popular product, and much information is available. It can download data, like fifty gigs per day. We can effectively handle it all as well. I never developed any issues.
View full review »The initial setup was a bit complex in terms of how to manage it across data centers. But once it was setup, we never faced issues.
View full review »The initial setup was straightforward.
View full review »NK
Nitin Kamble
Director at Tibco
The open-source version is relatively straightforward to set up and only takes a few minutes.
View full review »JA
Jack Angoe
Technical Lead at Interface Fintech Ltd
There was a little learning curve, but we managed it. I think it took us around six weeks to complete the deployment.
View full review »SK
SunilKalva
Barista Brewing Espresso at Linkedln
The initial setup was simple although we had a couple of hiccups. It took around a week but that was several years ago and we haven't had any problems since. Our team carried out the deployment and we currently have a few engineers who deal with maintenance.
It's very complex, like Spark.
Deployment took 50 minutes for all the Kubernetes ports, Spark, Kafka, and other components based on Sherlogic. In 30 minutes, we created an environment using this program to make installation easier.
View full review »ME
Mario Estrada
CTO at Estrada & Consultores
The initial setup is easy. The majority of the tools today are really very easy to configure and setup. Docker Containers and Kubernetes, actually, have made life easier for architects as well as developers.
Nowadays, you just install the container, and then you don't have to really manage the internals at libraries, OS levels, et cetera. You just run the container. Everything is containerized.
DZ
Dimitrios Zigkos
Enterprise Architect at Smals vzw
Setting up Apache Kafka is, is not an easy task, especially when trying to containerize it and make it controllable. This is because Apache Kafka has its own distributed mechanism for staying alive, checking readiness, replicating, and scaling. Ensuring that it complies with Kubernetes or OpenShift Orchestrator requires careful attention, as there is a risk of two masters attempting to perform the same task and ultimately undoing each other's work.
In comparison to Kubernetes, OpenShift is a highly skilled and advanced implementation infrastructure that automatically manages and orchestrates all the steps required for an application setup. It operates at a higher level of abstraction and eliminates the need for manual operations that are required with Kubernetes. While Kubernetes can run an application with some pipeline and configuration, OpenShift takes care of everything from finding the required images to creating ports and connecting databases. Although manual changes can be made, it's not necessary as OpenShift offers a much more course-grained management approach.
View full review »Apache Kafka was simple to install. If you have a complicated clustered production, it takes time. However, for the development, it doesn't take more than one or two hours.
View full review »KS
reviewer1421481
Solution Architect at a manufacturing company with 10,001+ employees
I don't think that the initial setup was a complex process.
View full review »The initial setup was simple.
View full review »The initial setup of Apache Kafka is easy to set up a cluster. I did the initial setup on my laptop and it is straightforward. I used the Confluent version, but even if you want to run native capabilities it's straightforward to do the implementation.
View full review »RP
Ravi-Patel
Assistant Professor at CHAROTAR UNIVERSITY OF SCIENCE AND TECHNOLOGY
The installation is fairly easy. Additionally, there is a cloud-based version available if a use case requires it.
View full review »JB
reviewer2000091
Software Support & Development Engineer at a computer software company with 501-1,000 employees
The initial setup of Apache Kafka took some time but after it was easy.
I rate the initial setup of Apache Kafka a three out of five.
View full review »DP
DustyPressley
Sr Technical Consultant at a tech services company with 1,001-5,000 employees
Apache Kafka on AWS is a bit complex. There is a third-party company called Confluent and they have the support that makes their installation much easier, especially for the on-premise deployment. You install Apache Kafka alone it can be a little complex compared to other queuing messaging solutions.
The on-premise deployment takes approximately a few days. The cloud or hybrid deployments including all the permissions, typologies, firewalls, and networking configuration can take weeks for all the accessibility issues to be resolved. However, the delay could have been client-related and not necessarily the solution.
View full review »LP
Lakshmanan Panneerselvam
Owner at Binarylogicworks.com.au
The initial setup was straightforward for me. However, it is not straightforward for everyone because there are some tricky things to implement. In single-mode it is a little bit easier, but when it is set up as a distributed system then it is more complex because there are a lot of things to be considered.
View full review »TM
reviewer1398480
Building Event-centric Data processing Architectures at a tech services company with 51-200 employees
It is super easy to set up. I rate the ease of setup a ten out of ten. However, building and administration get quite difficult. It takes three months to make things production-ready.
View full review »TM
reviewer971988
Assistant Student at a retailer with 5,001-10,000 employees
Implementations took half a year for everyone to learn the solution. It was quite lengthy.
View full review »Since it is open source, you are on your own for setup. However, the tutorials from the Apache foundation and online sources have been an immense help.
Getting started is very easy. The complexity of very large volumes of data and appropriate sharding, however, is difficult. There are fewer resources for tuning and best practices.
View full review »I initially found Kafka difficult to set up, so I would rate it about five out of 10 for ease of setup. After I learned more about the platform, I would rate it eight out of 10. It is deployed on-premises over a cluster of three or four PCs. You can deploy Kafka in a few hours with one person.
View full review »DS
reviewer1052868
Principal Technology Architect at a computer software company with 5,001-10,000 employees
We were able to deploy and use it with no problems for our use case. We didn't find it so complex. We work with so many applications, databases, Postgres, and so many other things, so we could manage it easily. We deployed Kafka in a few hours. We have an infrastructure team and DevOps. Those teams are pretty capable, and they've completely automated the whole deployment. It always takes time the first time you upgrade any application, not just Kafka. We might discover some issues, such as configuration, parameters, compatibility, etc. Once that becomes standard, it is stable, and then they only need to replicate it to the different environments or different developers groups. We have a sophisticated process.
View full review »MN
MoulaliNaguri
Project Engineer at Wipro Limited
We're continuously deploying the product. We're still in the process of deployment.
View full review »The initial setup was straightforward. We faced some issues during the development in areas such as message producer and consumer. We rectified those with the tweaking the producer and consumer configurations. The documentation is very good.
View full review »Initial setup wasn’t really complex. We use Kafka through Hortonworks Suite, which comes with many other big data tools. Ambari makes it easy to setup
View full review »KQ
Kevin Quon
Senior Technical Architect at a computer software company with 51-200 employees
Initial setup was straightforward. We simply hosted multiple Kafka brokers and ZooKeeper servers on AWS EC2 instances.
View full review »Initial setup was not easy. It requires intermediate skills.
View full review »YL
reviewer1128858
Vice President at a consultancy with 51-200 employees
The initial setup is straightforward. It's easy to set up.
It took a few days to get it up and running.
We only need one or two engineers to keep this solution running. We basically let it run and monitor what's going on. We usually don't touch it unless something goes wrong.
View full review »The setup is not a big deal for us. We can handle it. After the system is set up, the person who administers it has to do so with Apache Kafka.
Depending on the setup, it will usually take two weeks.
View full review »The setup was complex. One must consider setting up ZooKeeper, Kafka, multi-zone/region availability, as well as typical associated functions for running it all in production. This includes monitoring, message schema changes (consider Avro), encrypting messages if it's a concern, potentially authorization for different topics depending up on the sensitivity of data.
If an organization uses Kafka as the first messaging system, then the approach for application design must also shift significantly.
View full review »Setting up was easy with ZooKeeper.
View full review »The initial setup is straightforward, it's not complicated.
JJ
reviewer1247268
Technology Lead at a tech services company with 10,001+ employees
We had problems when we were setting up Kafka ourselves to conduct our PoC internally. Kafka would not start and it was related to parameters or property settings in Java. We were able to work around it, but we had problems like adding certificates.
View full review »The setup was pretty straightforward.
View full review »The initial setup was straightforward.
View full review »Initial setup was mostly intuitive (based on rsync).
View full review »The initial setup was a little complex. We carried out the development ourselves.
AM
AhmadMasamreh
IBMi/MIMIX Administrator at Arab Bank
The initial setup was straightforward.
View full review »It's wasn't so easy for me to deploy.
The installation was somewhat straightforward.
View full review »The basic configuration is quite good. We have built a Hadoop cluster and the Kafka service was included.
View full review »OT
OnurTokat
Senior Big Data Developer | Cloudera at Dilisim
The initial setup isn't too complex. I know Kafka very well and don't find it to be overly difficult. There's also very good documentation which users can take advantage of.
Deployment, including security integration, only took about one day.
Two people handled the deployment. One person created the authentification group and after creating groups and users, another handled topic authentification and user definition for the customer.
View full review »The setup was straightforward.
View full review »DR
FounderC32bc
Founder, CEO at a tech vendor with 1-10 employees
The setup was straightforward.
View full review »MS
reviewer1304505
Senior Consultant at a tech services company with 51-200 employees
The difficulty or simplicity of the initial setup varies. It really depends on the organization and its requirements and infrastructure.
Deployment times vary. It can be up to a week in production, however, with some products online, some services can be deployed within minutes.
When you have already deployed the solution, and it's installed, it doesn't require very much maintenance. If it needs any, my company handles it for our clients. We have an entire team that can work on it.
View full review »RH
reviewer1289778
freelance at a tech services company with 11-50 employees
For the most part, the initial setup is easy, but if you need a more sophisticated infrastructure or if you have to set up the topics, then you have to be careful and you have to be more knowledgable in Kafka. You will have to know the parameters for the rotations, the size of the message, and the timeouts, as an example.
For a developer it is easy, but for an administrator and production, it requires more.
View full review »RH
reviewer1289778
freelance at a tech services company with 11-50 employees
The first time we tried to deploy Kafka, it seemed a little complicated, but the second try went better. Sometimes it isn't easy to set up the necessary communication or estimate how many partitions we need. Some applications have a vast amount of data, so we have to consider how to improve the performance and not increase the transfer times.
View full review »The initial setup is easy.
View full review »The initial setup was straightforward. Just go through the Kafka documentation and it will be up and running in no time.
View full review »Setup is very much straightforward for development, and cluster setup is also easy. I am not aware of the production setup yet.
View full review »WG
reviewer1388343
Solutions Architect at a tech services company with 201-500 employees
Deploying Kafka is more complex than IBM MQ.
View full review »Buyer's Guide
Apache Kafka
April 2024
Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
768,415 professionals have used our research since 2012.