Apache Kafka Improvements to My Organization

Michael Silvi
Senior Software Engineering Consultant at a tech services company with 51-200 employees
I used Kafka with a client to decouple applications with different availability profiles. Before using a messaging-based architecture with Kafka as the messaging system, the client used a coordinator application to fire off various posts to as many as eight other applications. With an application that's impacting at least a customer a second in airports, where the customers demand that the system always works, there were issues with ensuring high availability. A typical way to calculate system availability is: Availability = Uptime/(Uptime + Downtime). Hence, where there are two applications involved with a 99% availability, the total system availability degrades quickly: 99% * 99% = 98.01%. With eight applications, total availability caused issues. However, only two systems needed to provide real-time responses, while other systems were for payment processing, CRM, promotions, etc. It was OK if those systems were not up to date in real time. Kafka allowed the client to have temporal decoupling for writes, i.e., the flaky third-party CRM system did not need to be available at the moment for us to respond to a user with a successful response. The availability concerns shifted to Kafka, which is a better trade off because it's built for this. Another benefit, though not required, was the addition of logical decoupling between applications. Additional consumers could be built to overlay concerns of analytics, but the systems responsible for creating the entities on a given topic did not need to be aware of the analytics applications. This simplifies the interaction between applications and concerns of an organization. Another benefit of this architecture is that testing is simplified. A given application needs to be tested to obey a contract of reading a message and producing another message. A Kafka topic acts as the boundary for an integration test. View full review »
Sean Hickey
Solutions Architect at a consultancy with 1,001-5,000 employees
Kafka has a guaranteed delivery mechanism that is very easy to set up. When starting out with minimal hardware, it can handle very large data volumes. When prototyping and creating a proof of concept, Kafka has helped to speed up the timeline from the prototype all the way to production volumes. View full review »
Kevin Quon
Technical Architect at a tech vendor with 51-200 employees
Through its publisher-subscriber pattern, Kafka has allowed our applications to access and consume data at a real time pace. View full review »
Founder, CEO at a tech vendor with 1-10 employees
We have used Kafka for streaming customer web clicks from live sessions to understand customer behavioral patterns. View full review »
Head of Engineering
Kafka was at the base of our system architecture. The system was designed as an event based architecture. Almost all the interactions among micro-services and the same data are used as input to our analytics infrastructure. View full review »
Ivan Dyachkov
Team Lead at a financial services firm with 1,001-5,000 employees
It has become dead simple to connect different application and services, saving a lot of development hours. View full review »
Java Developer at a media company with 1,001-5,000 employees
It will help us build a scalable platform. This will allow the company to provide better customer service. View full review »
Lead Engineer at a retailer with 1,001-5,000 employees
We were using another commercial messaging engine, which was not scalable unless you paid more. Each hub that we provisioned was expensive. This solution is open source, which is much easier to use and doesn’t cost us anything. View full review »
Technical Lead/Project Manager(Consulting Apple Inc) at a tech services company with 1,001-5,000 employees
My organization is transforming by using the new SOA/eventing-based architecture. The application depends on the employees’ information events. Kafka is very helpful in implementing this. It increases the performance and gives the details to multiple external/internal teams using Kafka topics in an asynchronous manner. For example, if someone is moving from one office to another one, we have to update the software. While updating it, the system puts that event in a topic so that all other consumers can update that person’s new location. This can include the payroll team, the insurance team, and the hospital network. The retention period helps us retain the data in the topic for the configured number of days. In this example, if any of the consumers fail to consume the message from the topic, then that message will be there until the retention period ends. View full review »
Chandra Keerthy
Principal Software Architect at a tech services company with 11-50 employees
We are using Kafka as part of our product. It is one of the messaging layers used to interact between various layers of software modules. This provides a clear separation of modules and leverages it for development and testing of different modules. View full review »
Dori Waldman
Big Data Lead at a marketing services firm with 51-200 employees
We are using Kafka as MQ; our servers generate events which are being sent to Kafka. From Kafka, we have several consumers like Secor that upload raw data to S3; Spark stream that is doing aggregations and saving the result in Cassandra; and Druid for OLAP. View full review »
Senior Java Consultant at a tech services company with 501-1,000 employees
We wanted to track the customer activities on our application and store those details on another system(RDBMS/Apache Hadoop). We do extensive analysis with that. This helps the company to analyze the customer activities, such as search terms, and do better. View full review »
Jyothish Kalavoor Parambil
Hadoop Technical Lead (Assistant Consultant) at a tech services company with 10,001+ employees
This is the base streaming component of our IoT platform. In case of disaster recovery, we mirror the data in the cluster by maintaining the offsets and store the data within Hadoop 2.8 HDFS. View full review »
Mehul Jani
Deputy General Manager, DevOps Manager at a comms service provider
Real-time log aggregation which was earlier done with rsync has been moved to Kafka infrastructure along with other real-time streams. View full review »
Enterprise Architect at a logistics company with 1,001-5,000 employees
We use Kafka for reactive architecture, track and trace, mail and parcel. View full review »

Sign Up with Email