We are still in the cluster database phase. Based on the use cases captured during the advisory phase, there will be a mix of 40 to 60% of users. 40% will be internal data science and IT teams, while 60% will be end users. So, the total number of users we have seen is 25. Out of these, around 15 business users will make decisions based on reports generated by Kafka analytic data. The remaining users are internal, who analyze this data daily to identify more use cases from a predictive and AI perspective for the future banking domain.

Moreover, our current client is an enterprise business. It is a globally renowned bank that has entered Saudi.

Apache Kafka has helped out the organization because we leverage it for all our eCommerce real-time analytics use cases.

So it is a good backbone for microservices. So basically you want to write microservices, which you can shut down and bring it up whenever you want. You want to be able to shut it down to actually replace it with a newer version and bring it up. The bottom line is you can kill the microservice and bring it back up and do all the things that you want to do with it. But whenever it comes back up, it should pick up and run from where it had left off. That is what everybody tries to do. And in order to build such a system, they have to write several logical pieces of code, and most of that code has already been built for in Kafka so that you don't have to do it yourself.

One example of how Kafka has been beneficial is in the context of financial trading. When a trade is executed, it generates an event. I used Kafka to create an application that captures these events and stores them in a topic, allowing for efficient processing in real time.

We had been using a lot of expensive licenses earlier, such as SOLEIL, as well as some legacy versions, which were not only costly but also caused memory issues and required highly technical personnel to manage. This posed a huge challenge in terms of resourcing and cost, and it simply wasn't worth investing more in. However, Kafka was comparatively free as it was open source, and we were able to build our own monitoring system on top of it. Kafka is an open-source platform that allows us to develop modern solutions with relative ease. Additionally, there are many resources available in the market to quickly train personnel to work with this platform. Kafka is user-friendly and does not require an extensive learning curve, unlike other tools. Furthermore, the configuration is straightforward. All in all, Kafka provides us with a great platform to build upon with minimal effort.

We built a solution for a client and the client was happy with the solution.

We used to lose some of our messages when we integrated them in bulk, this solution has stopped that happening.

Kafka allows you to handle huge amounts of data and classify it into different categories. If you have huge amounts of data, Kafka is a very good solution for data classification. When you need to route it in different directions, you have to take a look at the messages that you get, interfile them, and then send them to the correct place. Kafka is a good product to use in the backend.

Apache Kafka has helped our organization handle larger volumes without affecting the infrastructure load.

We were using another commercial messaging engine, which was not scalable unless you paid more. Each hub that we provisioned was expensive. This solution is open source, which is much easier to use and doesn’t cost us anything.

The solution has allowed us to take the use cases provided by another communication tool and resolve those issues.

Apache Kafka has became our main component on almost all our distributed solutions. It has helped us to delivery fast distributing messages to our customer's applications.

From my experience with Apache Kafka, one of the most notable advantages is its ability to maintain a comprehensive record of historical data that includes every update, alteration, and version of information, unlike a conventional relational database. This feature allows for seamless tracking and analysis of the progression and transformation of the data over time, enabling users to easily review and analyze the history of the information.

The solution has the capability for various systems to effortlessly interact with one another without prior knowledge of their existence, current operational status, or specific configurations. By utilizing service buses and dynamic integration, data can be distributed across networks and retrieved in a way that is most suitable for each system's requirements. In addition, Apache Kafka allows for the modification of data to provide diverse clients, consumers, or observers with unique and varying data. The replication of data can produce multiple versions, and this data can be adjusted to fit various needs. With the use of probes, one can alter the behavior of the transformation process, thereby changing the way in which data is transformed and the output produced. Overall, working with Apache Kafka has brought about an array of benefits, enabling seamless system interactions and allowing for the customization and modification of data to meet individual requirements.

Apache Kafka has helped our client's online restaurant company by allowing them to take any orders and send the notifications with some other details, such as logic commands, to the different microservices.

We implemented the notification system between our components, and we found that Apache Kafka performs well in scalability. It has improved our organization because of the scalability and the comfort of a fail-safe or disaster recovery it provides.

The solution can handle more speed and has horizontal scalability for both messaging, but more specifically stream processing and data enrichment. By using this solution it can reduce the number of components required in the tech stack. For example, we were taking data events out of SAP and sending them to consumers without having to go through multiple processors that were outside of the KAFKA space. Additionally, we are using Kafka from GoldenGate to propagate database updates in real-time.

Apache Kafka has improved our organization because it's more reliable than Rabbit. That's the whole point for us.

Kafka has a guaranteed delivery mechanism that is very easy to set up. When starting out with minimal hardware, it can handle very large data volumes. When prototyping and creating a proof of concept, Kafka has helped to speed up the timeline from the prototype all the way to production volumes.

It eases our current data flow and framework, which digests all types of sources regardless of it being structured or not.

We wanted to track the customer activities on our application and store those details on another system(RDBMS/Apache Hadoop). We do extensive analysis with that. This helps the company to analyze the customer activities, such as search terms, and do better.

It will help us build a scalable platform. This will allow the company to provide better customer service.

In my previous company, we had a proprietary implementation and we changed it with Kafka. We changed it because we had many different connectors available and it also allowed us to create a window to our products for the client. It was an on-premise product and it allowed the outline to take the data out, without us developing anything.

You can connect in any language and there are a lot of connectors available, it helps a lot. And it creates visibility into the data and stability. There are several alternatives but this is one of the best options for this.

Through its publisher-subscriber pattern, Kafka has allowed our applications to access and consume data at a real time pace.

We are using Kafka as MQ; our servers generate events which are being sent to Kafka. From Kafka, we have several consumers like Secor (https://github.com/pinterest/secor) that upload raw data to S3; Spark stream that is doing aggregations and saving the result in Cassandra; and Druid for OLAP.

I used Kafka with a client to decouple applications with different availability profiles. Before using a messaging-based architecture with Kafka as the messaging system, the client used a coordinator application to fire off various posts to as many as eight other applications. With an application that's impacting at least a customer a second in airports, where the customers demand that the system always works, there were issues with ensuring high availability.

A typical way to calculate system availability is: Availability = Uptime/(Uptime + Downtime). Hence, where there are two applications involved with a 99% availability, the total system availability degrades quickly: 99% * 99% = 98.01%.

With eight applications, total availability caused issues. However, only two systems needed to provide real-time responses, while other systems were for payment processing, CRM, promotions, etc. It was OK if those systems were not up to date in real time.

Kafka allowed the client to have temporal decoupling for writes, i.e., the flaky third-party CRM system did not need to be available at the moment for us to respond to a user with a successful response. The availability concerns shifted to Kafka, which is a better trade off because it's built for this.

Another benefit, though not required, was the addition of logical decoupling between applications. Additional consumers could be built to overlay concerns of analytics, but the systems responsible for creating the entities on a given topic did not need to be aware of the analytics applications. This simplifies the interaction between applications and concerns of an organization.

Another benefit of this architecture is that testing is simplified. A given application needs to be tested to obey a contract of reading a message and producing another message. A Kafka topic acts as the boundary for an integration test.

We are using Kafka as part of our product. It is one of the messaging layers used to interact between various layers of software modules. This provides a clear separation of modules and leverages it for development and testing of different modules.

It has become dead simple to connect different application and services, saving a lot of development hours.

Kafka was at the base of our system architecture. The system was designed as an event based architecture. Almost all the interactions among micro-services and the same data are used as input to our analytics infrastructure.

Real-time log aggregation which was earlier done with rsync has been moved to Kafka infrastructure along with other real-time streams.

My organization is transforming by using the new SOA/eventing-based architecture. The application depends on the employees’ information events. Kafka is very helpful in implementing this. It increases the performance and gives the details to multiple external/internal teams using Kafka topics in an asynchronous manner.

For example, if someone is moving from one office to another one, we have to update the software. While updating it, the system puts that event in a topic so that all other consumers can update that person’s new location. This can include the payroll team, the insurance team, and the hospital network.

The retention period helps us retain the data in the topic for the configured number of days. In this example, if any of the consumers fail to consume the message from the topic, then that message will be there until the retention period ends.

We use Kafka for reactive architecture, track and trace, mail and parcel.

We use Kafka as part of our services. Our product (cloud clusters) has many components and Kafka is one of them.

For example, we use Kafka as a data integration tool. If you take Oracle GoldenGate as a typical use case, what happens is GoldenGate collects the data for the replication and sends this data to the Kafka servers. We collect the data on the Kafka servers, and we create some transformations, some operations, from that data. We then copy the data to the HTTP or hub site.

Previously, when I worked at Nokia, we were collected data using Kafka and then we stored the data on the Kafka servers. We did all transformations through Kafka streaming. Later, Kafka moved data over to the HP site. 

This is the base streaming component of our IoT platform.

In case of disaster recovery, we mirror the data in the cluster by maintaining the offsets and store the data within Hadoop 2.8 HDFS.

We have used Kafka for streaming customer web clicks from live sessions to understand customer behavioral patterns.