Amazon Kinesis Reviews

I work as a senior software engineer in eCommerce analytics company, we have to process a huge amount of data.

Only a few people within our organization use Kinesis. My team, which includes three backend developers, simply wanted to test out different approaches.

We are now in the middle of migrating our existing databases in MySQL and Postgres, to Snowflake. We use Kinesis Firehose to ingest data in Snowflake at the same time that we ingest data in MySQL, without it impacting any performance.

If you ingest two databases in a synchronous way, then the performance is very slow. We wanted to avoid that so we came up with this solution to ingest the data in the stream.

We use Kinesis Firehose to send the data to the stream, which then buffers the data for roughly two minutes. Afterwards, it places the files in an S3 bucket, which is then loaded automatically, via an integration with Snowflake that's called Snowpipe. Snowpipe reads and ingests every message and every file that's in the S3 bucket. This stage doesn't bother us because we don't need to wait for it. We just stream the data — fire and forget. Sometimes, if the record is not ingested successfully, we have to retry. Apart from that, it's great because we don't need to wait and the performance is great.

There are some caveats there, but overall, the performance and the reality of it all has been great. This year, 100% of the time when there was an issue in production, it was due to a bug in our code rather than a bug in Kinesis.

We save a lot of time with Kinesis, but it's difficult to measure just how much. We actually have something similar regarding some other processes. We have developed somewhere else a tool that takes note of the contents of the stream, places them into a file, manually uploads them to the S3, and copies the files into Snowflake. That could be done with Kinesis, but it could take two weeks or 1 month less to get it production-ready.

The first would be the one found in the AWS SDK using the asynchronous client: put Record batch function which allows you to put a list of records in one put record request, which saves time and it's more efficient. Also, by using the asynchronous client, the records are sent in the background using an internal thread pool that can be configurable for your needs. In our performance testing, we came across this setting was the fastest solution. It didn't impact anything in the performance of the system process.

The second one would be the ability to link the stream to other places other than S3 via configuration of the stream and without changing a line of code.

Lastly, you can also link a lambda function to the stream to transform the data as it arrives in before writing it in S3, which is great to perform some aggregations or enrich the data with other data sources.

The default limit that they have, which at the moment is 5,000 records per second (I'm talking about Kinesis Firehose which is a specialized form of the Amazon Kinesis service) seems too low. Actually, on the first week that we deployed it into production, we had to roll it back and ask Amazon to increase the default limits.

It's mentioned in the documentation, but I think the default settings are far too low. The first week it was extremely slow because the records were not properly ingested in the stream, so we had to try it again. This happened the first week that we deployed it into production, but after talking with Amazon, they increased their throttling limits up to 10,000 records. Now it works fine.

We've been using this solution since September 2019.

The stability is great. I'd say that maybe we have it running 99% of the time, and nothing stops it.

Amazon Kinesis is definitely scalable. We have huge spikes of data that get processed around midnight and Kinesis handles it fine.

It automatically scales up and down, We don't need to compute it for that. It's great.

The only time that we needed to contact Amazon was to ask them to increase the throttling limit. They replied to us very quickly and did what we asked.

Initially, we were evaluating Kafka. I think Kafka is faster, but it's less reliable in terms of maintenance; however, when Kafka works, and you have it properly configured, it's much better than Kinesis, to be honest.

On the other hand, Kinesis provides us with better maintenance. Our DevOps team is already oversaturated, so we didn't want to increase the maintenance cost of the production environment. That's why we decided to go with Kinesis; because performance-wise, it's easy to configure and maintain.

I found this solution to be really easy to configure. The essential parts of the configuration include naming the stream and also configuring the buffering time that it takes for a record to get ingested into S3 (how long it will be in the stream until it's put into an S3). You also need to link the Amazon S3 buckets with the Amazon Kinesis stream. After you've completed these configurations, it's pretty much production-ready. It's very, very easy. That's a huge advantage of using this service.

Deployment took a few minutes.

You don't need a deployment plan or an implementation strategy because once you configure it, you can just use a stream. It's not an obligatory version that needs a library, etc. This stream is completely abstract in that way. You only need to configure it once, that's it.

We have seen a return on our investment with Amazon Kinesis. We are able to process data without any issue. It's our solution for ingesting data in other databases, such as Snowflake. 

Developing the stream process manually or using Kafka

If you want to use a stream solution you need to evaluate your needs. If your needs are really performance-based, maybe you should go with Kafka, but for near, real-time performance, I would recommend Amazon Kinesis.

If you need more than one destination for the data that you are ingesting in the stream, you will need to use Amazon Kinesis Data Streams rather than Firehose. If you only want to integrate from one point to another, then Kinesis Firehose is a considerably cheaper option and is much easier to configure. 

From using Kinesis, I have learned a lot about the synchronous way of processing data. We always had a more sequential way of doing things. 

On a scale from one to ten, I would give this solution a rating of eight.

My company primarily uses Amazon Kinesis within our data processing workflow to push the data into our data lake.

AI processing or cleaning up data would be nice since I don't think it is a feature in Amazon Kinesis right now. Amazon Kinesis has a specific purpose, and it may not fit into use cases anymore if it is loaded with too many features.

I have been using Amazon Kinesis for a year. My company has a partnership with Amazon. My company is also a customer of Amazon.

Stability-wise, I rate the solution a nine out of ten. I have not encountered any problems with the product's stability.

I have not encountered any problems with the product's scalability. Scalability-wise, I rate the solution a nine out of ten.

My company mostly deals with enterprise-sized businesses.

Whether I recommend the product to others depends on the applications, but moving a lot of data in real-time is a very good solution.

The technical support a user receives from Amazon depends on the kind of relationship that they have with AWS. If you are not known by Amazon, support would be low from AWS. If you have a specific relationship, then the support is okay. I rate the technical support an eight out of ten.

Positive

I rate the setup phase a seven on a scale of one to ten, where one is difficult, and ten is easy.

The solution is deployed on a public cloud.

The time required to deploy the solution is something that depends. A quick deployment can be done in a few minutes. If it is a production deployment where all the scripts and other related areas must be managed, it takes a couple of weeks. It is not the deployment itself of the tool alone that is involved since there is a need to prepare everything around it.

I rate the product price a five on a scale of one to ten, where one is cheap, and ten is expensive.

In terms of the most useful part of the tool for our company's data analytics, I think that because we decided to use AWS, Amazon Kinesis is present in our organization. It is easy to configure, highly available and offers high throughput. Amazon Kinesis is one of our company's main resources.

The scalability hasn't impacted our real-time data processing capabilities. Actually, users can scale up, and it doesn't impact our company. There might be a positive impact on scalability, so there are no issues.

The product has had a positive impact on operational efficiency because it actually allows our company to filter out some data and save costs.

The integration capabilities of the product are good. My company has integrated Amazon Kinesis with some APIs, AWS Lambda, S3, and Amazon EKS.

The adoption of data processing and data lakes is the purpose for which Amazon Kinesis is designed.

I rate the tool a seven to eight out of ten.

I work in a gaming company that builds games for the global market. We use Amazon Kinesis to stream events. 

I have worked in companies that build tools in-house. They face scaling challenges. 

Amazon Kinesis should improve its limits. 

I have been using the product for a month. 

I rate the tool's stability a ten out of ten. 

My company has two to three users for Amazon Kinesis. 

I rate the tool's deployment a nine out of ten. Deployment takes one day to complete. 

The tool's entry price is cheap. However, pricing increases with data volume. 

I rate Amazon Kinesis a seven out of ten. 

It's a kind of data stream tool that actually does kind of external serverless data streaming to different services at Amazon. Especially for the usage I've done, it's basically for getting data to Redshift. 

Kinesis can be one of the sources, and at the same time, Amazon Kinesis can be used to get the data out to other systems through other APIs. Kinesis is one of the tools I have used for typical data streaming.

I've utilized Kinesis Data Streams along with Kinesis Video Streams for different use cases. Again, these are required for working with data as part of Amazon Connect configurations.

Kinesis is fairly easy to use; it's kind of a service that is there and then fairly easy to configure. And it gathers lots of data while having proper authentication mechanisms. It's easy to activate and easy to test and set up as well. 

Typically, all the contact tracing codes that are there as part of Amazon Connect and also some of the real-time data can be passed with different sources of streaming. Like, Genesys also has data streaming and video streaming, which actually passes the real-time data as well. 

So it's pretty straightforward to configure, easy to understand. It's a typical service that is required in Connect integrations, specifically.

For me, especially with video streams, there's sometimes a kind of delay when the data has to be pumped to other services. This delay could be improved in Kinesis, or especially the Kinesis Video Streams, which is being used for different use cases for Amazon Connect. With that improvement, a lot of other use cases of Amazon Connect integrating with third-party analytic tools would be easier. 

It's a stable product overall. So far, I haven't seen any issues wherever I have used it.

This is kind of more like a backend tool used by our developers and the engineering team, including myself, to improve configurations and connectivity. Typically, it depends on the use cases and company requirements, but for my company,  three people generally do this work.

The initial setup is pretty easy and straightforward. It can be done pretty quickly with the direct documentation we have from AWS for Amazon Connect and Amazon Kinesis. That helps things to be configured very quickly.

The pricing depends on the use cases and the level of usage. 

If you wanted to use Kinesis for different use cases, there's definitely a cheaper base cost involved. However, it's not entirely cheap, as different use cases might require different levels of Kinesis usage. 

But, from an Amazon Connect perspective, where I have used this, it's quite good. It's pretty much cheaper compared to alternative methods.

I would recommend using Kinesis. Again, it depends on the use cases. One specific advantage of Kinesis is how straightforward it is, and it offers a lot of integration possibilities. So, for those wanting to use it with Amazon Connect, this is a good solution.

Overall, I would rate it between an eight out of ten because of its ease of use. I like the ease of use and how we can quickly get the configurations done, making it pretty straightforward and stable.

We use it for:

• Security 
• DDoS attacks
• Server application firewall
• Kinesis makes it easy to collect the process 
• Real-time analysis
• Streaming data

We can get insight and react quickly to new information.

Kinesis is in real-time. It enables you to process stream data in real-time. You can drive it in seconds or minutes instead of hours or days.

Kinesis is a fully managed program streaming application. You can manage any infrastructure. It is also scalable. Kinesis can handle any amount of data streaming and process data from hundreds to thousands of processes in every source with very low latency.

Kinesis is good for Amazon Cloud but not as suitable for other cloud vendors. 

I have been using Amazon Kinesis for two years. 

We have used their support before. They are quick to help. Support is very good. They can resolve an issue within ten minutes. 

The initial setup was straightforward. The deployment took two months. 

There are 15 employees who work on it in my company. Some roles include myself as the team lead, support, and solution architects. 

I have seen ROI. 

Kinesis has the best of Amazon: data streaming, building processes, data analytics, data in real-time are very good. The output and monitoring are easy. It has good performance.

I would rate it a nine out of ten. 

Amazon Kinesis is a queuing or buffering system that we use as a central place to buffer the incoming data we receive from the source. The actual destination is open-faced. Amazon Kinesis is used as a buffer in between to decouple the workload.

Amazon Kinesis's main purpose is to provide near real-time data streaming at a consistent 2Mbps rate, which is really impressive.

The solution currently provides an option to retrieve data in the stream or the queue, but it's not that helpful. We have to write some custom scripts to fetch data from there. An option to search for data in the queue can really help us in our day-to-day operations.

Since the solution is a buffer system, you write to it and read from it. The readers are called consumers. If you want to run multiple consumers reading from the queue, you have to enable the enhanced fan-out feature on Amazon Kinesis. This enhanced fan-out feature is quite costly.

There was a point when we had a huge budget increase in one week just because of the enhanced fan-out feature. This feature does not provide any special out-of-the-box functionality. Hence, we struggle to optimize multiple consumers reading from a single queue. We were charged high costs for the solution’s enhanced fan-out feature.

I have been using Amazon Kinesis for more than two years.

The solution is pretty good in terms of scaling. Amazon Kinesis has shards, which are the instances or units that the solution spins up for you. Depending upon your account quota, you can spin up as many shards as you want. You can even raise a request to increase that quota, which will be done sooner. Overall, Amazon Kinesis is a really scalable solution.

Our team, consisting of four to five people, uses the solution extensively in our organization.

We really struggle to get better support for Amazon Kinesis.

Amazon Kinesis is an expensive solution.

Amazon Kinesis is an AWS-managed service, just like S3 or EC. We don't have to deploy it; it is just there, and we spin it up. You must go to AWS' service page and click on Kinesis. Then, you can create it by clicking on Create and entering the name.

I would not recommend Amazon Kinesis to other users. Users can choose a cheaper alternative. They can use any other queuing system or in-house Kafka if they have a Kafka team. Amazon Kinesis provides near real-time read-and-write, but its cost is too high. Users can choose another option that provides the same functionality at less cost.

With Amazon Kinesis, you have to run a consumer who sees from Amazon Kinesis. AWS provides the Kinesis Client Library (KCL), which reads from the Kinesis stream. That library is also used in DynamoDB for data checkpointing. For example, if you have one day of data in Amazon Kinesis and started reading from 12 AM yesterday. The Kinesis Client Library (KCL) will check on the data in the DynamoDB. You get charged for the DynamoDB table out-of-the-box, along with Amazon Kinesis.

The DynamoDB table also costs a lot, which should not be the case. It is just read-and-write and is downloaded from the Kinesis Client Library (KCL). The DynamoDB table's cost should be very minimal, but that's not the case. The consumer is not optimal for efficient read-and-write, which further increases the cost. Both Amazon Kinesis and DynamoDB come into the picture.

Overall, I rate the solution a five or six out of ten.

We do data acquisition based on what is pumped from the remote data and process it centrally so that we may present to our customers meaningful reports, charts, additional layers of support, or alerts. 

At the moment, I am not using Amazon Kinesis, but Azure Event Hub, which I have found to be more meaningful and easier to use. 

I like the event bubbling feature of Amazon Kinesis, although I ultimately switched to Azure Event Hub. Both solutions have similar features, but the latter offers us certain operational advantages. 

Amazon Kinesis is not a bad product, but Azure Event Hub provides us with certain operational advantages, as our focus is on Microsoft related coding. This is why .NET is what we use at the backend. While we can use both Azure Event Hub and Amazon Kinesis towards this end, I feel the latter to be less customized or developed for use in connection with the server-less programming.

Amazon Kinesis has a less meaningful and easy use than Azure Event Hub. 

Amazon Kinesis involved a more complex setup and configuration than Azure Event Hub. 

I have been using Amazon Kinesis for the past year, although I have since switched to Azure Event Hub. 

The scalability is pretty good. One can have any number of nodes spawned or replicated on the primary. Any load can be handled, perhaps a few terabytes with ease in around 15 seconds. One can scale up to this. 

While we have not had occasion to contact Amazon tech support concerning the solution, we have in relation to other matters. We felt it to be good. 

The initial setup and configuration of Amazon Kinesis was more involved than that of Azure Event Hub. 

The solution's pricing is fair. The trick lies in Amazon's pricing. They charge according to the different layers of or types of data that is transfered.

In addition to Azure Event Hub, we also have experience with Apache Kafka, which I feel to offer greater power but more complex configuration. This solution has more features for a variety of purposes. 

The question of whether I would recommend Amazon Kinesis over Azure Event Hub is tricky. While both have their advantages and I consider them to be almost equal, we feel the latter to be better suited to our environment, which is why we went with it. The data transferring policies and associated costs of Amazon were the deciding factors for me.

I rate Amazon Kinesis as an eight or nine out of ten. 

We use this solution for quite large environments.

We use it to capture and process a lot of data. We use it, for example for data analytics and query and analyze a stream's data.

We are a sizable organization and as such, have a lot of data. The solution allows for real-time analysis and you can use a scaler to handle data flows. 

The solution is very flexible and allows for a lot of configuration. It just offers up a lot of possibilities.

I'm using Amazon S3 and Redshift using Amazon server. I can make large configurations and update in near real-time, so that we have real-time use for batch intervals. 

The solution is great for scanning in order to handle environmental data.

The data stream feature on offer is excellent. We use it quite extensively.

The solution works well in rather sizable environments. We deal with a lot of data and it handles it very well.

The solution has a very good alerts system to allow us to respond in real-time.

The dashboards are excellent.

The solution offers very good data capture and integrates well with Power BI and Tableau, for example.

The product makes it very easy to create jobs.

The automation could be better. The solution needs to be better at information capture.

Some jobs have limitations which can make the process a bit challenging.

In order to do a successful setup, the person handling the implementation needs to know the solution very well. You can't just come into it blind and with little to no experience.

I've used the solution for six or seven years or so.

We work with very large environments and haven't had any issues with feeling constricted by the solution.

Personally, based on my past experience and my long history with the solution, the initial setup was not complex. It was pretty straightforward. I find it very easy to use these tools.

A user will need to understand how to create analytics using processing a large amount of information. There may be legacy solutions in the mix as well. A new user will need to understand the environment and all of the requirements before really digging in.

What I will need, basically, is a data map, where I can find any legacy data. From there I can do the setup and it goes pretty smoothly.

I handle the implementation myself.

You can compare this solution to Data Factory and Hadoop. They have a few overlapping characteristics. However, for my industry, Hadoop, for example, wouldn't work as it was lacking some characteristics and parameters and some understanding of the industry itself.

I have a lot of experience in Kinesis and data analytics including in networking in the Amazon AWS environment. My experience is as a big data architect. I draw all environments in AWS. 

On a scale from one to ten, I would rate the solution at a six. It's pretty good, and great for big environments, however, you do need to be well versed in the product to set it up.