Amazon Kinesis Primary Use Case

Pablo Giner
Head of BI at Wind Mobility
In terms of use cases, it depends of which component we're talking about, as we use three of the 4 components. The only one we don't use is the Video Streams. Kinesis Data Stream is the module that we have been using the longest, essentially we use it to hold data which will be processed by multiple consumers. We have multiple data sources and we use Kinesis to funnel that data which is then consumed by multiple other consumers. We gather data coming from IoT devices, user phones, databases and a variety of other sources and then, as we have multiple consumers, we use Kinesis to actually gather the data and then we process it directly in Lambda, in Firehose, or in other applications. View full review »
Jesus Martin
Senior Software Engineer at a computer software company with 201-500 employees
I work as a senior software engineer in eCommerce analytics company, we have to process a huge amount of data. Only a few people within our organization use Kinesis. My team, which includes three backend developers, simply wanted to test out different approaches. We are now in the middle of migrating our existing databases in MySQL and Postgres, to Snowflake. We use Kinesis Firehose to ingest data in Snowflake at the same time that we ingest data in MySQL, without it impacting any performance. If you ingest two databases in a synchronous way, then the performance is very slow. We wanted to avoid that so we came up with this solution to ingest the data in the stream. We use Kinesis Firehose to send the data to the stream, which then buffers the data for roughly two minutes. Afterwards, it places the files in an S3 bucket, which is then loaded automatically, via an integration with Snowflake that's called Snowpipe. Snowpipe reads and ingests every message and every file that's in the S3 bucket. This stage doesn't bother us because we don't need to wait for it. We just stream the data — fire and forget. Sometimes, if the record is not ingested successfully, we have to retry. Apart from that, it's great because we don't need to wait and the performance is great. There are some caveats there, but overall, the performance and the reality of it all has been great. This year, 100% of the time when there was an issue in production, it was due to a bug in our code rather than a bug in Kinesis. View full review »
Jeff Levy
Senior Engineering Consultant at a tech services company with 201-500 employees
As part of my interest in obtaining Amazon certification and learning more about Kinesis, I am currently using it to capture streaming Twitter data. I get an avalanche of tweets and I need some technology to harness and capture them. I have used the streaming Twitter API to deal with it. Twitter is updated every half a second, so I'm tapping into the streaming API and capturing a lot of stuff. It has also been used for the Internet of Things (IoT), where there is a lot of streaming stuff that comes out and you need a mechanism to capture all of it from your devices. This includes things such as logs. My company was recently working on a project with Kinesis where we were capturing data from racecars. These racecars were emitting tons of data and it needed to be captured by some kind of tool for analytics. Kinesis was used to capture all of that information. The basic use case is just capturing the data. In the streams, you can do some sort of interim transformations but for the most part, the basic use case is just capturing data and persisting it in a data store like Amazon S3. Another example is Elastic MapReduce permanent storage. Once it lands in some kind of permanent store, further transformations or aggregations can be done at that point. View full review »
Learn what your peers think about Amazon Kinesis. Get advice and tips from experienced pros sharing their opinions. Updated: November 2020.
447,439 professionals have used our research since 2012.
Matt Newman
Principal Data Engineer at a transportation company with 1,001-5,000 employees
Our primary use case of this solution is for a streaming bus architecture, we get events and they come in through Kinesis like a Jason event. It's usually a change to a database, but it can be any event such as in our application, which feeds into the Kinesis and then we have a Lambda that consumes them and then finally it puts those into a data warehouse which is the ultimate goal. So it's a near real-time data warehouse. View full review »
Pagidal Venkata Subba Reddy
Data Engineer
One use case is consuming sales data and then writing it back into the S3. That's one small use case that we have; from data Shields to data Firehose, from data Firehose to Amazon S3. There are OneClick data streams that are coming in. For click streams data we established Kinesis data streams and then from Kinesis data streams, we dump data into the S3 using Kinesis Data Firehose. This is the main use case that we have. We did many POC's on Kinesis, as well. Also, one more live project using the DynamoDB database is running in Amazon. From DynamoDB we have triggers that automatically trigger to Lambda, and then from Lambda we call Kinesis then Kinesis writes back into the S3. This is another use case. Another thing that we did is called Kinesis data analytics where you can directly stream data. For that, we use a Kinesis data producer. From that, we establish a connection to the data stream and then from the data streams to the SQL, which is the Kinesis data analytics tool. From Kinesis analytics, we again establish a connection to the data Firehose and then drive data back into the S3. These are the main use cases that we have for working on Amazon Kinesis. View full review »
Akinola McLean
Chapter Lead - Data and Infrastructure (Head of Department) at a tech services company with 51-200 employees
Our primary use case of this solution is as an intricate part of our data pipeline to deal with all of our big data problems. The traffic in our industry is highly volatile. At any given time we could have 10,000 users, and five minutes later it could be 100,000. We need systems fast enough to deal with that elasticity of demand, and the ability to deal with all the big data problems. Volume, velocity, ferocity, things like that. That's where we use the Kinesis platform. They have different iterations of it. The normal Kinesis Stream, is a little bit more manual, but we use that for our legacy technology, and for the more recent ones, we use Kinesis Firehose. View full review »
Haja Najumudeen
IT Linux Administrator and Cloud Architect at Gateway Gulf
We use it for: * Security * DDoS attacks * Server application firewall * Kinesis makes it easy to collect the process * Real-time analysis * Streaming data We can get insight and react quickly to new information. View full review »
Learn what your peers think about Amazon Kinesis. Get advice and tips from experienced pros sharing their opinions. Updated: November 2020.
447,439 professionals have used our research since 2012.