What is our primary use case?
One use case is consuming sales data and then writing it back into the S3. That's one small use case that we have; from data Shields to data Firehose, from data Firehose to Amazon S3.
There are OneClick data streams that are coming in. For click streams data we established Kinesis data streams and then from Kinesis data streams, we dump data into the S3 using Kinesis Data Firehose. This is the main use case that we have. We did many POC's on Kinesis, as well. Also, one more live project using the DynamoDB database is running in Amazon. From DynamoDB we have triggers that automatically trigger to Lambda, and then from Lambda we call Kinesis then Kinesis writes back into the S3. This is another use case.
Another thing that we did is called Kinesis data analytics where you can directly stream data. For that, we use a Kinesis data producer. From that, we establish a connection to the data stream and then from the data streams to the SQL, which is the Kinesis data analytics tool. From Kinesis analytics, we again establish a connection to the data Firehose and then drive data back into the S3. These are the main use cases that we have for working on Amazon Kinesis.
How has it helped my organization?
In my client's company, there is one live database that comes into the DynamoDB. They want to replicate that in Amazon S3 for their data analytics and they do not want data to be refreshed every second. They want their data to be refreshed at a particular size, like five MBs. Kinesis provides data for that. That's the main improvement that we give to the client.
What is most valuable?
The features that I have found most valuable depend on the use case. I find data Firehose and data streams are much more intelligent than other streaming solutions.
There is a time provision as well as data size. Let's suppose you want to store data within 60 seconds, you can. Let's suppose you want to store data up to a certain size, you can, too. And then you can it write back to the S3. That's the beauty of the solution.
What needs improvement?
Kinesis Data Analytics needs to be improved somewhat. It's SQL based data but it is not as user friendly as MySQL or Athena tools. That's the one improvement that I'm expecting from Amazon. Apart from that everything is fine.
For how long have I used the solution?
I have two years of project experience on AWS, and around six months with Kinesis.
What do I think about the stability of the solution?
I am satisfied with Amazon Kinesis. It is pretty exiting to work on.
What do I think about the scalability of the solution?
Its scalability is very high. There is no maintenance and there is no throughput latency. I think data scalability is high, too. You can ingest gigabytes of data within seconds or milliseconds.
We are a team of five members using Amazon Kinesis. Two are working onshore and three of us are working offshore.
We are all developers implementing, developing, and designing the data pipeline, as well. The thing is we work in a startup company so we have to do all the things from our end on this.
How are customer service and technical support?
As of now we have not had any contact with customer support because we didn't face any complex types of problems while we were implementing our use cases.
How was the initial setup?
The initial setup is very straightforward. It is very well documented and anyone with simple knowledge or common sense can do it.
Implementing is very simple. You can just do it with your fingertips. There might be some improvements that can be made according to the requirements. For that, we do versioning. First we establish the pipeline from the data stream to the S3. That's very easy. You can do it within hours or within minutes. I can say the process is very simple and it's not as complex as it looks.
One more beauty is that Kinesis data Firehose will directly write to S3 in a partitioned way. Based on the timestamp it can directly write in the year, month, day and hour. That's the good thing I found about Amazon Kinesis.
We follow an implementation. We do the deployment directly on Dev. Once we get our results and our processes, and go through Q&A, we implement it directly throughout.
What was our ROI?
Our clients definitely see a return on their investment with Amazon Kinesis.
What's my experience with pricing, setup cost, and licensing?
The pricing depends on the number of shards that we are providing and the time the application is running.
We reduced the cost of the pipeline that we built. We built a generic type of pipeline so that two more times can use same data pipeline.
What other advice do I have?
My advice to anyone thinking about Amazon Kinesis, is that if they have ClickStream or any streaming data which varies from megabytes to gigabytes, they can definitely go for Amazon Kinesis. If they want to do data processing, or batch or streaming analytics, they can choose Amazon Kinesis. And if you want to enable database stream events in Amazon DynamoDB, then you can definitely go for Amazon Kinesis. I don't see any better option for these other than Amazon Kinesis. You can use Amazon Kinesis Data Analytics Tool to detect an anomaly before you process the data. That's one more beauty. The first things we need to determine are the source and the throughput of the data and the latency you want.
On a scale of one to ten I would rate Amazon Kinesis a nine.