PART I

For those not familiar, Simple Storage Services (S3), Glacier and Elastic Block Storage (EBS) are part of the AWS cloud storage portfolio of services. There are several other storage and data related service for little data database (SQL and NoSql based) other offerings include compute, data management, application and networking for different needs shown in the following image.

AWS%202.jpg" width="448" height="252" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;"> 

AWS Services Console via www.amazon.com

Simple Storage Service (S3) is commonly used in the context of cloud storage and object storage accessed via its S3 API. S3 can be used externally from outside AWS as well as within or via other AWS services. There are various S3 modes including standard, Reduced Redundancy (RR) and Infrequent Access (IA). For example with Elastic Cloud Compute (EC2) including via the Amazon Storage Gateway. Glacier is the AWS cold or deep storage service for inactive data and is a companion to S3.

S3 is well suited for both big and little data repositories of objects ranging from backup to archive to active video images and much more. In fact if you are using some of the different AaaS or SaaS services including backup or file and video sharing, those may be using S3 as its back-end storage repository. For example NetFlix leverages various AWS capabilities as part of its data and applications infrastructure.

AWS basics

AWS consists of multiple regions that contain multiple availability zones where data and applications are supported from.

AWS%203.jpg" width="435" height="250" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;">

Note that objects stored in a region never leave that region, such as data stored in the EU west never leave Ireland, or data in the US East never leaves Virginia.

AWS does support the ability for user controlled movement of data between regions for business continuance (BC), high availability (HA), and disaster recovery (DR). Read more here at the AWS Security and Compliance site.

PART II

For those not familiar, Simple Storage Services (S3), Glacier and Elastic Block Storage (EBS) are part of the AWS cloud storage portfolio of services. With S3, you specify a region where a bucket is created that will contain objects that can be written, read, listed and deleted. You can create multiple buckets in a region with unlimited number of objects ranging from 1 byte to 5 Tb in size per bucket. Each object has a unique, user or developer assigned access key. In addition to indicating which AWS region, S3 buckets and objects are provisioned using different levels of availability, durability, SLA’s and costs (view S3 SLA’s here).

AWS%208.jpg" width="465" height="250" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;">

Cost will vary depending on the AWS region being used, along if Standard or Reduced Redundancy Storage (RSS) selected. Standard S3 storage is designed with 99.999999999% durability (how many copies exists) and 99.99% availability (how often can it be accessed) on an annual basis capable of two data centers becoming un-available.

As its name implies, for a lower fee and level of durability, S3 RRS has an annual durability of 99.999% and availability of 99.99% capable of a single data center loss. In the following figure durability is how many copies of data exist spread across different servers and storage systems in various data centers and availability zones.

AWS%209.jpg" width="435" height="250" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;">

What would you put in RRS vs. Standard S3 storage'

Items that need some level of persistence that can be refreshed, recreated or restored from some other place or pool of storage such as thumbnails or static content or read caches. Other items would be those that you could tolerant some downtime while waiting for data to be restored, recovered or rebuilt from elsewhere in exchange for a lower cost.

Different AWS regions can be chosen for regulatory compliance requirements, performance, SLA’s, cost and redundancy with authentication mechanisms including encryption (SSL and HTTPS) to make sure data is kept secure. Various rights and access can be assigned to objects including making them public or private. In addition to logical data protection (security, identity and access management (IAM), encryption, access control) policies also apply to determine level of durability and availability or accessibility of buckets and objects. Other attributes of buckets and objects include life-cycle management polices and logging of activity to the items. Also part of the objects are meta data containing information about the data being stored shown in a generic example below.

AWS%2010.jpg" width="435" height="250" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;">

Access to objects is via standard REST and SOAP interfaces with an Application Programming Interface (API). For example default access is via HTTP along with a Bit Torrent interface with optional support via various gateways, appliances and software tools.

AWS%2011.jpg" width="435" height="250" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;"> 

Example cloud and object storage access

The above figure via Cloud and Virtual Data Storage Networking (CRC Press) shows a generic example applicable to AWS services including S3 being accessed in different ways. For example I access my S3 buckets and objects via Jungle Disk (one of the tools I use for data protection) that can also access my Rackspace Cloudfiles data. In the following figure there are examples of some of my S3 buckets and objects used by different applications and tools that I have in various AWS regions.

AWS%2012.jpg" width="453" height="253" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;"> 

AWS S3 buckets and objects in different regions

Note that I sometimes use other AWS regions outside the US for testing purposes, for compliance purpose my production, business or personal data is only in the US regions.

The following figure is a generic example of how cloud and object storage are accessed using different tools, hardware, software and API’s along with gateways. AWS is an example of what is shown in the following figure as a Cloud Service and S3, EBS or Glacier as cloud storage. Common example API commands are also shown which will vary by different vendors, products or solution definitions or implementations. While Amazon S3 API which is REST HTTP based has become an industry de facto standard, there are other API’s including CDMI (Cloud Data Management Interface) developed by SNIA which has gained ISO accreditation.

AWS%2013.jpg" width="435" height="250" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;"> 

Cloud and object storage access example via Cloud and Virtual Data Storage Networking

In addition to using Jungle Disk which manages my AWS keys and objects that it creates, I can also access my S3 objects via the AWS management console and web tools, also via third-party tools including Cyberduck.

PART III

AWS%2014.jpg" width="435" height="250" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;"> 

Cloud and object storage access example via Cloud and Virtual Data Storage Networking

AWS cloud storage gateway

In 2012 AWS released their Storage Gateway that you can use and try for free here using either an EC2 Amazon Machine Instance (AMI), or deployed locally on a hypervisor such as VMware vSphere/ESXi. In general, the gateway is an AWS alternative to using third product gateway, appliances of software tools for accessing AWS storage.

AWS%2015.jpg" width="465" height="220" style="cursor: pointer; max-width: 100%; height: auto; vertical-align: middle;"> 

Image courtesy of www.amazon.com

When deployed locally on a VM, the storage gateway communicates using the AWS API’s back to the S3 and EBS (depending on how configured) storage services. Locally, the storage gateway presents an iSCSI block access method for Windows or other servers to use.

There are two modes with one being Gateway-Stored and the other Gateway-Cached. Gateway-Stored uses your primary storage mapped to the storage gateway as primary storage and asynchronous (time delayed) snapshots (user defined) to S3 via EBS volumes. This is a handy way to have local storage for low latency access, yet use AWS for HA, BC and DR, along with a means for doing migration into or out of AWS. Gateway-cache mode places primary storage in AWS S3 with a local cached copy to reduce network overhead.

When I tried the gateway a month or so ago, using both modes, I was not able to view any of my data using standard S3 tools. For example if I looked in my S3 buckets the objects do not appear, something that AWS said had to do with where and how those buckets and objects are managed. Otoh, I was able to see EBS snapshots for the gateway-stored mode including using that as a means of moving data between local and AWS EC2 instances. Note that regardless of the AWS storage gateway mode, some local cache storage is needed, and likewise some EBS volumes will be needed depending on what mode is used.

When I used the gateway, a Windows Server mounted the iSCSI volume presented by the storage gateway and in turn served that to other systems as a shared folder. Thus while having block such as iSCSI is nice, a NAS (NFS or CIFS) presentation and access mode would also be useful. However more on the storage gateway in a future post. Also note that beyond the free trial period (you may have to pay for storage being used) for using the gateway, there are also fees for S3 and EBS storage volumes use.

How much do these AWS services cost'

Fees vary depending on which region is selected, amount of space capacity, level or durability and availability, performance along with type of service. S3 pricing can be found here including a free trial tier along with optional fees. 

Note that there is a myth that cloud vendors have hidden fees which may be the case for some, however so far I have not seen that to be the case with AWS. However, as a consumer, designer or architect, doing your homework and looking at the above links among others you can be ready and understand the various fees and options. Hence like procuring traditional hardware, software or services, do your due diligence and be an informed shopper.

Some more service cost notes include:

Note that with S3 Standard and RRS objects, there is not a charge for deletion of objects. 

As with Standard volumes, volume storage for Provisioned IOPS volumes is charged by the amount you provision in GB per month. With Provisioned IOPS volumes, you are also charged by the amount you provision in IOPS pro-rated as a percentage of days you have it in use for the month.

Thus important for cloud storage planning to know not only your space requirements, also IOP’s, bandwidth, and level of availability as well as durability. so for Standard volumes, you will likely see a lower number of I/O requests on your bill than is seen by your application unless you sync all of your I/Os to disk. Thus pay attention to what your needs are in terms of availability (accessibility), durability (resiliency or survivability), space capacity, and performance.

Leverage AWS CloudWatch tools and API’s to monitoring that matter for timely insight and situational awareness into how EBS, EC2, S3, Glacier, Storage Gateway and other services are being used (or costing you). Also visit the AWS service health status dashboard to gain insight into how things are running to help gain confidence with cloud services and solutions.

When it comes to Cloud, Virtualization, Data and Storage Networking along with AWS among other services, tools and technologies including object storage, we are just scratching the surface here.

Hopefully this helps to fill in some gaps giving more information addressing questions, along with generating new ones to prepare for your journey with clouds. After all, don’t be scared of clouds. Be prepared, do your homework, identify your concerns and then address those to gain cloud confidence.

We use S3 buckets to store logs from the application side. The logs stay on the server for seven days. After that, we transfer them to the S3 bucket, depending on the application, and the logs help us correctly troubleshoot problems. 

We can also host web apps in the historical bug bucket. If you have a domain name from Route 53, you can create a bug bucket over there and whatever is in the web applications folder. About 40 people at my company use S3. 

We use Amazon S3 for data storage purposes. We use the stored data for restoration or backup when there is a data leakage in websites and mobile applications. For enterprise-level applications, the stored data is also used for data analytics, backup, recovery, etc.

Amazon S3 is used for storage service. It's designed for 99.9% durability; hence, our data is highly protected against failures.

Our company uses the solution to store data from our analyses. More than 90% of our employees use the solution because our whole system and services are on the cloud. 

We use the solution to store BLOB data, analytical data, and large files.

We use Amazon S3 for storing images and documents. We use the solution for all file formats, and Amazon S3 gives us back URLs rendered on our end users' laptops.

I am using Amazon S3 for our retail portal sites and data scientist users are using it for analytics and Athena.

We use the solution to put our static web pages on the backend servers when we use Cloudfront if a transformation is required. When a request reaches Cloudfront, we have to mask some information in the response by transporting it.

The solution masks or transforms the request so the backend can understand them. For example, We have a proxy server in between the NGINX engine, and when a request hits the NGINX engine, it has a different format. The NGINX engine converts and secures the information before hitting the Cloudfront. We use the Lambda function which is a Python program so the code body stays in the Amazon S3 bucket and when a response is returned from the backend server we can mask some of the information, keeping only the required information for the end response.

I also use Amazon S3 for batch processing. Two gigabytes is a lot of data, especially if we are adding or updating information. If we have a Java application, it can take a long time to process two gigabytes of data. I split the data into multiple 50 or 30-MB files to make it easier to process.  I created a manifest file that specified which rows belonged to which files. Then I used the solutions batch processing with Lambda to create a pipeline that performs batch operations.

I use the solution for life cycle management. The information goes from the standard storage to the infrequent, and after 30 days it is moved to the glacier.

We have multiple use cases for Amazon S3. We use Amazon S3 for storing static data and as a static site load balancer.

We use this solution as readily available internet storage.

We use Amazon s3 to store objects like data in my environment.

Our company uses the solution for backups, data pumps, files, and logs. Our projects connect to the solution rather than individual users connecting to it. 

I'm a consultant who provides services for companies of various sizes here in South Africa. Our company is still small, with only about 20 clients. I use S3 for hosting my websites and other storage services. S3 can be used for static and dynamic websites as well as file storage. In terms of dynamic websites, I use S3 in addition to other technologies in a computerless environment. In other words, I don't need a virtual machine to have a dynamic website hosted on Amazon Web Services.

We use Amazon S3 for storing documents and hosting website content.

The primary use case of this solution is for storage.

The primary use case of this solution is for the storage of large amounts of data.

It's a public cloud deployment model that anyone can deploy.

There is no need to upgrade as they don't have versions, it's a managed service, it's hybrid. There are changes that we make internally. It's Amazon S3.

Amazon S3 can be used for central storage, backup, log, and storage.

We use Amazon S3 to store source files. We load the data we ingest from the source system into the solution, which is then consumed on Snowflake.

We have a data warehouse and a data lake using Amazon S3. Additionally, we use it for application streaming.

We have our computer systems that have storage but we do use the cloud storage available from Amazon S3. However, it depends on the type of data we have and whether we use the cloud or not.

I use Amazon S3 as storage and a platform for installing various types of software.

Most people who need to store files usually turn to Amazon S3. The use case for this is not only for me but for anyone building technology who needs to store files.

We are using Amazon S3 for storage. This is a Cloud solution. We have developed a new application which has partly been released to the end customer, but not the complete solution. In the housing market, we have a lot of buying and they use the application to manage their lectures and receipts. This is what we are using Amazon S3 for the storage.

Amazon S3 is our file storage solution. We use it for all of the backups of our storage and databases.

We use Amazon S3 for a file storing system. The solution is cloud-based.

We're working with 10-15 applications.

Our primary use of S3 is for storing data. I'm a senior associate. 

Our company has 30 staff members who use the solution for data storage. 

We use this solution for storing the input-output of our processing. We are customers of Amazon. 

Our company uses the solution for deep CI backups. We have thousands of customers who use the solution and access our application. 

When we develop applications, it generates files. We use Amazon S3 as a backup and storage solution. 

We use the solution to store deployment logs and tech information.

We are using the solution to secure the content. 

We are using Amazon S3 for image and video files.

My use case for Amazon S3 is storage, particularly for unstructured data.

We predominantly use this solution for our data storage, primarily making DB backups. I'm a lead data engineer.

We use it for Dollar Bank to provide them with Telematics and to transport IoT solutions. We host our full term on X-ray and the data we receive from the devices is put into the Amazon S3 bucket by using the Amazon Kinesis Data Firehose. The data in the Amazon S3 bucket is synced into Google BigFix.

We are very flexible with our use cases and it fits most clients.

Currently, I am a cloud solution architect. I consult with developers and set up the architecture needed to deploy a solution. If a solution needs to be deployed within AWS, we sit down with the client and discuss the services that are most appropriate for that solution and design the architecture, which is given to the review board for approval. The AWS services we consume are dependent on the solution. Users from various departments will use the solution, depending on what's being deployed there. 

It is mainly for transferring the files we are using from AWS, where we have to transfer the files from AWS to a non-cloud environment.

We use it for static website hosting, storing data, and second data of some of the websites. We can then easily sync the data whenever we have the need to do so.

Some of our websites are 2 or 4 pages and have static content. We can simply host it by creating a bucket, putting the content of the website there, and making it available to the public with the Make Public feature.

We use Amazon S3 to store data.

We are using Amazon services to create the SSX and other services.

We primarily use the solution for file storage. 

We are using Amazon S3  for short-term or long-term storage.

I am using Amazon S3 for storing all the organization data, which comes from a data pipeline.

Most of the files and some of the services connect, using the Amazon service. Our application foundation is on that and we are writing our solutions right now. After that, we will begin to sell it to other customers.

We use the solution for data blocking and analysis. It is also used as a storage solution, gene access and maintaining access.

I have used Amazon for my personal projects. I was using it for my personal things and for test purposes. I didn't use any use case. I just put some data in and read it back again.

Usually, I use it to make deployments and service solutions in the cloud.

Our primary use case of S3 is as an online storage solution. For example, in one of our projects, we were working on a document management system, so there were some documents that we processed and then stored on S3. We also used S3 to store some logos and images. 

This is a service on AWS, so it's cloud-based. 

I am managing the S3 storage solution from Amazon. From Microsoft, I'm managing the Azure administration. I handle that. As a part of that, I need to create the virtual machines, a point to site to VPN and Azure AD, AD synchronization with Windows server, a database server, mail server, and web server. I administrate all that.

We use Amazon S3 for storing files and documents on the cloud. I fill out the user whenever the files they use are downloaded. That is fully automated. After a few days, it can delete it. I've done the policy for the automated deletion.

Amazon S3 is great. We are using it for storage solutions.

We are making custom applications to the cloud.

We are service providers.

In terms of use cases, where a company wants to deploy their applications instead of building their servers on the premise, on the company premise. They can just, instead of buying servers and setting up networks and everything, buy space online on the AWS cloud. That way, you don't have to buy an expensive server. You already have servers online. You deploy your application. That's the main use case we are handling now. For example, if you have an in-house ERP, you want to move everything to the cloud. We migrate clients, for example. from on-premise to the cloud. If you have heavy tasks, there are computations that require heavy, powerful machines.

One of our clients required certain files to be processed and stored. As another separate service, they wanted a separate integration to capture those files or pull those files. As a storage location, we used S3. It was cloud storage between two services. It was an integration between two services.

We have deployed Amazon S3 in two places, in Ireland and Northern Ireland. These act as storage for all the data, which is there. We have up to three servers there, one of these is an SQL server and one is Windows.

We use Amazon S3 for data extraction because a few of the source information for our organization requires us to rely on it for any data point input.

Because it was only mined for the purpose of data extraction.

I work as an integrator of Amazon S3

I use Amazon S3 mainly for daily backups of our database.

Set up a performance and stress-based test suite to measure the endurance of a web application on Amazon AWS cloud system using S3 as the back-end storage infrastructure. 

We use this solution for cloud-based data storage.

We are using Amazon S3 mostly for block storage.

We primarily use the solution in order to share static files among web applications and mobile applications.

We use this solution to store the historical data of the customer. Additionally, we are creating JSON format data while running our application and we are pushing this JSON format data to the Amazon S3.

We haven't released the Amazon S3 platform in a production environment. It's more for prototyping and POC so far.

Normally we use it for the data dump. It's something similar to a data lake wherein we use it to obtain the data. 

The primary use case of this solution is for storage.