The Core Storage Services

Lesson purpose

For many organizations, data is the most valuable asset, and it is often split between the cloud and offices or data centers. This lesson covers where data lives in AWS and how it moves: Amazon S3 as the core object store, S3 Glacier for low-cost archives, Storage Gateway for connecting on-premises servers to cloud storage, the Snow Family for physically shipping huge data sets, and supporting services for file transfer, managed file systems, and disaster recovery. It supports CLF-C02 objectives 2.2, 2.3, 3.1, 3.3–3.6, and 3.8.

Learning objectives

Explain S3 objects, keys, buckets, bucket naming, Regions, and prefixes (“folders”).
Distinguish durability from availability.
Choose an S3 storage class based on access pattern, availability, resilience, and cost.
Control S3 access with bucket policies, user policies, ACLs, and Block Public Access.
Explain S3 encryption, versioning, and life-cycle rules.
Describe S3 Glacier archives, vaults, and retrieval options.
Compare the Storage Gateway file, volume (stored and cached), and tape gateways.
Explain when to use the Snow Family instead of a network transfer.
Identify AWS Transfer Family, Amazon FSx, and AWS Elastic Disaster Recovery.

1. Block storage vs. object storage

Lesson 7 introduced EBS, which is block storage: a virtual disk attached to one instance, formatted with a file system. Amazon S3 is object storage: whole files (objects) are stored and retrieved over the web through an API, with no disk to attach.

	EBS (block)	S3 (object)
Unit	Blocks on a volume	Whole objects (files)
Access	Attached to an instance in one AZ	Over HTTPS from anywhere, with permission
Typical use	OS drives, databases	Backups, media, logs, data lakes, static websites
Capacity	Fixed volume size you choose	Practically unlimited

2. Amazon S3 fundamentals

Amazon Simple Storage Service (S3) stores and retrieves practically unlimited amounts of data from anywhere. AWS calls it “storage for the internet,” but access is private unless you grant it. Many AWS services use S3 to store logs or read data for analytics, and S3 can even host static websites.

Objects

An object is any file — text, images, video, database backups — up to 5 TB each.
The object’s name is its key, up to 1,024 bytes. Keys can use letters, numbers, and characters such as ! - _ . * ( ).

Buckets

Objects live in a bucket, a flat container that can hold unlimited objects.
Bucket names are 3–63 characters and globally unique across all of AWS (tip: add a random string to a friendly name).
Each object key is unique within its bucket, so every object has its own URL, for example https://benpiper.s3.amazonaws.com/text.txt.
Each bucket lives in one Region. Data never leaves that Region unless you configure it to — for example, with cross-Region replication, which makes separate copies in another Region. This supports latency, cost, and data-residency requirements.

Prefixes (“folders”)

S3 has no real folders. A slash in the key creates the appearance of one: production/database.sql and development/database.sql are two unique objects in the same bucket, and the console displays them as folders.

Paying for S3

Uploading (data in) is free.
Downloading to the internet (data out) is charged after a small free allowance.
Storage is billed monthly per GB, based on the storage class.
Requests (PUT, GET, and so on) and some retrievals also carry small charges.

3. Durability vs. availability

Durability — the likelihood an object will not be lost over a year. S3 is designed for 99.999999999 percent (“eleven nines”). Store 100 billion objects and you would expect to lose about one per year. At 99.99 percent durability, you would lose about 10 million.
Availability — the percentage of time an object can be retrieved. A patient’s medical records need both high durability and high availability.

A quick way to remember it: durability is “will my data still exist?”; availability is “can I get to it right now?”

4. S3 storage classes

Every class except the legacy Reduced Redundancy option is designed for eleven nines of durability. The choice comes down to how often you access the data, how fast you need it back, and how many Availability Zones hold it.

Storage class	Access pattern	Availability (design)	AZs	Notes
S3 Standard	Frequent	99.99%	≥3	Default; millisecond access
S3 Intelligent-Tiering	Unknown or changing	99.9%	≥3	Moves objects between tiers automatically; small monitoring fee
S3 Standard-IA	Infrequent, but needed quickly	99.9%	≥3	Retrieval fee; 128 KB minimum billable size
S3 One Zone-IA	Infrequent, re-creatable	99.5%	1	Cheapest millisecond-access option; data can be lost if the AZ is destroyed
S3 Glacier Instant Retrieval	Rarely, but needed in milliseconds	99.9%	≥3	Medical images, news archives
S3 Glacier Flexible Retrieval (formerly “Glacier”)	Rarely; minutes to hours is fine	99.99% after restore	≥3	Must restore before reading
S3 Glacier Deep Archive	About once a year or less	99.99% after restore	≥3	Lowest cost; restore up to 12 hours (48 for bulk)
Reduced Redundancy (RRS)	Legacy	99.99%	≥3	Only 99.99% durability; AWS recommends against it

Storage prices change and vary by Region; for a sense of scale, Standard costs a few cents per GB-month and Deep Archive about one-tenth of a cent. Use the AWS pricing page for current numbers.

Key points by access pattern:

Frequent: Standard (RRS is legacy and should be migrated away from).
Infrequent: Standard-IA and One Zone-IA (millisecond access, lower availability), the Glacier classes for archives. IA and Glacier Instant Retrieval classes bill objects smaller than 128 KB as 128 KB.
Unknown or mixed: Intelligent-Tiering — an object not accessed for 30 days moves to a lower-cost tier and moves back when accessed. Objects under 128 KB stay in the frequent tier.

Modern note: S3 also offers S3 Express One Zone, a single-AZ class built for very high-performance workloads. It is newer than the book and rarely a cost-saving choice.

Check for understanding: Nightly thumbnails that can be regenerated, read about once a month. (One Zone-IA.) Seven years of tax records that must be kept but are almost never read. (Glacier Deep Archive.) A data set whose access pattern no one can predict. (Intelligent-Tiering.)

5. Controlling access to S3

By default, a new bucket and its objects are private — only the account that owns them can access them.

Three ways to grant access (they can be combined):

Bucket policies — resource-based JSON policies attached to a bucket. They can cover all or some objects, grant access to specific principals or other accounts, and can grant anonymous (public) read access, for example for website images.
User policies — IAM identity policies (Lesson 5) attached to users, groups, or roles. They grant access to principals in your account but cannot grant public access.
Access control lists (ACLs) — a legacy method that grants access to other AWS accounts or anonymous users, but not to specific IAM principals.

Modern practice, which differs from the book:

S3 Block Public Access is turned on by default for new buckets and blocks any public grant until it is deliberately turned off.
New buckets default to Object Ownership: bucket owner enforced, which disables ACLs entirely. AWS recommends policies instead of ACLs.

6. Encryption, versioning, and life cycles

Encryption at rest

Server-side encryption — S3 encrypts the object when saving it and decrypts it when an authorized user reads it. Easiest to use; AWS manages the keys (SSE-S3) or they are managed in KMS (SSE-KMS).
Client-side encryption — you encrypt before upload and decrypt after download. More work, and a lost key means lost data, but AWS can never read the content.

Correction to the book: the book says uploaded objects are stored unencrypted unless you choose encryption. Since January 2023, S3 automatically encrypts every new object with SSE-S3 (Lesson 5).

Versioning

Without versioning, uploading an object with an existing key overwrites the original.
With versioning enabled, S3 keeps every version. Deleting an object adds a delete marker that hides it, and earlier versions can still be restored. (The book says this happens when versioning is disabled; it is the reverse — without versioning, a delete removes the object permanently.)
Versioning is off by default, applies to the whole bucket, and once enabled can only be suspended, not removed.
There is no limit on versions, and each version is billed as stored data, so pair versioning with life-cycle rules.

Caution: deleting a bucket deletes everything in it.

Object life-cycle configurations

Life-cycle rules control cost automatically:

Transition actions move objects to a cheaper class after a set age (for example, Standard → Standard-IA after 90 days).
Expiration actions delete objects, or old versions, after a set age (for example, after 365 days).

Classic example: web server logs start in Standard, move to Standard-IA at 90 days, and are deleted after another year.

Exercise 8.1: create a versioned bucket

In the S3 console, choose Create bucket.
Enter a globally unique name and choose a Region.
Enable Bucket Versioning; leave Block Public Access on and other settings at their defaults.
Create the bucket, open it, upload a file, upload a changed copy with the same name, and use Show versions to see both.
Delete the object and observe the delete marker. Empty and delete the bucket when finished.

7. S3 Glacier archives

S3 Glacier provides extremely low-cost, eleven-nines-durable storage for data you hope never to need, such as long-term backups.

The original, standalone Glacier service uses its own vocabulary:

Archive — a block of data from 1 byte to 40 TB, often a ZIP or TAR file of many files.
Vault — a Regional container for archives (like a bucket). Vault names must be unique within a Region, not globally.
Vaults can be created in the console, but archives are uploaded, downloaded, and deleted with the CLI, SDKs, or third-party tools.

Modern note: most people now use Glacier through the S3 Glacier storage classes (Section 4) in ordinary S3 buckets with life-cycle rules, and AWS steers new customers toward those classes instead of the older vault-based service.

Retrieval is a two-step process

First start a retrieval (restore) job, then download the data when the job finishes.

Option	Flexible Retrieval	Deep Archive	Cost
Expedited	1–5 minutes (archives over 250 MB may take longer)	Not available	Highest
Standard (default)	3–5 hours	Within 12 hours	Middle
Bulk	5–12 hours	Within 48 hours	Lowest

Provisioned capacity can be purchased to guarantee expedited retrievals are available.

8. AWS Storage Gateway

Storage Gateway connects on-premises servers to AWS storage using standard storage protocols, so no special software is installed on those servers. You run the gateway as a virtual machine on VMware ESXi or Microsoft Hyper-V (a hardware appliance and an EC2 version also exist), and it handles transfers to AWS.

Gateway type	Protocol	How it works	Use case
File gateway	NFS, SMB	Files stored as S3 objects; recent data cached locally	On-premises file shares backed by S3 (versioning, life cycles, encryption, replication all apply)
Volume gateway — stored	iSCSI	All data kept locally; backed up asynchronously to S3 as EBS snapshots; volumes 1 GB–16 TB	Need uninterrupted local access to all data
Volume gateway — cached	iSCSI	All data in S3; only frequently used data cached locally; volumes 1 GB–32 TB	Limited local storage; accept that some data depends on the connection
Tape gateway	iSCSI	Virtual tapes (100 GB–2.5 TB) in an S3-backed virtual tape library; archive to a Glacier-backed virtual tape shelf	Replace physical tape with existing backup software (Veritas, Commvault, Microsoft DPM)

Volume snapshots can be restored to an on-premises gateway volume or to an EBS volume for an EC2 instance. Archived tapes take hours to retrieve back to the library.

9. The AWS Snow Family

When a network transfer would take days or weeks, AWS ships a rugged physical device. You load the data locally, ship the device back, and AWS imports the data into S3. Import to S3 carries no data transfer charge.

Why it matters: moving 40 TB over a 1 Gbps link takes more than four days even at full speed.

Book content, with modern status:

Snowcone — the smallest device (about 9×6×3 inches), with a small amount of compute and storage. AWS has since discontinued Snowcone.
Snowball — a storage appliance for migrating data centers, backups and recovery, and distributing large content sets. The book describes 50 TB and 80 TB models, 10 Gbps network ports, a 10-day included use period, and extra daily fees after that.
Snowball Edge — adds local compute and storage at the edge:
- Storage Optimized — the most storage, moderate compute.
- Compute Optimized (and a GPU version in the book) — more vCPUs and memory for local processing and machine learning.
- Runs EC2 instances and Lambda functions locally, provides local S3 buckets and NFS file shares, and can be clustered for high availability. Local instances use IP addresses on your network, not a VPC.
Snowmobile — a 45-foot shipping container for exabyte-scale migrations. AWS has retired Snowmobile.

Feature	Snowball	Snowball Edge
Transfer data to and from S3	Yes	Yes
Local EC2 instances and Lambda	No	Yes
NFS file server	No	Yes
Local S3 buckets	No	Yes

Security features to teach:

Tamper-resistant enclosure with a Trusted Platform Module (TPM); AWS checks for tampering before importing data.
Data is encrypted in transit and always encrypted at rest with 256-bit AES, enforced by the Snowball client or S3 adapter, so it is never stored unencrypted.
After each job, AWS erases the device following NIST SP 800-88 media sanitization guidelines.

Snow device availability has changed over time; check the Snow Family pages for what can be ordered today. For network-based migration, AWS DataSync moves data online between on-premises storage and AWS.

AWS Transfer Family

A managed file-transfer service for moving data into and out of S3 and Amazon EFS (a managed Linux file system, covered later) using familiar protocols and tools such as FileZilla, OpenSSH, and WinSCP:

SFTP — SSH File Transfer Protocol (encrypted); internet-facing or VPC endpoints.
FTPS — FTP over TLS/SSL (encrypted); internet-facing or VPC endpoints.
FTP — unencrypted, so only VPC-hosted endpoints are allowed.

Access can be limited by IP address, and users can authenticate locally or through identity providers such as Active Directory.

Amazon FSx

Fully managed file systems using industry-standard protocols:

FSx for Windows File Server — SMB shares with NTFS and Active Directory integration (the Windows counterpart to EFS).
FSx for Lustre — a high-performance file system for compute-intensive Linux clusters (machine learning, HPC).
FSx for OpenZFS and FSx for NetApp ONTAP — bring those file systems’ features to AWS.

AWS Elastic Disaster Recovery (AWS DRS)

Install the replication agent on source servers — physical, virtual, or EC2 — and DRS continuously replicates them to AWS for point-in-time recovery.
Failover: when disaster strikes, DRS launches recovery instances in EC2 from the replicated data.
Drills: test the plan ahead of time by launching drill instances (up to 100 per drill) and checking the results.
Failback: return to the original site — to another AZ or Region in AWS, or to on-premises servers using the Failback Client boot ISO (or the mass failback client for VMware vSphere).

Classroom application

Lab: Complete Exercise 8.1, then add a life-cycle rule that transitions objects to Standard-IA after 30 days and expires noncurrent versions after 90 days.
Storage class sort: Give students a list — daily sales reports, a 10-year legal archive, product images for a website, regenerable thumbnails, unpredictable research data — and have them choose a class and justify availability, AZ count, and cost.
Public access check: Show that a new object URL returns “Access Denied,” and discuss why Block Public Access exists (many data leaks came from public buckets).
Transfer math: Estimate the time to send 100 TB over a 100 Mbps and a 1 Gbps link, then discuss when a Snow device or DataSync makes sense.
Hybrid scenario: A clinic wants to keep its Windows file share but back it up to the cloud and eventually retire tape. Which gateway types fit? (File gateway for the share; tape gateway for existing backup software.)

Common misconceptions

“S3 is a disk I can attach to EC2.” That is EBS; S3 is object storage accessed over the network.
“Bucket names only need to be unique in my account.” They must be unique across all AWS accounts.
“S3 has real folders.” Folders are just key prefixes containing slashes.
“Durability and availability mean the same thing.” Durability is about not losing data; availability is about reaching it now.
“One Zone-IA is as safe as Standard-IA.” Same designed durability within the zone, but the data lives in one AZ and can be lost if that AZ is destroyed.
“Glacier data can be opened instantly.” Only Glacier Instant Retrieval allows that; Flexible Retrieval and Deep Archive require a restore job first.
“Versioning is on by default.” It must be enabled, and every version costs storage.
“Uploading to S3 costs money.” Data in is free; storage, requests, retrievals, and data out are charged.

Lesson summary

S3 stores objects (up to 5 TB each) identified by keys in globally uniquely named buckets that live in a single Region, with prefixes providing folder-like organization. S3 is designed for eleven nines of durability, and storage classes trade availability, AZ count, retrieval speed, and price: Standard for frequent access, Intelligent-Tiering for unknown patterns, Standard-IA and One Zone-IA for infrequent access, and Glacier Instant Retrieval, Flexible Retrieval, and Deep Archive for archives. Buckets are private by default; bucket policies, user policies, and legacy ACLs grant access, and Block Public Access guards against accidental exposure. Objects are encrypted at rest by default, versioning protects against overwrites and deletes, and life-cycle rules transition and expire objects to control cost. Glacier stores archives in vaults and returns data through Expedited, Standard, or Bulk retrieval jobs. Storage Gateway links on-premises servers to AWS through file, volume (stored or cached), and tape gateways. The Snow Family ships encrypted, tamper-resistant devices for offline transfer and edge computing. AWS Transfer Family provides managed SFTP, FTPS, and FTP; Amazon FSx provides managed Windows, Lustre, OpenZFS, and NetApp ONTAP file systems; and AWS Elastic Disaster Recovery replicates servers for failover, drills, and failback.