2.4.3. S3 Performance: Transfer Acceleration, Multipart Uploads, and Lifecycle
💡 First Principle: S3 is not just object storage — it's a globally distributed system with multiple performance optimization levers. The right combination depends on whether your bottleneck is upload speed, download latency, storage cost, or data retrieval frequency.
Request Rate Performance: S3 scales horizontally based on key prefixes. Each prefix supports 3,500 PUT/COPY/POST/DELETE requests and 5,500 GET/HEAD requests per second. There is no limit on the number of prefixes, so to go beyond these rates, spread requests across more prefixes and parallelize them (10 prefixes can serve up to 55,000 GET/s). Since 2018 you no longer need to randomize key names: what matters is how many prefixes the requests are spread across, not how the names are formed.
❌ Anti-pattern:
logs/2025/01/01/server1.log,logs/2025/01/01/server2.log(same prefix, requests concentrate) ✅ Better:logs/server1/2025-01-01.log,logs/server2/2025-01-01.log(one prefix per server, read and written in parallel; no random hashing needed)
S3 Transfer Acceleration: Routes uploads through AWS CloudFront edge locations using optimized network paths. Instead of uploading directly from your location to the S3 region, your upload goes to the nearest edge location and then travels through AWS's internal backbone. Best for:
- Large files (>1GB)
- Uploads from geographically distant clients
- Cross-continent transfers
Transfer Acceleration uses a special endpoint: bucket-name.s3-accelerate.amazonaws.com
Multipart Upload: Required for objects >5GB; recommended for objects >100MB. Benefits:
- Improved throughput (parallel part uploads)
- Ability to restart individual failed parts without restarting the entire upload
- Begin uploading before you know the final file size
- Parts can be 5 MB to 5 GB (up to 10,000 parts), so mid-size files also gain from parallel parts; combine this with many concurrent file uploads for aggregate throughput
⚠️ Incomplete multipart uploads accumulate cost. If a multipart upload is initiated but never completed (e.g., application crash), the parts remain in S3 and you're billed for them. Use an S3 Lifecycle policy to automatically abort incomplete multipart uploads after a defined period (e.g., 7 days). In a versioned bucket, old versions need their own noncurrent-version expiration rule (see 3.3.4).
S3 Lifecycle Policies automate transitions between storage classes based on object age or other criteria:
| Storage Class | Access Pattern | Min Storage Duration | Cost Relative to S3 Standard |
|---|---|---|---|
| S3 Standard | Frequent access | None | Baseline |
| S3 Standard-IA | Infrequent, but rapid access needed | 30 days | Cheaper storage, retrieval fee |
| S3 One Zone-IA | Infrequent, single AZ acceptable | 30 days | Cheapest IA option |
| S3 Glacier Instant Retrieval | Archive with ms retrieval | 90 days | Low storage, ms retrieval |
| S3 Glacier Flexible Retrieval | Archive, minutes–hours retrieval | 90 days | Very low storage |
| S3 Glacier Deep Archive | Long-term archive, 12h retrieval | 180 days | Lowest storage cost |
| S3 Intelligent-Tiering | Unknown or changing patterns | None (no retrieval fee) | Monitoring fee per object |
AWS DataSync is a managed service for large-scale data transfer between on-premises storage and S3 (or EFS, FSx). It's faster than using the AWS CLI or custom scripts because it handles parallelization, checksumming, and retry automatically. Use DataSync for migration projects or continuous sync of on-premises data to AWS. For scripted jobs, aws s3 sync copies only new or changed files (compared by size and last-modified time) and skips the rest, whereas aws s3 cp --recursive re-uploads everything on every run.
⚠️ Exam Trap: S3 Standard-IA and Glacier Flexible Retrieval have minimum storage duration charges. If you store an object for 15 days and delete it, you're still billed for 30 days (Standard-IA) or 90 days (Glacier). The exam may test whether you know when these classes are cost-effective.
Reflection Question: An application generates 10,000 log files per second. After 7 days, logs are queried rarely; after 90 days, they're almost never accessed; after 365 days, they must be retained for compliance but never accessed. Design the S3 Lifecycle policy.