3.1.5.1. Design for Data Lifecycle Management
3.1.5.1. Design for data lifecycle management
💡 First Principle: Automating data movement and retention across storage tiers based on predefined rules is essential for optimizing costs, ensuring compliance, and reducing manual operational overhead.
Scenario: You are managing a large volume of IoT telemetry data stored in Azure Blob Storage. This data is actively analyzed for the first 90 days, then accessed only occasionally for reporting for the next year, and finally needs to be retained for 5 years for auditing before being permanently deleted.
Data lifecycle management (DLM) is a feature in Azure Blob Storage that automates the process of tiering and deleting data based on predefined rules.
Key Design Considerations:
- Policy Definition: Define lifecycle rules using data age, last access time, or tags to trigger transitions or deletions.
- Tiering: Automate movement between Hot (frequent), Cool (infrequent), and Archive (rare) tiers to match usage and minimize costs.
- Deletion: Set policies for automatic deletion of expired or unneeded data, reducing storage bloat and compliance risks.
- Compliance: Ensure lifecycle policies align with legal and regulatory requirements (e.g., GDPR, HIPAA) for retention and deletion.
- Cost Savings: Automated tiering and deletion lower storage expenses by keeping only active data in premium tiers and removing obsolete data.
Design decisions in practice:
| Decision | Guidance |
|---|---|
| What drives the rule | Blob age since creation, since last modification, or since last access. Last-access tracking must be enabled explicitly and adds a small cost, but it is the only signal that reflects real usage rather than assumed usage |
| Where rules apply | Scope by prefix and by blob index tag, so a policy can treat raw/ differently from curated/ in the same account |
| Snapshots and versions | Have their own rules. Forgetting them is how an account keeps paying Hot prices for prior versions of data whose current version was tiered long ago |
| The delete action | Lifecycle can delete as well as tier. Combined with a time-based immutability policy, the delete simply fails until retention expires (a legal hold has no expiry and blocks it until someone explicitly clears the hold) — which is the intended interaction, not a conflict |
| Account type | Lifecycle management runs on general-purpose v2 accounts (and legacy Blob Storage accounts) — a GPv1 account must be upgraded first. Premium block blob accounts support lifecycle deletion but not tiering |
| Proving it ran | Each run raises a LifecyclePolicyCompleted event with per-action counts, and the blob resource logs — sent to Log Analytics through a diagnostic setting — record each operation the policy performed with a timestamp, under the ObjectLifeCycleScanner user agent. Azure Policy can show that a lifecycle policy is configured, never that it executed |
Tier by the read pattern, and know the two costs that bite. Each cooler tier has a minimum retention period, and moving data out early incurs an early deletion charge for the remainder — so tiering data that turns out to be read repeatedly costs more than leaving it alone. And rehydrating from Archive takes hours, at standard or high priority, which is why any requirement mentioning immediate access rules Archive out. Standard priority can take up to 15 hours; high priority costs more and can finish in under an hour for blobs under 10 GB, so a deadline of a few hours needs high priority. Moving between the online tiers (Hot, Cool, Cold) needs no rehydration — only Archive is offline.
Changing a tier is an in-place operation. Set Blob Tier changes an existing blob's tier without copying it, and can be applied in bulk through a blob batch request or a script — or by correcting the lifecycle rule. The account's default access tier is different: it applies only to blobs with no explicitly set tier, cannot be Archive, and never moves archived blobs.
⚠️ Exam Trap: a lifecycle policy is evaluated once a day, not continuously. A scenario expecting a blob to move tiers the moment it stops being read is describing behaviour the feature does not have. Rules apply to every existing blob that matches their filters, not only to blobs created afterwards, and a new or changed policy can take up to 24 hours to run for the first time — so a policy that seems idle is usually disabled, filtered to the wrong prefix or blob type, or simply has not run yet.
⚠️ Common Pitfall: Creating lifecycle policies based on creation date when access patterns are the more relevant factor. For unpredictable access, using the "last accessed" condition in a lifecycle rule is more effective for cost optimization.
Key Trade-Offs:
- Rule-based (Lifecycle Policy) vs. Access-based (Intelligent Tiering): Lifecycle policies are ideal for predictable aging patterns. For unpredictable access, Azure Storage's Intelligent Tiering feature (which automatically moves data based on actual access) might be a better, though slightly different, approach.
Practical Implementation: Conceptual Lifecycle Rule
// A rule to move blobs with prefix 'iot-data/' to Cool after 90 days,
// then to Archive after 365 days, and delete them after 1825 days (5 years).
{
"rules": [
{
"name": "iot-data-lifecycle",
"enabled": true,
"type": "Lifecycle",
"definition": {
"filters": {
"blobTypes": [ "blockBlob" ],
"prefixMatch": [ "iot-container/iot-data/" ]
},
"actions": {
"baseBlob": {
"tierToCool": { "daysAfterModificationGreaterThan": 90 },
"tierToArchive": { "daysAfterModificationGreaterThan": 365 },
"delete": { "daysAfterModificationGreaterThan": 1825 }
}
}
}
}
]
}
Reflection Question: How does designing for data lifecycle management (DLM), using policy definitions to automate data movement across storage tiers and eventual deletion, fundamentally optimize costs and enforce data retention and compliance by reducing manual effort?
