2.1.2. Data Tiers and Retention
💡 First Principle: Security data has two jobs with opposite economics. Detection needs a small window of recent data, queried constantly and fast. Investigation and compliance need years of data, queried rarely. Paying real-time prices to keep years of logs wastes money; keeping detection data in cold storage blinds the SOC. Sentinel's tiers let you set retention per table so each data set lives where its job is cheapest.
There are three tiers to keep straight:
| Analytics tier | Data lake tier | XDR default tier | |
|---|---|---|---|
| Purpose | Real-time detection, hunting, investigation | Low-cost long-term storage and large-scale analysis | Defender XDR advanced hunting |
| Default retention | 30 days (Sentinel solution tables can be kept 90 days at no extra charge) | Same as analytics retention by default (mirrored) | 30 days, included with XDR licensing |
| Maximum | Analytics retention up to 2 years | Total retention up to 12 years | Extend by ingesting into Sentinel tables |
| Analytics rules, workbooks, playbooks | ✅ Full support | ❌ Not for real-time rules | Custom detections in Defender XDR |
| How you query | Interactive KQL, hunting, rules | KQL (incl. joins and unions), KQL jobs, notebooks, search jobs | Advanced hunting |
| Cost shape | Higher ingestion, queries included | Minimal ingestion, pay for queries and storage | Included for 30 days |
Two settings define each table: analytics retention (how long data stays hot) and total retention (how long it's kept at all). By default the lake mirrors analytics data for the same period, so total retention can stretch past analytics retention: for example, 90 days hot and two years total means days 91 through 730 live only in the lake.
The consequences of changing settings are testable. Shortening analytics retention moves older data out of the hot tier immediately (it stays in the lake if total retention covers it). Shortening total retention gives you a 30-day grace period before deletion, so mistakes are reversible. Switching a table from analytics to lake-only stops every real-time analytics rule and hunting query against it. XDR tables such as DeviceProcessEvents keep 30 days in advanced hunting; to hunt further back or run Sentinel rules against them, you ingest them into Sentinel with the Defender XDR connector and set retention there (days 31–90 in the analytics tier carry ingestion charges with free storage).
Choosing a tier is a requirements exercise. High-value detection sources (sign-in logs, endpoint and identity alerts, firewall denies you alert on) belong in the analytics tier. High-volume, low-detection-value sources (verbose firewall allow logs, NetFlow, proxy logs you only search during investigations) are classic data lake candidates. Retention mandates measured in years point at total retention in the lake.
⚠️ Exam Trap: "Keep firewall logs for seven years at the lowest cost, and continue to detect command-and-control traffic in real time" needs both tiers: analytics retention for the detection window, total retention in the lake for the mandate. "Move the table to the data lake" alone breaks the detection requirement.
Reflection Question: A team moves its CommonSecurityLog table to lake-only to cut costs, and a week later a scheduled analytics rule for beaconing stops producing alerts. Explain why, and describe a tiering configuration that keeps most of the savings without losing the detection.
