4.3.3. Notebooks and the Sentinel MCP Server

💡 First Principle: Some analysis outgrows a single KQL query: machine learning over months of sign-ins, graph analytics, enrichment from external APIs, or repeatable investigation playbooks written as code. Notebooks bring Python and Spark to the data lake for that work. The MCP server takes a different route to the same data, giving AI agents and assistants a standard, governed way to query security data in natural language.

Data lake notebooks are Jupyter notebooks run from Visual Studio Code with the Microsoft Sentinel extension. After signing in, you browse lake tables and jobs, create a notebook, and choose a Spark runtime pool: small for exploration, medium for joins and aggregations, large for heavy machine learning. Python code uses the MicrosoftSentinelProvider class to read tables (read_table) into Spark DataFrames, transform them, and write results (save_as_table) to lake or analytics-tier tables. Notebooks can also be scheduled as jobs that summarize, transform, or analyze data on a recurring basis. Their limits differ from KQL jobs in a testable way: notebook jobs time out after 8 hours, at most three run concurrently, and extra runs are queued, whereas KQL jobs cap at five concurrent runs and reject the extras. Practical constraints: data lake onboarding is required, sessions time out after a period of inactivity, only supported libraries are available (no arbitrary pip installs), and writing new analytics-tier tables requires the data lake managed identity to have Log Analytics Contributor on the workspace. Classic Sentinel notebooks with MSTICPy (Microsoft Threat Intelligence Center's Python library) remain an option for workspace-based investigations.

The Microsoft Sentinel MCP server implements the Model Context Protocol, an open standard for how AI applications connect to tools and data. In MCP terms, the host is the AI application (for example, Visual Studio Code with an AI assistant), the client maintains the connection, and the server provides tools and context. Sentinel's MCP server is fully hosted by Microsoft (nothing to deploy), uses Microsoft Entra identity, and exposes scenario-focused tool collections:

• Data exploration: find relevant tables and query data lake data in natural language without knowing schemas or writing KQL, plus an entity analyzer that enriches users and URLs with verdicts.
• Triage and threat hunting: tools that support incident triage and hunting workflows.
• Agent creation: describe an automation in natural language to build Security Copilot agents that reason over your security data.

For hunting, the two meet naturally: an analyst working in Visual Studio Code can use an AI assistant connected to the Sentinel MCP server to find the right tables and draft queries, then move heavy analysis into a notebook. Because the MCP server acts under the signed-in identity, existing permissions still bound what any agent can see.

⚠️ Exam Trap: "Run a Python-based machine learning analysis over 18 months of sign-in data stored in the data lake" is a data lake notebook with an appropriately sized Spark pool, not an advanced hunting query (30-day XDR range and KQL only) and not a workbook.

Reflection Question: What does the MCP server change about how an AI assistant gets security data, compared with pasting query results into a chat? Consider identity, permissions, and freshness.