4.1.2. Azure Event Grid: Filters, Custom Events, and Retries

💡 First Principle: Event Grid is a routing fabric, not a store: its job is to notice that something happened and get a small JSON fact to every interested party, fast. Because it pushes rather than stores-until-asked, its reliability tooling is all about delivery persistence — retries with backoff, then dead-lettering to storage — and its power tooling is all about filtering, so parties hear only what interests them.

Events flow from topics to subscriptions. System topics publish Azure's own facts (blob created, resource changed, Key Vault secret near expiry — the hook 5.1.1 will use); custom topics carry your application's facts:

from azure.eventgrid import EventGridPublisherClient, EventGridEvent

client.send(EventGridEvent(
    event_type="MindMesh.Document.Ingested",
    subject="/documents/whitepapers/doc-42",
    data={"docId": "doc-42", "pages": 12},
    data_version="1.0"
))

Each event subscription binds a topic to a handler — a Function, a webhook, or notably a Service Bus queue — and declares filters so the handler sees only relevant events: event-type filters (MindMesh.Document.Ingested only), subject prefix/suffix filters (subject BeginsWith /documents/whitepapers), and advanced filters on data payload fields (data.pages > 10). Filtering at the subscription is the architectural win: publishers stay ignorant of consumers, and consumers subscribe to precisely their slice — new consumers appear without touching the publisher, the loose coupling promised in the parent section.

Delivery is push, with retry as the safety net: failed deliveries retry with exponential backoff for up to 24 hours, after which the event can be dead-lettered to a storage account (opt-in — configure it or undeliverable events are dropped). Delivery is at-least-once, so handlers must be idempotent — the third appearance of that rule, after the change feed (2.1.4) and Service Bus (4.1.1); by now it should feel like a law of distributed nature.

The subscription-to-Service-Bus handler deserves special attention because it composes the two species: Event Grid announces the fact, the subscription drops a message on a queue, and KEDA-scaled workers (3.2.2) process with full custody-chain guarantees — fact in, command out. "Blob uploaded → embedding pipeline" is exactly this composition, and the exam loves it.

⚠️ Exam Trap: "Event Grid guarantees exactly-once delivery" is always false — at-least-once, with duplicates by design during retries. And if a scenario mentions losing undeliverable events, the missing configuration is dead-lettering to a storage account, which is not enabled by default.

Delivery format and the webhook handshake round out Event Grid. Events ship in either the classic EventGridEvent schema or the CNCF CloudEvents 1.0 schema — subscriptions choose, and CloudEvents is the interoperable pick when non-Azure consumers subscribe. Webhook endpoints must prove ownership before delivery begins: Event Grid sends a validation event containing a code the endpoint echoes back (or a validation URL a human clicks). Functions and Service Bus handlers skip the handshake because trust is established platform-side — one more reason those destinations are smoother than raw webhooks. The retry schedule itself backs off from seconds toward hourly attempts as failures persist across the 24-hour window.

Size matters too: events cap at kilobytes, not megabytes — another reason payloads carry identifiers while blobs and documents stay in their durable stores.

Reflection Question: Why does filtering belong on the subscription rather than inside the handler code — what architectural property does subscription-side filtering preserve?