Copyright (c) 2026 MindMesh Academy. All rights reserved. This content is proprietary and may not be reproduced or distributed without permission.
4.2.2. Dataflow and Notebook Errors
💡 First Principle: Dataflow errors typically stem from data quality or connection issues. Notebook errors range from code bugs to resource exhaustion. Each requires different troubleshooting approaches.
Common Dataflow Errors
| Error Type | Cause | Resolution |
|---|---|---|
| Connection failed | Credentials expired, network issue | Refresh connection credentials |
| Type mismatch | Source data type changed | Update data type mapping |
| Query timeout | Source system slow or large dataset | Optimize query or paginate |
| Expression.Error: column wasn't found | A source column was renamed or removed (schema drift) | Update the steps that reference it, or tolerate missing columns (e.g., MissingField.Ignore) |
| DataFormat.Error: couldn't convert | A new value doesn't fit the column's type (e.g., "N/A" in a number column) | Clean or replace errors before the Changed Type step |
- Staging: with staging on, a query's output is first written to the dataflow staging lakehouse and later steps read it from there; with staging off, the dataflow's own mashup engine extracts, transforms and loads the data directly. If a query fails only when staged (for example a step or data type the staging path can't handle, or a gateway that can't reach the staging lakehouse), disabling staging for that query is the fix.
Common Notebook Errors
| Error Type | Cause | Resolution |
|---|---|---|
| Out of memory | Dataset too large for executor | Increase executor memory, reduce data |
| AnalysisException: cannot resolve column | The name doesn't match the DataFrame schema: case (when case sensitivity is on), spaces or special characters, or an upstream rename | Check df.printSchema(); wrap unusual names in backticks |
| Error must be logged but the run must continue | A bare except: pass hides the failure; re-raising stops the run | try: ... except Exception as e: logger.error(f"Failed: {e}", exc_info=True) |
| Schema mismatch | Expected columns missing | Validate schema before processing |
| Permission denied | Insufficient access to data | Check workspace and item permissions |
| Filter suddenly returns 0 rows | Source changed the filtered column's values, type or name (e.g. ACTIVE became A) | Profile distinct values and schema before filtering; add row-count checks |
Enroll to unlock the 11 practice questions written for this section, so you can test what you just read while it is fresh.
Enroll to unlock the 3 flashcards for this section and review them on a spaced-repetition schedule.
Written byAlvin Varughese
Founder•20 professional certifications