5.1.1. Image Q&A, Captioning, and Multimodal Prompting
💡 First Principle: Visual understanding via a multimodal model is grounding-by-attachment — the image is the context the model reasons over, exactly like a retrieved chunk is context for text. The same prompt-engineering levers (clear instructions, structured output) apply, just with an image in the message.
You send an image (as a URL or base64) in the message alongside your text instruction, and the model reasons over both. This handles captioning ("describe this image"), visual Q&A ("how many people are in this photo?"), document/chart reading, and accessibility alt-text generation. Because it's a generative model, you can request structured output (return findings as JSON) and combine it with grounding (answer about this image using our product manual). For agentic scenarios, an agent can take an image input and reason about it as one step in a larger task.
⚠️ Exam Trap: Don't reach for a specialized image-classification service when the requirement is open-ended understanding ("summarize what's happening in this image"). Multimodal prompting handles open understanding; a narrow classifier is for predefined categories. Matching an open task to a closed-category tool is the mismatch.
Reflection Question: Why is "describe the contents of this uploaded photo for an accessibility caption" a multimodal-prompt task rather than a custom-vision-model task, and what would change the answer?
