2.2.1. Computer Use
💡 First Principle: Computer use gives an agent hands and eyes instead of an API: a vision-plus-reasoning model looks at a screen, decides where to click and what to type, and works the same interfaces a human would — websites and desktop applications — on a dedicated Windows machine you configure for it.
Because it perceives the UI rather than binding to selectors, it adapts when layouts shift — the property that separates it from brittle screen-scraping. But everything about its design says fallback, not first choice: it requires generative orchestration enabled on the agent, a configured Windows machine, stored credentials (Power Platform internal storage or Azure Key Vault), and it bills Copilot Credits per step — a multi-step form fill is a multiple of that per run, with premium models costing more per step.
Configuration mirrors the risk profile: you provide a natural-language description (so the orchestrator knows when to reach for it) and step-by-step instructions (specific URLs, explicit actions, expected outputs), then optionally lock it down with website/app allow lists, HTTPS enforcement, least-privilege machine accounts, and human supervision alerts for potentially harmful instructions. Monitoring is built into the experience — runs show the model's reasoning and screenshots step by step, which is how you debug instructions before publishing.
⚠️ Common Misconception: Computer use is the default way to integrate any website or app. It is the last resort when no API or connector exists — slower and costlier per step than any API call, and dependent on a machine you must secure and maintain.
⚠️ Exam Trap: Computer use requires generative orchestration. An agent running classic orchestration cannot use it — if a scenario pairs classic orchestration with a computer-use requirement, the orchestration mode has to change first.
Reflection Question: An internal legacy app has no API and no budget for one, and staff spend an hour a day retyping data into it. Walk the decision table: why does computer use win here, and which two security controls would you insist on before publishing?
