3.3.2. Computer Use for UI Automation
Computer Use is a Copilot Studio capability that enables agents to interact with websites and desktop applications by directly manipulating the user interface — clicking buttons, filling forms, selecting menus, and reading screen content. It's designed specifically for scenarios where no API or MCP server exists.
Think of Computer Use as giving your agent the same ability a human has to operate software: it can see what's on screen, understand the interface layout, and interact with UI elements using mouse clicks and keyboard input.
When to Use Computer Use:
| Scenario | Why Computer Use? |
|---|---|
| Legacy application with no API | The only way to interact is through the UI |
| Third-party SaaS without connectors | No connector exists and building one is impractical |
| Complex multi-step UI workflows | Automating repetitive UI tasks (data entry, form filling) |
| Cross-application workflows | Agent needs to interact with multiple desktop/web apps in sequence |
When NOT to Use Computer Use:
- An API or connector exists — always prefer API-based integration for reliability and speed
- The target application changes its UI frequently — Computer Use adapts to changes but frequent major redesigns increase failure risk
- High-throughput scenarios — UI interaction is slower than API calls
- Security-sensitive operations — UI automation may bypass application-level audit logging
How Computer Use Works:
Computer Use in Copilot Studio works across Edge, Chrome, Firefox browsers and desktop applications. The agent:
- Receives a task instruction ("Enter this invoice data into the ERP system")
- Navigates to the target application
- Uses visual understanding to identify UI elements (buttons, fields, menus)
- Interacts with elements through simulated user actions
- Adapts in real-time if the UI layout changes — built-in reasoning handles variations
Design Best Practices:
- Define clear task boundaries. "Process this invoice" is better than "do whatever is needed." The agent needs a specific scope for UI interaction.
- Implement verification steps. After filling a form, the agent should verify the entered data before submitting — just as a human would review before clicking "Submit."
- Design for failure recovery. If the agent encounters an unexpected dialog box or error message, it should know how to recover or escalate rather than getting stuck.
- Consider security implications. Computer Use operates with the credentials of the session it's running in. Ensure the session has appropriate (minimum necessary) permissions.
Exam Trap: Computer Use is NOT a replacement for proper API integration. It's a last resort for scenarios where APIs don't exist. If the exam presents a scenario where both an API and Computer Use could solve the problem, the correct answer is always the API.
Reflection Question: A finance team processes vendor invoices by manually entering data into an ERP system that has an API but the API lacks certain data entry capabilities available only through the UI. How would you design a solution that uses both the API and Computer Use?
