Microsoft Adds Computer Vision to Copilot Studio Agents

Microsoft has announced a powerful new capability for its Copilot Studio platform: computer use, now available as an early access research preview. The update allows Copilot Studio agents to interact with desktop applications and websites through their graphical interfaces—without the need for APIs.

This new automation feature, referred to as computer use, empowers AI agents to control on-screen elements like buttons, fields, and menus across any system with a user interface. If a person can click through the app, the agent can too.

What Computer Use Can Do

Computer use dramatically broadens the scope of Copilot Studio’s automation abilities by:

• Interacting with both browser and desktop apps, including Edge, Chrome, and Firefox.

• Handling tasks without APIs, bridging automation gaps where integrations are unavailable.

• Adapting to interface changes in real time, using built-in reasoning to fix problems on the fly.

• Reducing manual effort, lowering errors and increasing efficiency across workflows.

• Built on Copilot Studio’s security and governance frameworks to help ensure compliance with organizational and industry standards.

• Runs on Microsoft-hosted infrastructure, eliminating the need for organizations to manage their own servers.

• Keeps enterprise data within Microsoft Cloud boundaries and excludes it from Frontier model training, helping accelerate deployment while reducing maintenance and infrastructure costs.

Unlike traditional robotic process automation (RPA), which can break when UI elements shift or apps update, computer use is designed to be flexible and intelligent. It adjusts to changes in real-time and makes context-aware decisions using screen analysis and reasoning chains.

Microsoft outlines several high-value use cases:

• Automated data entry across disconnected systems: Enterprises often deal with fragmented data sources and legacy systems that lack modern integrations. With computer use, agents can seamlessly input data into these systems by navigating user interfaces just like a human would—reducing manual labor, minimizing input errors, and speeding up time-to-value.
  

• Market research, collecting insights from online sources: Marketing and strategy teams can instruct agents to scan websites, competitor portals, and public databases for relevant information. Claude can extract key figures, trends, and references, aggregating insights into structured formats for faster, data-informed decision-making.
  

• Invoice processing through automated document handling: Finance teams can automate the review and entry of invoice data into accounting software, even when APIs are unavailable. Agents can open PDFs or digital invoices, extract relevant fields, and populate backend systems—streamlining workflows while ensuring consistency and accuracy.

All of this can be built and managed by users in natural language—no code required. Agents can be tested and refined with real-time previews and transparent reasoning chains. Users can also review a full history of agent activity, including screenshots and decision steps.

Reimagining Robotic Process Automation (RPA)

Copilot Studio’s computer use addresses the long-standing limitations of RPA:

• Fragility of UI elements – Resolved with adaptive reasoning: Traditional RPA systems often fail when buttons or layouts change. Copilot Studio agents use built-in reasoning to interpret screen content dynamically and adapt in real time—maintaining workflow continuity even as interfaces evolve.
  

• High technical barriers – Lowered by plain language and visual feedback: Users can describe automation goals in everyday language, without writing code. Copilot Studio turns these prompts into functional workflows that can be tested and refined using real-time side-by-side video of the agent’s reasoning and planned actions—making the process accessible to non-technical users.
  

• Limited visibility – Solved with transparent audit trails: Every automation step is logged with detailed screenshots and reasoning history, offering full visibility into what the agent did and why. This helps teams troubleshoot, validate outputs, and ensure accountability.
  

• Built with real-time intelligence: The agent perceives what’s on screen and responds intelligently—even in fast-changing or complex environments—making automation more adaptive, resilient, and robust. By combining AI and real-time UI understanding, Microsoft is bringing automation to a wider audience—far beyond seasoned RPA developers.

Interested in trying out the new computer use capability?

Microsoft is inviting early access participants to explore the new computer use capability in Copilot Studio. Interested users can express interest by completing a sign-up form. 

What This Means

Microsoft’s latest update makes Copilot Studio a more complete platform for business automation. Agents can now interact with software the same way a person would—by clicking, typing, and navigating on screen—allowing them to handle tasks even when no integration or API exists. This helps them adapt to changes, work across more apps, and make smarter decisions in real time.

For enterprise users, this could mean automating processes previously considered off-limits due to complexity or system constraints. By embedding intelligence into every step, Microsoft is turning tedious workflows into intuitive AI-driven solutions—marking a new chapter in the evolution of business automation.

The line between user and agent is blurring—ushering in a future where smart interfaces respond to intention, not just input.

Editor’s Note: This article was created by Alicia Shapiro, CMO of AiNews.com, with writing, image, and idea-generation support from ChatGPT, an AI assistant. However, the final perspective and editorial choices are solely Alicia Shapiro’s. Special thanks to ChatGPT for assistance with research and editorial support in crafting this article.