Microsoft Copilot Adds Screen Reading, Deep Thinking, & Voice Features

Microsoft is expanding the reach of its AI-powered Copilot, adding new features to enhance user experience on Windows, iOS, Android, and the web. The latest update introduces a tool that understands and responds to what's on your screen, bringing a more interactive and intuitive approach to AI assistance.

New Copilot Features for Enhanced User Interaction

Refreshed Copilot apps are rolling out today, complete with a more “warm” and “distinct” style, according to Microsoft. In addition to this, Microsoft is launching Copilot on WhatsApp, allowing users to chat directly with Copilot via DM. This functionality is similar to other chatbot experiences on Meta's messaging platforms, but with a distinct Microsoft twist.

Copilot Vision: Your PC's New Lens

One of the standout features is Copilot Vision, which gives Copilot a lens into what you’re viewing on your PC. Through Microsoft Edge, Vision analyzes text and images on websites, offering answers to questions like “What’s the recipe for the food in this picture?”

Accessible by typing “@copilot” in Edge’s address bar, Vision aims to provide a privacy-conscious alternative to existing screen-analyzing tools. Microsoft claims that Vision was designed to delete data immediately after use, ensuring that processed images, audio, or text are not stored or used for training AI models.

Currently, Copilot Vision is only available to users who opt into Copilot Labs, an exclusive program for Copilot Pro subscribers, giving them access to experimental features.

Privacy and Legal Concerns Surround Copilot Vision

While Copilot Vision is limited to analyzing pre-approved websites, its potential to circumvent paywalls or access sensitive content has raised questions. Microsoft has faced legal challenges in the past for allegedly allowing users to bypass paywalls using Copilot chatbot on Bing. In response, Microsoft claims that Vision respects "machine-readable controls" that govern AI interactions with websites, though they haven't specified which controls will be honored.

Introducing Think Deeper: Tackling Complex Problems

Another major feature is Think Deeper, designed to help Copilot tackle more complex problems with greater depth. Using reasoning models—fine-tuned by Microsoft and based on OpenAI’s o1 model—Think Deeper provides step-by-step answers to intricate questions. This feature aims to assist users with tasks ranging from solving difficult math problems to managing home improvement costs. “Think Deeper can help with anything from solving tough math problems to weighing up the costs of managing home projects.”

While Microsoft has yet to clarify which specific enhancements were made to Think Deeper, it emphasizes the model’s potential to solve practical challenges in everyday life. Starting today, Think Deeper will be available to a limited number of Copilot Labs users in Australia, Canada, New Zealand, the U.S., and the U.K.

Copilot Voice: Speaking in Natural Language

Launching today in select countries, Copilot Voice introduces synthetic voices, allowing users to converse with Copilot and hear its responses aloud. Available in English in the U.S., U.K., Australia, Canada, and New Zealand, Voice uses advanced technology to adjust responses based on the user's tone, and allows users to interrupt at any point. This feature mirrors OpenAI’s Advanced Voice Mode for ChatGPT but with its own Microsoft-specific refinements.

However, there are usage limits tied to time, with Copilot Pro subscribers receiving more minutes based on demand, but Microsoft said the number of minutes allowed is "variable".

Personalization Settings: Tailoring Copilot to Your Preferences

In another update, Copilot will soon offer personalization settings that allow it to adapt to your preferences based on past interactions and usage history across Microsoft apps. This feature is designed to create a more tailored Copilot experience, offering suggestions and conversation starters.

However, users in the U.K. and EU will have to wait longer for these settings due to Microsoft’s cautious approach to AI rollouts in regions governed by stricter data privacy laws.

Privacy and Compliance Challenges in the EU

Microsoft’s relationship with the EU has been rocky regarding AI product rollouts. Earlier this year, the company faced the possibility of a fine under the Digital Services Act for failing to provide information about its generative AI tools. Like other tech giants, Microsoft is treading carefully in the EU, ensuring that its AI tools comply with local data privacy and model deployment regulations.