Microsoft gives Copilot a voice and vision in its biggest redesign ever

Microsoft is today unveiling a major overhaul of its Copilot solution, adding voice and vision capabilities to transform it into a more personalized AI assistant. As I exclusively revealed in my Notebook last week’s newsletter, Copilot’s fresh capabilities include a virtual news anchor mode that reads headlines, the ability for Copilot to see what you’re looking at, and a voice feature that lets you talk to Copilot in a natural way, similar to OpenAI’s Advanced Voice Mode.

Copilot is being redesigned on mobile, on the web, and in a dedicated Windows app to make the user interface more tab-based and look very similar to the work Inflection AI did with its Pi personalized AI assistant. Earlier this year, Microsoft hired a group of employees from Inflection AI, including Google DeepMind co-founder Mustafa Suleyman, who is now CEO of Microsoft AI. This is Suleyman’s first major change at Copilot since taking over the consumer side of the AI ​​assistant.

“At Microsoft AI, we are creating an AI companion for everyone,” Suleyman says in today’s open letter. “I truly believe we can create a calmer, more helpful and supportive era of technology, unlike anything we have seen before.”

Copilot now looks different than anything I’ve seen before from Microsoft, and its interface is vastly different from what exists today. It’s much sweeter with a personalized Copilot Discover page that’s more useful and inviting than a text input prompt for a chatbot. Microsoft is customizing the entire Copilot home page based on your conversation history, and over time will include useful searches, tips, and relevant information.

Earlier this year, Microsoft released its consumer version of Copilot to Suleyman’s team, which clearly allowed the company to further experiment with personality and customization. “What we learned from the Pi i team [Inflection AI] people who have come to us is that they have always paid attention to the details of customer needs,” says Yusuf Mehdi, executive vice president and chief consumer marketing officer at Microsoft, in an interview Edge. “The way they listen and what they learn from long conversations during research has certainly influenced what we have done here.”

Beyond the look and feel of this fresh Copilot, Microsoft is also accelerating work on its vision of an AI companion for everyone by adding voice features very similar to what OpenAI introduced with ChatGPT. You can now talk to the AI ​​assistant, ask it questions and interrupt it just like you would when talking to a friend or colleague. Copilot now has four voice options to choose from, and we encourage you to select one the first time you employ the updated version of Copilot.

“We place a huge emphasis on voice,” says Mehdi. “When you use it the way we designed it, you really start to allow yourself to have conversations. Then you’ll see glimpses of where we’re going in the long term, with a vision where AI can help you and see what you see if that’s what you want.”

Copilot Vision is Microsoft’s second massive step with this redesign, allowing its AI assistant to see what you see on the web page you’re viewing. You can ask it questions about the text, images and content you are viewing, and combined with Copilot Voice’s fresh features, it will respond in a natural way. You can employ this feature when shopping online to find product recommendations, so Copilot will assist you find different options.

Copilot Vision sessions are ephemeral and voluntary, and Microsoft says that no content accessed by Copilot Vision is stored or used for training purposes. This fresh experience won’t work on all websites yet, as Microsoft has placed restrictions on the types of websites Copilot Vision works with. “We’re starting with a limited list of popular websites to keep everyone safe,” says the Copilot team. In preview, Copilot Vision will also not work for paid and sensitive content.

Despite the reservations, Microsoft clearly has a long-term vision for the fresh voice and video capabilities in Copilot. One demonstration shows how Copilot Vision is used to view photos of elderly, handwritten recipes, which helps explain what a food is and gives an indication of how long it takes to prepare a recipe. Microsoft demonstrated similar assist with Xbox games earlier this year, showing how Copilot can assist you navigate Minecraft.

The next phase of Copilot also includes Copilot Daily, an audio summary of news and weather that Copilot reads as if it were a CNN anchor. It is designed as a miniature clip that you can listen to in the morning and only uses content from news and weather providers who have authorized Copilot to employ their content. Microsoft cooperates ReutersAxel Springer, Hearst and Financial Times. initially, with plans to add more sources over time.

Copilot also handles more intricate questions thanks to the latest OpenAI models. Think Deeper is a fresh feature in Copilot that allows the assistant to spend more time responding, so it can provide detailed answers to intricate questions. It’s designed to work best when you’re trying to compare two options side by side, for example: “Should I move to New York or San Francisco?”

Think Deeper is still in its early stages of development, and Microsoft is putting it into Copilot Labs, a fresh way of testing experimental features that the company is still working on. Copilot Vision will also initially be part of the Lab feature, and participants will be able to share their views on the experience. Microsoft is apparently taking a cautious approach to Copilot Vision after the backlash surrounding initial security and privacy issues surrounding the product recall. Last week, Microsoft revealed that the Recall feature has been modernized and includes improved security and privacy options. You can even uninstall this feature completely or not enable it at all.

This updated Copilot solution will be available today on iOS and Android mobile apps on the web at copilot.microsoft.comand via the Copilot application for Windows. Copilot Voice will initially be available in English in Australia, Canada, Fresh Zealand, the UK and the US, and will expand to more regions and languages ​​in the future. Copilot Daily is only available in the US and UK before expanding to other countries, and Copilot Vision will only be available to a confined number of Copilot Pro subscribers in the US.

If you’re like me and wondering where Copilot is going next, Microsoft’s fresh CEO of artificial intelligence has some great ideas. “Over time, it will adapt to your mannerisms and develop capabilities based on your preferences and needs. We are not creating a static tool, but establishing a dynamic, emergent and evolving interaction,” says Suleyman. “It will accompany you during your doctor visits, allowing you to take notes and monitor your health at the right time. She will share with you the responsibilities of planning and preparing for your child’s birthday party. And he will be there at the end of the day to assist you think through a tough life decision.