Here’s a cool Windows Copilot AI trick you might have missed

Last week’s Windows 11 update puts its AI assistant front and centre in the operating system. And there’s a neat little Windows Copilot AI trick that you might not have noticed during your early experimentation with the chatbot.

The Windows Copilot AI has fairly limited interaction with Windows apps, but it does work smartly with the revamped Snipping Tool – the app used to take screenshots when you press the Print Screen (Prtsc) key.

If you take a screenshot and open the Windows Copilot, you’ll be asked if you want to add the image to the chat. In the image below, for example, I took a screenshot of the photo of the car being driven by Rowan Atkinson, then added it to the chat.

I then asked Windows Copilot: “What car is in the picture?”

It took a few seconds to think about it before correctly identifying it as a McLaren F1 Sports Car, as well as providing supplementary information, such as the car’s price and when it was made.

I’ve also used it to identify breeds of dogs, artworks and musical instruments.

Windows Copilot AI limitations

There are certain types of images that Windows Copilot AI won’t identify or interact with.

For example, I took a screenshot of a Manchester United footballer and asked the AI to identify him, but the AI automatically blurs faces in uploaded images for privacy reasons so couldn’t name the footballer. That said, the AI added: “I can only tell the person is wearing a Manchester United jersey”, which is a handy pointer for anyone trying to identify the footballer.

There are other things that have been deliberately disabled. The AI won’t try and solve the CAPTCHA images used to prove that visitors to a website are human. It won’t comment on someone’s body shape. Neither will it try and identify the precise location of a property in a photo.

As with any new AI development, it’s highly likely that someone will find a way to put the feature to malicious use, but at least the developers are trying to thwart obvious cases of misuse.

This new ability to answer questions about images is part of the newly released GPT-4V facility, the V stands for vision. GPT-4 is the AI model that powers Microsoft’s AI services and is the same technology that’s behind ChatGPT.