Trending Articles
Trending Topics
Trending
A 14-week Microsoft 365 Copilot trial conducted by the Australian government found that 200+ Treasury staff used the AI assistant much less than was expected.
The results of the trial are detailed in an extensive report published by the Australian Centre for Evaluation and parts of it will make uncomfortable reading for Microsoft, which is is trying to drive corporate uptake of its AI assistant.
The most damning finding is how little staff ended up using Copilot in their day-to-day work. Ahead of the trial, 75% of the staff expected that the AI would be useful for some of their work tasks, with only 6% saying it would be of little or no use.
After the trial, 59% of found they had little or no use for it, and only 38% said they used it for some of their tasks. Most participants only used it two or three times per week.
The report identifies why Copilot didn’t prove to be a widespread hit with Treasury staff. To be fair to Microsoft, some of the findings are more institutional than technological.
For example, trialists complained that Microsoft 365 Copilot “did not perform as well as generative AI products that staff had used elsewhere”, however this was in part “due to restrictions imposed by the Treasury’s IT security environment”.
There are also question marks over how well staff were trained to use the AI before it was unleashed on them. “Staff required time to learn how to use Copilot effectively, which they found challenging to fit into their workload,” the report states. “A common request from participants throughout the trial was for more tailored and targeted education and training to support their use of Copilot.”
However, there were also criticisms of the AI’s capabilities. Four use cases were identified for the AI:
However, Treasury staff found that “Copilot was not appropriate for more complex tasks, mostly due to the limitations of the product itself”.
That’s not to say there weren’t benefits of the Microsoft 365 Copilot trial. One of the biggest plus points was the increased inclusion for neurodivergent or part-time staff. Features such as automatic summaries of missed meetings or structured starting points for tasks proved beneficial.
Junior staff also felt the AI helped close the knowledge gap on their more senior colleagues, “levelling the playing field for those who struggle to navigate workplace norms or culture”.
Other staff reported feeling less overwhelmed by routine admin tasks that the AI could assist with.
Overall, the report claims the cost of the Copilot licences could be justifiable if it helped staff “redirect approximately 13 minutes of time from low-value to high-value tasks per week”.
Microsoft has claimed more positive results from Microsoft 365 Copilot trials in the past. Last year, for example, the company claimed its partnership with Vodafone resulted in the AI being rolled out to 68,000 of the mobile network’s staff, with 90% of staff who participated in a trial saying they benefited from the AI.
It should be noted, however, these findings were reported by Microsoft itself, and not in an independent report such as the Australian trial.
