A report commissioned by Australia’s Department of Employment and Workplace Relations, which was paid AUD 440,000 (around 231,200 Swiss francs), looked into an IT system intended to automate sanctions in the country’s welfare framework. The original report was published in July and was quietly updated on the ministry’s website last week. The updated version featured over a dozen deletions of non-existent references and footnotes, a refreshed reference list, and various typographical corrections.
Christopher Rudge, a researcher at the University of Sydney, discovered several inaccuracies in the report. He hypothesized that these inaccuracies could be due to instances of AI hallucinations, which occur when generative models fabricate information that appears to be factual.
While the report now includes a disclaimer that generative AI was utilized to address “gaps in traceability and documentation”, the trust in the report’s findings has been undermined as a result of the reported inaccuracies. “The firm acknowledged the use of generative AI for a significant analytical task but did not initially disclose it,” said Rudge. He expressed concern that the recommendations made in the report could not be fully trusted.
Nonetheless, the ministry maintained that “the essence of the independent review remains unchanged and there will be no alterations to the recommendations”. There are also reports that Deloitte forfeited the final payment for the report, although the amount was not specified.
This incident does more than just create a financial dent; it also represents a significant blow to Deloitte’s reputation. The firm, which advises corporations, governments, and institutions worldwide on the proper utilization of AI, is now facing scrutiny for the responsible use of these technologies.
What inaccuracies were found in the report?
Christopher Rudge from the University of Sydney found several inaccuracies in the report, suggesting they could be the result of AI hallucinations – when generative models create information that appears factual.
What was the initial reaction to these inaccuracies?
While Deloitte admitted to using generative AI and added a disclaimer in the updated version of the report, there has been a decline in trust regarding the report’s findings. The ministry, however, maintained that the core of the review and its recommendations remain unaltered.
What impact has this had on Deloitte?
Beyond the financial implications, this incident represents a significant reputational setback for Deloitte, a firm that advises on the responsible use of AI worldwide.
