Microsoft says its Agent Mode in Excel has an accuracy rate of 57.2 percent in SpreadsheetBench, a benchmark for evaluating an AI model’s ability to edit real world spreadsheets.
It generates 42.8% bullshit.
They probably view that as a statistic worth bragging about. It’s not. If Excel got calculations right 57.2% of the time it would be completely worthless.
Did you read the next sentence? Humans only get like 72% right. It’s not far off at all.
I wonder where that “human accuracy” statistic is coming from. Plenty of people don’t know how to read and interpret data, much less use excel in the first place. There’s a difference between 1/4 of people in the workforce not being able to complete a task, and a specialized AI not being able to complete a task. Additionally, this is how you get into the KPI as a goal rather than a proxy issue. AI will never understand context isn’t directly provided in the workbook. If you introduced a new drink at your restaurant in 2020 AI will tell you that the introduction of the drink caused a 100% decrease in foot traffic since there’s no line item for “global pandemic”. I’m not saying AI will never be there, but people using this version of AI instead of actual analysis don’t care about the facts and just want an answer and for that answer to be cheap.
