World student math competition disses OpenAI over AI claims

The head of an international children’s mathematics competition at the centre of a controversy in the AI world has poured cold water on OpenAI’s claims that its AI model blitzed the event.

Saturday marked the end of the 66th International Mathematical Olympiad held in the Sunshine Coast, Australia. It’s an annual math competition that brought together more than 600 students to compete to solve a series of problems.

This year marked the first time that AI companies were invited to test their own technology on the same problems given to the students.

On Saturday, when the competition had concluded, an OpenAI researcher Alexander Wei announced that one of its LLM models had “achieved a longstanding grand challenge in AI”: a gold medal at the Olympiad.

This announcement kicked off an online furore over two things: whether the OpenAI claim about its model’s performance was reliable and whether the company had ignored a request from the Olympiad organisers to hold off publishing its results over fears it would eclipse the students’ performance.

The host organisation, the Australian Maths Trust, directed me to a statement by International Mathematical Olympiad president Gregor Dolinar about the AI controversy.

Dolinar said that IMO “cannot validate the methods, including the amount of compute used or whether there was any human involvement, or whether the results can be reproduced.”

Respected mathematician and IMO attendee Terence Tao voiced his concern about comparing an AI model’s performance on the Olympiad problems with the participants’ performance in the real world competition.

“So, in the absence of a controlled test methodology that was not self-selected by the competing teams, one should be wary of making overly simplistic apples-to-apples comparisons between the performance of various AI models on competitions such as the IMO, or between such models and the human contestants,” he posted on Mastodon.

OpenAI was not part of the Olympiad’s official competition for AI companies, and instead graded its models’ performance using its own panel of three former Olympiad medalists. The company reportedly declined to take part in the official process earlier this year and claims it was given permission to publish its own results on Saturday.

When Google, which did officially participate, announced its own model’s results on Tuesday morning, its DeepMind CEO Demis Hassabis posted “We didn’t announce on Friday because we respected the IMO Board's original request that all AI labs share their results only after the official results had been verified by independent experts & the students had rightly received the acclamation they deserved.”

Dolinar seemed to hint at this attempt to avoid overshadowing contestants. His statement finished with a message to the participants: “we hope that you enjoy your day in the spotlight.