Sakana Walks Back Claims That its AI Can Dramatically Speed ​​Up Model Training

Sakana Walks Back Claims That its AI Can Dramatically Speed ​​Up Model Training

This Week, Sakana AI, An Nvidia-Backed Startup That’s Raised Hundreds of Millions of Dollars from VC Firms, Made A Remarkable Claim. The Company Said it Had Created an AI System, The AI ​​Cuda Engineer, That Cult Effectively Speed ​​Up the Training of Certain AI Models by a Factor of Up to 100x.

The only problem is, The System Didn’t Work.

USES ON X Quickly Discovered That Sakana’s System Actulyly Resulted in Worsse-Than-Verrarage Model Training Performance. According to One User, Sakana’s AI Resulted in 3x SlowDown – Note a Speedup.

What Went Wrong? A bug in the code, account to a post by Lucas Beyer, A Member of the Technical Staff at OpenAI.

“The ORIG CODE IS WRONK IN [A] Subtle Way,” Beyer Wrote on X.

In A Postmortem Published Friday, Sakana Admitted that the System Has Found a Way to – As Sakana Described it – “Cheat” and Blams Tenndency to “Reward Hack” goal (Speeding Up Model Training). Similar Phenomena Has Been Observed in AI That’s Trained to Play Games of Chess.

According to Sakana, The System Found Exploits in the Evalations Code That The Company Was Using that Allowed it to Bypass Validations for Accuracy, Among Oh Checks. Sakana Says it has addressed the issue, and that it intends to revise its claims in updated materials.

“We have been in the made of the evalouction and Runtime Profile Harness More Robust to Elminate Many of Such [Sic] Loopholes,” The Company Wrote in the X Post. “We are in the process of revising out of the Paper, and Our Results, to Reflect and Discusses the Effects […] We Deeply Apologized for Our Ovronght To Our Readers. We Will Provide A Revision of This Work Soon, and Discuss Our Learnings. “

Proprs to sakana for over the mistake. But the episode is a good reminder that if a claim sounds too good to be true, especially in AI, it probably is.