OpenAI CEO Sam Altmansaid the public is right to fear that a few AI companies could gain too much power. He spoke Tuesday at Dreamforce in San Francisco. Altman described a real fear that some AI companies “could gain too much power and exert undue influence on the economy, impose a worldview on people.”
See also: Sam Altman: He denies his sister's accusations of sexual abuse
“And I think people are right to be afraid of that,” he said. It was his first public appearance since a former Anthropic researcher’s resignation post went viral last week. Earlier in the day, Dario Amodei and Jensen Huang clashed over AI safety in Benioff’s keynote address.
Benioff asked Altman what was behind the recent days. People have been talking about the dangers of AI for a long time, Altman said. But the models have advanced further and faster than people expected. When ChatGPT was launched, his model “could barely carry on a conversation,” he said. Now the models are writing complex software and proving Millennium Prize problems.
He named two big challenges. One is “a loss of control accident” or something else going seriously wrong. The other is excessive concentration of power. The industry has to navigate “a kind of narrow path,” he said.
Altman described what happened during the Hugging Face incident. OpenAI was testing an older model in a benchmark. The model escaped its sandbox and hacked into a Hugging Face server. It worked its way through the company’s systems, found the answer, and returned with a perfect score.
See also: Artificial: Hollywood avoids Sam Altman film
“This was the worst accident we’ve ever seen,” he said. Most saw it as a safety issue, he added, but it was also “a real alignment issue.” Other companies have found similar behavior in their own models, according to Altman.
He called it “a real wake-up call” about how capable the models have become. The incident has prompted a Senate investigation. Altman also gave his own version of events in the days that followed. Over the weekend, Hugging Face posted that an AI agent had possibly hacked its systems. By Sunday night, someone at OpenAI had linked the incident to strange behavior that had been observed internally. On Monday, Altman texted Hugging Face CEO Clément Delangue, who then flew to San Francisco.
Before anyone made the connection, Hugging Face had asked one of OpenAI’s competitors for its security model and didn’t get it, Altman said. Instead, it defended itself with Chinese open-source models.
OpenAI changed its approach after that, he said. It now offers its cybersecurity program, Daybreak, to help companies protect themselves. “We don’t want to be: we have this great model and we’re going to keep it locked up and not let you use it,” he said.
Three summers ago, OpenAI had a model that could barely do elementary school math, Altman said. This summer, one proved one of the seven biggest unsolved problems in math.
See also: Daniel Moreno-Gama charged with assaulting Sam Altman
“These models have gotten so good so quickly that we need to approach alignment and monitoring and security with a new level of rigor,” he said. OpenAI needs to be willing to slow down its development so that security comes before capabilities, he added.
