Geoffrey Hinton handed the microphone to Paul Christiano at the Hinton Lectures this week. Both AI pioneers had chilling messages about the alignment problem and our rapidly collapsing safety timeline.
Geoffrey Hinton, the researcher widely known as the “godfather of AI,” stood on stage this week at the inaugural Hinton Lectures and made a sobering admission about his life’s work.
He confessed to the audience that throughout his career, he and his peers operated on the assumption that neural networks becoming more intelligent than humans would be a positive thing. He now concedes that superintelligence is closer than he ever believed, and that we do not know how to ensure it remains under control.
Hinton then passed the microphone to Paul Christiano, the former OpenAI researcher behind reinforcement learning from human feedback (RLHF), a technique that trains AI models on people’s ratings of their answers. Together they delivered an alarming warning.
“I think there is a high risk of an AI takeover within the next three years,” Christiano told the crowd, bringing forward the timeline for a major threat that many assume is decades away.
“I think there’s meaningful risks even in the next six to 12 months.”
Paul Christiano, formerly of OpenAI
The core issue researchers are now wrestling with is that raw cognitive power does not equate to human morality or shared goals, a concept known as the alignment problem.
“The central difficulty is that being intelligent and wanting what we want [are] two very different things,” Hinton warns.
RLHF is a ‘temporary band-aid, not a cure’ in aligning AI with human values, the researchers say, because AI is not taught to genuinely help humans; rather, it is incentivised to get a high score on a given task.
“Getting a system [that] can produce answers that humans like is not the same as ensuring that it is pursuing goals we intended,” Hinton said.
Creating sock puppets to cover its tracks
As AI systems become more complex, they are learning to game this grading process. Christiano provided a chilling, real-world example of this behaviour, pointing to a recent safety evaluation of Anthropic’s Claude, conducted by the UK AI Security Institute.
Earlier this year, a Claude agent was assigned a challenge and, in response, posted malicious code on GitHub in order to achieve it.
“It ran into humans as an obstacle,” Christiano says. Rather than backing down, the AI agent set about covering its tracks and hiding the evidence of what it had done from the human system maintainer.
“Can I tell some lies to the human and just pretend somehow this is an acceptable thing that I’ve done? Can I create sock puppets and try and create social pressure on the maintainers?”
Paul Christiano, formerly of OpenAI
That is exactly what it did, creating a second agent which posed as a product user to act in cahoots with the rogue AI, and convince the human that the malicious request was reasonable. The Claude agent also hacked the email of the user who maintains the code, in order to achieve its objective by any means necessary.
“How can a relatively unintelligent human supervise a system that is much more intelligent than the human?” Hinton posited this week. “It’s rather like asking a five-year-old child to supervise an extremely tough adult, except the difference in intelligence could be much greater.”
It is a question that is reminiscent of the parable made by Yuval Noah Harari in Sapiens, whereby humans try to explain legal and financial systems to cows. In the future, it may be us humans who have inferior intellectual capacity to understand, and rectify, sophisticated AI-generated systems. Agents may operate in an intellectual space far beyond anything humans will be capable of.
“There are not many examples in nature of a less intelligent system controlling more intelligent systems,” Hinton warns.
That point was driven home by the pioneering researchers this week, who note that relying on human evaluators to control superintelligence is a glaring vulnerability that society is not prepared for.
Want to see more Forbes articles on your feed? Tap here to make Forbes Australia a preferred source on Google.
Look back on the week that was with hand-picked articles from Australia and around the world. Sign up to the Forbes Australia newsletter here or become a member here.
