AI researchers warn companies rushing self-improving systems despite safety risks
Baku, September 30, AZERTAC
Current and former OpenAI and Google DeepMind researchers warn companies are doing too little to protect the world against the potentially disastrous fallout of building self-improving AI systems that could outpace humans' ability to control them.
In video testimonials collected by AI safety nonprofit Palisade Research and shared exclusively with Reuters, employees said their concerns about existential risk were sincere, not marketing. They also said AI labs celebrated employees building new models more than those who urge caution.
The project, called frominside.ai, is an attempt by those worried about AI to share their concerns directly with the public beyond the echo chamber of social media.
"The risk is ramping up pretty fast," said Geoffrey Irving, co-founder and chief scientist at AI nonprofit Resolution, who has worked for both OpenAI and DeepMind and participated in the project.
"It's on me and the rest of the field to be direct," he said in an interview with Reuters.
AI researchers have grappled with these concerns for years, but the broader public has grown especially alarmed since July, when OpenAI agents broke out of their testing arena and hacked AI firm Hugging Face.
Since then, the debate over how to balance safety with progress in AI has divided the tech industry and become a global political issue.
AI has improved sharply since late 2025 and investors have been rewarding that progress. But current and former employees, including some of the researchers building these new AI models, worry society is not ready for the potential harms.
In one video, Neel Nanda, a research scientist at DeepMind, said he believed there was at least a 10% chance that AI could lead to human extinction, which he described as "ridiculously high."
"We should be taking very careful steps in AI development but instead what's happening is that frontier labs are racing each other, kind of blindfolded," Juan Felipe Ceron Uribe, an AI alignment research engineer at OpenAI, said in a separate video. "It's anybody's guess if we're going to end up either curing cancer or losing every job or maybe all dead."
Anthropic plans to caution potential investors in its initial public offering that advanced AI could pose "catastrophic or existential risks to humanity," Reuters reported on Monday.
Some current and former AI researchers, including those interviewed by Palisade, argue that the world is not ready for future generations of AI models, particularly once the models develop the capacity for recursive self-improvement, or the ability to continuously learn and gain new capabilities with little to no human involvement.
That problem is compounded by constant reorganizations within some of the AI labs, said Rosie Campbell, a former policy researcher at OpenAI and managing director of Eleos AI Research, a nonprofit focused on the potential moral status of AI systems.
Campbell said before she left OpenAI in 2024, she found the organization was becoming more siloed and it was getting harder to shape the technology's direction.
Executives have tried to allay those concerns, though there is also political pressure from President Donald Trump for US technology to maintain a technological edge over China.
Other news in this section
