AI Apocalypse: The problem is policy

Something has changed in the AI conversation. For years, warnings about artificial intelligence destroying jobs, disrupting society, or eventually threatening humanity were easy to dismiss as science fiction. That is becoming harder to do. Over the past week, some of the people closest to the frontier have been publicly wrestling with whether the industry is moving too quickly. Dario Amodei of Anthropic has called for greater caution around the pace of frontier development, while other technology leaders have increasingly acknowledged that increasingly autonomous systems create risks that cannot be managed simply by adding another product safeguard. When the people building the technology start asking whether they should slow down, the rest of us should probably start listening.

I am not predicting an AI apocalypse; in fact, I spent the summer of 2019 advocating for people not to fear superintelligent AI running roadshows in Nigeria and the United States. I don’t think anyone honestly knows whether the most extreme scenarios will ever happen.

What concerns me is something much more immediate: we are developing systems whose capabilities are advancing faster than our institutions are learning how to govern them. The transition from AI that generates an answer to AI that can conduct research, write and execute code, use software, interact with other systems and operate with increasing autonomy is a fundamental change. At some point, the question stops being how useful AI is and becomes how much authority we are willing to give it.

The incentives are also difficult to ignore. OpenAI, Anthropic, Google, Meta and others are competing for leadership in the same technological race, while the United States and China are competing for strategic advantage. Investors want returns, companies want market share and governments do not want their countries to fall behind. Every participant therefore has a legitimate reason to keep moving faster. That is precisely why relying on individual companies to decide when the race has become too dangerous is not enough. We have seen this problem before in nuclear technology, cybersecurity and other areas where the consequences of failure extend far beyond the company that made the mistake. That’s why the recent Hugging Face and Open AI incident has been a reference point for autonomous agents and their capability to carry out autonomous vulnerability detection among other concerns.

The most important development may therefore not be the next model release. It may be the emergence of AI systems that can materially contribute to the development of better AI. Anthropic has said that Claude is already being used extensively across its own research and development process. That creates a feedback loop worth taking seriously. If increasingly capable systems can accelerate the work required to build the next generation of increasingly capable systems, the pace of technological progress could eventually become less dependent on human engineering capacity. We should not assume that this automatically leads to catastrophe, but we should understand the implications before the feedback loop becomes too difficult to control, and it might be.

This is why AI safety cannot remain something that happens after the model is built. We need independent testing of frontier systems, serious standards for autonomous agents, stronger security around models with dangerous capabilities and clearer rules about when human approval must remain in the loop. Governments also need technical capacity of their own. Regulation written by people who do not understand the technology will fail, but leaving governance entirely to the companies developing the technology creates its own obvious problem. The objective should not be to stop innovation. It should be to make sure that innovation does not outrun our ability to manage its consequences.

• 2027: AI misuse no longer a distant risk – INEC
• US Treasury chief says meeting with China on trade, AI ‘very successful’
• Top US, Chinese officials meet ahead of Trump-Xi summit

There is also a major opportunity here for Africa. The continent should not approach the AI era simply as a market for technologies developed elsewhere. African universities, researchers, engineers and entrepreneurs need to participate in frontier research and in the global conversation about how these systems are governed. We have an opportunity to build AI infrastructure around our own realities, while learning from the mistakes being made elsewhere. The goal should not be to reproduce Silicon Valley at African scale. It should be to build systems that expand human capability while keeping human agency and safety at the center.

The uncomfortable truth is that the AI apocalypse, if it ever comes, probably will not arrive looking like an apocalypse. It is more likely to emerge from a series of individually rational decisions: giving an AI agent access to more systems, allowing it to operate with less supervision, connecting it to physical infrastructure, allowing it to write and deploy its own code, or permitting it to participate in building the next generation of models. None of these decisions necessarily looks catastrophic on its own. The risk comes from what happens when they are connected.

We should keep building AI. But we should stop pretending that speed is the only measure of progress.

The people building these systems, the governments regulating them and the societies that will live with them have a responsibility to put guardrails in place before they are needed, not after something goes wrong. And this brings me to the question I think we need to confront next: what authority should we actually give AI? Should an AI be allowed to make a medical decision, approve a loan, control critical infrastructure, write and deploy production code, conduct scientific research, make immigration decisions, or act on behalf of a government without a human making the final call? Those are not merely technical questions. They are questions about power, accountability and human agency. I will continue with that question in my next op-ed.

Olusola Amusan is an AI evangelist, founder and operator, working between Lagos, Dallas and San Francisco