When AI safeguards fail, users and essential services could bear the cost

On Monday, the UK government said it would explore strengthening parts of its Cyber Security and Resilience Bill and wider regime to protect essential services, including hospitals and utilities, from AI-related risks. It also said it would consider ways to identify and report near misses involving AI agents. These are areas for further work, not new legal requirements already in force.

That distinction matters to people who rely on those services. A breach or failure involving a hospital or utility could put data or operations at risk, but the reporting does not establish that such harm occurred in the examples discussed. The government says its AI Security Institute is evaluating advanced models. It also reports that two pilots identified and helped fix more than 1,000 potential cyber vulnerabilities across government systems. Those figures show government activity, not independent confirmation that safeguards are sufficient.

An incident involving an AI agent and an Australian government website offers another reason to focus on response, not slogans. ITV reported that UK AI Minister Kanishka Narayan said the Australian government reported no known access to personal medical records and no evidence of wider compromise. A near miss is not proof of injury. It does, however, make practical questions urgent: who detects a problem, who is told, who can stop a system and what happens next?

Are voluntary promises enough?

The United States offers a different test of whether stated safeguards have force. A June executive order set a policy against overly burdensome regulation while also directing federal agencies to strengthen cybersecurity and develop a voluntary framework with AI developers for certain advanced models. On Sept. 29, the administration and AI-company leaders signed a voluntary accord calling for internal controls, external audits and board-level review. The accord’s signing does not show that those measures work. The administration had not specified how they would be implemented, AP reported.

There is a fair case for voluntary steps: they can put safety reviews and oversight on companies’ agendas without waiting for a law. University of Southern California professor Shri Narayanan said the accord could balance innovation and regulation and encourage safety work. That possibility deserves consideration; not every useful safeguard must begin as a legal mandate.

But discretion is the weak point. Former OpenAI safety-team leader David Robinson warned that a voluntary agreement leaves companies substantial room to police themselves and is not a substitute for binding law. Robin Jia, an assistant professor of computer science at the University of Southern California, questioned whether leading labs’ goals align with the public interest. Alex Pascal, executive director of the Berkman Klein Center for Internet and Society, argued that liability and regulation matter because incentives can drive risky decisions. These concerns do not prove voluntary audits fail. They identify what a promise cannot settle on its own: who checks the checkers, whether findings are disclosed and what consequences follow when a company falls short.

Who should be accountable when safeguards fail?

The strongest case against rushing into broad rules is that poorly designed requirements could burden useful development. The US executive order reflects that concern. But rejecting rules because milk is regulated differently would be just as weak: AI systems are used across sectors, and risks and oversight may vary by use. The choice is not between copying food inspections and having no safeguards.

A sound policy should specify which systems and uses trigger duties, what counts as an incident or near miss, who receives reports and when affected people must be notified. It should also identify who can restrict a system during an emergency and who is accountable if required protections are missing. Independent evaluation needs a clear scope and credible independence; an audit label alone cannot establish that the public is safe.

Companies can test systems and adopt internal controls, but they should not be the sole judges of whether protections meet the public interest. Legislatures and regulators should set enforceable duties, while independent evaluators check whether those duties are met. The UK government has announced exploration, not enacted a new regime. Its next steps should be judged by whether people and essential services gain clear protections and a route to accountability—not by whether AI rules resemble those for a carton of milk.