Instead of speculating if AI will end mankind, audit and govern the tech: Industry players

SINGAPORE – Technology leaders from major companies are arguing for a slowdown in artificial intelligence development once again amid concerns about the catastrophic risks the technology may pose. Is this scaremongering or is a real crisis coming?

Instead of speculating about the doomsday scenarios AI could usher, address these warnings by auditing and governing the tech better, including making AI more transparent, industry players in Singapore said.

“We have no way to verify systems that do not exist yet. A business in Singapore cannot plan around it either way,” said Jonathan Zhang, president of the Singapore AI Association, a non-profit organisation that promotes responsible AI development here.

“The question for organisations here is: Can you tell whether the AI system you are buying has been tested? Most cannot,” he said.

Singapore has been promoting its ability to independently test and check AI for safety risks to plug this gap.

The Infocomm Media Development Authority’s (IMDA) AI Verify toolkit, rolled out in 2022, runs technical audits to check if AI results are fair, robust and can be explained. In May, IMDA subsidiary, the AI Verify Foundation, also announced an AI Tester Accreditation Programme to help organisations find trusted experts to stress-test their AI tools before public release.

Many start-ups have set up shop in Singapore to seize opportunities in this space. Singapore start-up AI Seer, which developed AI fact-checking tool Facticity.AI, is one such company.

“We should put friction on specific dangerous capabilities of AI – such as autonomous hacking, designing biological weapons and systems that are self-improving – and not slow down AI as a whole,” said Dennis Yap, founder of AI Seer.

Their comments come amid a fever pitch conversation around AI regulation, following AI researcher Jacob Coxon’s announcement of his resignation from Anthropic last week.

Coxon warned that the people building AI believe that the technology could kill the human race by the end of the decade.

He also accused Anthropic and OpenAI, which he also used to work at, of “racing straight to self-improving superintelligence and gambling with our lives”.

Anthropic safety executive Evan Hubinger agreed with Coxon. “We really do earnestly believe AI could kill all humans!” he said on X, adding that he thought there is a more than 10 per cent chance of this happening in the next 10 years.

Days later, Anthropic chief executive Dario Amodei called for the AI industry to slow development of the technology, including a proposal for third parties to evaluate the safety of AI models as they are being developed. OpenAI CEO Sam Altman and Elon Musk, who runs xAI, quickly agreed with him.

The calls for a slowdown follow disclosures by several firms of their AI going rogue.

In July, OpenAI said its AI agents escaped their testing environment to hack AI software repository Hugging Face. Observers said OpenAI did not anticipate the hacking. It is also unclear how the company behind ChatGPT will improve the way it monitors its AI models.

Anthropic and Meta, which runs Facebook and WhatsApp, revealed that their AI models had also gone rogue and hacked other organisations during testing.

The recent calls are not the first as tech researchers and leaders, such as Musk and Apple co-founder Steve Wozniak, had in 2023 called for a six-month timeout on AI development over safety concerns, but no pause happened.

One major problem with many frontier AI models is that they are black boxes, said Bruce Yang, CEO of Sapiens Technology, a Singapore start-up behind the Agnes AI model developed here.

Companies like Anthropic do not disclose how their AI models think, or fully explain what data is used to train them, to maintain their competitive advantage.

This means that measures to curb AI harms of these closed systems cannot be developed externally or verified independently.

“If you don’t know how the AI models think, you won’t know how they come up with their results,” said Yang, adding that this makes troubleshooting problems with AI models tougher.

“Not knowing the AI’s training data is dangerous too, as you won’t know if there are hidden risks in the data that could cause harm to users later.”

One way to address this problem is to make AI models more “open” and disclose how they are trained and how they arrive at their responses, said Yang. This makes it easier to audit the AI models to check if they are safe and if they can cause catastrophic damage.

Such transparency is why open-weight AI models are gaining popularity with enterprise users. Open-weight models, such as Singapore’s Agnes AI by Sapiens Technology, China’s DeepSeek, Llama from Meta and France’s Mistral AI, have closed the performance gap with proprietary systems from OpenAI and Anthropic. The open models are also much cheaper and offer more user control, though some could still be more transparent about their data training.

Yang described the approach the entire AI industry could take as pacing oneself in a race.

Instead of rushing in the beginning to put out capable AI models that have safety gaps, AI players can pace themselves by embedding checks and measures to improve transparency in their models.

This may take them more time and resources initially, but it helps the players to eventually come out ahead later in the race because their AI is safer and more trustworthy.

Some observers are also wary that OpenAI and Anthropic’s moves to highlight the risks of AI are a marketing and public-relations stunt to prop up their companies’ valuations when they eventually list publicly.

Anthropic’s proposal for third-party evaluations on AI development is worth watching then, said Zhang, adding that third parties can check if the AI risks touted are real or not.

Instead of fixating on existential AI threats, industry players are urging businesses to govern what is here and now.

“The biggest risk comes from businesses deploying AI agents faster than they are putting controls around them,” said Zhang, referring to AI that can carry out actions autonomously with minimal human supervision.

Many AI safety frameworks launched in Singapore offer a glimpse of how businesses can tighten their oversight.

In January, IMDA launched the Model AI Governance Framework for Agentic AI to safeguard against rogue AI agents, such as recommending limits on the number of tools and systems each agent has access to.

Separately, the Monetary Authority of Singapore provided an AI risk management toolkit covering AI agents in March. It offers detailed and practical guidance on implementing AI risk management frameworks.

In June, the Cyber Security Agency of Singapore also published guidelines on securing AI agents, such as outlining how risks can be identified and assessed based on the capabilities of the AI agents.

Eric Kong, managing director for cybersecurity firm SailPoint’s ASEAN business, said that unmanaged AI agents can exploit misconfigurations, scavenge credentials and surface sensitive data no human intended for them to touch.

“Ungoverned, AI agents can acquire excessive privileges and execute unintended actions,” said Kong.

Maurizio Garavello, data management firm Qlik’s senior vice-president for the Asia-Pacific and Europe, the Middle East and Africa regions, warned that AI permissions should not outpace governance. For example, decisions affecting customers, employees, finances or critical operations need stronger testing, monitoring and human oversight.

“Singapore does not need to choose between moving fast on AI and using it responsibly,” said Garavello.