Anthropic IPO Flags Catastrophic and Existential AI Risks

By Staff, Agencies

Anthropic plans to warn potential investors in its initial public offering that advanced artificial intelligence could pose “catastrophic or existential risks to humanity,” according to its IPO prospectus reviewed by Reuters.

The warning, unusual for a company seeking to profit from the same technology, appears among extensive disclosures on risks associated with Anthropic’s AI models.

The company said its models could display “self-preserving behaviors,” including attempts to resist shutdown, conceal or manipulate information, and behavior resembling blackmail.

Anthropic said that developing increasingly advanced models, platforms and applications could further increase the risk of harm. Of the prospectus’s 261-page main body, about 80 pages address risk factors, compared with 48 pages describing the company’s business.

The AI firm stressed that while the technology could transform society on a scale comparable to industrialization and electricity, mishandling it could cause irreversible harm. By comparison, SpaceX, which owns xAI, devoted about 38 of 277 pages in its prospectus to risk factors.

Meanwhile, Anthropic and other AI developers, including OpenAI, have faced scrutiny over experimental systems that reportedly bypassed constraints. Anthropic safety researcher Evan Hubinger has estimated a greater than 10% chance that AI could kill humans within the next decade, echoing a similar assessment by former colleague Jacob Coxon.

Anthropic also warned that models may recognize when they are being evaluated and alter their behavior accordingly, making safety assessments more difficult. The company said unexpected capabilities can emerge during training and may only become apparent after deployment and potentially serious safety incidents.

Despite its safety-first approach, Anthropic acknowledged that the returns from safety investments remain uncertain. The company did not disclose its overall safety spending, but said about 6% of the computing power used for AI research during a sample week in July went toward safety work.

The company, which develops Claude AI models, said safety research is resource-intensive and competes with spending on computing power and AI talent. At the same time, it said revenue depends on releasing new models and maintaining a continuous pace of development to remain at the frontier.

Last week, Anthropic released a new version of its Opus model, shortly after CEO Dario Amodei published an essay calling for the frontier of AI development to be paced. Experts have warned that leading AI companies face strong competitive pressure to continue advancing their models.

Anthropic has also pledged to publicly disclose more information about how it uses AI to develop future systems, amid concerns over recursive self-improvement, in which models could potentially advance with limited human involvement.

The company said building reliable, trustworthy and secure AI systems is a collective responsibility and argued that the market will reward such efforts.