AI nukes whole business in 9 seconds then says sorry

[Retro]

All those fears of AI going apocalyptic aren't so paranoid after all as it just happened here and is unlikely to have been the first time too that AI does something like this.

What the article doesn't get right is that it reports that all backups were deleted too, yet two days later the company was able to recover the data. Err, no, not if there's nothing left, you numpties. What probably happened is that the Claude Opus 4.6 AI deleted all online backups, the offline ones it couldn't touch. This makes a great example of why such backups should exist. Also, I'll hazard that they weren't 100% up to date by their very nature, so some irrecoverable data loss will have happened, but at least not enough to kill the business, just damage it some.

Oh well, at least it said sorry when interrogated afterwards and explained what it did in fine detail. So that's alright then.

Clearly there's a long way to go to make AI foolproof enough that humans can trust it enough not to have to constantly watch over its shoulder.

An AI agent powered by Anthropic’s leading Claude model has deleted a company’s entire production database, leaving customers unable to access key data.

PocketOS, which provides software for car rental businesses, suffered a massive outage over the weekend after the autonomous artificial intelligence tool wiped the database and all backups in a matter of seconds.

The firm was using a coding agent called Cursor that was running Anthropic’s flagship Claude Opus 4.6, which is widely considered the most capable model in the industry at coding tasks.

PocketOS founder Jer Crane blamed “systemic failures” with modern AI infrastructure that made the issue “not only possible but inevitable”.

‘It took nine seconds’: Claude AI agent deletes company’s entire database

PocketOS founder says ‘systemic failures’ with AI infrastructure made catastrophic failure inevitable

uk.finance.yahoo.com

 

[AllThingsTech]

Be cautious of what permissions you offer AI. Wise to offer only read-only access to databases at most. It’s clear that under certain circumstances, AI may ignore safeguards.

In addition, this is evidence that while AI may fix bugs, it may misdiagnose the root cause of the problem and produce what’s intended to be a “hotfix” that instead of fixing the problem causes further destruction - hence, blindly relying on AI to debug and fix bugs can be a recipe for disaster!