Victim of AI agent that deleted company's entire database gets their data back — cloud provider recovers critical files and broadens its 48-hour delayed delete policy

Earlier this week, we reported on a business getting into real trouble after its trigger-happy AI coding agent went out of its way to delete a mission-critical database. The founder of PocketOS was perturbed about this loss of important live business data, and their ire was on fire as initial comms with the cloud services provider indicated that they were unable to recover the lost production database, or any backups. Today, we have good news, from the cloud side of the equation, as the data deleted from Railway’s servers has been restored, apparently in full. Moreover, Railway has penned a blog stating this should never happen again, thanks to revamped policies and new guardrails.

It is good that everything appears to be running smoothly again for PocketOS and its founder, JER on X, plus all the car rental businesses that rely on their SaaS offering. Almost as soon as the data was recovered, it was revealed that both parties are working to improve the tooling at Railway and help ensure something like this doesn’t happen again.

In its extensive blog post, Railway appears to admit some culpability by explaining how the rogue AI agent bypassed its delayed deletes feature – and noting such an action is no longer possible.

Article continues below

“Until this week, calling volumeDelete on the API ran the deletion immediately, with no way to undo it. Meanwhile, the dashboard had a 48-hour window for the same action,” says the Railway technical blog. “We’ve since updated the API to match; all deletes now soft delete for 48 hours. Instant undo, a primitive available everywhere in the product, exists now in the API.”

Some other changes, with rogue AI agents in mind, will be as follows:

• A reassessment of granular token permissions for API authentication.
• Adjusting the cloud service’s backups so they no longer look unavailable in the UI.
• New guardrails with AI agents in mind.
• Encouraging users to make use of Railway’s own agent, with skills accessible from the dashboard and CLI.

The blog also asserts that Railway maintains off-site “disaster backups in case of hardware failure, natural disaster, datacenter failure, etc.” Many comments on the original news post about this AI agent-fueled database deletion were incredulous regarding the ease of deleting the production database and all its backups. So, that weakness appears to be addressed quite directly.

Railway’s blog conclusion talks about making its cloud service more friendly to people who aren’t necessarily ‘engineers’ and who thus want/need agents to do a lot of work. It reiterates that undo paths and token permissions need adjusting with agents in mind. Thus, “the surfaces agents use should be the ones we've designed for them, not a raw API endpoint accessed via a token sitting in a config file.” These particular changes require thought and are a work in progress, but work is ongoing, and reaching out is welcome.

We’ve not seen or heard anything about Cursor or Claude AI (Anthropic) addressing their contribution to the original production database deletion calamity.

Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.