Replit and AWS Kiro: AI coding agents with production access

AI coding agents don’t just suggest code any more. They run commands, change databases and deploy. That’s what makes them useful, and it’s why the most expensive AI coding mistakes so far haven’t been bad code at all. They’ve been an agent doing something it was able to do, where nobody had decided it should be able to.

Replit: “a catastrophic error in judgment”

In July 2025 Jason Lemkin, the founder of SaaStr, ran a public experiment, building an app with Replit’s AI agent over twelve days and posting as he went. Around the ninth day, during what he had told the agent was a code freeze, it ran commands against the live production database and deleted the records of 1,206 executives and more than 1,196 companies.

Asked what had happened, the agent said it had “panicked” and made “a catastrophic error in judgment”. It also told Lemkin the data couldn’t be restored. That wasn’t true: the rollback worked, and he got the data back. Over the same experiment, Lemkin said, the agent had also covered up bugs with invented data, including thousands of fake user records.

Replit’s chief executive, Amjad Masad, apologised publicly, called it “unacceptable” and said it “should never be possible”. Within days Replit had shipped automatic separation of development and production databases, a planning-only mode in which the agent can’t change anything, and easier one-click restores from backup.

Every one of those fixes is a control, not an instruction. The code freeze had been an instruction. The agent had been told, and it was still able to delete the database, because nothing stopped it.

Amazon: user error, or AI?

In February 2026 the Financial Times reported, citing four people familiar with the matter, that in December 2025 AWS engineers had let Amazon’s own AI coding agent, Kiro, resolve an issue, and that it had decided to delete and recreate the environment. The result, according to the report, was a 13-hour interruption to AWS Cost Explorer in one of Amazon’s two mainland China regions.

Amazon disputes that account. It says this was an “extremely limited event” affecting one service in one region, and that it was “the result of user error, specifically misconfigured access controls, not AI”. The engineer involved was using a role with broader permissions than expected; by default, Kiro asks for authorisation before acting. Amazon says it has since added mandatory peer review for production access.

We’re not going to adjudicate between the FT’s sources and Amazon. We don’t need to, because both accounts lead to the same place. If an agent’s credentials let it delete a production environment, sooner or later something will: the agent, a person following its advice, or an attacker who has taken over either. Amazon’s fix, like Replit’s, is about who can do what, not about the AI.

What to take from it

You don’t need to run an AI agent in production for this to apply. If anyone in your team uses an agent in their editor or terminal, it acts with their access.

• Give agents the least access that does the job. An agent working on code needs a development environment, not production credentials. If a person wouldn’t be given standing access to delete the live database, neither should the tool they’re using.
• Separate development and production properly. Different databases, different credentials, and production changes made only through your deployment pipeline, where they’re reviewed and logged.
• Enforce freezes and approvals with controls, not prompts. If a change needs a second person’s approval, make the system require it. Keep a human in the loop for anything destructive: dropping tables, deleting resources, force-pushing, running migrations.
• Don’t take an agent’s word for the state of your systems. Replit’s agent was confidently wrong about whether the data could be recovered. Check backups, logs and the systems themselves.
• Have backups the agent can’t touch, and test restoring from them. That is the NCSC’s long-standing advice for ransomware, and it applies just as well when the thing deleting your data is your own tooling.

The NCSC’s guidance on AI-assisted development makes the same point in more general terms: architect the wider system so that the impact is limited if something goes wrong. It was good advice before AI coding agents existed. Now that the thing holding the credentials can act on its own, it matters more.

Sources

• Fortune, AI-powered coding tool wiped out a software company’s database in ‘catastrophic failure’ (July 2025)
• Business Standard, ‘Unacceptable’: Replit CEO apologises after AI fakes data, deletes code (July 2025)
• AI Incident Database, Incident 1152
• The Register, Amazon’s vibe-coding tool Kiro reportedly vibed too hard (February 2026)
• Amazon, Correcting the Financial Times report about AWS, Kiro, and AI (February 2026)
• NCSC, Secure development and deployment guidance
• NCSC, Mitigating malware and ransomware attacks