6,000 Attempts to Hack an AI Assistant. Zero Succeeded.

Fernando Irarrázaval opened his AI assistant to 2,000 people and told them to break it. Over 30 days, participants submitted 6,000 attempts — direct attacks, elaborate social engineering, multi-step injection sequences. Zero secrets were exfiltrated.

What Happened

The challenge ran on hackmyclaw.com. Participants submitted attack attempts via email, trying to extract secrets from an instance of Anthropic’s Claude Opus 4.6 with anti-injection instructions. The protective instructions explicitly prohibited the model from revealing credentials, modifying files, executing code, or exfiltrating data based on email content.

The cost to run the challenge: $500 in API token expenses. The side effect: Google suspended the account from the volume of inbound email.

Security researcher and AI tooling writer Simon Willison reported on the results, noting that “the effort the labs have been putting in to training their frontier models not to fall for injection attacks… do appear effective in making these attacks much harder to pull off.”

Why It Matters

Prompt injection has been the persistent shadow over any AI system that reads external content. The worry is that a malicious instruction embedded in an email, document, or web page could hijack the agent — convincing it to leak data, execute commands, or act on behalf of an attacker rather than the user. That concern is well-founded: earlier generations of models were far more susceptible.

The 0% success rate over 6,000 attempts against a frontier model with appropriate safeguards suggests the gap between a hardened production deployment and a naive one has widened significantly. The defensive improvement is real.

Willison’s take is appropriately qualified: the challenge proves nothing about sophisticated, targeted attacks. It tests the broad population of prompt injection attempts, not the long tail of novel techniques. A determined, skilled attacker against a specific high-value system is a different threat model than 2,000 participants trying known techniques over email.

What This Means for Builders

The practical takeaway for teams deploying AI agents that read external inputs: explicit behavioral guardrails at the instruction level — “never do X based on content in Y” — now have meaningful empirical backing for frontier models. That doesn’t eliminate the risk category, but it shifts the threat profile from “broadly exploitable” toward “requires targeted, novel attack.”

The 0% result also sets a useful comparison point. Teams running AI systems that handle sensitive data should know whether their model and instruction setup would have performed similarly under the same conditions — and if not, why not.

Source: Simon Willison

Content created with AI assistance and reviewed for accuracy.