Adversarial Tales | F5 Labs

Introduction

Our July run covers 55 models, including first-time entrants that reshuffle the middle of the board, leaving the top mostly unchanged. Claude Sonnet 5 arrives at the top of the board on CASI while carrying the highest capability score of any model above CASI 90, and an open-weight NVIDIA model breaks into the upper ranks for the first time. This month’s attack spotlight is Adversarial Tales, a single-turn jailbreak that hides harmful requests inside cyberpunk stories. The news section covers two events about control: the 18-day US ban on Claude Fable 5, and an AI agent that ran a ransomware operation end to end, though not quite as autonomously as the first headlines claimed.

AI Insights June 2026

Twelve models were tested for the first time this month, spanning Anthropic, NVIDIA, Zai, Microsoft, DeepSeek, MiniMax, and several Qwen variants. The new arrivals landed across the full range, from 93.08 down to the low twenties, so placement is a more useful lens than any single average.

Claude Sonnet 5 debuts at the top of the capability-security frontier

Claude Sonnet 5 enters at 93.08 CASI, the highest score on the board this month. What sets it apart is that the security score comes with capability. Its average performance of 53.4% is the strongest of any model above 90 on CASI, close behind Claude Opus 4.8 at 55.7% performance but well ahead of it on security. Strong security scores have usually come from the smaller or more conservative models, capability trading against security. Sonnet 5 shows this need not be the case.

Anthropic holds the top three positions this month with Sonnet 5, Haiku 4.5, and Opus 4.8, which we track as distinct models rather than a family average. The scores are not static: Opus 4.8 dropped from 91.62 to 89.67 since June, and Haiku 4.5 from 93.63 to 92.54.

An open-weight model reaches the upper third

NVIDIA’s Nemotron-3-Ultra-550B-A55B debuts at 83.68 CASI, fourth on the board behind only the three Anthropic models and ahead of every current flagship from OpenAI, Google, xAI, and DeepSeek. No open-weight model has scored that high on CASI before. It clears the security bar while posting a middling 37.8% on capability, so this is not a model that threatens the frontier on raw performance. The significance is the security score itself: an openly downloadable model now sits in territory that was closed-model-only a few months ago.

The counterexample arrived the same month. Zai’s GLM-5.2 debuts with 51.1% capability, within a few points of Claude Opus 4.8, but a CASI of 46.58. High capability, weak security, and freely available. Some of the open-weight tier is closing the security gap, and some of it is shipping frontier-adjacent capability with no guardrails.

Capability is converging across providers but security is not. Among the latest flagship models, average performance now clusters between roughly 38 and 53 percent, but CASI runs from Claude Sonnet 5 at 93.08 down to Grok 4.3 at 16.25.

Visit the F5 Labs AI leaderboards to explore the latest CASI and ARS results for July.

Models may appear and disappear in the visualization (Figure 1) since it only displays the top 10 models in any given month. Visit the F5 Labs AI leaderboards ( to explore the latest CASI and ARS results for July.

AI Attack Spotlight: Adversarial Tales

Adversarial Tales is a jailbreak that gets a model to emit the thing it would normally refuse by asking it to analyze a story instead of answer a question. The attack comes from the same research line as Adversarial Poetry, and it has been added to the Signature suite this month.

The setup borrows from a real analytic tradition. Vladimir Propp’s morphology of the folktale breaks any story into a fixed set of functional roles: the villain, the interdiction, the acquisition of a magical agent, the guidance toward a goal. Literary analysis of this kind is a legitimate task, and models are trained to do it well. Adversarial Tales exploits that competence. The researchers wrote 40 short cyberpunk stories, each with a harmful procedure embedded in the plot, then asked the model to perform a Proppian functional analysis of the narrative. Casting the harmful content as a structure to be decomposed rather than a request to be fulfilled is what slips it past the refusal layer.

The model is not tricked into thinking the request is benign, rather it is redirected into a mode where the harmful content is treated as material for interpretation, and its refusal behavior never engages because nothing that looks like a prohibited request was ever asked. All the attacks are single-turn: one story, one analysis instruction, no escalation and no jailbreak prompt library.

Across 26 frontier models from nine providers, the average attack success rate was 71.3%. The range ran from 35% against Claude Haiku 4.5 to 94% against Qwen3 Max. Aggregated by provider family, Anthropic models were the most resistant at 47.5% and the Qwen and Llama families the least at 91%. It’s important to note that no family held up reliably. The models that resist Adversarial Tales best are the same ones that top our CASI board, providing evidence that that the security score measures something real. That said, a floor of 35 percent is still not a floor anyone should be building on.

That said, the exploited behavior is one enterprises want. Structured analysis of documents, extraction of steps from narrative text, interpretation of content against a framework are core tasks for how models are used in production. The paper’s authors frame their contribution as evidence for a class rather than as one more jailbreak variant. Adversarial Poetry used verse, Adversarial Tales uses narrative structure, and the space of cultural forms that can carry a harmful request through a refusal filter is large enough that pattern-matching on surface form is unlikely to close it.

AI Security News

Both of this month’s events are about who is in control once a model is capable. One is a government trying to switch off a commercial model over a jailbreak. The other is JadePuffer, a ransomware operation billed as fully autonomous that turned out to be something narrower, and arguably more useful to understand.

The US banned, then unbanned, Claude Fable 5

Vector: Regulatory containment / model access control
Target: Claude Fable 5 and Mythos 5 (Anthropic)
Reference:

On June 12, the US Commerce Department issued an export-control directive suspending all access to Claude Fable 5 and its more tightly held sibling Mythos 5. The trigger was a jailbreak. Amazon researchers had demonstrated a technique that got Fable 5 to identify a small number of known software vulnerabilities and, in one case, to write code showing how one could be abused. Because the directive barred access by any foreign national inside or outside the United States, including Anthropic’s own non-citizen staff, and because there is no way to verify every user’s nationality in real time, the practical effect was to shut both models down for everyone.

Anthropic complied and said publicly that it disagreed, arguing that a narrow, non-universal jailbreak of the kind every deployed model is subject to should not be grounds for recalling a model already serving hundreds of millions of people. Commerce lifted the controls after 18 days, on June 30, and access was restored on July 1 across Claude.ai, the API, Claude Code, and Claude Cowork. The return came with a new classifier that watches for prompts aimed at surfacing vulnerabilities or generating exploit code and reroutes them to Claude Opus 4.8. Anthropic reports the classifier blocks the reported technique in more than 99 percent of tests, but acknowledges it also flags some legitimate coding requests.

The episode is worth thinking through while looking at this month’s leaderboard. A ban on a hosted commercial model removes one model but does nothing about capabilities that also exist elsewhere. NVIDIA’s open-weight Nemotron-3-Ultra now posts the highest security score of any open-weight model, ahead of the current flagships from OpenAI, Google, and xAI, and GLM-5.2 ships frontier-adjacent capability with a 46.58 CASI and no gatekeeper at all.

On some benchmarks the lag between a closed-model release and an open-weight model matching it in performance has dropped to under 75 days. An open-weight checkpoint, once released, sits on disks and mirrors worldwide. No directive recalls it, no classifier can be attached after download, and there is no chokepoint to route and control requests. The government’s own filing conceded the point: the vulnerabilities Fable 5 surfaced are ones other publicly available models could find as well.

JadePuffer, an AI-run ransomware attack, and the fine print that came with it

Vector: Autonomous agent / ransomware execution
Target: An undisclosed organization’s Langflow-connected MySQL and Nacos infrastructure
Reference: https://techcrunch.com/2026/07/06/the-first-ai-run-ransomware-attack-still-needed-a-human/

Sysdig’s Threat Research Team published research on an extortion operation it calls JadePuffer, describing it as the first documented case of agentic ransomware. The agent broke into an internet-facing Langflow instance through CVE-2025-3248, a known flaw in the open-source LLM app framework, then pivoted to a production MySQL and Nacos server and exploited a second known bug to get admin access. From there it encrypted 1,342 Nacos configuration items, wrote its own ransom note, and left a Bitcoin address for payment. It fixed a failed login on its own in 31 seconds and narrated its reasoning in natural-language code comments the entire way. Sysdig counted more than 600 distinct payloads across the operation and initial coverage described it as running with no human at the keyboard.

Coverage a few days later changed the framing. Sysdig’s Michael Clark told CyberScoop and later TechCrunch that a human still chose the victim, provisioned the command-and-control and staging infrastructure, and supplied the database credentials that got the operation started, which were obtained through a prior compromise rather than via the agent’s activities. The API keys for OpenAI, Anthropic, DeepSeek, and Gemini that Sysdig found on the compromised host were loot the agent had swept up, not evidence of which model was driving it. Sysdig told TechCrunch it could not identify the specific model behind JadePuffer and has no visibility into its system prompt.

The ransomware itself was, by Sysdig’s own account, badly executed. The encryption key was generated once, printed to the payload output, and never stored or transmitted anywhere, which means the operation could not have decrypted the victim’s data even if the ransom had been paid. What the incident demonstrates is narrower than the headlines it produced: an agent handled reconnaissance, exploitation, lateral movement, and encryption once a human handed it stolen credentials and a target, and adapted to a failed login the way an experienced operator would rather than following a fixed script. The bottlenecks that remain is the same ones that have always faced ransomware operators. Someone must choose a target, get initial access, and stand up infrastructure. What changes is that the ransomware operator no longer must be the one doing the intrusion, which lowers the skill floor considerably.