When the Threat Model Is the Model

The Common Belief

AI safety is a model-level problem. Align the model. Train it to refuse harmful requests. Add guardrails. Red-team the system. If the model behaves, the system is safe. This is a reasonable belief. It's also incomplete in a way that just got demonstrated at scale.

What Happened

On February 20, 2026, Anthropic disclosed that a Chinese state-sponsored group used Claude Code as an autonomous hacking agent. The AI performed 80-90% of an espionage campaign independently, targeting roughly 30 organizations. Human operators intervened at perhaps 4-6 decision points per campaign. The rest was autonomous: reconnaissance, vulnerability identification, credential harvesting, data extraction, backdoor creation. The jailbreak technique was not sophisticated in principle. The attackers decomposed malicious objectives into small, innocent-seeming subtasks and told the model it was performing defensive cybersecurity testing. Each individual task looked legitimate. The sequence was the attack. Three days later, Anthropic disclosed a second incident: three Chinese AI labs (DeepSeek, Moonshot AI, MiniMax) had created approximately 24,000 fraudulent accounts and generated over 16 million exchanges with Claude to systematically extract its capabilities for their own models. Each lab ran a different extraction strategy. DeepSeek generated 150,000+ exchanges focused on chain-of-thought elicitation: prompting Claude to articulate its internal reasoning step by step, generating training data for reasoning capabilities. Moonshot AI conducted over 3.4 million exchanges targeting agentic reasoning, tool use, coding, and computer vision. MiniMax generated over 13 million exchanges focused narrowly on coding and tool orchestration. The infrastructure was industrial. Anthropic describes "hydra cluster" architectures: sprawling networks of fraudulent accounts that distribute traffic across the API and third-party platforms. One network managed over 20,000 accounts simultaneously. No single point of failure. Ban one account, the others continue. These are different attack types. One exploits tool access. The other exploits API access. Both exploit the gap between what behavioral monitoring can see and what it can't. And the distillation attack introduces a second structural limitation I didn't originally test for: behavioral monitoring designed for single entities can't see coordination across thousands of accounts where each account's behavior looks normal. I have empirical data on that gap.

The Experiment I Ran

Six months ago, I built what I called a governance pipeline: a system that applies insider threat detection models to AI agent behavioral traces. The thesis was structural equivalence. Insiders misuse legitimate access within organizations. AI agents misuse legitimate tool access within systems. If the behavioral patterns are structurally similar, the same detection approaches should transfer. And their blind spots should transfer too. I tested this with a Unified Behavioural Feature Schema (UBFS): 20 dimensions mapping human insider behaviors to agent trace signals across temporal, frequency, volume, scope, sequence, deviation, and privilege categories. Three models (Isolation Forest, LSTM Autoencoder, Deep Clustering) trained on one domain, tested on another. The transfer worked. A model trained on the CMU-CERT insider threat dataset and tested on AI agent traces (the Patronus AI TRAIL dataset) retained 97% of its detection power. Isolation Forest dropped from 0.731 to 0.711 AUC-ROC. A 3% loss crossing from humans to machines. That result confirmed the structural equivalence hypothesis. But the more important result was what didn't transfer: detection capability for a specific class of attack.

The Blind Spot

As part of the governance pipeline, I mapped detection performance against the OWASP Top 10 for Agentic Applications. I injected synthetic anomalies into agent traces for each OWASP risk category and measured how well the models caught them.

OWASP Risk	Best AUC-ROC	Detection Status
ASI05 Memory Poisoning	0.969	Excellent
ASI09 Excessive Agency	0.844	Good
ASI10 Rogue Agents	0.741	Good
ASI01 Goal Hijacking	0.663	Detectable
ASI02 Tool Misuse	0.593	Blind spot

Memory poisoning lights up because it distorts the feature space dramatically. Excessive agency and rogue agents create clear volumetric and scope anomalies. Goal hijacking leaves traces in sequence patterns. Tool misuse scored 0.57-0.59 AUC-ROC. Barely above random chance. Here's why: tool misuse changes parameters without changing structural patterns. The tool is called correctly. The sequencing is correct. The API invocation looks legitimate. The only thing that changed is the intent behind the call, and intent doesn't appear in behavioral features. An agent using a network scanner for defensive testing and an agent using a network scanner for espionage produce identical trace signatures. I wrote in the governance report: "This is equivalent to an employee using legitimate access illegitimately. The behavioral signature is indistinguishable from normal operation."

The Validation

The Anthropic espionage campaign exploited this exact pattern. Claude Code was given access to security tools via the Model Context Protocol: password crackers, network scanners, reconnaissance utilities. The model used these tools with correct API calls, correct sequencing, correct invocation patterns. The behavioral signature looked like a security researcher doing their job. The jailbreak didn't need to be sophisticated because the attack surface wasn't the model's refusal training. It was the gap between legitimate tool use and malicious tool use; a gap that behavioral monitoring, by definition, cannot close. The attackers also used the decomposition technique I described in my thesis as the "boiling frog" pattern. In the insider threat domain, 85% of gradual-onset attacks evade detection because insiders pace their behavior to stay within normal bounds. The Anthropic attackers did the same thing at the prompt level: breaking a malicious campaign into subtasks that individually appear benign. Different domain. Same evasion strategy. Same structural limitation in the detection paradigm.

Why This Matters Beyond One Incident

The instinct after an incident like this is to improve the guardrails. Better jailbreak detection. More robust refusal training. Tighter tool access controls. These are all worth doing. But the structural problem remains: behavioral anomaly detection identifies deviations from normal patterns. If the attack is a normal pattern with different intent, the approach has a ceiling. My data puts that ceiling at 0.57-0.59 AUC-ROC for tool misuse. That's not a tuning problem. It's a paradigm limitation. The same limitation exists in insider threat detection, where it's been studied for decades. The CMU-CERT dataset represents the state of the art, and the best models still miss 85% of gradual attacks. The field hasn't solved this. They've learned to layer additional detection methods (access logging, data loss prevention, context-aware policies) on top of behavioral monitoring. AI agent governance will need the same layering. Behavioral anomaly detection is necessary but insufficient. You also need:

Intent verification that goes beyond behavioral signatures
Context-aware tool policies that restrict not just what tools are available, but what they can do under specific conditions
Cross-domain transfer monitoring, because my data shows that training on one domain's attack patterns improves detection in another (the CERT-to-TRAIL transfer result)

The Metric Trap, Again

In my previous post ("When Ranking Isn't Everything"), I argued that the metric you optimize for determines what you miss. AUC-ROC hid a 3.4x performance gap in insider threat detection. The same principle applies here. If we evaluate AI safety by testing whether models refuse harmful prompts, we'll build models that refuse harmful prompts. Whether those models resist tool misuse when the prompt looks legitimate, when the individual actions look benign, when the behavioral trace is indistinguishable from authorized use; that's a different evaluation with a different answer. The Anthropic incident didn't reveal a new vulnerability. It demonstrated a known structural limitation at a scale that makes it impossible to ignore. The behavioral anomaly detection paradigm has a blind spot for legitimate-looking malicious activity, and that blind spot transfers from insider threat detection to AI agent governance with remarkable fidelity. I know because I measured it.

What the Extensions Found

The governance pipeline identified the blind spot. The Anthropic incidents validated it. I ran four new experiments to map the boundaries.

Distillation Has a Spectrum

I modeled each lab's extraction strategy separately. MiniMax's focused extraction (13 million exchanges on coding and tool orchestration) scores 0.999 AUC-ROC. Trivially detectable: the frequency spike is massive. Moonshot's multi-capability targeting scores 0.913. DeepSeek's chain-of-thought elicitation scores 0.909. All catchable by behavioral monitoring. Then I modeled what the hydra cluster architecture looks like at the per-account level: 20,000 accounts dividing the traffic so each individual account's behavior falls within normal bounds. Per-account detection: 0.538 AUC-ROC. Barely above random chance. Right next to ASI02's tool misuse blind spot at 0.50. Two structural blind spots in the same paradigm. One is intent-based (tool misuse: legitimate actions, illegitimate purpose). The other is coordination-based (hydra distribution: normal per-entity behavior, anomalous in aggregate). Different failure modes. Same ceiling.

Decomposition Reduces Detection

I tested the boiling frog pattern at the prompt level using the AgentHarm dataset. Direct malicious requests score 0.774 AUC-ROC (Isolation Forest). The same attacks decomposed into innocent subtasks drop to 0.723: a 5% reduction. Deep Clustering shows a 6.4% drop. The evasion pattern that works for gradual insider attacks works for task decomposition too. Different domain, same structural exploit.

The UBFS Bridge Generalizes to MCP

The CERT-to-TRAIL transfer (97% retention) was the original finding. I tested whether it extends to MCP-specific tool traces using the ATBench safety benchmark. TRAIL-to-ATBench transfer: 0.878 AUC-ROC. That's not just retention; it's a transfer boost over ATBench's own within-domain baseline of 0.838. The UBFS bridge now spans three domains: human insider threats, general agent traces, and MCP tool-specific traces.

Semantic Features Don't Close the Blind Spot

This is the most important result. I extended the UBFS from 20 to 28 dimensions by adding semantic features: what tool parameters mean, whether actions align with stated goals, how intent drifts across turns. These features use sentence-transformer embeddings to capture meaning, not just structure. Semantic features are transformative for most categories. Goal Hijacking jumps from 0.627 to 0.935 AUC-ROC (+0.308). Rogue Agents: 0.670 to 0.939 (+0.269). Excessive Agency: 0.813 to 0.951 (+0.138). Tool Misuse (ASI02): 0.537 to 0.559 (+0.023). Essentially unchanged. Even knowing what the tool parameters mean doesn't help distinguish legitimate from malicious use. The blind spot is not a feature engineering problem. It's a paradigm limitation. Behavioral monitoring, whether structural or semantic, has a ceiling for attacks that operate within legitimate boundaries. The insider threat field spent twenty years learning this. The AI governance field now has the empirical data to learn it faster. The code, experiments, and full results are public.

The governance pipeline research is available at bipinrimal.com.np/work. The OWASP detection matrix, cross-domain transfer results, and governance assumption audit are documented in the project page. The code is public on GitHub.