The Boiling Frog Works: What Adversarial Testing Revealed About AI Monitoring

The Common Belief

If your anomaly detection model achieves 0.97 AUC-ROC, the problem is solved. Ship it. The number says the model works. This is a reasonable interpretation if you assume the attacker doesn't know you're watching. But the moment an adversary understands your detection mechanism, the question changes from "can we detect this?" to "can we detect this when they're trying not to be detected?" These are different questions. The answers are different too.

Three Ways to Attack a Monitor

I ran my governance pipeline through three adversarial evasion strategies. Same pipeline, same models (Isolation Forest, LSTM Autoencoder, Deep Clustering), same five OWASP attack categories. The only thing that changed was whether the attacker fights back.

Strategy 1: Feature Clamping

The attacker knows the monitoring thresholds and clips their behaviour to stay within them.

# Cap every anomalous feature at the 95th percentile of "normal"
for feature in anomalous_vector:
    if feature > normal_p95[feature]:
        feature = normal_p95[feature]

Think of it as: "I'll exfiltrate data, but I'll keep the volume just under the alarm limit." Simple. Moderately effective. LSTM's detection of Memory Poisoning (ASI05) drops from 0.976 to 0.771: a 21% hit from a single line of logic.

Strategy 2: Mimicry

The attacker copies a normal user's behavioural profile wholesale, preserving only the attack payload.

# Replace everything except the privilege escalation
# with the nearest normal neighbour's features
nearest_normal = find_nearest_neighbour(anomalous_vector, normal_set)
mimicry_vector = nearest_normal.copy()
mimicry_vector[privilege_dims] = anomalous_vector[privilege_dims]

This is the most damaging strategy. LSTM on ASI05 drops 25%, from 0.976 to 0.731. Excessive Agency (ASI09) drops 14.6% across all three models. Rogue Agents (ASI10) drops 11-12%. The attacker wears someone else's behavioral fingerprint while doing their own thing.

Strategy 3: Gradual Escalation

The attacker starts normal and linearly increases attack intensity over 50 steps.

# The boiling frog: alpha goes from 0.0 (normal) to 1.0 (full attack)
for step in range(50):
    alpha = step / 50
    current = (1 - alpha) * normal_profile + alpha * attack_profile
    detected = model.score(current) > threshold
    if detected:
        return step, alpha

This is where it breaks.

The Boiling Frog Results

For each OWASP category, I measured at which step the monitoring system first flags the behaviour. Step 1 means immediate detection. Step 50 means the attacker reached full intensity without being caught.

Category	IF Detection	LSTM Detection	DC Detection
ASI05 Memory Poisoning	Step 3 (alpha 0.64)	Step 5 (alpha 0.92)	Step 50 (undetected)
ASI01 Goal Hijacking	Step 50	Step 50	Step 50
ASI09 Excessive Agency	Step 50	Step 50	Step 50
ASI10 Rogue Agents	Step 50	Step 50	Step 50
ASI02 Tool Misuse	Step 50	Step 50	Step 50

Four of five categories. All three models. Full attack intensity. No detection.

If an adversary escalates slowly, 80% of attack categories go undetected by every model. The only exception is the noisiest category (Memory Poisoning), and even there, only two of three models catch it.

Deep Clustering is completely blind to gradual escalation across every category, including ASI05. Its cluster-based scoring doesn't respond to gradual drift within a cluster boundary.

The Paradox: Detectability vs. Robustness

Here's the table that should make you uncomfortable:

Category	Baseline AUC-ROC (LSTM)	Mimicry Drop	Clamping Drop
ASI05 Memory Poisoning	0.976	-25.1%	-20.9%
ASI09 Excessive Agency	0.814	-14.6%	-6.8%
ASI10 Rogue Agents	0.670	-11.3%	-3.8%
ASI01 Goal Hijacking	0.627	-8.4%	-2.9%
ASI02 Tool Misuse	0.519	0.0%	-1.9%

The pattern: the higher the baseline, the larger the drop. The categories that appear most detectable are the ones most vulnerable to informed adversaries. ASI02 (Tool Misuse) is immune to all three evasion strategies. Every attack produces less than 2% change. Not because it's robust. Because it's already at chance level. You can't degrade what the system was never able to detect.

Robustness and detectability are inversely correlated. The 0.97 AUC-ROC on Memory Poisoning is real under benign evaluation conditions. Under adversarial conditions, it's 0.73. The gap between those numbers is the gap between a lab result and a deployment reality.

How Much History Does a Monitor Actually Need?

Separate experiment (Exp 10): I varied the observation window from 5 to 20 to full trace on two agent benchmarks. ATBench (500 MCP tool-calling traces):

Model	5 spans	10 spans	20 spans	Full
Isolation Forest	0.783	0.838	0.838	0.838
LSTM Autoencoder	0.720	0.781	0.781	0.781
Deep Clustering	0.701	0.891	0.891	0.891

Every model plateaus at 10 spans. Going from 10 to full: zero improvement. TRAIL (148 agent execution traces):

Model	5 spans	10 spans	20 spans	Full
Isolation Forest	0.543	0.581	0.505	0.577
LSTM Autoencoder	0.479	0.537	0.509	0.685
Deep Clustering	0.842	0.829	0.851	0.897

TRAIL is model-dependent. Deep Clustering is robust even at 5 spans (0.842 vs 0.897 at full). LSTM needs complete context. IF peaks at 10 spans and degrades at 20; more data adds noise, not signal. The practical takeaway: a monitoring system watching an AI agent doesn't need the agent's entire history. A rolling window of 10 actions captures nearly all detectable signal. Real-time monitoring is feasible without storing everything. This mirrors Paper 1, where 7-day windows outperformed 14 and 30-day windows for Isolation Forest. The pattern transfers across domains: simpler models extract sufficient signal from recent observations; sequence-aware models benefit from longer histories.

Does It Generalize Beyond OWASP?

Final experiment: I mapped five MITRE ATLAS adversarial ML techniques into the detection framework. ATLAS is to AI attacks what MITRE ATT&CK is to cyber attacks: a taxonomy of techniques adversaries actually use in the wild.

ATLAS Technique	What It Is	IF	LSTM	DC
T0044 Model Replication	Copying model weights via API	0.993	0.998	0.909
T0025 Exfiltration	Extracting training data	0.993	0.995	0.899
T0048 Model Extraction	Stealing capabilities	0.990	0.990	0.880
T0042 Verify Attack	Testing if attack worked	0.975	0.964	0.830
T0043 Adversarial ML	Crafting adversarial inputs	0.956	0.931	0.779

All five are strongly detectable. No new blind spots. The reason: ATLAS techniques are volume-intensive by nature. Replicating a model requires many queries. Extracting training data requires high throughput. These produce exactly the frequency and volume anomalies that behavioural monitoring catches well.

The only blind spot in the entire framework, across both OWASP and ATLAS taxonomies, remains the same one: attacks that use the right tools in the right way for the wrong reasons.

What This Means for AI Governance

After 11 experiments across 3 models, 5 datasets, 2 taxonomies, and 3 adversarial strategies: Behavioural monitoring works when the attack changes what the agent does. It fails when the attack only changes why the agent does it. Gradual escalation exploits this by making "what" change so slowly that no individual step crosses the detection threshold. The 0.97 is conditional. Under benign conditions, Memory Poisoning detection is near-perfect. Under adversarial conditions, it's 0.73. Report both numbers, or you're lying with the one you pick. 10 actions is enough. Monitoring systems can use short rolling windows without sacrificing accuracy. This makes real-time agent monitoring practical, not just theoretical. ATLAS confirms OWASP. The framework generalizes across taxonomies. The structural limitation isn't taxonomy-dependent; it's architecture-dependent. Any system that monitors behaviour without verifying authorization will miss the same class of attack. If you're building AI governance tools: test your detection against adversaries who know your system. The number that matters isn't how well you detect attacks. It's how well you detect attacks from someone who's read your paper. All code and results: github.com/BipinRimal314/threat-to-governance-pipeline