When Ranking Isn't Everything

The Best Model Lost

For my MSc thesis, I compared four unsupervised anomaly detection models on the CMU-CERT insider threat dataset. The question was straightforward: which approach best identifies malicious insiders from behavioral data when you don't have labeled examples to train on? By the standard ranking metric (AUC-ROC), the answer was clear. Isolation Forest scored 0.799. LSTM Autoencoder scored 0.770. PCA Reconstruction and Dense Autoencoder trailed further behind. If I'd stopped there, the conclusion writes itself: tree-based methods outperform neural approaches for this task. Paper done. I didn't stop there.

What Happens at the Threshold

AUC-ROC measures how well a model separates threats from normal behavior across all possible thresholds. It's the area under the entire curve. It's a useful summary statistic. And it's almost completely irrelevant to how security teams actually work. A security operations center doesn't investigate every alert. They can't. At a large organization, even a 5% false positive rate generates hundreds of cases per day. A 10% false positive rate is already pushing the limits of what analysts can handle before alert fatigue sets in and they start ignoring everything. So the real question isn't "which model ranks threats best overall?" It's "which model catches the most real attacks within the budget of alerts my team can actually investigate?" That question has a different answer.

Model	AUC-ROC	Recall @ 5% FPR	Recall @ 10% FPR
Isolation Forest	0.799	0.044	0.220
LSTM Autoencoder	0.770	0.149	0.254
PCA Reconstruction	0.612	0.049	0.129
Dense Autoencoder	0.659	0.048	0.118

At 5% false positive rate (the threshold where a security team might realistically investigate every flagged case), the LSTM Autoencoder detected 14.9% of insider threats. Isolation Forest detected 4.4%. That's 3.4 times more attacks caught. Same data. Same operating conditions. The model that lost on the ranking metric won where it mattered.

Why Temporal Modeling Mattered

The difference comes down to what each model can see. Isolation Forest treats each data point independently. It asks: "Is this single observation unusual compared to the distribution of all observations?" It's excellent at catching statistical outliers: the employee who suddenly downloads 500 files at 3am. But most insider threats aren't single-event anomalies. They're behavioral shifts over time. An employee gradually increases their access scope. Downloads tick up slowly over weeks. Login patterns drift. Any single data point looks plausible. The sequence is what's suspicious. The LSTM Autoencoder learns temporal patterns (normal sequences of behavior) and flags deviations from those sequences. It catches the slow drift that Isolation Forest can't see because each individual point in the drift looks normal. At low false positive rates, where security teams actually operate, this temporal awareness makes the difference between detecting an insider and missing one.

The Boiling Frog Problem

I should be honest about what none of the models solved. The CMU-CERT dataset includes what I started calling "boiling frog" attacks: insiders whose malicious behavior is nearly indistinguishable from their normal workflow. No sudden spikes. No obvious anomalies. Just a slow, deliberate escalation that stays within the noise of legitimate activity. Across all four models, roughly 85% of these gradual-onset threats went undetected. This isn't a failure of the models specifically. It's a fundamental limitation of the unsupervised anomaly detection paradigm. If the attack looks like normal behavior, a system trained to identify deviations from normal behavior won't catch it. The approach has a structural blind spot, and no amount of architectural sophistication closes it entirely. Knowing where your method fails is as important as knowing where it succeeds. Most papers don't report this. They should.

The Metric Trap

Here's the broader pattern: the metric you optimize for determines what you miss. AUC-ROC is the gold standard for binary classification evaluation. It's in every textbook. It's the first number reviewers look for. And it averages performance across a range of operating conditions that most real systems will never encounter. If your security team can tolerate a 50% false positive rate, AUC-ROC is highly informative. If they can tolerate 5%, you need a different lens. This isn't unique to security. In machine learning research, we optimize for benchmark scores. A model that achieves state-of-the-art on ImageNet gets published. Whether that model works reliably in a hospital, under different lighting, on patient populations underrepresented in the training data: that's a different question with a different answer. The benchmark becomes a proxy for quality, then the proxy becomes the goal, then we optimize for the proxy at the expense of the thing it was supposed to measure. In hiring, the ranking metric is the resume. Keywords, credentials, years of experience. The operational metric is whether someone can actually do the job. Everyone knows these aren't the same thing. We use the ranking metric anyway because it's measurable, and measurable feels objective, and objective feels fair. In AI governance (and this is where the pattern gets consequential), how we choose to evaluate AI systems will determine what behaviors those systems exhibit. If we measure safety by benchmark performance on curated test sets, we'll get systems that pass benchmarks. Whether those systems behave safely in deployment, under adversarial conditions, in contexts the benchmarks didn't anticipate; that depends on whether our evaluation metrics capture what actually matters at the thresholds where the systems actually operate.

The pattern is always the same: a summary metric that's easy to compute and compare becomes the standard, and the gap between that metric and operational reality becomes invisible precisely because nobody is measuring it.

The Reframe

I started this thesis trying to find the best model for insider threat detection. I ended it with a different question: best according to whom, under what constraints, at what threshold? The model that ranks highest isn't always the one that works best. The metric that's easiest to compute isn't always the one that matters. The evaluation framework that everyone uses isn't always the one that captures the thing you're trying to protect against. It's not that AUC-ROC is wrong. It's that AUC-ROC is incomplete. And the distance between a good ranking and a good outcome is exactly the space where threats (and opportunities) hide. The question isn't just "how do we score?" It's "what does the score miss?" That's the question worth optimizing for.