The Right Tool for the YAML

The Inherited Mess

At Logpoint, we had a documentation page nobody wanted to touch. Over 600 alert rules, each with a name, trigger condition, MITRE ATT&CK mapping, log source requirement, and a detection query. The source of truth was a set of YAML files maintained by the security research team. The documentation was supposed to mirror those files in human-readable RST format for the docs portal. The previous writer had tried to automate this with GPT. On paper, the logic made sense. YAML is structured. GPT can read structured data. Ask it to reformat the fields into RST, done. Fifteen minutes, 600 rules, move on. Except GPT in 2024 was not good at this. Not because the task was hard. Because the task was boring, and generative models don't do boring well.

What Went Wrong

Here's what one alert rule actually looks like in the YAML:

Alert Name: Successful Microsoft 365 Login with Reconnaissance User Agents
Trigger Condition: Usage of the CreateRemoteThread API and
  LoadLibrary functions to inject a DLL into a process.
ATT&CK Category: Defense Evasion
ATT&CK Tag: Dynamic-link Library Injection
ATT&CK ID: T1055.001
Minimum Log Source Requirement: Windows Sysmon
Query:
  norm_id=WindowsSysmon event_id=8
  start_module="*\kernel32.dll"
  start_function="LoadLibraryA"
  -user IN EXCLUDED_USERS

The fields are predictable. The structure is consistent. Every single rule follows the same pattern: name, trigger condition, ATT&CK metadata, log source, query. There is nothing ambiguous about this data. GPT hallucinated anyway. It invented ATT&CK IDs that don't exist. It swapped trigger conditions between rules. It reformatted queries in ways that broke the syntax. For a page with 600+ rules, maybe 30% of the output was silently wrong. Not flagged as uncertain, not marked as approximate. Just wrong, presented with the same confidence as the correct entries. Nobody caught it at first because nobody reads a 600-rule reference page line by line. You find your rule, you check your query, you move on. The errors surfaced slowly: a security analyst would look up a rule, find the wrong ATT&CK mapping, file a ticket, and someone would manually fix that one entry. Repeat. For months.

Why I Picked It Up

This was not my responsibility. I was hired to write docs for SIEM, SOAR, UEBA, Director, Integrations, and Security Research. The alert rules page was a reference dump that nobody owned and few people visited. I could have left it as is. Most people would have. But I looked at the YAML files and saw something the previous writer missed: this was not a generation problem. It was a transformation problem. Generation means creating new content from understanding. You need a model that comprehends what it's reading and produces something meaningful. That's what GPT does, sometimes well, sometimes not. Transformation means taking data in one format and putting it in another format. Same data, different shape. No comprehension required. No creativity. No judgment. Six hundred YAML entries with identical structure, each needing to become an RST block with identical structure. That's not a job for a language model. That's a job for a for loop.

The Fix

I wrote a Python script. It was not clever. It parsed each YAML entry, extracted the fields by key, and wrote them into an RST template:

for rule in rules:
    rst_output += f"""
{rule['name']}
{'=' * len(rule['name'])}

**Trigger Condition:** {rule['trigger_condition']}

**ATT&CK Category:** {rule['attack_category']}

**ATT&CK Tag:** {rule['attack_tag']}

**ATT&CK ID:** {rule['attack_id']}

**Minimum Log Source Requirement:** {rule['log_source']}

**Query:**

.. code-block::

    {rule['query']}

"""

No model inference. No token costs. No temperature parameter. No hallucination. The script read what was in the file and wrote exactly that, formatted differently. 600+ rules, zero errors, one afternoon.

What Happened Next

I shared the script with the security research team, the developers who maintained the YAML files in the first place. They appreciated it, which in engineering culture means they actually started using it. The workflow changed. Instead of handing YAML files to a writer and hoping the conversion was accurate, the devs ran the script themselves whenever they updated the alert rules. The output went directly to the documentation portal in the correct RST format. Writers only needed to review and upload, not recreate. A documentation task that had been a recurring source of inaccuracies became a solved problem. Not through a more sophisticated AI. Through a script that did exactly one thing and did it correctly every time.

The Lesson I Keep Coming Back To

There's a reflex now, in 2024, to throw a language model at every text problem. Structured data? GPT. Reformatting? GPT. Extraction? GPT. The hammer is so impressive that everything starts looking like a nail. But language models are generative. They produce plausible text based on patterns. When your task requires plausibility (writing, summarization, creative reframing), they're extraordinary. When your task requires exactness (data transformation, format conversion, field extraction from structured files), they're the wrong tool. Not because they can't do it. Because they can't guarantee they'll do it right. And for 600 security alert rules that analysts rely on to detect real threats, "probably right" is not a standard. The right question is never "can GPT do this?" The right question is "does this task require generation, or transformation?" If the answer is transformation, write the script. It'll be faster, cheaper, and correct.

A 40-line Python script will always beat a billion-parameter model at reading a file and writing down what it says.