Three Industries That Don't Talk to Each Other

Two weeks ago I wrote about the AI Trace Auditor: a CLI that checks whether your LLM traces satisfy EU AI Act Article 12. That was one article, one command, one report. It wasn't enough. Article 12 (record-keeping) tells you whether your logging is compliant. But the EU AI Act has two more requirements for high-risk AI systems that hit the same August 2026 deadline: Article 11 (technical documentation) and Article 13 (transparency). GDPR Article 30 adds a fourth: a record of every processing activity. Companies need all four. Consultants charge $30K-$500K to produce them. The market is $492 million this year, projected to exceed $1 billion by 2030. So I extended the tool.

What Changed

The original tool did one thing: map trace data against regulatory requirements. The new version does three things in one pass:

aitrace comply ./your-codebase --traces traces.json --split -o compliance/

That command scans your codebase, analyzes your traces, and produces a directory with four files: article-11-docs.md — Annex IV technical documentation. The scanner walks your Python and JavaScript/TypeScript files using AST parsing and regex. It detects AI SDK imports (10 providers), model identifiers (14 patterns), vector databases (8), training data loading, evaluation metrics, deployment configs, and API endpoints. Each detection populates the appropriate Annex IV section. Sections that can't be inferred from code are flagged [MANUAL INPUT REQUIRED]. article-12-audit.md — The compliance gap report from the original tool. 18 requirements from EU AI Act Article 12 and NIST AI RMF. Per-requirement coverage scores. article-13-flows.md — A data flow diagram showing every external service your application sends data to. The diagram is Mermaid, color-coded by GDPR role: green for controller (you own the data), blue for processor (third party processes on your behalf), yellow for sub-processor. Each flow is annotated with PII risk. data-flow.mermaid — Raw diagram source for embedding in documentation or dashboards. One codebase. One command. Three articles.

Why This Shape

I researched 130+ tools across five categories before deciding what to build. The findings were counterintuitive. Prompt regression testing: 30+ tools. DeepEval has 13.9K GitHub stars. Promptfoo had 10.8K before OpenAI acquired it on March 9. RAGAS has 12.9K. This space is saturated and consolidating. Skip. Cross-provider LLM evaluation: 38 tools. Promptfoo was the neutral standard; OpenAI bought it. Humanloop was the alternative; Anthropic bought it. But DeepEval and Opik fill the gap well enough. Skip. Shadow AI detection: 30+ enterprise tools, $2K-$10K/month. The gap exists for open-source (only VerifyWise, and it's BSL-licensed). But building it requires network monitoring, endpoint agents, browser extensions. Different domain, different skillset. Skip. AI technical documentation generation: Clear whitespace. No tool auto-generates Annex IV from code. Every tool (AiActo, ComplyAct, Credo AI, Holistic AI) uses questionnaires. The closest open-source attempt is a 72-hour proof-of-concept. Build. AI data flow mapping: Relyance AI ($105K/year) is the only tool combining static code analysis with runtime traces. No open-source alternative exists. Build. The two gaps share the same architecture (YAML-driven regulatory mapping), the same buyer (compliance teams), and the same deadline (August 2026). They're not separate products. They're modules in the same tool.

The data flow scanner does something I haven't seen elsewhere: it auto-classifies GDPR roles based on service architecture. When your code imports chromadb (self-hosted vector database), the tool marks it as "controller" — you control the data, no third-party transfer. When it imports pinecone (cloud-hosted), it's marked "processor" — Pinecone processes embeddings on your behalf, which means you need a Data Processing Agreement under GDPR Article 28. When it detects anthropic or openai, it marks the flow as "processor" with "PII likely" — because user prompts sent to cloud APIs probably contain personal data, and you need to document that. The GDPR Article 30 Record of Processing Activities is generated from these classifications. Each detected flow becomes a processing activity with purpose, data categories, recipients, and transfer description. Retention periods and security measures are flagged as manual input. This matters because GDPR is already enforced. The EU AI Act adds new requirements on top. A tool that handles both from the same scan is worth more than two tools that each handle one.

The Structural Argument

The question I keep coming back to: why doesn't this already exist? It's not technical. The scanner is Python AST parsing and regex. The Mermaid diagram is string concatenation. The Jinja2 templates are straightforward. Anyone could build this. The answer is structural. Three parties could build it, and each has a reason not to. AI providers (Anthropic, OpenAI) won't interpret their own trace data against regulations. That's liability. Observability tools (Langfuse, Arize) sell to engineers, not compliance teams. Adding regulatory interpretation changes the buyer, the sales motion, and the support burden. GRC platforms (OneTrust, Vanta) start from policy and work toward data; adding AI-specific trace ingestion requires understanding OTel GenAI conventions and the gap between what LLMs log and what regulators need. The gap persists not because it's hard, but because it falls between three industries that don't share buyers, expertise, or incentive structures. An independent, open-source tool that sits at the intersection has no natural competitor.

180 Tests

The compliance suite has 180 tests covering the full pipeline: AST parsing edge cases (syntax errors, empty files, broken imports), cross-language detection (Python + JS/TS), GDPR role classification, Mermaid diagram generation, Annex IV section building, RoPA generation, and the unified compliance runner. All passing in 0.43 seconds. No new dependencies. The entire tool uses stdlib (ast, re, pathlib, time) plus five packages: pydantic, typer, rich, jinja2, pyyaml. Zero cloud. Apache 2.0.

Running It on Real Projects

Theory is easy. Here's what the tool actually finds when pointed at real codebases: three of my own projects and five major open-source frameworks.

My research: neurodivergent-prompting

This project runs 7,800 API calls across Anthropic, OpenAI, and Google to test how identity prompts change LLM behavior. Ten Python files.

aitrace comply ./neurodivergent-prompting

The tool found: 2 AI providers (Anthropic, OpenAI), 2 model identifiers (claude-sonnet-4-20250514, gemini-2.5-flash), 3 external services, 3 data flows, Dockerfiles with AI dependencies. Auto-populated 44% of Annex IV sections. The data flow diagram:

graph LR
    USER((User)) -->|user input| APP[Application]
    APP -->|prompts, PII likely| Anthropic_API([Anthropic API])
    Anthropic_API -->|responses| APP
    APP -->|prompts, PII likely| OpenAI_API([OpenAI API])
    OpenAI_API -->|responses| APP

Both cloud APIs are classified as "processor" under GDPR. The prompts contain identity-framed system messages ("You are autistic," "You have antisocial personality disorder") — that's PII-adjacent data flowing to two US-based cloud providers. If this were a production system under EU AI Act, that data flow would require documentation, a DPIA, and processor agreements with both Anthropic and OpenAI. For a research tool, this is informational. For a company deploying something similar, it's the starting point for regulatory compliance.

My speech coach: DAWYM

DAWYM transcribes speech using Whisper, analyzes it locally, and optionally sends transcripts to Claude for feedback. Thirty-four files across TypeScript and Python. The tool found: 1 AI provider (HuggingFace — the Whisper model), 3 model identifiers (whisper-tiny.en, whisper-base.en, whisper-small.en), 1 external service, 1 data flow. Auto-populated 33% of Annex IV. The interesting finding: Whisper runs entirely in-browser via transformers.js. No audio leaves the user's device. The data flow diagram shows only a HuggingFace connection for the initial model download; actual inference is local. Under GDPR, that's a controller relationship with no third-party data transfer for the core analysis. The Claude API connection (for "The Verdict" feature) is BYOK — the user provides their own API key. The tool doesn't detect this as a built-in flow because it's user-configured at runtime, not hardcoded in source. That's a limitation: the scanner sees code-level connections, not runtime configuration.

My piano app: Chopin's Touch

Forty TypeScript files. A piano learning app with no AI APIs. The tool found: nothing. Zero AI providers, zero models, zero external services. 11% auto-populated (only the deployment section, because there's a build config). This is the correct result. Not every app needs AI Act compliance. The tool confirming "you don't have AI-specific regulatory obligations" is as useful as finding problems. A negative result from an automated scan is evidence you can keep on file.

Five open-source frameworks

I ran the same command on Haystack (deepset, Berlin, 24K stars), CrewAI (44K stars), LiteLLM (39.5K stars), n8n (Berlin, 180K stars), and Dify (100K stars).

Project	Files	Providers	Models	Services	Flows	Auto-populated
Haystack	573	openai, huggingface	15	3	3	33%
CrewAI	1,027	anthropic, openai, google, langchain	28	17	17	44%
LiteLLM	4,861	7 providers	112	12	12	56%
n8n	10,634	openai, google, huggingface, langchain	40	10	10	44%
Dify	8,275	huggingface	21	11	11	33%

CrewAI's data flow diagram was the richest: 3 AI providers, 3 vector databases (ChromaDB as controller, Pinecone and Qdrant as processors), 6 traditional databases, 5 cloud services (AWS S3, Bedrock, Lambda, Azure OpenAI, GCP Vertex AI). All classified by GDPR role automatically. The initial scan on LiteLLM produced 10,453 model references — the tool was matching model names in test fixtures, routing tables, and cost maps. Three fixes (skip test files, cap high-density files, tighten regex) reduced it to 112. Real-world testing on real codebases found real bugs. That's why you test against projects you didn't build. None of these five projects have any EU AI Act compliance documentation. All of them have EU users or are EU-based companies. The August 2026 deadline applies to all of them.

What the tool does and doesn't do

It doesn't replace a lawyer. It doesn't replace a compliance consultant. It doesn't tell you whether your system is high-risk. It replaces the blank page. It reads your code, inventories your AI systems, maps your data flows, classifies your GDPR roles, and gives you a structured document that's 33-56% populated from your actual codebase. The remaining sections require human judgment: risk assessment, intended purpose, monitoring plans, legal basis. The difference between this and a questionnaire-based tool: this is connected to your code. When you add a new AI provider, re-running the scan picks it up. When you switch from a cloud vector database to a self-hosted one, the GDPR role classification changes automatically.

pip install ai-trace-auditor
aitrace comply .

Source: github.com/BipinRimal314/ai-trace-auditor Stack: Python >=3.11, Pydantic v2, Typer, Rich, Jinja2. 12 commands. 180 tests. Apache 2.0.