Why We Stopped Trying to Be Smart: Building AI Systems the Feynman Way

See It in Action

This is what the system looks like in production: trigger Docs Agent from Slack, it drafts content using context from PRs and Linear tickets, you review and edit, then create a PR. EkLine's GitHub integration automatically checks style guide compliance and accuracy.

And here's the other side of the workflow: EkLine detects a new feature PR, automatically generates a parallel documentation PR, and lets developers request edits by mentioning @ekline in a comment. Docs stay in sync as the feature code evolves.

The Problem

At EkLine, we were building AI-powered tools for technical documentation. Our customers needed reliable, consistent output from language models. We had the latest AI tools at our disposal, and we were going to use all of them.

The Wrong Approach: Starting Complex

We did what felt intelligent. We built a system with many agents and many skills: multi-step chains, sophisticated orchestration, models coordinating with models. The architecture looked impressive. It felt like real engineering. But we were building something we didn't understand. Not because agentic systems are bad. Because we started with them before we had intuition for how LLMs actually behaved. We didn't know their failure modes, their edge cases, their quiet ways of being confidently wrong. We treated complexity as a starting point instead of a destination.

The Simple Loop

In parallel, we also built something deliberately simple. A basic loop: one or two agents, one or two skills. It essentially asked Claude what to do and acted on the response. No sophisticated orchestration. No impressive architecture diagrams. We ran both systems side by side.

90 Evaluations in Docker

To get an honest comparison, we ran evaluations inside Docker containers so each run was isolated. We tested across various open source projects, examining the documentation output from each system. Over 90 evaluation runs, the result was consistent: the simple loop outperformed the complex agentic system every time. Not sometimes. Not on average. Consistently, across different projects and different types of documentation. The complex system we'd spent so much effort designing was being beaten by something we'd built in a fraction of the time.

The Feynman Lesson

Richard Feynman didn't start with quantum electrodynamics. He started with radios. He built understanding from the ground up: simple experiments, clear observations, incremental complexity. Each layer of sophistication was earned through genuine comprehension of the layer below it. We did the exact opposite. We started at the top (orchestrating systems we didn't understand) and wondered why things were fragile. The realization was simple but painful: you cannot build intuition for a system by starting with its most sophisticated configuration. If you don't understand what a single LLM call does well and does poorly, you have no business chaining five of them together. You're not building intelligence. You're building a Rube Goldberg machine out of black boxes.

Growing Complexity One Layer at a Time

Once we accepted the simple loop as our baseline, we changed our approach completely. Instead of designing complexity upfront, we earned it. We took the simple scaffolding and added one thing at a time. A single new skill, a small behavioral change, one additional agent capability. After each addition, we ran five evaluation runs and compared against our existing results from the 90-run baseline. If the addition showed a positive delta, we kept it. If it didn't, we removed it. This was a fundamentally different process from what we'd started with. We weren't guessing what the system needed. We were learning what it needed, one change at a time. Each round gave us intuition: not just about what worked, but about why it worked. We started to understand what was better to encode directly into the scaffolding (because we knew our customers) versus what to leave for the agent to figure out on its own. The system that emerged was more complex than the original simple loop, but every piece of that complexity had earned its place. Nothing was there because it seemed smart. Everything was there because it proved useful.

What I Learned

Simple systems you understand will always beat complex systems you don't. This isn't anti-intellectualism. It's engineering discipline. Complexity is a cost, and it should only be paid when the return is proven. Smartness is compounded over time, not front-loaded. The team that starts simple and builds understanding will outperform the team that starts complex and spends months debugging what they don't understand. Production AI is about reliability, not impressiveness. No customer cares about your architecture diagram. They care about whether the output is right and whether it's right consistently. Know what to encode and what to infer. Some things you should tell the system directly because you know your domain. Other things you should let the model figure out. The only way to know the difference is to start simple and pay attention.

The gap between demo and deployment is where most AI projects die. Demos are friendly. Production is adversarial. If your system can't handle the adversarial case, it doesn't matter how good the demo was.