What I Learned Containerizing a Research Pipeline

The Common Belief

Researchers don't need to know infrastructure. Write your Python script. Run it on your machine. If it works, it works. This is a reasonable position if your code only ever runs on your laptop. But my thesis pipeline runs on two machines (a MacBook for development, an Ubuntu PC with an RTX 4060 for training), pulls datasets from three sources (CMU-CERT locally, TRAIL and TRACE from HuggingFace), and produces results that need to be accessible to anyone who wants to verify them. "It works on my machine" is the research equivalent of "trust me." Containerization is the research equivalent of "here, run it yourself."

What I Actually Did

I took my cross-domain threat transfer pipeline (11 experiments, 3 anomaly detection models, 5 datasets) and built three layers of infrastructure around it: Layer 1: Docker. A multi-stage Dockerfile that separates dependencies from source code. The first stage installs Python packages and only rebuilds when pyproject.toml changes. The second stage copies the experiment code. The third adds a FastAPI server for serving results. A rebuild after a code change takes 30 seconds, not 10 minutes, because Docker caches the dependency layer. Layer 2: Docker Compose. Four services:

An experiment runner (CPU or GPU)
A results API (FastAPI exposing experiment data as REST endpoints)
A static dashboard (Nginx serving figures and result tables)
Shared volumes for results, dataset caches, and CMU-CERT data

Layer 3: Kubernetes manifests. A namespace, a Deployment for the API (2 replicas, auto-restart, health checks), a Job for running experiments (run once to completion, don't retry blindly), PersistentVolumeClaims for data that survives pod restarts, and an Ingress for routing external traffic.

The Decisions That Matter

The infrastructure choices aren't interesting. What's interesting is WHY each choice was made, because the reasoning transfers to any system.

Multi-stage builds exist because rebuilds are expensive

If you put COPY . . before pip install, Docker rebuilds the entire dependency layer every time you change a single line of code. Put the dependency installation first, and Docker's layer cache means changing run_experiments.py doesn't trigger a 10-minute pip install. The principle: things that change frequently go in later layers.

Jobs are not Deployments

My experiment runner is a Kubernetes Job, not a Deployment. The difference: a Deployment says "keep this running forever; if it crashes, restart it." A Job says "run this once to completion, then stop." Experiments have a defined end state. An API server doesn't. Using the wrong abstraction means either your experiment restarts infinitely after an error (bad) or your API server stops after serving one request (worse).

Volume mounts separate state from code

The CMU-CERT dataset is 5GB. The HuggingFace model cache is 2GB+. Baking these into the Docker image would make it 7GB+ and require re-downloading on every build. Instead, they're volume-mounted: the container accesses them from the host filesystem. The image stays small (~500MB). The data persists across container restarts. The same principle applies to results. Experiment outputs go to a mounted volume, not the container's filesystem. When the container stops, the results remain. When the API server starts, it reads from the same volume. State and computation are decoupled.

Health checks are not optional

The API Deployment has both a livenessProbe and a readinessProbe. Liveness: "is this pod alive?" If it fails three times, Kubernetes kills and restarts it. Readiness: "is this pod ready to receive traffic?" A pod can be alive but not ready (still loading data, for example). Without readiness probes, Kubernetes routes traffic to pods that aren't ready, and users get errors.

Resource limits prevent starvation

Each pod declares requests (guaranteed minimum) and limits (hard ceiling). The API server gets 100m-500m CPU and 128-512Mi memory. The experiment runner gets 2-4 CPUs and 8-16Gi memory. Without limits, one runaway experiment could starve the API server of resources. With limits, Kubernetes schedules pods on nodes that can actually handle them.

What This Has to Do With AI Governance

My research pipeline transfers insider threat detection models to AI agent monitoring. The monitoring systems I study are deployed in exactly the kind of containerized, orchestrated infrastructure I just described. Understanding how containers work isn't separate from understanding how AI monitoring works. The container is the deployment environment for the detection system. The Kubernetes Job that runs an experiment is the same abstraction that would run an inference pipeline. The PVC that stores results is the same concept that would store agent behavior logs. If you're building AI governance tools and you don't understand the infrastructure they run on, you're building for an environment you haven't seen.

The Practical Takeaway

Containerizing a research pipeline forces you to make every dependency explicit, every data path clear, and every runtime assumption documented. If your experiment only works because of something implicit in your local environment, Docker will tell you. Loudly. The Dockerfile is the most honest piece of documentation in any project. It can't lie about what the code needs to run.