Contents
People talk a lot about the best artificial intelligence model for coding. They talk far less about the environment in which that model is expected to work.
RSS
RSS sends new articles to the reader of your choice, without an algorithm or newsletter.
Yet two teams can use the same model and get radically different results. In one team, the agent produces a coherent feature, respects the architecture, runs the right tests and fixes its own mistakes. In the other, it edits the wrong files, invents conventions, generates superficial tests and leaves behind technical debt that someone will have to pay for later.
The model has not changed.
The working environment has.
That is the core idea behind Harness Engineering. The term refers to all the mechanisms surrounding an AI model that allow it to understand its environment, use tools, produce verifiable changes and correct its own errors.
For a developer, this means designing a reliable execution loop around the agent. For a product team or an executive, it means turning an impressive generation capability into a controlled delivery capability.
The question is therefore not only: “Which model should we use?” It becomes: In what environment can this model produce software we are willing to maintain?
TL;DR
- A language model generates proposals. An agent acts within an environment.
- A harness brings together the rules, tools, memory, sandbox, tests and feedback loops that govern that action.
- The prompt gives direction. The context provides information. The harness organizes the complete working system.
- Deterministic controls remain the first line of defense. Semantic controls complement them without replacing them.
- The value lies in producing code that is not only fast, but also verifiable, traceable and reversible.
1. Generating code is not software development yet
An AI demo often creates a misleading impression. You ask a model to create a page, an API or a component; a few seconds later, the code appears and looks convincing. Then you place the same request inside a real project: the repository contains years of history, implicit conventions, external dependencies, environment variables, security constraints and tests that do not all run at the same speed.
In that context, producing code is only the beginning. The agent must understand the request, identify the relevant part of the system, check existing conventions, choose the right files, ask a question when necessary, modify the code, run the appropriate checks, interpret the errors and explain the result.
A simple text completion cannot handle that entire sequence on its own. A language model does not automatically have the company’s memory, product responsibility or the intuition of a developer who understands the consequences of a bad decision. It works with the information available to it at execution time. If an important rule is not visible, it may effectively not exist for the agent.
The trap is to respond to every failure with a longer prompt. If the agent uses the wrong command, the tool may need to be improved. If it changes the wrong architectural layer, the project may need a testable architecture rule. If it repeats the same mistake, the feedback may need to be improved.
The problem is not always the model’s reasoning.
Often, it is the absence of a system around the model.
2. What is Harness Engineering?
The word harness comes, among other places, from physical engineering. In a vehicle, a wiring harness distributes power and signals. In testing, a test harness allows a component to be driven, its dependencies to be simulated and its behavior to be measured in a controlled environment.
Classic software testing uses the same logic with drivers, test data, dependency stubs, simulators, an execution engine and a reporting system. The component under test may believe it is talking to a database or a remote service, while it is actually receiving a simulated and reproducible response.
With AI agents, the concept becomes broader. The harness organizes the work of the system that produces the software. It defines what the agent can see, do and modify, which tools it can call, which checks it must trigger and how errors are returned to it.
An operational definition would be:
Harness Engineering is the design of the environment, tools, rules and control loops that allow an AI agent to act reliably inside a real software system.
This definition is broader than an agent framework. It covers understanding, action, verification, correction, traceability and stopping. In his analysis of the subject, Martin Fowler distinguishes between the harness built into an agent by its provider and the harness built by the team using it. The repository, scripts, rules and checks form that project-specific layer.
3. Prompt Engineering, Context Engineering and Harness Engineering
These three approaches all aim to improve a model’s results, but they operate at different levels.
| Approach | Main question | Example |
|---|---|---|
| Prompt Engineering | What should I ask the model to do? | “Add pagination to this list.” |
| Context Engineering | Which information should I provide? | Architecture, business rules, relevant files and documentation. |
| Harness Engineering | In what system can the agent act, and how do we verify its work? | Tools, permissions, sandbox, tests, feedback and correction loops. |
The prompt states the objective or expected result. It clarifies human intent, but it cannot carry an entire project’s architecture, deployment procedure, conventions and historical exceptions on its own.
Context makes the information required for a decision available: repository files, API contracts, schemas, tickets, architectural decisions or previous errors. But a window filled with irrelevant information increases noise and cost. The right context is not the maximum amount of information. It is the relevant information, at the right time, in a format the agent can navigate and verify.
The harness organizes the working system: operation order, permissions, tools, isolation, artifact persistence, checks, error reinjection, time and cost limits, and human validation when required.
A prompt can ask an agent to run the tests; a harness can make that step automatic and block the workflow if the tests fail. A prompt can ask it not to modify a given layer; the harness can check dependencies and reject the change. A prompt can ask it not to use real data; the harness can run the agent in an isolated environment. The prompt explains the rule. The harness makes the rule observable and, whenever possible, executable.
4. Three analogies for understanding the concept
The flight simulator
A pilot does not practice an engine failure with passengers on board a commercial aircraft. They use a simulator that reproduces enough of the environment to make the exercise useful, while allowing dangerous situations to be created without exposing anyone to the real-world consequences.
A development agent can work in the same way: an isolated repository, a test database, simulated services and controlled commands. It can reproduce a bug and verify behavior without touching production. The harness provides isolation and repeatability.
The horse and the harness
A powerful model is like an engine capable of producing a great deal of energy, but not necessarily knowing where that energy should go. The harness transmits commands and makes that power usable in a given direction. Without control, an agent can execute commands and consume tokens without getting closer to the right result. The goal is to make its actions observable, correctable and proportionate to the task.
The industrial workshop
Imagine an extremely fast apprentice who can read thousands of pages in a few seconds, but does not know the workshop’s habits, which machines are fragile or which tolerances are critical. You organize the workstation, provide suitable tools, install physical stops, inspect the parts and immediately explain the cause of an error.
Harness Engineering is the design of that software workshop. The agent still makes many local decisions, but it works in a space where costly errors are detected early and important rules no longer depend only on an expert’s memory.
5. The anatomy of a harness for a development agent
A harness is not just an AGENTS.md file, an MCP server or a test script. Those elements are only components of a larger system.
5.1. Guides: acting before the error
Guides, or feedforward controls, provide the information and tools that increase the probability of a good decision on the first attempt.
They include contribution conventions, architecture rules, acceptance criteria, bootstrap procedures, test commands, security rules and API contracts.
The AGENTS.md file is useful as an entry point: where to find the information, how to start the project and which commands to use. It becomes less useful when it turns into an endless manual. In its engineering report, OpenAI explains that it preferred a short file used as a table of contents pointing to a structured knowledge base inside the repository.
5.2. Tools and their boundaries
An agent becomes operational when it can act. But every action also increases the risk surface: reading a file, calling an API, pushing a branch or running a migration do not have the same impact.
The harness must therefore define the available tools and parameters, visible resources, irreversible actions that require confirmation, and the formats of outputs and logs to retain.
Tool design matters as much as model choice. A vague error forces the agent to guess; a precise message, probable cause and verification command provide actionable feedback. A limited, idempotent and testable command is sometimes better than a general-purpose interface.
5.3. The sandbox and reproducibility
An agent should not need to touch production to demonstrate that a change works. The harness can use containers, Git worktrees, temporary databases, synthetic data and simulated services. The goal is to make behavior sufficiently close, controlled and repeatable to detect relevant errors.
Reproducibility is a condition of trust. A well-designed harness should tell you which version of the code was tested, with which dependencies, in which environment and with which checks.
5.4. The file system as durable memory
Agents often work in long loops: they plan, explore the repository, run commands, receive errors and resume their work later. Keeping everything in the context window is inefficient. Information gets mixed together and inference costs increase.
The file system can serve as durable working memory: an execution plan, a decision log, command results, test reports, progress state, intermediate diffs and conclusions passed to a sub-agent.
This memory can be inspected by a human and versioned with the project. It is not about recording everything. It is about preserving the artifacts needed for resumption, auditing and understanding the work.
5.5. Feedback sensors
Sensors, or feedback controls, observe the agent’s actions. They can validate a result, report an error or suggest a correction: compilation, type checking, linters, tests, dependency analysis, security checks or module-boundary validation.
A useful sensor does not simply return “failure”. It provides a signal that both the agent and the human can understand. A custom linter can identify the violated rule, the affected file, the expected example and the command to run next. Without feedback, the agent repeats its mistakes; without prior guidance, it discovers important constraints too late. A balanced harness combines anticipation and observation.
6. Two control axes: deterministic and inferential
A practical way to analyze a harness is to cross the timing of the control with the nature of its mechanism.
| Computational control | Inferential control | |
|---|---|---|
| Before the action | Templates, bootstrap scripts, structural rules, codemods | Architecture principles, documentation, examples, semantic instructions |
| After the action | Compilation, types, linters, tests, static analysis, dependency rules | Model-based review, consistency analysis, business-intent verification |
Computational controls
Computational controls are fast, reproducible and inexpensive. They are suitable for rules that can be expressed without interpretation:
- a file must follow a given syntax;
- a type must be compatible;
- a module must not import a forbidden layer;
- a test must produce an expected result;
- a vulnerable dependency must not be introduced;
- a secret must not appear in the repository.
They do not always know whether a feature actually meets the business need. But whenever a rule can be expressed deterministically, it is better to give it to a specialized machine than to a probabilistic model.
Inferential controls
Inferential controls use a model to analyze intent or consistency that is difficult to reduce to a formal rule. A critic agent can look for a business inconsistency, an oversized solution or a divergence from the request.
This capability is useful, but it is not an absolute guarantee. The evaluator model can be wrong or miss a problem. It should be used as an additional layer, not as the sole arbiter.
A good practical rule is this: deterministic controls should close the doors that can be closed mechanically. Inferential controls can help explore the areas where human interpretation is still required.
7. The agent steering loop
The harness turns a request into a controlled sequence:
- The agent checks the state of the repository.
- It identifies the relevant documents, files and tools.
- It restates the task and flags ambiguities.
- It creates an execution plan.
- It makes a limited change.
- The harness runs the appropriate checks.
- Errors are returned in an actionable form.
- The agent corrects the change or requests a human decision.
- The result is recorded, presented and possibly submitted for review.
This loop resembles a classic development workflow, but the agent can chain research, modification and verification cycles without waiting for human intervention after every command. Autonomy should not be confused with the absence of limits: the harness must stop loops that run too long, limit retries, request approval for destructive actions and escalate when signals conflict.
An endlessly repeating loop is not an autonomous agent.
It is automated spending.
Feedback reduces mechanical review work. If the agent can detect a type error, a boundary violation or a failing test, the developer can focus on the need, the architecture and the trade-offs.
8. Why this is becoming a strategic business issue
The point of Harness Engineering is not to build complex machinery for the sake of complexity. Agents reduce the marginal cost of producing code, but they can increase the cost of verification and maintenance. Teams can quickly create several variants, open many pull requests and run tasks in parallel.
The bottleneck moves. The question is no longer only “How many lines can we produce?” It becomes “How many changes can we verify, understand and maintain?”
Shift-Left applied to agents
The Shift-Left principle moves checks toward the beginning of the development cycle. The earlier an error is detected, the less context is usually needed to fix it. An immediate correction can be handled while the agent still has the context of the change; a late error requires the reasoning to be reconstructed.
Cost ratios should not be presented as universal laws. The relationship depends on the project and the team’s diagnostic capability. The robust idea is simple: errors detected close to where they were introduced are generally easier to understand and less expensive to fix.
The example published by OpenAI
In its report on a product developed with Codex, OpenAI says that the team produced roughly one million lines of code with three engineers at the start, without manually writing code, and estimated that construction took about one tenth of the conventional time. These figures describe a specific experiment, not a promise that can be transferred to every team.
The more interesting lesson is elsewhere: humans spent their time designing environments, abstractions, rules and feedback loops. Because agents can reproduce imperfect patterns, the team also introduced a form of technical “garbage collection”. AI does not remove the need for engineering. It moves engineering toward the system that makes code production reliable.
Metrics to track
An organization should measure more than lines of code or merged pull requests. More useful indicators include:
- the delay between a change and its feedback;
- the average number of iterations before validation;
- the percentage of tasks completed without human intervention;
- regressions detected before integration;
- the average cost of a task in tokens and machine time;
- the time spent correcting architectural drift.
The useful metric is not “the agent wrote a lot of code”. It is “the team delivers faster without increasing the risk and debt it will have to support tomorrow”.
9. Two concrete examples of harnesses
SWE-bench: evaluating an agent in a reproducible environment
SWE-bench evaluates the ability of models to solve real software issues drawn from GitHub repositories. The model receives a repository and an issue description, produces a patch, and the patch is then applied and evaluated against the project’s tests.
This benchmark shows why evaluating an agent cannot be reduced to asking the model whether its code looks correct. You need an initial state, an environment, a patch-application procedure and a validation method. The project uses Docker to make evaluations more reproducible and retains execution logs. Without a controlled environment, a score may reflect dependency or configuration differences rather than progress in the model.
Teams can apply the same logic on a smaller scale: select real tasks, freeze a repository state, define acceptance criteria, run the tests and retain the traces.
Making an enterprise repository legible to agents
Another example is to treat the repository as the team’s system of record. Important decisions, architecture rules, execution plans and procedures required to change the product should be discoverable, versioned and connected to the code.
A possible structure might look like this:
AGENTS.md
ARCHITECTURE.md
docs/
├── design-decisions/
├── product-specs/
├── execution-plans/
├── references/
└── quality/
scripts/
├── check-architecture
├── check-docs
└── test-affected
The entry file stays short. Detailed rules live in specialized documents, and scripts verify links and critical constraints. The repository becomes easier for agents to navigate, but also easier for new developers to understand. An environment designed to be legible to a machine can improve human onboarding as well.
10. Deploying a harness through maturity levels
Do not start with a gigantic architecture of sub-agents, vector memory and multiple evaluators. Maturity is built progressively.
Level 1: make the rules explicit
Identify the rules already present in team conversations: commands, conventions, forbidden dependencies, testing procedures, architecture decisions and known traps. Document them in the repository, with an owner and a review date.
Level 2: automate mechanical checks
Automate formatting, linting, compilation, type checking, fast tests, secret detection, dependency checks and module-boundary validation. Select the checks that actually protect the project and make their messages understandable.
Level 3: isolate tasks
Run agentic changes in isolated environments: dedicated branches or worktrees, temporary databases, synthetic data and minimal secrets. The agent must be able to fail without making the shared environment unusable.
Level 4: introduce the correction loop
Configure the agent to run relevant checks after each significant step. Return useful errors, limit retries and preserve work artifacts. Improving feedback quality is often more valuable than adding another evaluator model.
Level 5: measure real tasks
Build a small set of representative tasks: adding a route, fixing a regression, changing a screen, performing a controlled migration or evolving a module. Run them regularly and measure the results.
Level 6: increase autonomy gradually
Autonomy should be earned through evidence. A low-impact task can be automated more broadly than a data migration or security change. Define levels: proposal, supervised execution, automatic pull request, conditional merge and recurring execution for low-risk tasks.
The goal is not to remove humans. It is to involve them where their judgment adds real value.
11. Mistakes to avoid
The endless instruction file
A huge document gives the impression that the team has formalized everything, but it becomes difficult to read, maintain and verify. Information starts to contradict itself and important rules get lost. Prefer a short entry point, structured documentation and testable rules.
Delegating everything to another LLM
An agent that writes the code, a second that reviews it and a third that validates the review can be useful. But three probabilistic opinions do not become proof. Evaluator models complement deterministic controls; they do not replace functional tests, security or business validation.
Granting too many permissions
An agent that can delete a database, modify secrets and deploy to production has disproportionate power for an ordinary development task. Permissions should be limited by environment, tool and step, and irreversible operations should be explicit.
Measuring speed instead of flow quality
Producing ten pull requests instead of two is not necessarily progress. If developers then spend their days understanding, fixing and cleaning up those changes, the apparent productivity is misleading. Measure feedback time, regression rate, rework cost and the team’s ability to maintain the result.
Believing generated tests prove the feature
An agent can generate a test suite that validates its own incorrect behavior. Tests must therefore be reviewed, diversified and compared with business criteria. A green test means that the tested scenario is green. It does not automatically mean that the product meets the real need.
12. What the harness cannot solve
Harness Engineering improves the working environment, but it does not turn an unclear specification into a clear need. If nobody can define the feature, the agent may produce a clean but useless implementation. If the architecture or test data is poor, the harness can slow down the degradation without eliminating the debt or the false confidence.
External data, dependencies and documents must also be treated as potentially untrusted sources. Least-privilege permissions, input validation, privilege separation, logs and revocation capabilities remain essential.
The harness is not a magic guarantee. It is a risk-reduction system: it turns unpredictable errors into detectable errors, implicit conventions into visible rules and repetitive manual corrections into reusable controls.
Conclusion: the next competitive advantage will be the working environment
The race for the most powerful model will not disappear, but it will not be enough to create a lasting difference between teams. What becomes difficult to reproduce is the environment that makes these models usable inside a real project: domain knowledge, conventions, tools, test data, guardrails, feedback loops and measurement methods.
Harness Engineering gives this idea a name. An agent is not simply a model with a few tools attached. It is a system that must perceive an environment, act within its limits, receive signals, correct its trajectory and leave verifiable traces.
For developers, architecture, scripts, tests and documentation become even more important. For decision-makers, AI investment should not be assessed only through model price or token usage. It should also be assessed through the framework that turns this capability into maintainable software.
The real question is no longer “Which AI will write our code?” It is: What system will we build so that this AI can work quickly without degrading the software it produces? The most effective teams will not necessarily be the ones with access to the most spectacular model. They will be the ones that build the best environment for it to work in.
