Towards a Harness That Can Do Anything

I've been thinking about how to free LLMs from the chat pane for a few years now. Watching what people have tried and seeing what worked (and didn't), I've formed my own opinions on the right way to approach it. Here are my thoughts, and what I've been working on recently.

What Makes a Good Harness

• I. Should be naturally intuitive to the Agent.
• II. Everything should be transparent so that the Agent can self-develop or heal (or audited postmortem).
• III. Must be as lean as possible and flexible.
• IV. Error survival, update survival, no memory corruption or degradation over time.

With the advancing intelligence of LLMs, harnesses will eventually be reliable.
The real matter at hand is reducing how much cognitive load (measured in tokens) you are putting on your bot.

Preliminary Truths

The most important things we've learned over the years.

• Determinism as much as possible. The LLM should choose what goal to pursue, but the deliberation towards that goal should be well-defined or at least a collection of well-defined steps.
• The core prompt should be as small as possible, and the LLM should then choose what skills to load into context at runtime.
• LLMs start going crazy as you approach context limits.

Don't Play the Odds, Play the Bot

A good harness MUST make use of the a priori coding knowledge of the LLM. Coding and systems administration are heavily overrepresented in the LLM's training data, so give it an environment it is already comfortable in; wrangling it through a novel one ultimately wastes tokens. Similarly, precious context should not be wasted on things like file discovery, traversal, etc. -- good harnesses make delegation easy and efficient. At the same time, a harness should feel light to the LLM, but actually do a lot of things in the background. These include logging, sanity checks, failsafes, sanitizations, etc.

Auditability, Logging, and Self-Healing

Everything is vulnerable, and all agents eventually fail. There are two types of Agent failures:

• LLM Level
• Harness Level

Failures at the LLM level cannot be directly patched, but the risk of such a failure can be mitigated by the harness.
Harness-level failures can be recovered from, and should be fixable at runtime due to the turn-based nature of the LLM.

In order to fix a defect, the Agent needs two things: great logging, and a clear error message.

A Unified Data Layer That Can Do It All

Most of these requirements are age-old questions; only the verbiage has shifted from Users to Agents. So it's worth asking: what can we learn from the before-fore times, when people used to actually write code?

My Hypothesis: the Unix / Linux Environment is a natural candidate, and could be turned into an Agentic harness with a few modifications.

I am, by no means, an expert on Unix or any of its descendants, but I have spent the last decade learning about its history, design choices, and usage. A lot of it maps beautifully onto our predicament; plenty of it doesn't. Think of U/L as a motivating analogy rather than a direct comparison.

Mapping Concepts of Old to New, and the Unix Philosophy

In case you might have forgotten, or not seen it before, here is the creed of our forefathers Ritchie and Thompson:

1. Write programs that do one thing and do it well. To do a new job, build afresh rather than complicate old programs by adding new "features".
2. Write programs to work together. Expect the output of every program to be the input for another program.
3. Write programs to handle text streams, because that is a universal interface.

These perfectly encapsulate the issues with harnesses today: they're overly complicated, attempting to do a lot of things, and the trajectories our Agents are expected to take are often ill-defined. These are symptoms of the same issue: there is no way for the Agent to hold agency over itself. In many cases today, tools are directly loaded into context, and system prompts are preemptively filled with warnings and specific rules and guidelines that developers have (which eventually fade turn-over-turn).

We can then derive our own set of principles for designing our harness:

1. Write modular, transparent tools that do one thing and do it well; ensure they fail loudly.
2. Write tools, skills, and connectors that work together. Skills dictate workflows, tools are the means to execute them, and connectors are the data which the Agent manipulates.
3. Text streams are a universal interface, and a Language Model has home-court advantage. Everything should be a flat text file.

Ambiance: My Take on Harnesses

Everything is a File

Have you ever manually done JSON wrangling? What about constructing heavy curl commands to query an endpoint? Complicated regex? I haven't, because they are a pain in the ass. An LLM might have an easier time with them than you, but rest assured that they prefer plaintext too. Thus, whenever dealing with external data sources, your harness should perform whatever manipulation might be necessary to clean it up before it reaches your LLM.

You need a place to store all this data. By categorizing everything into directories, you can save your Agent a lot of wasted tokens.

Consider the Filesystem Hierarchy Standard (FHS)

The FHS just outlines what a fresh Linux install should look like. Naturally, LLMs are experts at navigating the Linux FS, thus strategically planting our external data sources and routing into the VFS will allow the Agent to feel at home (e.g. your logs go into /var, configuration files go into /etc, Agent workspace at /home, etc.). An additional benefit is that both you and the Agent can easily audit, track, and search for things via grep, find, which, and even non-GNU programs such as rg and fzf.

The heavy lifting lies in making the messy, cluttered outside world fit into the faux-VFS.

I've been using the following mapping. Note that lots of things do directly translate 1:1.

The "Kernel"

Now that we've established what our filesystem should look like, let's think about how the Agent will interact with the environment (or rather, when). The industry standard for always-on Agents, OpenClaw, pairs event-driven message handling with a heartbeat: a full agent turn on a fixed interval (30 minutes by default) to check whether anything needs attention. The trouble is that everything that isn't a pushed message, like file changes or external state, only gets noticed at heartbeat granularity. Tighten the interval and you burn a full LLM turn on every empty check; loosen it and the agent is up to an hour behind the world.

Thus, I decided to build Ambiance around an event bus that I misnomerously call "the Kernel". It watches our FS for changes with cursors on textfiles, and then invokes the LLM accordingly (with a few coalescence strategies to handle high throughput). That way you can ensure that the Agent never misses a single notification. Additionally, you can also wire up different 'users' (LLM instances) to respond to different events.

Kernel Does the Heavy Lifting

A real kernel acts as the middle layer between software and hardware. The Ambiance Kernel acts as the middle layer between the LLM and the outside world. The Kernel checks and ensures everything the LLM does is safe and not obviously harmful.

The "Users"

There are three default users I've developed so far:

1. root, which handles all system-level stuff, especially coding new drivers, binaries and fixing old ones.
2. pai, which is the human-facing LLM that actually interacts with the outside world
3. librarian, which journals what pai is good at, what it's bad at, and what the system did for the day.

All three are constantly communicating with each other via the event bus and a send-message binary.

Try It

The idea behind Ambiance is simple: the model's priors are the cheapest resource you have, so a harness built out of things the model already knows (files, users, logs, docs) will always beat one it has to be taught. Everything else here is just in service of that.

Ambiance is a work in progress, but you can try it today at whitematterlabs.ai, or skip straight to:

curl -fsSL https://raw.githubusercontent.com/whitematterlabs/ambiance/main/install.sh | sh

PS: Do projects like this interest you? Are you working on projects like this yourself? You should totally check out the Recurse Center, which is where I started building this out!