How to Run TrueForge with Local Models on Your Own Hardware
A practical guide to installing TrueForge, an open-source agent harness, and wiring it up to locally hosted models instead of cloud APIs.
What is TrueForge and why does it need a “harness”?
TrueForge is an open source, MIT licensed agent harness built by Two Foundry. It’s the infrastructure layer that sits around a language model so it can run as a durable, production-grade agent instead of a fragile script. A basic agent loop (model calls a tool, output gets stuffed back into the prompt, repeat) can be written in about 15 lines of Python. That works for a two or three turn task. It falls apart once you need retries on flaky APIs, state that survives an hour-long user session, protection against a model dropping a database table, or context windows that don’t fill up with noise after 40 tool calls. TrueForge ships that missing scaffolding: deferred tool loading, file offloading, code execution in isolated sandboxes, sub-agents, context compaction, and human-in-the-loop approval gates, all exposed through a dashboard, an HTTP server, and an SDK. Because it’s model agnostic, it can point at OpenAI-compatible endpoints, including models running entirely on your own hardware.
TL;DR
- TrueForge is an open-source agent harness (MIT licensed, from Two Foundry) that provides the retry logic, sandboxing, memory management, and approval gates that a raw agent loop doesn’t have.
- It’s model agnostic, meaning it talks to any OpenAI-compatible API, so you can swap in locally hosted models via a custom provider entry (base URL plus name) instead of relying on a cloud vendor.
- Deferred schema loading means connected MCP servers only register their name up front; the model discovers tool schemas on demand instead of burning thousands of tokens loading every tool definition before a conversation even starts.
- Code mode lets the agent write and run small scripts inside an isolated sandbox (Daytona by default) rather than making raw tool calls and doing math in its own context, which keeps token usage and error rates down.
- Sandboxes are treated as ephemeral tools, not as containers holding the whole agent hostage, so the core orchestration loop stays on your server and API keys never touch the execution environment.
- Human-in-the-loop gates can be applied per tool, so read-only operations run automatically while destructive actions (like changing a config) pause for explicit approval.
- Every agent run is traced end to end, which is useful for debugging tool schemas and spotting where an agent’s reasoning went wrong, whether it’s talking to a cloud model or a local one.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
How do you install TrueForge and connect it to local models?
TrueForge installs via a single setup command (covered in the source walkthrough) and comes up as a local server with a web UI, a dashboard, and an API. Once it’s running, local model configuration happens in the settings panel:
- Open Settings and look for provider configuration.
- Choose Custom Provider instead of one of the built-in cloud options.
- Enter a name for the provider and the base URL of your local inference server.
- Save it, and the model becomes selectable anywhere TrueForge lets you pick an LLM for an agent.
Because TrueForge speaks the OpenAI-compatible API format, any local serving stack that exposes that interface, such as popular self-hosted inference servers running open-weight models, can be plugged in this way. The setup demonstrated in the source walkthrough used two local models running across a pair of DGX Spark machines, showing that TrueForge doesn’t assume any particular cloud backend and doesn’t require rewriting agent logic to switch providers.
How does TrueForge manage context so agents don’t get dumber over time?
Long agent runs have a well-known failure mode: every turn replays the full conversation history, so the context window fills with old tool outputs and the signal-to-noise ratio collapses. Models get measurably worse at reasoning as the window fills with noise. TrueForge addresses this with a few concrete mechanisms rather than one silver bullet:
- Deferred tool loading: instead of loading every MCP server’s full tool schema before the first message, TrueForge passes just server names plus a handful of small meta-tools, so the model requests schema detail only when it needs it. This trades a small latency cost for a large reduction in upfront token spend.
- File offloading: when a tool call returns a large payload, TrueForge streams the raw response to disk inside the sandbox rather than dumping it into the prompt. The model gets a short preview and a file path, and it can use bash-style tools to grep for exactly what it needs.
- Sub-agents: noisy, multi-step research can be delegated to a sub-agent with its own fresh context window, which reports back a distilled answer instead of polluting the main thread.
- Compaction: when the context window crosses a token threshold, a background model summarizes older turns and swaps out the raw history. This is explicitly lossy: fine detail gets traded for the ability to keep going.
What is “code mode” and why does it matter?
In a typical tool-calling loop, a model requests an API call, waits for JSON, reads it into context, and decides on the next call, one round-trip at a time. Most of the time the agent only needs a specific number or pattern out of that response, not the entire payload, and asking a language model to do arithmetic over a wall of JSON in its own context is both expensive and error-prone.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
Code mode flips this: the agent writes a short script that runs inside the isolated sandbox, and any tool calls made from within that script get bridged back to the harness rather than exposed directly to the sandbox. This keeps credentials and API keys out of the execution environment entirely, since only the final result comes back to the model. It also cuts token usage sharply, since intermediate data never has to pass through the model’s context at all.
How does sandboxing and human-in-the-loop approval work in practice?
TrueForge treats code execution as an on-demand tool rather than baking the whole agent into a single long-lived container. A sandbox spins up only when code actually needs to run and tears down right after, which avoids the inefficiency of a container sitting idle for an entire user session and reduces the exposure of sensitive tokens.
For tool permissions, TrueForge distinguishes between read-only and destructive operations. Read-only tools run automatically. Destructive ones, such as modifying a configuration or applying a fix, can be gated behind human approval. In a demonstrated example, an “on-call engineer” agent connected to a custom MCP server monitoring infrastructure was able to confirm an alert, pull recent deploys, metrics, and logs, and identify a root cause (in that case, a changed environment variable) entirely on its own. When it came time to actually change the configuration, the harness paused and required explicit human approval before proceeding, with the same approval prompt visible both in the TrueForge dashboard and in a custom external application calling the same agent via API.
Is running TrueForge with local models worth it?
For teams already running local models on their own GPUs, TrueForge offers a way to get harness-level infrastructure (retries, sandboxing, tracing, approval gates) without being locked into a specific model vendor’s managed agent product. The tradeoff is speed: local models, especially on modest hardware, respond more slowly than hosted frontier models, and a multi-step agent run will feel that latency more than a single chat completion would. What you gain is full control over where data and code execution happen, no per-token cloud billing, and an audit trail of every tool call and agent decision stored on infrastructure you control. Because TrueForge is open source and MIT licensed, there’s no vendor lock-in on the harness side either, so switching models, swapping providers, or self-hosting the whole stack doesn’t require rebuilding the agent logic.
Frequently Asked Questions
Does TrueForge require a cloud model provider to work?
No. TrueForge is model agnostic and connects to any OpenAI-compatible API endpoint, which includes locally hosted models configured as a custom provider with a name and base URL.
What sandbox does TrueForge use for code execution?
By default it uses Daytona as the sandbox environment, and it requires an API key for that service. Sandboxes are spun up on demand for code execution and torn down afterward rather than running as persistent containers.
Can TrueForge connect to my own tools and services?
Yes. It supports MCP servers as connectors, including custom ones you build yourself, alongside built-in connectors for things like web search. It’s designed as a general purpose harness, not a coding-specific one.
How does TrueForge prevent an agent from taking destructive actions unsupervised?
Tools can be marked for human-in-the-loop approval. Read-only actions execute automatically, while destructive operations pause and wait for a human to approve them before the agent proceeds.
Why does deferred tool loading matter for cost?
Loading every connected MCP server’s full tool schema upfront can burn thousands of tokens before a user even sends a message. Deferred loading only passes server names and a few meta-tools initially, letting the model request full schemas on demand, which cuts upfront token cost at the expense of a small latency hit during tool discovery.
