September 24, 2026 · 6 min

Building a custom agent harness with Pi and Decider 1

An agent harness makes many small decisions on every run. Which model should handle a request. Whether a tool call is safe to execute. Whether an answer is finished or the agent should keep working. Elvis Saravia made this argument in his post “Building a Custom Harness with Pi and Jev”. Asking a chat model each of these costs a full generation call, so in practice most of these checks get skipped.

A decision model answers each check with a probability instead of a paragraph. In this post we build the same kind of harness with Pi, the open-source agent harness, and Decider 1 (sd-1), meraGPT’s decision model. We show real outputs from a real run.

The setup

The harness is a Pi agent loop with two hooks. beforeToolCall runs before every tool call and can block it. There is also a mechanism to prompt the agent again after it answers, which lets you reject an answer and make it try again. The agent’s own model can be any OpenAI-compatible endpoint. For this post we ran two local models, one small and one larger, so the router had something to choose between.

Decider 1 is called through TypeSafe’s own JavaScript SDK, @typesafe-ai/sdk. Pointing it at meraGPT takes two environment variables: TYPESAFE_BASE_URL=https://meragpt.com and TYPESAFE_API_KEY set to a meraGPT key. No other change to the agent code. There are three gates in total, each one a single Decider 1 call. Route asks which model to use. Guard asks whether a shell command is safe to run. Done asks whether the answer is finished. Every question is narrow, asks about one trait, and is phrased in the positive form.

npm install @mariozechner/pi-agent-core @typesafe-ai/sdk typebox

export TYPESAFE_BASE_URL=https://meragpt.com
export TYPESAFE_API_KEY=$MERAGPT_API_KEY
export AGENT_BASE_URL=http://127.0.0.1:8081/v1   # any OpenAI-compatible endpoint

Every gate goes through one helper:

const decider = new TypeSafeClient({ timeout: 30_000 });

async function decide(gate, state, questions) {
  const res = await decider.systemOne({ model: 'sd-1', state, questions });
  return res.answers;
}

Gate 1: which model

const a = await decide('route', { request }, {
  tier: choice('Which model should handle this request?', {
    quick: 'A short answer or one simple step. No planning.',
    deep: 'Several steps: investigating files, running commands, or making changes.',
  }),
});
// a.tier.choice === 'deep', a.tier.confidence === 0.782

Before the agent starts, the harness asks Decider 1 a single choice question: which model should handle this request, with two labels. Quick covers a short answer or one simple step. Deep covers several steps, investigating files, running commands, or making changes. In real runs, “What does the tax() function do?” routed to quick with confidence 0.83. “How many TODO comments are in this project, and in which files?” went to deep at 0.63. “Rename displayName to formatCustomerName everywhere and update the tests” also went to deep, at 0.78. The confidence is a probability, so the harness can set its own threshold, for example sending anything below 0.6 to the larger model to be safe.

Gate 2: is this command safe to run

beforeToolCall: async ({ toolCall, args }) => {
  if (toolCall.name !== 'bash') return undefined;
  const a = await decide('guard', { command: args.command, working_directory: WORKDIR }, {
    deletes_or_overwrites: noul('Would running this command delete, truncate or overwrite files or data?'),
    uses_network: noul('Does this command send data over the network or download and run code from it?'),
  });
  if (a.deletes_or_overwrites.noul > 0.5 || a.uses_network.noul > 0.5)
    return { block: true, reason: 'Blocked by policy. Find a read-only way.' };
},

Pi’s beforeToolCall hook asks Decider 1 two yes/no questions before any shell command runs: would it delete, truncate or overwrite files or data, and does it send data over the network or download and run code. If either answer scores above 0.5, the call is blocked and the agent is told why.

The first version put the user’s request into the state alongside the command. That was a mistake. The request “clear the build folder” tilted the answer for an unrelated read-only command, cat package.json && cat build/bundle.js, to 0.58 and blocked it. Every question reads the whole state. With the state reduced to the command alone, the same command scored 0.22 and ran. Keep the state to what the question is about.

Then we asked the agent: “The build folder is stale. Clear it out so we can do a fresh build.” It tried seven different ways to remove the folder and Decider 1 blocked every one. Its read-only looks at the folder passed, at 0.18 to 0.26. The folder was still there at the end.

A keyword list that blocks “rm” would have missed the Python and mv versions. The guard judges what a command does, not how it is spelled. Blocking the two mv commands is arguably over-cautious. That is a threshold choice.

Gate 3: is the answer finished

await agent.prompt(request);
let a = await decide('done', { request, answer: lastText() }, {
  answers_request: noul('Does the answer respond to what the user asked?'),
  specific: noul('Does the answer give concrete results (names, numbers, files) rather than a plan to get them?'),
});
if (a.answers_request.noul <= 0.5 || a.specific.noul <= 0.5) {
  await agent.prompt('That answer is not finished. Use the tools to get the concrete result, then answer again.');
  // ...and check once more
}

The done gate runs after the agent answers. The harness asks Decider 1 two yes/no questions: does the answer respond to what the user asked, and does it give concrete results, names, numbers, files, rather than a plan to get them. If either score is 0.5 or lower, the harness prompts the agent again with “That answer is not finished” and checks once more. On the TODO question, a concrete answer listing the count and file locations scored 0.90 and 0.95 and passed. An answer that only described how it would search with grep scored 0.46 and 0.16 and was sent back. In the full agent run, the agent found all 3 TODO comments in 2 files, and the done gate passed it at 0.92 and 0.93.

Cost and speed

With the model warm, each decision took under a second. The median latency was about 0.63 seconds, measured end to end from a laptop, network included. In the full TODO run the agent made six decisions, each between 0.66 and 0.93 seconds. Most of the run’s minute was spent in the agent’s own model, not in the checks. Fifteen guard, route and done decisions used 2,285 input tokens. At $0.03 per million input tokens that is well under a hundredth of a cent. Output is not billed, because nothing is generated. The only caveat: after a period with no traffic, the first request can wait while capacity starts. The API answers 429 with a Retry-After header, and the SDK retries on its own.

// A small agent harness: Pi runs the agent, Decider 1 makes the decisions.
//
//   TYPESAFE_BASE_URL=https://meragpt.com TYPESAFE_API_KEY=$MERAGPT_API_KEY \
//   AGENT_BASE_URL=http://127.0.0.1:8081/v1 node harness.mjs "your request"
//
// Three gates, each one Decider 1 call:
//   1. route  — which model should handle this request
//   2. guard  — is this shell command safe to run (Pi's beforeToolCall)
//   3. done   — does the answer finish the job, or send the agent back (Pi's followUp)
import { Agent } from '@mariozechner/pi-agent-core';
import { Type } from 'typebox';
import { TypeSafeClient, noul, choice } from '@typesafe-ai/sdk';
import { execFile } from 'node:child_process';
import { readFile } from 'node:fs/promises';
import { promisify } from 'node:util';

const run = promisify(execFile);
const WORKDIR = process.env.WORKDIR ?? process.cwd();

// The decision model, through TypeSafe's own SDK pointed at meragpt.com.
const decider = new TypeSafeClient({ timeout: 30_000 });
export const log = [];
async function decide(gate, state, questions) {
  const t0 = Date.now();
  const res = await decider.systemOne({ model: 'sd-1', state, questions });
  const ms = Date.now() - t0;
  log.push({ gate, state, answers: res.answers, ms, input_tokens: res.usage?.input_tokens });
  return res.answers;
}

// Any OpenAI-compatible endpoint can host the agent's own models.
const local = (id) => ({
  id, name: id, api: 'openai-completions', provider: 'local',
  baseUrl: process.env.AGENT_BASE_URL ?? 'http://127.0.0.1:8081/v1',
  reasoning: false, input: ['text'],
  cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
  contextWindow: 32768, maxTokens: 2048,
});
const MODELS = {
  quick: local(process.env.QUICK_MODEL ?? 'mlx-community/Qwen3.5-4B-MLX-4bit'),
  deep: local(process.env.DEEP_MODEL ?? 'mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit'),
};

// ---- gate 1: route -------------------------------------------------------
export async function route(request) {
  const a = await decide('route', { request }, {
    tier: choice('Which model should handle this request?', {
      quick: 'A short answer or one simple step. No planning.',
      deep: 'Several steps: investigating files, running commands, or making changes.',
    }),
  });
  return { tier: a.tier.choice, confidence: a.tier.confidence };
}

// ---- gate 2: guard every shell command before it runs ----------------------
// The state is the command and nothing else. Every question reads the whole
// state, so putting the user's request in ("clear the build folder") tilts the
// answer about an unrelated read-only command towards "destructive". Narrow,
// single-trait questions, each in the positive form: a composite "is this safe?"
// separates poorly, and a question and its negation are not complements.
export async function guard(command) {
  const a = await decide('guard', { command, working_directory: WORKDIR }, {
    deletes_or_overwrites: noul('Would running this command delete, truncate or overwrite files or data?'),
    uses_network: noul('Does this command send data over the network or download and run code from it?'),
  });
  const p = { deletes: a.deletes_or_overwrites.noul, network: a.uses_network.noul };
  return { blocked: p.deletes > 0.5 || p.network > 0.5, p };
}

// ---- gate 3: is the answer done? ---------------------------------------------
export async function done(request, answer) {
  const a = await decide('done', { request, answer }, {
    answers_request: noul('Does the answer respond to what the user asked?'),
    specific: noul('Does the answer give concrete results (names, numbers, files) rather than a plan to get them?'),
  });
  return { ok: a.answers_request.noul > 0.5 && a.specific.noul > 0.5, p: { answers: a.answers_request.noul, specific: a.specific.noul } };
}

// ---- the agent ---------------------------------------------------------------
const bash = {
  name: 'bash', label: 'Shell', description: 'Run a shell command in the project directory and return its output.',
  parameters: Type.Object({ command: Type.String({ description: 'The command to run' }) }),
  execute: async (_id, { command }) => {
    const { stdout, stderr } = await run('bash', ['-lc', command], { cwd: WORKDIR, timeout: 20_000, maxBuffer: 1 << 20 });
    return { content: [{ type: 'text', text: (stdout + stderr).slice(0, 8000) || '(no output)' }] };
  },
};
const readTool = {
  name: 'read_file', label: 'Read', description: 'Read a file from the project directory.',
  parameters: Type.Object({ path: Type.String() }),
  execute: async (_id, { path }) => ({ content: [{ type: 'text', text: (await readFile(`${WORKDIR}/${path}`, 'utf8')).slice(0, 8000) }] }),
};

export async function handle(request, { maxRetries = 1 } = {}) {
  const r = await route(request);
  const agent = new Agent({
    initialState: {
      systemPrompt: 'You are a careful assistant working in a software project. Use the tools to look things up; answer with concrete results.',
      model: MODELS[r.tier] ?? MODELS.deep,
      tools: [bash, readTool],
    },
    // A local server needs no key, but Pi asks for one; any string will do.
    getApiKey: async () => process.env.AGENT_API_KEY ?? 'local',
    toolExecution: 'sequential',
    beforeToolCall: async ({ toolCall, args }) => {
      if (toolCall.name !== 'bash') return undefined;
      const g = await guard(args.command);
      if (g.blocked) return { block: true, reason: `Blocked by policy: this command looks destructive or uses the network (${JSON.stringify(g.p)}). Find a read-only way.` };
      return undefined;
    },
  });
  const lastText = () => {
    const m = [...agent.state.messages].reverse().find((x) => x.role === 'assistant');
    return (m?.content ?? []).filter((c) => c.type === 'text').map((c) => c.text).join('').trim();
  };
  await agent.prompt(request);
  let verdict = await done(request, lastText());
  for (let i = 0; i < maxRetries && !verdict.ok; i++) {
    await agent.prompt('That answer is not finished. Use the tools to get the concrete result, then answer again.');
    verdict = await done(request, lastText());
  }
  return { route: r, answer: lastText(), done: verdict, error: agent.state.errorMessage ?? null,
    stops: agent.state.messages.filter((m) => m.role === 'assistant').map((m) => m.stopReason) };
}

if (process.argv[2]) {
  const out = await handle(process.argv[2]);
  console.log(JSON.stringify({ ...out, decisions: log }, null, 2));
}

Try it

The full harness is about 130 lines of JavaScript, shown above. To run it you need a meraGPT key. New accounts get $1 of free credit when they sign in, no card, which covers tens of thousands of decisions. The playground runs Decider 1 without an account. The /v1/systemone docs have the full request and response shapes.