Note: This is a re-post of my LinkedIn article.

Overview

I’ve written about using OpenClaw, Hermes, and experimenting with other autonomous agent harnesses (like Claude Desktop). There is no “best architecture” because these things should be purpose-built for your way of working. In this article, I’m sharing how my setup has evolved using a few more tools and I’ll explain what they do.

The Harness

Hermes wins here because it’s self-improving, has great speed and writes its own skills. My agent needs to act like an employee or business partner, OpenClaw forgets too much. I still run an agent on OpenClaw because that’s how I started but going forward, new agents are built on Hermes Agent. An agent is only as good as the models it has access to. I’ve tried “free” tiers of models, and it makes the agent feel dumb. It will trick you because some things will work, you’ll get good responses, even a website built out for you, but you’ll end up regretting it – the quality is terrible.

The Memory

If I had a new install of Hermes, one of the first things I would do is give it better memory. It’s OK out of the box but there are plugins that make it better. I’ve heard honcho is a great choice but I’m running Mnemosyne. Mnemosyne is built for Hermes and allows you to dig into the memory system – not by analyzing a bunch of files or database tables, visually too:

With memory handled, it’s time to review models.

The Model

When using OpenClaw at first, I was switching between Claude Sonnet and Claude Opus, depending on my needs. These models are expensive, or if you’re using a subscription plan, time-consuming because you have to wait a while for quota resets. I’ve tried Z.AI subscription and then OpenAI Codex, too. I actually had the best experience with Z.AI‘s GLM-5-Turbo but sometimes I needed to switch. All of this switching is painful. I also tried various ways to incorporate a “model router” within OpenClaw and found them to be very clunky. I often got errors. So here’s my recommendation: use a multi-model subscription. There’s a few good subscription and pay-as-you-go options:

• OpenRouter – PAYG but every model is available.
• OpenCode – Subscription with 12 curated models.
• GitHub Copilot – Subscription (currently paused) with models from Anthropic, Google, OpenAI and more.
• Together AI – PAYG High-performance open-source models from Gemma, Mixtral, Qwen, and Llama.
• Fireworks AI – PAYG Fastest multi-model, similar models to Together AI
• Groq Cloud – PAYG Low latency Llama and Mixtral.

Best of all, you can use multiple of them but then switching between models and now model providers becomes really hard. We have a solution for this, that I explain in the next section. A quick note on privacy: some people distrust Chinese models, others distrust all cloud-hosted models. I get it and it may depend on the kind of work you’re doing. You can adapt these learnings to your posture.

The Routing

This was the key unlock for me most recently. We need a fool-proof way to route. Enter, Manifest. Manifest is an open-source LLM router for agents. It sends each query to the cheapest model that can handle it, saving up to 70% on inference costs. You can control the routing and set limits. But, best of all, it will automatically fallback to a different model on failure.

When I was using just OpenRouter, I often had model failures. OpenClaw and Hermes would try their best to re-try with their configured fallback model, but it was hit-or-miss (mostly miss). I was hand-editing config files to change the model and restart my work. Manifest fixes this. Here is my current routing config on Manifest:

Manifest can connect to your OpenCode subscription, OpenRouter, Anthropic, GitHub, Google, Minimal, Ollama Cloud, OpenAI, xAI, Z.AI, Deepseek, Moonshot, Nvidia and others (including custom). Then you can configure the routing, limits, which agents use what models, and other setting. Manifest also gives you a nice overview, here’s my token usage for the past few days:

I can see every query, which model was used, its cost, token usage, latency, and cache hit. In the next image, you can see one of the rows shows “Handled” meaning the model (Kimi K2.6, in this case) failed but Manifest sent it to Deepseek V4 Pro (fallback).

In Hermes, no errors, maybe a delay that I didn’t notice. Distrust cloud? You can run your own Manifest instance, using local models if you want.

The Architecture

Now that we have the pieces here’s the end-to-end architecture, to make this really simple.

1. Run Hermes in a VM, VPS, or on your own computer. Since I already had a computer lab, I just spun up a VM to host it. I like that all my files (memory, skills, etc.) are all saved locally.
2. The ONLY endpoint (cmd: hermes model) that Hermes needs is the one to Manifest and your API key. Manifest will handle the rest.
3. Within Manifest, setup any one of the subscription or pay-as-you-go plan or better yet, a combination of them. Setup routing, limits, and other settings. I’m using OpenCode Go.
4. Do yourself a favor and avoid trying to juggle free models. Pay for OpenCode Go, GitHub Copilot (when its available again) or add credits to OpenRouter to access frontier models (even if they are Chinese or open source).
5. For extra monitoring, use Langfuse. I have it setup but find that I rarely need to go there anymore since I am using Manifest.

If I started heavier work and ran into subscription limits within OpenCode, my next step is either a subscription for MiMo, Kimi or GLM – all are low cost and really good. I mean, just look at this 99% price reduction from Xiaomi:

The Wrap

Claude Opus and GPT-5.5 get all the hype but I see people struggling with Hermes (or OpenClaw) because they picked one of these models and think they’re set. Here’s an example:

Is Hermes the problem? Is GPT-5.5 the problem? There will be problems, build a resilient architecture!