Hermes Agent looked like the fast way in
#ai-agents #copilot #microsoft-agent-framework
I am experimenting with building agents that use open weight models I host myself, with a vague idea of building a threat analyst flow. I had heard great things about Hermes Agent, so I thought I would check it out. After spinning up my sandbox and a bit of battle, I got it to connect to my local models.
What I liked about the Hermes blurb was its built-in learning/skills loop and the number of plugins it seemed to have, promising the ability to connect to many platforms and tools.
The 1980s text adventure
I am sure it is a great tool for those that built it, but to say I was underwhelmed would be an understatement. It looked like a 1980s text adventure (I love 1980s text adventures!), but it had the feel of something vibe coded. Setting it up to run local models was endless YAML config changes. I got it to call my model, but it would not run tools.
The Haskell loop
In attempting to fix the tools, I ended up in what I can only describe as a “Haskell loop.” This is interesting, because it shows how poor these agents can be the moment you move a little outside the norm. Hermes promises to support local models, but what I actually saw was the model just giving me Haskell docs, even when I asked it a plain “Hello!”
Turns out there was a keyword collision: Hermes is also a popular Haskell library for JSON parsing. On top of that there was context collapse, because Hermes Agent requires a highly descriptive prompt to stay on track, and my smaller local models lost their identity context. The model was tripping over its own hallucinations. Because the prompts from Hermes had “json”, “schema”, “parser” and “hermes” all sitting together, it just gave me Haskell docs every time.
How I got there was even more interesting: it took just one changed value in my model’s response format to trigger the whole loop.
So it turns out agents and LLMs are no different from anything that came before: just a mess of YAML config. The only new problem is that this time the YAML was not written deliberately, it was generated by an LLM. Maybe you need an agent to set up your agent.
Why I went back to a framework
The honest version of this post is not “framework good, lightweight tool bad.” It’s that the upfront cost of a framework buys you things you don’t notice you need until you’re missing them, and for a threat analyst flow, what I was missing was a boundary.
Hermes Agent gave an unconstrained loop raw shell access and let it reach straight into whatever my prompt happened to wire up. That is exactly the wrong shape for something that is meant to be reasoning about untrusted input. I settled on the Microsoft Agent Framework instead, because it pushes you toward strict, isolated environments: tools are called behind an explicit API boundary rather than a model freely shelling out, and there is a real notion of governance and permissions around what an agent is allowed to touch. That is not glamorous, but it is the difference between “the model can call a function I defined” and “the model can do anything my user account can do.”
Trade-offs
- Pros of the framework route: a defined, auditable boundary between the model and my tools/shell, rather than an agent with the run of the sandbox. Sensible defaults for the state and tool-calling plumbing I was otherwise hand-rolling around Hermes.
- Cons of the framework route: more upfront structure to learn before you get anything running, and it is a heavier lift than spinning up a YAML file and pointing it at a model.
- Where Hermes Agent still makes sense: quick, throwaway experiments where you trust the environment completely and don’t mind babysitting it, not for anything that touches real credentials or untrusted input.
References
- Hermes Agent — the lightweight agent tool this post started with.
- Microsoft Agent Framework — the framework I moved to, for its stricter, sandboxed approach to tool calling.