OpenClaw and Hermes Are Cool, Not Useful
Like a lot of the internet, I was enraptured when OpenClaw blew up on X. It was the dream come true - AI that went from being smart best friend to AI doing useful work and freeing up all my time to play more tennis. Agents would not only code for me, but find bugs, think of ways to improve metrics, answer customer queries, run marketing and outreach campaigns, deal with investors, and everything in between.
But despite the hype, neither OpenClaw nor its followers (Hermes, nanoclaw, etc) have fulfilled that promise. I’m a Silicon Valley techie, so of course everyone I know has heard of OpenClaw, and half my friends have tried running it. And yet I know 0 people who rely on any of these agents IRL, but know a lot more people with idle Mac Minis. If they delivered on the promise, adoption would be a little better than that.
It doesn’t always not work
The thing is, OpenClaw is clearly a revolutionary product idea possible for the first time. An AI agent that can not only respond to queries, but actually have a cron job, memory you have control over, access to lots of APIs, and the ability to do browser use - the recipe is a really good one. And there’s some pretty cool success stories out there - a weekly friends planning agent, the car negotiation that saved its owner $4,200, an agent that booked a kid’s weekly swim class by filling out the signup forms itself, and one that watches for daycare waitlist openings and stroller tune-up cancellations and grabs the slot.
Unfortunately, all these turn out to be one-time successes. In the language of the τ-bench paper, they are high pass@1, but very low pass^8: meaning there’s a decent chance they’ll do the right thing when asked to do it once, but very low chance they’ll do the right thing 8 times in a row. A job, inconveniently, is pass^250.
When I’ve tried all the personal assistance agents, I’ve found the same failure mode - sometimes it wows by actually successfully booking a tennis court for me. But that turns out to have been a one-time fluke, rather than repeated success. And when I’ve tried it for work, it’s been a nonstarter - these agents can’t help me write essays like this one (they are shallow thinkers by default, their logic contradicts itself all the time, they use nonsense metaphors - any nonsense metaphors in this essay are handmade), nor do product work for me (they can’t research how other products do things by signing up, can’t watch product demo videos, can’t run customer research, they don’t know whom to call, they don’t know how to weigh different types of evidence they’re collecting, etc).
An unkind world
At least part of the reason the agents can’t succeed is not their fault: the world is not very friendly to them. Signup flows, CAPTCHAs, demo videos - these are interfaces built for human eyes and hands, not for machines. It doesn’t have to be that way - an agent-friendly world would sport APIs everywhere, and it’s telling that the one place agents shine is code, the one domain that already is all APIs. But an API fixes “can’t sign up.” It doesn’t fix “doesn’t know whom to call” or “can’t tell which evidence matters.” Even in a world paved with APIs, the failures below would remain.
Why they fail
When it comes to actually behaving like an employee would, the agent harnesses are hard/impossible to calibrate around a few things:
Moral judgment: on the one hand, Claude refuses to do calorie tracking for a friend because it has determined that counting calories eaten is the sign of an eating disorder, and on the other hand OpenAI’s model hacked into HuggingFace to pass a test. The judgment required to do real work is somewhere between staging an intervention for a hallucinated problem and committing cyber crimes. Real humans also have judgment errors, but for any person with task-relevant skills, their pass^8 is ~99% and not ~10%.
Persistence: we’re not able to get the models to try the right amount of hard at the right tasks. On one extreme is the Meta AI security researcher whose agents started deleting all her emails and didn’t take no for an answer, and on the other extreme is a coding agent that gives up on fixing all the tests and instead assigns the human the task of fixing them - or worse yet just comments the test out.
Context: despite increasing context windows, models can’t hold the type of context an employee has after months in the job in their heads - what happens isn’t that the employee has memorized a ton of stuff, but rather they have applied judgment day in and day out to compress what’s important into their brain in a way that’s extremely memory efficient. The model just doesn’t have enough judgment to do that, even if the context window itself is much larger than a person’s working memory.
In many ways - the model is smarter than any human you can hire. The billions of dollars spent pretraining the model means it has more knowledge than any human you will ever meet. It can absorb very large chunks of materials in seconds, and can produce a shocking amount of text extremely quickly. However, the unpredictability matters way more than you think it will - because a model that does the right thing occasionally and never many times in a row turns out to be pretty useless as a stand in for an employee.
This isn’t just a bug, it’s endemic to the large language model itself. Better engineering and better models can help us work around it, but we can never get away from it entirely, because inference from a large language model is sampling from a distribution. Whereas a human brain is a system that converges (do a task 50 times, and it settles into a groove), the model is a system that doesn’t converge - it draws fresh from the distribution every run. And small per-step wobbles compound across a long trajectory: an agent that’s 99% right at every step completes a 100-step job about a third of the time.
Code: the exception that’s not an exception
As a lifelong programmer, I continue to be blown away as the models get better and better at programming. And if there’s any place where it can almost sort of do the work of a full person, it’s in coding. Famously, the creator of OpenClaw, Peter Steinberger said “I ship code I don’t read.” That’s pretty amazing - being able to build such a powerful product without reading the code for it.
But note, OpenClaw didn’t make OpenClaw. It didn’t make Hermes or NanoClaw, or any of the other products either. I don’t know of a single product or repo that’s end-to-end run by only agents - where agents look at support tickets, think of product innovations, think of all the tests, do all the deployment work, and actually run the whole thing. They’re still very much assistants, albeit incredibly good ones.
The failure modes are the same as before, the agent can write an openclaw skill, but can’t think of which one, can’t quite get the relevant context right, and will probably not have it work on the first try. And even if it does get it working on the first try, it certainly won’t get the next one on the first try, and probably not after many retries either.
Code turns out not to be an exception to agent-behaving-as-employees, but actually a best-case demonstration - there’s maximum AI ability, somewhat lowered stakes (because of tests, code review, and a staged deployment process), and they still can’t be given the type of autonomy even a junior employee would have.
Better models
People on the internet are fond of saying “this is the worst it’ll ever be at task X.” I love that saying - I think it’s both true and pregnant with so much promise of the times that are to come. And that may be the appropriate rejoinder for this essay as well.
Models are, undoubtedly, getting better fast. The time horizon that agents can work for is doubling every seven months (METR’s measurement, and the single most-cited capability trend in AI), and there doesn’t appear to be a plateau in sight. They’re also getting smarter - scoring better and better on benchmarks, doing work that wasn’t previously possible (frontier models gained 30 points in a single year on Humanity’s Last Exam, a test written by a thousand experts specifically to stump them, and Gemini went from silver to gold on the International Math Olympiad; benchmarks designed to stay hard for years now saturate in months). But the kinds of problems that make OpenClaw underdeliver are not the ones that seem to be improving.
Agents still don’t know when to stop or keep going. They get better at hacking into systems, but don’t know that mostly they shouldn’t. They are as susceptible to prompt injection as ever, but the consequences are more devastating than before (when Varonis phished an OpenClaw agent they’d wired to a mock company inbox, a single email from an outside Gmail address, posing as a team lead named “Dan” with an urgent production incident, was enough: the agent dug up the AWS keys, database connection strings, and SSH credentials, and forwarded them all to the attacker in plaintext).
On top of that, there are new kinds of headwinds that agents are facing - the world is closing what kind of access agents can get. Cloudflare, which fronts roughly a fifth of the web, now sorts AI traffic into Search, Agent, and Training, and as of September 15 blocks Agent traffic by default on ad-supported pages (search stays allowed - looking is fine, acting on your behalf is not). And these aren’t safety measures - they’re revenue protection.
I don’t think these things mean that the promise of OpenClaw will never come true, I just think that it will take longer than most suspect.
Signs I’m looking out for
I’m watching for one sign in particular: the path to a fully operational agent that behaves like a real human worker goes through a fully operational agent that can do just a couple of things really well, reliably. For example, if I can actually trust an agent to do status chasing for me with a vendor - why is my ticket still open, when will it close, follow up - all at a cadence I would never be bothered to sustain, I would think we’re getting closer.
The thing it would prove to me is that it can make a series of small decisions autonomously - when to follow up, what tone to use, where to push back, whether to look for alternatives, whom else to bring into the conversation.
In the meantime, OpenClaw promised an employee, but really gave us ChatGPT with a cron job. If you consider the list of use cases people like OpenClaw for, you’ll see that a preponderance of them are something you’d use ChatGPT for, but with a cron job (which ChatGPT now kind of supports). But it doesn’t come with the venture backed subsidization of ChatGPT, so it costs you a whole lot more (or risks getting your account banned).
I don’t run OpenClaw / Hermes anymore, but whenever a new harness starts showing up in my X or WhatsApp feeds, I try to book tennis courts again. I can’t wait for them to succeed in booking me the courts I want, twice in a row.