agentos.guide › blog

Hermes Computer Use Skill: The Quick Start, And How To Fit It Into Your Agent OS

By Julian Goldie · 23 October 2026 · agentos.guide

The Hermes computer use skill is Hermes Agent's bundled Computer Use skill, which teaches the model to drive your desktop through the built-in computer_use tool.

The honest answer to "skill or tool" is that it's both, because the toolset does the clicking and the skill tells the model how to click well.

Hermes Agent is the free, open-source agent from Nous Research, and it works with any tool-capable model on macOS, Windows and Linux.

This page is the quick start, plus how I'd fit computer use into each profile of an Agent OS.

The Hermes computer use skill quick start

This is the shortest path to an Agent OS profile that can drive desktop apps, with the safety settings switched on.

The rest of this page explains the skill-or-tool question, each step and how I'd map computer use to Agent OS jobs.

Skill or tool: the honest answer

Computer use in Hermes is a tool first, and the skill sits on top of it.

The skill page lists its authors as Francesco Bonacci (f-trycua) and Hermes Agent, under the MIT licence, for macOS, Windows and Linux.

Francesco Bonacci is also the founder and CTO of Cua, according to Cua's Y Combinator page, and cua-driver comes from Cua's trycua/cua project.

The skill can't click anything without the tool, so switching computer use on really means enabling the toolset.

What computer use does

Hermes drives your desktop in the background on macOS, Windows and Linux.

The docs say your cursor doesn't move, your keyboard focus doesn't change and your virtual desktops don't switch.

It works with any tool-capable model, including Claude, GPT, Gemini and local models on an OpenAI-compatible endpoint.

The old way of desktop automation grabbed your mouse, so you couldn't work while it ran.

The new way lets you and the agent co-work on the same machine.

Steps 1 to 3: check, grant and switch on

The docs say the driver ships with Hermes, because cua-driver is pinned and installed by default by the installers, hermes pm install and hermes update.

hermes computer-use status
hermes computer-use doctor
hermes -t computer_use chat

If the driver is missing, hermes computer-use install installs it, or you can pick Computer Use in hermes tools.

On macOS, grant Accessibility and Screen Recording to CuaDriver, and hermes computer-use permissions grant asks macOS for both.

On Linux you need a reachable display server with AT-SPI on, and Windows needs nothing at install time.

You can also add computer_use to your enabled toolsets in ~/.hermes/config.yaml.

Since v0.22.0, the release notes say computer_use sits behind tool_search by default and loads on demand, which keeps each profile's prompt lighter.

🔥 Want the exact Hermes Agent setup I run inside my Agent OS?

Inside the AI Profit Boardroom you get the installable Agent OS, step-by-step video tutorials, four coaching calls a week and 3,400+ members building real automations.

→ Get access here

How the skill makes the model work

The skill teaches one habit above all others, which is capture first, click by element number and verify.

computer_use(action="capture", mode="som", app="Mail")
computer_use(action="click", element=7, capture_after=True)

The som capture returns a screenshot with a numbered element list, vision returns a plain screenshot and ax returns the list with no image.

Element numbers only last until the next capture, and a stale number is refused rather than clicking the wrong thing.

When a click doesn't land, the skill climbs a ladder from a background element click to a pixel click and then a foreground click with its own approval.

For web page content, the skill points the model at the separate browser tools, because they are more reliable for pages.

On Telegram, Discord or Desktop you can ask "Send me a screenshot of my screen" and get the real image, which suits an Agent OS that lives in chat.

Step 4: pick the permission mode

The default standard mode keeps your normal Hermes approvals for every click and keystroke.

For scheduled jobs, the docs describe bounded mode with a capability manifest you review once at launch.

computer_use:
  permission_mode: bounded
  capability_manifest: ~/.hermes/cua-manifest.yaml

Anything outside the manifest fails closed, and a missing manifest fails loudly at session start.

--yolo, /yolo or approvals.mode: off launch an unrestricted runtime, and the docs warn that it doesn't protect against prompt injection.

Hard blocks still apply, including empty trash, log out, lock screen and typed patterns such as sudo rm -rf /.

Where nobody can approve, such as a cron job or a headless run, actions are refused rather than auto-approved.

Step 5: give a server bot its own screen

A bot on a headless Linux gateway has no monitor, so v0.22.0 added Bot Screen.

Each bot gets its own Xfce desktop over TigerVNC, streamed live into Hermes Desktop, and the CLI help calls the same feature Bot Desktop.

You click Take over for a login, 2FA, CAPTCHA or payment step, then Hand back, and the bot carries on with your session.

It's off by default, and you start it from the Screen pane or with hermes computer-use screen start.

Every profile has its own screen, browser profile and cookies, which maps neatly onto one bot per Agent OS profile.

The docs plan on about 1.1 to 1.5 GB of memory per open screen with a browser.

If bots are new to you, start with my Hermes Agent bot mode guide.

Step 6: keep the screen off your own desktop

The docs say computer_use acts on the same machine the bot's screen lives on.

With a Docker, SSH or Singularity backend, the driver runs inside the sandbox on its own display, so it can only touch what the terminal can.

Modal, Daytona and Vercel sandboxes can't host a display yet, so computer_use refuses there unless bot_desktop.placement: gateway opts into the host.

The newest option is Cua's cua-spaces plugin, which the catalog says lets Bot Desktop, computer_use and the browser follow the terminal into a Cua Space on Hermes 0.22.3 and later.

That works in Linux Spaces with the cua command at 0.7.0 or newer and the desktop stack installed in the Space.

That's the update in my video at the top, which walks through Cua's demo footage rather than my own test.

The full setup is in the sibling post on Hermes Cua, and the backends are covered in my Hermes Agent sandbox quick start.

🔥 Want the exact Hermes Agent setup I run inside my Agent OS?

Inside the AI Profit Boardroom you get the installable Agent OS, step-by-step video tutorials, four coaching calls a week and 3,400+ members building real automations.

→ Get access here

How I'd fit computer use into an Agent OS

I'd roll it out one profile at a time, starting with a low-risk helper in standard mode on a machine you can watch.

Which models drive it well

Nous's compatibility table rates Anthropic's Claude Sonnet and Opus models best overall, with raw coordinates and the numbered list both supported.

OpenAI, Gemini, OpenRouter vision models and local vision models on vLLM, LM Studio or Ollama also work.

Text-only models work in a degraded mode with mode="ax", or Hermes routes screenshots through the auxiliary vision model set in auxiliary.vision.

The docs say a 20-action session on a 1568×900 display typically costs about 30,000 tokens of screenshot context rather than about 600,000.

Fixes for the first run

• If anything is off, run hermes computer-use doctor first, because it prints a check for every part of the setup.
• If you see "cua-driver is not installed", run hermes computer-use install or enable the toolset in hermes tools.
• If clicks seem to do nothing, re-capture, because a hidden modal may be blocking input.
• If captures are empty on Linux, check that DISPLAY is set or that you have an XWayland bridge.
• If macOS grants look on but doctor disagrees, reset them with tccutil and run hermes computer-use permissions grant again.

Limits to know

• Some apps expose a sparse accessibility tree, so the agent falls back to pixel clicks.
• A normal Windows agent can't drive elevated admin windows.
• The type action won't enter passwords, so use your system's password autofill.
• macOS support relies on private interfaces that Apple can change in any update.

For the wider list of add-ons I'd look at next, see the best Hermes Agent tools.

If you want to talk through where desktop agents fit your SEO business, you can book a free SEO strategy session with my team.

FAQ

Is Hermes computer use a skill or a tool?

It is both. The computer_use toolset performs the actions through cua-driver, and the bundled Computer Use skill, version 2.1.0, teaches the model the actions, the workflow and the safety rules.

How do I turn on the Hermes computer use skill?

Check the driver with hermes computer-use status, grant any platform permissions, then enable the toolset in hermes tools or start a session with hermes -t computer_use chat.

Do I need to install cua-driver myself?

Usually not. The docs say it is pinned and installed by default by the installers, hermes pm install and hermes update, and hermes computer-use install repairs it if it is missing.

Which models work with Hermes computer use?

Any tool-capable model works. Nous rates Claude Sonnet and Opus best overall, and text-only models work in a degraded accessibility-tree mode.

Can each Agent OS profile have its own screen?

Yes. With Bot Screen on a Linux gateway, every Hermes profile has its own screen, browser profile and cookies, and you can take over for logins and hand back.

Is it safe to let Hermes use my desktop?

Input goes through the same approval gate as dangerous shell commands, risky key combos are hard-blocked and actions are refused where nobody can approve. A sandbox backend or a Linux Cua Space keeps the screen off your own desktop.

Final word on the Hermes computer use skill

Switch the toolset on, let the bundled skill guide the model, keep standard or bounded mode and move risky jobs into a sandbox.

If you run an agency and want help planning agent work for clients, you can book a free strategy session with my team.

If you want the whole Agent OS with the Hermes tab, the profiles and the training already set up, it lives inside the AI Profit Boardroom, with 3,400+ members and four coaching calls a week.

That is how I'd put the Hermes computer use skill to work.

Related guides on agentos.guide

→ Hermes Cua: The Quick Start

→ Hermes Agent Bot Mode

→ Hermes Agent Sandbox: The Quick Start

→ Hermes Agent v0.22: The Quick Start

📺 Video notes + links to the tools 👉

🎥 Learn how I make these videos 👉

🆓 Get a FREE AI Course + Community + 1,000 AI Agents 👉

← all guides on agentos.guide