Your agent should not take the cursor

On headless browser automation, the walls it runs into, and what we had to build to get past being stuck. · November 2026

Every computer-use agent I tried had the same shape: it moves your mouse, it clicks your windows, it types into whatever has focus. Watching it work is the demo. The demo is good.

Then you try to use it for something that takes half an hour, and you discover the actual cost. You cannot use your computer while it runs. Not "it is a bit annoying" — you literally cannot type, because your keystrokes land in whatever window the agent just brought forward. So you only run it when you are away from the machine. And an assistant you can only run when you are not working is not an assistant. It is a batch job with a nicer interface.

So we did the obvious thing and drove the browser headlessly instead. The agent gets its own Chrome, with its own profile and its own cursor, and it never touches yours. I kept working through a 36-minute run while it compared nine hotels and wrote 29 calendar events. The whole thing is on video, uncut, partly because that was the only way I could think of to prove the screen was never taken.

That part worked on the first try. Everything after it is the interesting half.

Headless is not free

The moment you stop pretending to be a human at a keyboard, a large part of the web stops letting you in. Not because of anything clever you did wrong — because headless Chrome has a fingerprint and plenty of sites just decline it.

Here is a measurement from this week, trying to post to Reddit:

howresult
curl — no cookies200 OK
curl — with a logged-in session cookieblocked
headless Chrome, same cookiesblocked
headless Chrome, a different account's sessionblocked
/api/*.json endpoints403

Four days earlier, the same code worked. Nothing on our side changed. That is the texture of this problem: it is not a bug you fix once, it is weather.

And you cannot engineer around it, because the correct response to "this site does not want automated access" is to stop, not to disguise yourself better. We decided early that we would not touch captcha solving. Once you start down that road you are building a different product, and you are building it against the people whose sites you depend on.

Which leaves you with the real problem

If you are not going to defeat the wall, you have to notice it and hand it to a person. That sounds trivial. It is not, and the way it failed for us is worth writing down.

Our wall detector had four cases: a captcha on screen, a visible password field, a one-time-code field, and nothing. It worked for months.

Then Google served a passkey prompt. A passkey screen has no password field — that is the entire point of passkeys. So it fell through every case and landed on "nothing". The detector reported no wall found. The agent, believing there was nothing in its way, did nothing at all and reported success-shaped silence.

No error. No log line. No exception to catch. The run just quietly stopped meaning anything, and the one code path that exists to hand a wall to a human never opened, because as far as the system was concerned there was no wall.

I want to be precise about why this class of bug is nasty. It was not that the detector was wrong about something. It was that the absence of a case looked exactly like a successful check. Everything downstream behaved correctly given its input. The failure had no surface.

Two decisions that turned out to carry weight

Fixing it meant adding a fifth case, and deciding where in the order it goes. That ordering is now the most important line in the file.

Password is checked before passkey. If a visible password field exists, the agent can fill it from the credential store and keep going — no human needed. Only when there is no password path does someone actually have to put a finger on a sensor. Check passkey first and you interrupt a person for a wall the machine could have cleared by itself. We interrupt people rarely enough that each interruption has to be real.

Only visible elements count. Hidden password inputs are everywhere; plenty of ordinary homepages ship one so the browser's autofill behaves. Count those and every page on the internet looks like a login wall. And the second your detector cries wolf a few times, you stop trusting it — at which point it is worse than not having one, because now you have a false sense of coverage.

The same logic says a one-time-code wall needs both a phrase and a field, since the word "verification" appears on every help page ever written. And passkey detection does not match the bare word: "What is a passkey?" in a blog post is not a wall.

Testing the thing that has no surface

A detector that fails by returning "nothing" cannot be tested by checking that it does not crash. So two things changed in how we test it.

First, the detector is a string of JavaScript that runs inside the page, and for a long time that meant it could only be exercised by launching a real browser — which is exactly why nobody noticed the missing case. We pulled it out of the module so a test can run it against a fake document. The thing that runs in production and the thing under test are now the same string.

That immediately caught a second bug, and it is a funny one. The detector lives inside a template literal, so \/ in a regex gets eaten during interpolation and the pattern silently becomes invalid. In the file it looked fine. Evaluated, it threw.

Second, and more useful generally: every test is verified by breaking the thing it tests.

reverttests that catch it
put passkey before password2
drop the visibility guard3
widen passkey to the bare word3
let the OTP case fire on the phrase alone2
let canAutomate accept captcha2

A test that still passes after you break its subject is worse than no test, because it reports coverage you do not have. The first version of one of these tests passed happily with the entire passkey case deleted — it was reading a string from the file instead of evaluating it. That is the same species of bug as the original one, found the same way.

Where it still loses

I would rather say this myself than have someone find it:

The code

The detector is open source, MIT, about a hundred lines with no dependencies: github.com/sujipar/wallcheck

import { checkWall, canAutomate } from 'wallcheck';

// runs inside the page
const { kind, why } = checkWall();

if (kind && !canAutomate(kind)) {
  // why === "Needs a fingerprint, Face ID or a security key."
  await askAHuman(why);
}

It returns one of captcha / password / passkey / otp / null. And canAutomate is true only for password. When it returns false, that is the signal to stop and ask — not to try harder.

It does not solve captchas, and that is deliberate. A captcha you defeated is a captcha you were not supposed to defeat.