# Agent computers

A sandbox with a desktop: Linux, a window manager and Google Chrome, 1280x800 by default. Your computer-use agent drives it with screenshots, clicks and keys. Your users watch it live in your own app, and take over when the agent needs a person. Groundbox supplies the computer; the agent and the product around it are yours. Billed like any sandbox, on every plan.

## Start a desktop

```
# A desktop: Linux, a window manager, Google Chrome. 1280x800 by default.
curl https://api.groundbox.dev/v1/sandboxes \
  -H "Authorization: Bearer $GROUNDBOX_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"desktop": true, "timeout_minutes": 60}'
```

`desktop` takes `true` or `{width, height, browser_profile_dir, start_url, recording}`: a screen up to 2560x1600, the folder Chrome keeps its profile in (default `/home/runner/.groundbox-browser`), the page Chrome opens first, and recording. The default size, 2 vCPU and 4 GB, is a good start; give Chrome-heavy work more. The sandbox's `desktop` field says when it is `ready` and who has `control`. Audio is off.

## Computer-use API

One endpoint, `POST /v1/sandboxes/{id}/computer`, takes the actions computer-use models emit, in their shape. Pass the model's tool input straight through. Any action takes `screenshot: true` to return a screenshot of what happened in the same answer (after `screenshot_delay_ms`, default 300).

```
# A screenshot (PNG, base64 in the answer)
curl https://api.groundbox.dev/v1/sandboxes/sbx_.../computer \
  -H "Authorization: Bearer $GROUNDBOX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action": "screenshot"}'

# Click, and get a screenshot of what happened in the same answer
curl https://api.groundbox.dev/v1/sandboxes/sbx_.../computer \
  -H "Authorization: Bearer $GROUNDBOX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action": "left_click", "coordinate": [640, 63], "screenshot": true}'

# Type, press keys, scroll
-d '{"action": "type", "text": "groundbox.dev"}'
-d '{"action": "key", "text": "Return"}'
-d '{"action": "scroll", "coordinate": [640, 400], "scroll_direction": "down", "scroll_amount": 5}'
```

| action | Takes |
| --- | --- |
| `screenshot` | PNG by default; format: "jpeg", quality, max_width / max_height to scale it down |
| `left_click, right_click, middle_click, double_click, triple_click` | at coordinate [x, y] (or where the pointer is); text: modifiers held, like "ctrl+shift" |
| `mouse_move, left_mouse_down, left_mouse_up, left_click_drag` | drag takes start_coordinate and coordinate |
| `scroll` | scroll_direction up, down, left or right; scroll_amount in wheel clicks |
| `key, hold_key` | text: "Return", "ctrl+l", "ctrl+shift+t", several separated by spaces; hold_key takes duration |
| `type` | text, typed as keystrokes |
| `cursor_position, wait` | wait takes duration in seconds |
| `clipboard_get, clipboard_set` | the desktop's clipboard as text |
| `open_url` | a new tab in Chrome |
| `wait_for_screen_change` | until the screen differs from the start of the call (or from since: an earlier screen_hash), up to timeout_s |

Scaled screenshots: when you show the model a screenshot scaled with `max_width`, send its coordinates back with `coordinate_space: {width, height}` (the size the screenshot reported) and they are scaled to the screen. Each desktop takes up to 30 input actions and 10 screenshots a second; more answer 429.

The same actions are in the SDKs (`computer()`, `computer_screenshot()`, `computer_click()`, ...), the CLI (`groundbox computer screenshot|click|type|key|scroll`) and the MCP server (`computer_screenshot`, which returns an image the model sees, `computer_click`, `computer_type`, `computer_key`, `computer_scroll`, `computer_action`).

## Live view in your app

A link to the live view goes in an iframe in your app. The page has nothing of ours on it: just the screen and, unless you hide it, a thin bar with Take control and I'm done. Links are watch-only, or with `control: true` their holder can take control. They last up to 24 hours, and revoking one (`DELETE .../desktop/links/{link_id}`) cuts open viewers off at once. `frame_origins` limits which pages may frame it.

```
curl https://api.groundbox.dev/v1/sandboxes/sbx_.../desktop/links \
  -H "Authorization: Bearer $GROUNDBOX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"control": true, "expires_in_minutes": 60, "label": "ana@example.com", "frame_origins": ["https://app.example.com"]}'
```

The token is in the URL's fragment, so it never reaches a server log. Add `&ui=none` to hide the bar and draw your own buttons: the viewer posts events to your page, and takes `take_control` and `return_control` from it. Or call `POST .../handoff/take` and `.../handoff/return` from your backend.

```
<!-- link.url from POST .../desktop/links. Add &ui=none to draw your own buttons. -->
<iframe src="https://api.groundbox.dev/desktop/view/5b0e...#gbd_..."
        style="width: 100%; aspect-ratio: 1280 / 800; border: 0"
        allow="clipboard-read; clipboard-write"></iframe>

<script>
  const frame = document.querySelector("iframe");
  window.addEventListener("message", (event) => {
    if (event.source !== frame.contentWindow || event.data?.source !== "groundbox-desktop") return;
    // ready, connected, disconnected, control, handoff_requested, handoff_returned, error
    if (event.data.type === "handoff_requested") showTakeControl(event.data.reason);
  });
  // Your own buttons
  const send = (type) => frame.contentWindow.postMessage({ type }, "https://api.groundbox.dev");
  takeButton.onclick = () => send("take_control");
  doneButton.onclick = () => send("return_control");
</script>
```

Only the person in control can type: a watch-only link, or anyone while the agent has control, sees the screen and nothing they press reaches it.

## Human handoff

When the agent hits a sign-in, a password, a 2FA prompt or a challenge, it stops and asks for a person. Control moves `agent` → `requested` → `human` → `agent`. From the request until the person gives control back, the agent's input actions answer 409 with `reason: "human_in_control"`; screenshots still work. A person can also take control without being asked.

```
# The agent stops and asks for a person
curl https://api.groundbox.dev/v1/sandboxes/sbx_.../handoff \
  -H "Authorization: Bearer $GROUNDBOX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Sign in to the bank and approve the 2FA prompt"}'

# Wait for a change (up to 25 seconds; call again with the new version)
curl "https://api.groundbox.dev/v1/sandboxes/sbx_.../handoff?after=2&wait_s=25" \
  -H "Authorization: Bearer $GROUNDBOX_API_KEY"

# Your app's buttons: Take control, and I'm done
curl -X POST https://api.groundbox.dev/v1/sandboxes/sbx_.../handoff/take -H "Authorization: Bearer $GROUNDBOX_API_KEY"
curl -X POST https://api.groundbox.dev/v1/sandboxes/sbx_.../handoff/return -H "Authorization: Bearer $GROUNDBOX_API_KEY" \
  -H "Content-Type: application/json" -d '{"note": "Signed in"}'
```

**Passwords and 2FA codes are always typed by a person**, in the live view. Do not give them to the agent, and do not send them through the API. The SDKs' `wait_for_handoff()` returns when control is back, with the person's note. Every handoff is kept: who took control, when, and when they gave it back.

## Sign in once, fork

A fork copies the running VM, desktop and browser included: fork a signed-in desktop and each copy is signed in. Or keep Chrome's profile in `browser_profile_dir` on a volume, so the next desktop starts with its cookies.

```
import { Groundbox } from "@groundbox/sdk";

const groundbox = new Groundbox();

// Sign in once, by a person
const desktop = await groundbox.create_sandbox({ desktop: { start_url: "https://app.example.com/login" } });
const link = await groundbox.create_desktop_link(desktop.id, { control: true });
console.log("Open and sign in:", link.url);
await groundbox.request_handoff(desktop.id, "Sign in to app.example.com");
await groundbox.wait_for_handoff(desktop.id); // back when they press I'm done

// Then fork the signed-in desktop for each run
const runs = await Promise.all([1, 2, 3].map(() => groundbox.fork_sandbox(desktop.id)));
for (const run of runs) {
  const shot = await groundbox.computer_screenshot(run.id, { max_width: 1280 });
  await groundbox.computer(run.id, { action: "left_click", coordinate: [640, 400] });
}
```

## An agent loop

Claude through OpenRouter, with a `computer` tool whose calls go straight to the API. OpenRouter does not pass Anthropic's own computer-use tool through, so this declares it as a regular tool in the same shape; with an Anthropic key, use Anthropic's computer toolset and map each call's name to `action`. Cap the turns: each one sends a screenshot.

```
import Anthropic from "@anthropic-ai/sdk";
import { Groundbox, GroundboxError } from "@groundbox/sdk";

const groundbox = new Groundbox();
// OpenRouter's Anthropic-compatible endpoint. With an Anthropic key, use new Anthropic() instead.
const claude = new Anthropic({ baseURL: "https://openrouter.ai/api", apiKey: null, authToken: process.env.OPENROUTER_API_KEY });
const MODEL = "anthropic/claude-opus-5.5"; // "claude-opus-5-5" on Anthropic's own API
const MAX_TURNS = 30; // a cap on what one task can cost

// The tool's input is the API's action as it is: the model's call goes straight through.
const computer: Anthropic.Tool = {
  name: "computer",
  description:
    "Use a 1280x800 Linux desktop with Google Chrome. Every call returns a screenshot; coordinates are pixels in it. " +
    "Never type passwords or 2FA codes: call request_handoff and a person will do it.",
  input_schema: {
    type: "object",
    properties: {
      action: { type: "string", enum: ["screenshot", "left_click", "double_click", "right_click", "left_click_drag", "mouse_move", "type", "key", "scroll", "wait", "open_url"] },
      coordinate: { type: "array", items: { type: "integer" }, minItems: 2, maxItems: 2 },
      start_coordinate: { type: "array", items: { type: "integer" }, minItems: 2, maxItems: 2 },
      text: { type: "string", description: "type: the text. key: keys like Return or ctrl+l" },
      scroll_direction: { type: "string", enum: ["up", "down", "left", "right"] },
      scroll_amount: { type: "integer" },
      duration: { type: "number" },
      url: { type: "string" },
    },
    required: ["action"],
  },
};
const requestHandoff: Anthropic.Tool = {
  name: "request_handoff",
  description: "Ask a person to take over: sign-ins, passwords, 2FA, challenges. Returns when they are done, with their note.",
  input_schema: { type: "object", properties: { reason: { type: "string" } }, required: ["reason"] },
};

export async function runTask(task: string) {
  const sandbox = await groundbox.create_sandbox({ desktop: true });
  const messages: Anthropic.MessageParam[] = [{ role: "user", content: task }];
  try {
    for (let turn = 0; turn < MAX_TURNS; turn++) {
      const response = await claude.messages.create({ model: MODEL, max_tokens: 8000, tools: [computer, requestHandoff], messages });
      messages.push({ role: "assistant", content: response.content });
      const calls = response.content.filter((block) => block.type === "tool_use");
      if (calls.length === 0) return response.content;

      const results: Anthropic.ToolResultBlockParam[] = [];
      for (const call of calls) {
        if (call.name === "request_handoff") {
          const { reason } = call.input as { reason: string };
          await groundbox.request_handoff(sandbox.id, reason); // your app shows Take control
          const back = await groundbox.wait_for_handoff(sandbox.id);
          results.push({ type: "tool_result", tool_use_id: call.id, content: back.handoff?.note ?? "Done. Take a screenshot." });
          continue;
        }
        const action = { ...(call.input as object), screenshot: true } as Parameters<typeof groundbox.computer>[1];
        try {
          const done = action.action === "screenshot" ? await groundbox.computer(sandbox.id, { action: "screenshot" }) : await groundbox.computer(sandbox.id, action);
          const shot = done.screenshot!;
          results.push({ type: "tool_result", tool_use_id: call.id, content: [{ type: "image", source: { type: "base64", media_type: shot.media_type, data: shot.data } }] });
        } catch (error) {
          // A person has control, a bad coordinate, ...: the model reads it and goes on.
          results.push({ type: "tool_result", tool_use_id: call.id, is_error: true, content: error instanceof GroundboxError ? error.message : String(error) });
        }
      }
      messages.push({ role: "user", content: results });
    }
  } finally {
    await groundbox.delete_sandbox(sandbox.id);
  }
}
```

## Network and safety

- The open internet by default, with the same fences and abuse scoring as every sandbox; set `allow_hosts` to narrow it.
- Traffic leaves from our servers' addresses, not a home connection. Some sites challenge or block sign-ins from data centers: that is what the handoff is for. We do not route through residential proxies.
- Downloads stay in the VM. Recording is off unless you ask: with `recording: true`, a frame each second the screen changed, in `/home/runner/.groundbox-desktop/recording`.
