Agent computers
A sandbox with a desktop: Linux, a window manager and Google Chrome, 1280x800 by default. Your computer-use agent drives it with screenshots, clicks and keys. Your users watch it live in your own app, and take over when the agent needs a person. Groundbox supplies the computer; the agent and the product around it are yours. Billed like any sandbox, on every plan.
Start a desktop
# A desktop: Linux, a window manager, Google Chrome. 1280x800 by default.
curl https://api.groundbox.dev/v1/sandboxes \
-H "Authorization: Bearer $GROUNDBOX_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"desktop": true, "timeout_minutes": 60}'desktop takes true or {width, height, browser_profile_dir, start_url, recording}: a screen up to 2560x1600, the folder Chrome keeps its profile in (default /home/runner/.groundbox-browser), the page Chrome opens first, and recording. The default size, 2 vCPU and 4 GB, is a good start; give Chrome-heavy work more. The sandbox's desktop field says when it is ready and who has control. Audio is off.
Computer-use API
One endpoint, POST /v1/sandboxes/{id}/computer, takes the actions computer-use models emit, in their shape. Pass the model's tool input straight through. Any action takes screenshot: true to return a screenshot of what happened in the same answer (after screenshot_delay_ms, default 300).
# A screenshot (PNG, base64 in the answer)
curl https://api.groundbox.dev/v1/sandboxes/sbx_.../computer \
-H "Authorization: Bearer $GROUNDBOX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"action": "screenshot"}'
# Click, and get a screenshot of what happened in the same answer
curl https://api.groundbox.dev/v1/sandboxes/sbx_.../computer \
-H "Authorization: Bearer $GROUNDBOX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"action": "left_click", "coordinate": [640, 63], "screenshot": true}'
# Type, press keys, scroll
-d '{"action": "type", "text": "groundbox.dev"}'
-d '{"action": "key", "text": "Return"}'
-d '{"action": "scroll", "coordinate": [640, 400], "scroll_direction": "down", "scroll_amount": 5}'| action | Takes |
|---|---|
screenshot | PNG by default; format: "jpeg", quality, max_width / max_height to scale it down |
left_click, right_click, middle_click, double_click, triple_click | at coordinate [x, y] (or where the pointer is); text: modifiers held, like "ctrl+shift" |
mouse_move, left_mouse_down, left_mouse_up, left_click_drag | drag takes start_coordinate and coordinate |
scroll | scroll_direction up, down, left or right; scroll_amount in wheel clicks |
key, hold_key | text: "Return", "ctrl+l", "ctrl+shift+t", several separated by spaces; hold_key takes duration |
type | text, typed as keystrokes |
cursor_position, wait | wait takes duration in seconds |
clipboard_get, clipboard_set | the desktop's clipboard as text |
open_url | a new tab in Chrome |
wait_for_screen_change | until the screen differs from the start of the call (or from since: an earlier screen_hash), up to timeout_s |
Scaled screenshots: when you show the model a screenshot scaled with max_width, send its coordinates back with coordinate_space: {width, height} (the size the screenshot reported) and they are scaled to the screen. Each desktop takes up to 30 input actions and 10 screenshots a second; more answer 429.
The same actions are in the SDKs (computer(), computer_screenshot(), computer_click(), ...), the CLI (groundbox computer screenshot|click|type|key|scroll) and the MCP server (computer_screenshot, which returns an image the model sees, computer_click, computer_type, computer_key, computer_scroll, computer_action).
Live view in your app
A link to the live view goes in an iframe in your app. The page has nothing of ours on it: just the screen and, unless you hide it, a thin bar with Take control and I'm done. Links are watch-only, or with control: true their holder can take control. They last up to 24 hours, and revoking one (DELETE .../desktop/links/{link_id}) cuts open viewers off at once. frame_origins limits which pages may frame it.
curl https://api.groundbox.dev/v1/sandboxes/sbx_.../desktop/links \
-H "Authorization: Bearer $GROUNDBOX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"control": true, "expires_in_minutes": 60, "label": "ana@example.com", "frame_origins": ["https://app.example.com"]}'The token is in the URL's fragment, so it never reaches a server log. Add &ui=none to hide the bar and draw your own buttons: the viewer posts events to your page, and takes take_control and return_control from it. Or call POST .../handoff/take and .../handoff/return from your backend.
<!-- link.url from POST .../desktop/links. Add &ui=none to draw your own buttons. -->
<iframe src="https://api.groundbox.dev/desktop/view/5b0e...#gbd_..."
style="width: 100%; aspect-ratio: 1280 / 800; border: 0"
allow="clipboard-read; clipboard-write"></iframe>
<script>
const frame = document.querySelector("iframe");
window.addEventListener("message", (event) => {
if (event.source !== frame.contentWindow || event.data?.source !== "groundbox-desktop") return;
// ready, connected, disconnected, control, handoff_requested, handoff_returned, error
if (event.data.type === "handoff_requested") showTakeControl(event.data.reason);
});
// Your own buttons
const send = (type) => frame.contentWindow.postMessage({ type }, "https://api.groundbox.dev");
takeButton.onclick = () => send("take_control");
doneButton.onclick = () => send("return_control");
</script>Only the person in control can type: a watch-only link, or anyone while the agent has control, sees the screen and nothing they press reaches it.
Human handoff
When the agent hits a sign-in, a password, a 2FA prompt or a challenge, it stops and asks for a person. Control moves agent → requested → human → agent. From the request until the person gives control back, the agent's input actions answer 409 with reason: "human_in_control"; screenshots still work. A person can also take control without being asked.
# The agent stops and asks for a person
curl https://api.groundbox.dev/v1/sandboxes/sbx_.../handoff \
-H "Authorization: Bearer $GROUNDBOX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"reason": "Sign in to the bank and approve the 2FA prompt"}'
# Wait for a change (up to 25 seconds; call again with the new version)
curl "https://api.groundbox.dev/v1/sandboxes/sbx_.../handoff?after=2&wait_s=25" \
-H "Authorization: Bearer $GROUNDBOX_API_KEY"
# Your app's buttons: Take control, and I'm done
curl -X POST https://api.groundbox.dev/v1/sandboxes/sbx_.../handoff/take -H "Authorization: Bearer $GROUNDBOX_API_KEY"
curl -X POST https://api.groundbox.dev/v1/sandboxes/sbx_.../handoff/return -H "Authorization: Bearer $GROUNDBOX_API_KEY" \
-H "Content-Type: application/json" -d '{"note": "Signed in"}'Passwords and 2FA codes are always typed by a person, in the live view. Do not give them to the agent, and do not send them through the API. The SDKs' wait_for_handoff() returns when control is back, with the person's note. Every handoff is kept: who took control, when, and when they gave it back.
Sign in once, fork
A fork copies the running VM, desktop and browser included: fork a signed-in desktop and each copy is signed in. Or keep Chrome's profile in browser_profile_dir on a volume, so the next desktop starts with its cookies.
import { Groundbox } from "@groundbox/sdk";
const groundbox = new Groundbox();
// Sign in once, by a person
const desktop = await groundbox.create_sandbox({ desktop: { start_url: "https://app.example.com/login" } });
const link = await groundbox.create_desktop_link(desktop.id, { control: true });
console.log("Open and sign in:", link.url);
await groundbox.request_handoff(desktop.id, "Sign in to app.example.com");
await groundbox.wait_for_handoff(desktop.id); // back when they press I'm done
// Then fork the signed-in desktop for each run
const runs = await Promise.all([1, 2, 3].map(() => groundbox.fork_sandbox(desktop.id)));
for (const run of runs) {
const shot = await groundbox.computer_screenshot(run.id, { max_width: 1280 });
await groundbox.computer(run.id, { action: "left_click", coordinate: [640, 400] });
}An agent loop
Claude through OpenRouter, with a computer tool whose calls go straight to the API. OpenRouter does not pass Anthropic's own computer-use tool through, so this declares it as a regular tool in the same shape; with an Anthropic key, use Anthropic's computer toolset and map each call's name to action. Cap the turns: each one sends a screenshot.
import Anthropic from "@anthropic-ai/sdk";
import { Groundbox, GroundboxError } from "@groundbox/sdk";
const groundbox = new Groundbox();
// OpenRouter's Anthropic-compatible endpoint. With an Anthropic key, use new Anthropic() instead.
const claude = new Anthropic({ baseURL: "https://openrouter.ai/api", apiKey: null, authToken: process.env.OPENROUTER_API_KEY });
const MODEL = "anthropic/claude-opus-5.5"; // "claude-opus-5-5" on Anthropic's own API
const MAX_TURNS = 30; // a cap on what one task can cost
// The tool's input is the API's action as it is: the model's call goes straight through.
const computer: Anthropic.Tool = {
name: "computer",
description:
"Use a 1280x800 Linux desktop with Google Chrome. Every call returns a screenshot; coordinates are pixels in it. " +
"Never type passwords or 2FA codes: call request_handoff and a person will do it.",
input_schema: {
type: "object",
properties: {
action: { type: "string", enum: ["screenshot", "left_click", "double_click", "right_click", "left_click_drag", "mouse_move", "type", "key", "scroll", "wait", "open_url"] },
coordinate: { type: "array", items: { type: "integer" }, minItems: 2, maxItems: 2 },
start_coordinate: { type: "array", items: { type: "integer" }, minItems: 2, maxItems: 2 },
text: { type: "string", description: "type: the text. key: keys like Return or ctrl+l" },
scroll_direction: { type: "string", enum: ["up", "down", "left", "right"] },
scroll_amount: { type: "integer" },
duration: { type: "number" },
url: { type: "string" },
},
required: ["action"],
},
};
const requestHandoff: Anthropic.Tool = {
name: "request_handoff",
description: "Ask a person to take over: sign-ins, passwords, 2FA, challenges. Returns when they are done, with their note.",
input_schema: { type: "object", properties: { reason: { type: "string" } }, required: ["reason"] },
};
export async function runTask(task: string) {
const sandbox = await groundbox.create_sandbox({ desktop: true });
const messages: Anthropic.MessageParam[] = [{ role: "user", content: task }];
try {
for (let turn = 0; turn < MAX_TURNS; turn++) {
const response = await claude.messages.create({ model: MODEL, max_tokens: 8000, tools: [computer, requestHandoff], messages });
messages.push({ role: "assistant", content: response.content });
const calls = response.content.filter((block) => block.type === "tool_use");
if (calls.length === 0) return response.content;
const results: Anthropic.ToolResultBlockParam[] = [];
for (const call of calls) {
if (call.name === "request_handoff") {
const { reason } = call.input as { reason: string };
await groundbox.request_handoff(sandbox.id, reason); // your app shows Take control
const back = await groundbox.wait_for_handoff(sandbox.id);
results.push({ type: "tool_result", tool_use_id: call.id, content: back.handoff?.note ?? "Done. Take a screenshot." });
continue;
}
const action = { ...(call.input as object), screenshot: true } as Parameters<typeof groundbox.computer>[1];
try {
const done = action.action === "screenshot" ? await groundbox.computer(sandbox.id, { action: "screenshot" }) : await groundbox.computer(sandbox.id, action);
const shot = done.screenshot!;
results.push({ type: "tool_result", tool_use_id: call.id, content: [{ type: "image", source: { type: "base64", media_type: shot.media_type, data: shot.data } }] });
} catch (error) {
// A person has control, a bad coordinate, ...: the model reads it and goes on.
results.push({ type: "tool_result", tool_use_id: call.id, is_error: true, content: error instanceof GroundboxError ? error.message : String(error) });
}
}
messages.push({ role: "user", content: results });
}
} finally {
await groundbox.delete_sandbox(sandbox.id);
}
}Network and safety
- The open internet by default, with the same fences and abuse scoring as every sandbox; set
allow_hoststo narrow it. - Traffic leaves from our servers' addresses, not a home connection. Some sites challenge or block sign-ins from data centers: that is what the handoff is for. We do not route through residential proxies.
- Downloads stay in the VM. Recording is off unless you ask: with
recording: true, a frame each second the screen changed, in/home/runner/.groundbox-desktop/recording.