Computer use

Computer use lets an agent operate a Tektona sandbox the way a human would: see the screen, move the mouse, type, run a browser. Tektona ships an official desktop image with a full Linux desktop, a ready-to-run Chrome browser, and the in-sandbox tektonactl tool that exposes a small API for screenshots, mouse and keyboard input, the clipboard, window listing, and named PTY sessions.

ghcr.io/tektona-ai/desktop-x11:0.4.3

The newest tag is published at ghcr.io/tektona-ai/desktop-x11. Floating tags like :latest are rejected by the API, so always pin to a specific tag.

Spin up a desktop sandbox

tektona sandbox create \
  --image ghcr.io/tektona-ai/desktop-x11:0.4.3 \
  --egress-policy tektona/open \
  --cpu 1 --memory 4 \
  --vnc --browser

--vnc opens a VNC viewer after the sandbox is created. If the desktop is not already running, start it with tektona sandbox desktop start <id> or connect with tektona vnc <id> --start-desktop (see VNC access for everything VNC-specific).

If you'd rather wire it up step-by-step:

ID=$(tektona sandbox create -i ghcr.io/tektona-ai/desktop-x11:0.4.3 --egress-policy tektona/open -o json | jq -r .id)
tektona sandbox desktop start "$ID"
tektona vnc "$ID" --browser

Drive the desktop with tektonactl

tektonactl is on PATH inside every sandbox started from the desktop image. From your laptop you wrap calls with tektona ssh:

tektona ssh <sandbox-id> -- tektonactl <command> [args]

The rest of this page uses $ID as a stand-in for the sandbox ID.

Open Chrome on a URL

tektona ssh "$ID" -- tektonactl pty create chrome -- \
  google-chrome --no-first-run --start-maximized https://tektona.ai

pty create launches Chrome under a named PTY so it survives across SSH invocations. Subsequent commands can stream its log:

tektona ssh "$ID" -- tektonactl pty logs chrome --tail 50

When you're done:

tektona ssh "$ID" -- tektonactl pty kill chrome

Capture a screenshot

# Save locally
tektona ssh "$ID" -- tektonactl desktop screenshot > shot.png

# Or write to the sandbox's filesystem and pull it later
tektona ssh "$ID" -- tektonactl desktop screenshot -o /tmp/shot.png

tektona sandbox screenshot <id> does the same thing without needing an SSH round trip, useful for one-off captures.

Click and type

Coordinates are in pixels from the top-left of the desktop. Check the current resolution with tektonactl desktop display.

tektona ssh "$ID" -- tektonactl desktop click 600 400
tektona ssh "$ID" -- tektonactl desktop click 600 400 --double
tektona ssh "$ID" -- tektonactl desktop click 600 400 --button right

tektona ssh "$ID" -- tektonactl desktop type "hello world"
tektona ssh "$ID" -- tektonactl desktop key Return
tektona ssh "$ID" -- tektonactl desktop hotkey ctrl+s

For text input that triggers IME or rapid-repeat issues (some GTK widgets drop characters), bump the per-key delay:

tektona ssh "$ID" -- tektonactl desktop type "slowly" --delay 100

Closed-loop automation

Every input action accepts --screenshot to return a PNG of the post-action screen on stdout. One round trip per "act, then look":

tektona ssh "$ID" -- tektonactl desktop click 600 400 --screenshot > after.png
tektona ssh "$ID" -- tektonactl desktop hotkey ctrl+t  --screenshot > new-tab.png

Useful in agent loops where you want to confirm a click landed before deciding the next action.

Inspect windows and the clipboard

tektona ssh "$ID" -- tektonactl desktop windows     # list open windows
tektona ssh "$ID" -- tektonactl desktop clipboard   # read clipboard text

Long-running processes

tektonactl pty is a named-PTY runner. Think of it as tmux for single-command sessions. Use it when the process needs to outlive the SSH connection (dev servers, watchers, REPLs, browsers).

tektona ssh "$ID" -- tektonactl pty create web --max-log-size 10MB -- npm run dev
tektona ssh "$ID" -- tektonactl pty list
tektona ssh "$ID" -- tektonactl pty logs web --tail 50
tektona ssh "$ID" -- tektonactl pty send web $'rs\n'   # nodemon "restart"
tektona ssh "$ID" -- tektonactl pty kill web --timeout 5

--max-log-size accepts human sizes (10MB, 1GB). Older output past that limit is trimmed from a ring buffer.

Stop the desktop

tektona sandbox desktop stop "$ID"

Or pause the whole sandbox to free the host:

tektona sandbox pause "$ID"
tektona sandbox resume "$ID"      # later

Quick reference

Where to go next

• VNC access to watch the desktop in your browser while the agent drives it.
• HTTP previews to share a port from the sandbox externally.
• Agent Skills to install the tektonactl skill so an AI agent can drive computer use with these same commands.