Let the agent drive

The agent performs the demo itself on your real desktop: write a take.json, rehearse it, perform it while recording, then edit and render.

Instead of asking you to perform the demo, the agent can do it. It looks at your screen, moves the real mouse, clicks, types and scrolls, and ScreenPilot records everything while it does. The result is a recording plus a starter storyboard.json that goes into the normal edit and render loop.

This page is for developers and for anyone who wants to read or edit the file the agent writes. The file is called a take: take.json.

The agent uses your real mouse and keyboard. Keep your hands off the mouse while it runs: moving the pointer stops the action. Read Safety before you let an agent run unattended.

The loop#

Every driven demo follows five steps. The agent runs them; you can also run them yourself from the command line.

Explore. Look before you act: screenshot (with a 0..1 grid), windows, browser_open for web apps. Act with click, type_text and the other drive tools, and take a new screenshot after each action to check it worked.
Write take.json. Read the schema first: take_schema (MCP) or screenpilot take-schema.
Rehearse. Run perform with dryRun (CLI: screenpilot perform take.json --dry-run). It drives the desktop with the same actions but does not record. When a step fails, the error names the step. Fix it and run again until it is clean.
Perform. Run perform without dryRun. ScreenPilot records, drives, and writes a session folder with the video, events.jsonl and a starter storyboard.json.
Review and render. Look at the result with analyze and get_frames, edit the storyboard, validate it, render a draft, check it, then render the delivery quality. See Storyboard reference.

A dry run still makes the voice clips for your say lines, so the first one needs internet. The clips are cached next to the take, so later runs reuse them.

Commands#

screenpilot screenshot [--screen eDP-1] [--grid]   # what each monitor shows
screenpilot windows                                # open windows and their geometry
screenpilot browser --screen HDMI-A-1 --url https://example.com
screenpilot drive move eDP-1,0.5,0.5               # also: drive click, type, key, scroll
screenpilot take-schema                            # print the take.json schema
screenpilot perform take.json --dry-run            # rehearse: same actions, no recording
screenpilot perform take.json                      # record and drive; writes the session folder

In an AI app, the same steps are MCP tools. See Tools for the full list.

What a take looks like#

A take is one JSON file. Only steps is required.

Key	Default	What it does
`name`	none	A label for you.
`screens`	`"all"`	Which monitors to record: `"all"` or a list of monitor names.
`voice`	`en-US-AndrewNeural`, rate `+0%`, gap `0.35`	The voice for `say` lines (`name`, `rate`, `gap` in seconds between lines).
`setup`	`[]`	Steps that run before the recording starts: open apps, place windows. Use it for `browser`, `open`, `run`, `focus` and `waitForWindow`.
`steps`	required	The demo itself, at least one step.
`tail`	`1`	Seconds to keep recording after the last step or voice line.
`camera`	`"window"`	How the starter storyboard frames the video: `"window"` follows the app in use, `"screen"` follows the monitor, `"all"` shows every monitor side by side. `view` steps always win.

Paths in a take (a browser URL or goto given as a file) are relative to the take file. Write site/index.html, not an absolute path from your machine.

Targets#

Steps that point at something (click, move, drag, scroll) take a target. There are three forms.

Form	Means	Use it when
`{ "web": "selector" }`	An element in the demo browser, found by a Playwright selector: CSS, `text=Submit`, `role=button`.	The demo is a web app. This is the exact, preferred form: it finds the element wherever it is on screen.
`{ "window": "match", "x": 0.5, "y": 0.5 }`	A point inside the first window whose class or title contains `match`. x and y are 0..1 inside that window.	A native app, when the desktop can list windows. The point stays right even if the window moves.
`{ "screen": "eDP-1", "x": 0.5, "y": 0.5 }`	A point on one monitor, 0..1 (x from the left, y from the top). Without `screen` it is the monitor under the pointer.	Everywhere. It is the only form that works on desktops with no window control (GNOME on Wayland).

To read screen coordinates, take a screenshot with grid on and read x and y off the grid. To check that a {web} target exists before you click, call web_locate.

Every step#

Each step is an object with one main key. Unknown keys are rejected, so a typo fails validation instead of being ignored.

Step	Options	What it does
`say`	`voice`, `wait`	A narration line at this moment. A line never starts before the previous one ends. With `wait: true` the take pauses until the line finishes.
`caption`	`dur` (3), `position` (`bottom`, `top`, `center`)	Show a short text on screen for `dur` seconds.
`view`		Point the camera at a monitor name, or `"all"` for every monitor. Without `view` steps the camera follows the pointer.
`browser`	`screen`, `url`, `app`	Open the demo browser on a monitor. `app: true` hides the address bar and tabs, so the video shows only the page.
`goto`		Send the demo browser to a URL or a file path.
`move`	`dur`	Move the pointer to a target, like a hand would.
`click`	`button` (`left`, `right`, `middle`), `count` (1 to 3)	Move to a target and click. `count: 2` is a double click.
`drag`	`dur`	Press at `from`, move to `to`, release. Both are targets.
`type`	`cps`	Type text into the focused field at human speed. `cps` is characters per second. Give a string, or `{ "env": "NAME" }` for a secret.
`key`		Press a combo such as `ctrl+l`, `enter` or `alt+tab`. On macOS use `cmd+l`, not `ctrl+l`.
`scroll`	`at`, `horizontal`	Scroll by whole steps. Positive goes down (or right with `horizontal`), negative goes up. `at` is a target to move to first.
`wait`		Pause for a number of seconds.
`waitFor`	`web`, `timeout` (30)	Wait until an element shows up in the demo browser. Fails if it does not appear in time.
`waitForWindow`	`match`, `timeout` (15)	Wait until a window whose class or title contains `match` exists.
`focus`	`match`, `screen`	Raise a window, and optionally move it onto a monitor.
`open`		Open a file or URL with the system default app.
`run`		Start a program from a list: `["xdg-open", "~/Downloads/x.pdf"]`. There is no shell, so no pipes or quotes. Prefer `open`.

say, caption and view do not touch the desktop. They are written into the starter storyboard as narration, captions and camera switches, at the moment they happen. Inside setup they do nothing.

A complete take#

This take fills in a web form, downloads a file, opens it in a native app on the laptop screen, and scrolls it. The last voice line overrides the voice. The monitor names (HDMI-A-1, eDP-1) are examples: use the names that screenshot or doctor report on your machine.

{
  "name": "Submit a support ticket",
  "screens": "all",
  "voice": { "name": "en-US-AndrewNeural", "rate": "+0%" },
  "camera": "window",
  "setup": [
    { "browser": { "screen": "HDMI-A-1", "url": "https://app.example.com/support", "app": true } }
  ],
  "steps": [
    { "view": "HDMI-A-1" },
    { "say": "Let's open a support ticket.", "wait": true },
    { "wait": 0.6 },
    { "click": { "web": "input[name=subject]" } },
    { "type": "Cannot export my report" },
    { "click": { "web": "textarea[name=body]" } },
    { "type": "The export button spins forever on large reports." },
    { "caption": "Describe the problem", "dur": 3, "position": "top" },
    { "say": "Then we send it.", "wait": true },
    { "click": { "web": "text=Submit" } },
    { "waitFor": { "web": "text=Download receipt", "timeout": 20 } },
    { "click": { "web": "text=Download receipt" } },
    { "wait": 1.5 },
    { "open": "~/Downloads/receipt.pdf" },
    { "waitForWindow": { "match": "receipt", "timeout": 15 } },
    { "view": "eDP-1" },
    { "focus": { "match": "receipt", "screen": "eDP-1" } },
    { "say": "ورقة التأكيد وصلت وبتتفتح على طول.", "voice": "ar-EG-SalmaNeural", "wait": true },
    { "scroll": 3, "at": { "window": "receipt", "x": 0.5, "y": 0.5 } },
    { "key": "ctrl+w" }
  ],
  "tail": 1.5
}

What perform gives you#

A performed take produces a session folder with the video, events.jsonl (the clicks come from what the driver did, so they are exact) and a storyboard.json. The storyboard already has zooms placed on the clicks, your narration and captions at the right times, and the camera views. Edit it like any storyboard.

From the CLI the folder is sessions/<timestamp> unless you pass --dir. From an AI app it is a new folder in the ScreenPilot folder inside your Videos folder (Movies on a Mac).

When a step fails, the take stops, keeps whatever was recorded so far, and reports which step failed. In an AI app the result has failedStep, error and partial. Look at the partial recording to see where it went wrong.

Tips#

Say it, then click. Put a say with wait: true before an important click so the viewer hears what is about to happen first.
Wait for things, not for time. After a click that loads something, use waitFor or waitForWindow instead of a long wait. Keep plain waits to 0.3 to 1 second after a page change.
Keep takes short. Stay under about 90 seconds. A shorter take is faster to rehearse and easier to edit.
Keep lines short. One sentence, under about 12 words. The render freezes the video when a line is longer than its slot, so long lines add dead time.
Prefer {web} targets inside the demo browser. They find the element, so they do not break when the layout changes.
Secrets go in the environment. Use { "type": { "env": "NAME" } }. The value is read from the environment when the step runs and is never logged or printed. Never write a password into take.json.
Always rehearse. A dry run costs seconds. A failed real take costs a recording.
Show nothing private. Open pages with "app": true so no address bar, tabs or local paths appear.

What each desktop supports#

Driving works on KDE Plasma, any X11 desktop, GNOME on Wayland, Windows and macOS. It does not work on sway or Hyprland, where recording still works.

KDE Plasma, X11 desktops, Windows, macOS: everything in this page works, including {web} and {window} targets, focus and waitForWindow.
GNOME on Wayland: no window control. Use only { "screen", "x", "y" } targets. The focus and waitForWindow steps and the {window} and {web} targets fail with an error.
sway, Hyprland: driving is not possible.

On Wayland, the first drive asks you to approve a remote control dialog once. On macOS the AI app needs Accessibility permission. On Windows, apps that run as administrator cannot be driven unless the AI app runs as administrator too. See Platforms for the full table.