MCP tools
The tools ScreenPilot gives an AI app, grouped by job: recording, footage, storyboard, render, combining recordings, desktop control and diagnostics.
ScreenPilot runs as an MCP server. Your AI app sees the tools below and calls them for you, so you ask for a video in plain words and the app picks the tools. This page is for developers who want to know exactly what each tool does. The server also sends the workflow to every client when it connects, so apps without a skill know the loop too.
Times in storyboard tools are source seconds (the raw recording). Positions are normalized 0..1, with x from the left and y from the top. Only check_output uses output seconds.
Recording#
| Tool | What it does | Key inputs |
|---|---|---|
record_start | Starts recording the desktop with the built-in recorder (no OBS needed), plus pointer, click and active-window events. Only one recording at a time. Ask the person to do the demo, then call record_stop. | dir, screens ("all" or monitor names), screen, fitWindow, recorder (native or obs), input, obsUrl, obsPassword, map |
record_stop | Stops the running recording. Returns the session folder, video path, events.jsonl path, duration and warnings. | none |
remap | Re-maps a session's pointer events to another monitor, when record_stop warned that it guessed which monitor OBS records. | dir, screen |
If the server's client closes during a recording, the server stops the recording and saves it.
Looking at footage#
| Tool | What it does | Key inputs |
|---|---|---|
analyze | Analyzes a recording: duration, idle spans, clicks, suggested segments and zooms. Also returns a contact sheet image of labelled thumbnails. Look at the contact sheet first. | target (session folder or video) |
get_frames | Extracts frames of the source recording at the given source seconds and returns them as images, at most 12 per call. Frames are also saved in a frames/ folder next to the video. With grid on, a 0..1 grid lets you read zoom, highlight and callout positions off the image. | video, times, grid, width (160..3840, default 1280) |
Storyboard#
| Tool | What it does | Key inputs |
|---|---|---|
storyboard_schema | Returns the JSON Schema of storyboard.json. Read it before you write a storyboard. | none |
storyboard_init | Writes a starter storyboard.json for a recording or a session. With suggest, it pre-fills segments (idle time cut) and zooms (from clicks). provider is edge (default), local or command, and voice is the voice name for it, for example voicetut:Sarah. | target, suggest, out, voice, provider, force |
storyboard_validate | Validates a storyboard against the schema and the source video. Returns valid with warnings, or the schema issues. It never throws. Run it before every render. | path |
voices | Lists voices. With provider edge (default) it lists Edge voices and needs a network connection; filter by locale, for example ar-EG or en-US. With local it lists the installed local models. With command it lists the voices defined in voices.json. Installing a local model has no tool: run screenpilot voice-setup in a terminal. | lang, provider |
The file format is in the storyboard reference. The voice providers are in Voices.
Render and check#
| Tool | What it does | Key inputs |
|---|---|---|
render | Renders a storyboard to MP4. It takes minutes. When a narration line is longer than its slot, the video freezes automatically (see autoHolds in the result). Use draft while iterating, looks for a normal render, and delivery only for the final file. Pass from and to (output seconds) to re-render only that window for a fast review: the file is <storyboard>.part-<from>-<to>.mp4. Pass formats to render one file per format, narration generated once. | storyboard, output, quality (draft, looks, delivery; default looks), from, to, formats |
check_output | Returns frames of a rendered video as images, at most 12 per call. Times here are output seconds. Without times you get 8 evenly spaced frames. Look for clipped captions, zooms on the wrong spot and awkward freezes. | video, times |
build | Generates the voice, cuts the video and writes the HyperFrames project without rendering. Reports the output duration, automatic freezes and warnings. | storyboard |
preview | Builds the storyboard and opens HyperFrames Studio so a person can scrub through it. | storyboard |
The formats input is a list of strings, each "16:9", "9:16", "1:1" or "WxH" such as "1280x720". It writes <storyboard>.<format>.mp4 (labels 16x9, 9x16, 1x1, WxH), and then output becomes the base name. The result is one entry per format. A portrait export crops landscape views that frame a window. See Several formats.
Combine recordings#
These tools join several storyboards, each with its own recording, into one video, for example to insert a feature clip into an existing demo without re-recording it. The file format is in Combine recordings.
| Tool | What it does | Key inputs |
|---|---|---|
sequence_schema | Returns the JSON Schema of sequence.json. Read it before you write a sequence. | none |
sequence_validate | Validates a sequence and every storyboard it uses. Returns valid with warnings, or the issues. It never throws. | path |
sequence_render | Renders each part like a storyboard (unchanged parts are reused), all at one output size, with the intro of the first part and the outro of the last, then joins them into one MP4. It takes minutes. Follow it with check_output. | sequence, output (default <sequence>.mp4), quality |
See and drive the desktop#
These tools let the agent look at the screen and act on it. Drive tools use the real mouse and keyboard. Read Agent drive for the loop and Safety for what the agent can touch and how to limit it. Every drive action is logged to audit.jsonl in the config folder. Typed text is never logged.
See#
These tools only read.
| Tool | What it does | Key inputs |
|---|---|---|
screenshot | Returns a picture of each monitor. With grid on, a 0..1 grid shows the x and y to pass to pointer_move and click. Take one before an action and one after, to verify. | screen, grid, width |
windows | Lists open windows: id, title, class or app, pid, whether active, geometry and monitor. Empty on desktops without window control (GNOME, sway); use screen targets there. | none |
browser_open | Opens, or reuses, the clean demo browser (its own profile) and places it on a monitor. app hides the address bar and tabs, so nothing about the machine shows. After that, use {web: selector} targets and web_locate. | screen, url, app |
web_locate | Finds an element in the demo browser by Playwright selector (CSS, text=..., role=...) and returns its rectangle in screen pixels. Use it to check that a target exists before you click. | selector |
wait_for | Waits until a window or a browser element shows up, so you can verify an action without guessing with sleeps. Give exactly one of window or web. It fails when the timeout passes. | window or web, timeout (seconds, default 15, max 120) |
page_text | Reads the demo browser's current tab as data: URL, title and visible text. Use it to check what a page says instead of reading a screenshot. | maxChars (default 4000) |
Act#
These tools move the real pointer and press real keys.
| Tool | What it does | Key inputs |
|---|---|---|
focus_window | Raises and focuses the first window whose class or title contains match. It can also move the window onto a monitor. | match, screen |
pointer_move | Glides the pointer to a point on a monitor. | screen, x, y, dur |
click | Moves the pointer to a target and clicks. | target, button (left, right, middle), count (1 to 3; 2 is a double click) |
drag | Presses at from, moves to to, releases. | from, to |
scroll | Scrolls the wheel, optionally after moving to a target. Positive steps scroll down or right. | steps, at, horizontal |
key | Presses a key combination in the focused window, such as ctrl+l, enter or alt+tab. On macOS use cmd, not ctrl. | combo |
type_text | Types text into the focused field like a person. For secrets pass {env: NAME}: the value is read from the server's environment and never shown. Typed text is never echoed back. | text (a string or {env: NAME}) |
Targets#
click, drag and scroll take a target. There are three forms:
{web: selector}: an element in the demo browser, by Playwright selector. This is the most reliable form.{screen, x, y}: a point on a monitor, normalized 0..1.screenis optional.{window, x, y}: a point inside the window whose class or title contains the text, normalized 0..1.
{screen, x, y} targets only there. Run doctor to see what the desktop supports.Takes#
A take is a take.json that the agent performs on the real desktop while recording.
| Tool | What it does | Key inputs |
|---|---|---|
take_schema | Returns the JSON Schema of take.json: say, caption, view, browser, goto, move, click, drag, type, key, scroll, wait, waitFor, waitForWindow, focus, open and run steps. Read it before you write a take. | none |
perform | Performs a take: records with the built-in recorder while driving the mouse and keyboard, and writes a storyboard from what happened. dryRun rehearses the same actions without recording; always rehearse first. If a step fails, the result is an error with failedStep, error and partial. Only one take runs at a time. | take, dir, dryRun |
Abort and resume#
If a person moves the mouse while the agent drives, the running action stops with a message. This protects the person at the machine. For unattended runs, start the server with SCREENPILOT_INTERRUPT=off.
| Tool | What it does | Key inputs |
|---|---|---|
resume | Carries on after a click, move or type stopped because someone else moved the pointer. It adopts the pointer where it is now and keeps the driver open, so no new approval dialog appears. Take a screenshot afterwards: the screen may have changed. | none |
abort | Stops whatever the agent is driving: the live driver and a running perform. Safe to call any time. The next drive tool opens a fresh driver. | none |
Diagnose#
| Tool | What it does | Key inputs |
|---|---|---|
doctor | Checks the machine and returns {summary, checks}: node, ffmpeg, desktop, monitors, windows, OBS, demo browser and voices, including a "local voice" check that is skipped when no local voice is set up. It never fails, so read the report. With input on, it also tests input injection and briefly moves the real pointer. | input |
Run doctor first on a new machine.