Record desktop workflows once. Compile them into reusable agent skills.
Record → Compile → Give the skill to your agent
Install · Features · How It Works · Docs · Report Bug
An MCP server that records a human-operated desktop workflow and compiles it into reusable skill files. The generated SKILL.md can then be given to an agent so the agent can perform the workflow later.
- Record — Captures native accessibility events, typed values, target metadata, and optional screen video to VideoDB for visual reference.
- Compile — An LLM transforms the event log and scene descriptions into reusable
SKILL.jsonand human-readableSKILL.mdfiles. - Use — The server installs the generated
SKILL.mdinto the agent's global skills directory, where future agent runs can use the skill instructions, inputs, verification checks, and execution guidance.
Demonstrate a task once on screen, and the server produces a self-contained, versioned skill artifact. This repo does not include a replay engine; replay is performed by the agent that consumes the generated skill.
Record_and_Replay.mp4
git clone https://github.com/video-db/open-record-replay.git
cd open-record-replay
uv syncCreate a .env file in the project root:
VIDEODB_API_KEY=your_VIDEODB_API_KEYAdd to your MCP config (claude_desktop_config.json, VS Code MCP settings, etc.):
{
"mcpServers": {
"videodb-record-replay": {
"command": "uv",
"args": ["run", "python", "server.py"],
"cwd": "/path/to/open-record-replay"
}
}
}Five tools and two resources will appear. You're ready to record.
Platform-specific setup
macOS — Requires Screen Recording, Microphone, Accessibility, and Input Monitoring permissions. Run the permission helper first:
uv run python scripts/smoke_macos_hook.py --prompt-permissionsIf ready_for_event_recording is false, enable the terminal process in System Settings > Privacy & Security > Accessibility and Input Monitoring, then rerun.
Windows — Uses UI Automation. No additional setup required beyond the standard install.
Linux — Uses AT-SPI. Ensure at-spi2-core is installed and your desktop environment has accessibility enabled.
Recording is human-in-the-loop. The agent starts recording, announces that recording is active, then waits. The human operator performs the UI workflow being captured.
record_skill_tool("my-workflow", lead_in_seconds=5)
→ Agent tells the operator recording is active
→ Operator switches to the target app and performs the workflow
→ Operator returns to the MCP client and says "stop"
→ Agent calls stop_recording_tool(trim_end_seconds=10)
→ Agent calls compile_skill_tool(video_id, "my-workflow")
→ Agent verifies/reports global_skill_md_path for future use
lead_in_seconds
The recorder starts capture immediately, then the compiler ignores events before the effective workflow start. With lead_in_seconds=5, the operator can switch from the MCP client to the target app and should begin the demonstrated workflow after 5 seconds.
trim_end_seconds
Discards events at the tail of the recording. Use when the operator must switch back to the MCP client to say "stop". For example, trim_end_seconds=10 ignores the final 10 seconds so the generated skill does not include the operator returning to the terminal or chat window.
Events-only mode
If VideoDB screen capture is unavailable, the system falls back to recording native accessibility events only. Call compile_skill_tool with video_id="" or video_id="none" to compile from events alone.
Human performs workflow
|
v
+------------------------------------------------------------------+
| Native AX/UIA/AT-SPI hooks -> events.jsonl |
| VideoDB Capture SDK -> video_id, when capture succeeds |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| Compiler |
| events.jsonl + optional scene summaries/transcript -> SKILL.json |
| SKILL.json -> SKILL.md |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| Agent skill install |
| ~/.mcp-videodb/skills/<name>/SKILL.md |
| ~/.codex/skills/<name>/SKILL.md, unless overridden |
+------------------------------------------------------------------+
The accessibility hooks provide the action log. VideoDB scene indexing adds visual context when screen capture is available. The compiler combines those signals into skill files that a future agent can use to perform the workflow.
| Feature | Description |
|---|---|
| Dual recording | Captures native accessibility events and, when full capture is available, screen video for visual reference |
| LLM compilation | VideoDB scene indexing and generate_text compile event logs, scene descriptions, and transcript text into structured SKILL.json |
| Graceful degradation | Falls back to events-only recording when screen capture is unavailable |
| Cross-platform | Native accessibility hooks for Windows (UIA), macOS (AX), and Linux (AT-SPI) |
| Skill versioning | Auto-increments on recompile; archives old versions as SKILL.vN.json |
| Variable templating | Prompts the compiler to turn recorded literals such as search queries, dates, dropdown choices, and file paths into reusable inputs |
| Human-in-the-loop | Recording is operator-driven, not agent-driven — the human demonstrates, the AI learns |
| Tool | Parameters | Description |
|---|---|---|
request_capture_permissions_tool |
— | Request microphone and screen capture permissions before recording |
record_skill_tool |
name: str, lead_in_seconds: float = 0.0 |
Start a human-in-the-loop workflow recording |
stop_recording_tool |
trim_end_seconds: float = 0.0 |
Stop the active recording and export video to VideoDB when capture is available |
compile_skill_tool |
video_id: str, name: str |
Compile a recording into SKILL.json and SKILL.md, then install SKILL.md globally for future agent use |
list_skills_tool |
— | List all skills generated through this MCP |
| Resource | Description |
|---|---|
skills://list |
List all available skills as JSON |
skills://{name}/content |
Load a skill's SKILL.md into the agent context |
Compiled skills land in ~/.mcp-videodb/skills/<name>/:
| File | Purpose |
|---|---|
SKILL.json |
Structured skill definition with steps, inputs, verification checks, recorded surface data, and execution strategy |
SKILL.md |
Human and agent-readable markdown following the agentskills.io standard |
SKILL.vN.json |
Archived previous versions on recompile |
Every generated SKILL.json includes an execution_strategy — web_browser, desktop_app, hybrid, terminal, file_system, or unknown — so the agent consuming the skill knows which tool path to prefer. Every SKILL.md includes execution guidance, continuous improvement guidance, and agent tool-priority guidance.
After SKILL.md is created, compile_skill_tool also installs it into the
agent's global skills directory, ~/.codex/skills/<name>/SKILL.md by default,
and returns global_skill_md_path plus an agent_instruction reminding the
agent to verify or report the global install. Agents should ensure this global
install step has happened so the skill is available in future runs. Set
CODEX_HOME to change the base Codex directory, or set
AGENT_GLOBAL_SKILLS_ROOT to override the global skills directory directly.
open-record-replay/
├── server.py # FastMCP entry point, tool and resource definitions
├── state.py # Shared server state singleton
├── config.py # Constants, .env loading
├── registry.py # Skill CRUD + versioning
│
├── capture/
│ ├── recorder.py # Records native accessibility events + optional VideoDB capture
│ ├── ax_client.py # JSONL IPC wrapper for native accessibility companion
│ ├── capture_client.py # VideoDB Capture SDK wrapper
│ └── native/
│ ├── ax_hook_win32.py # Windows: UI Automation + keyboard polling + TCP IPC
│ ├── ax_hook_darwin.py # macOS: Accessibility API + pynput + pipe IPC
│ └── ax_hook_linux.py # Linux: AT-SPI + pynput + pipe IPC
│
├── compiler/
│ ├── compiler.py # LLM compilation: index scenes → match events → prompt → normalize
│ ├── prompts.py # LLM system prompt for structured skill generation
│ ├── md_generator.py # Converts SKILL.json to agent-readable SKILL.md
│ ├── tool_manifest.py # Surface-to-tool mapping for replay guidance
│ └── recommended_tools.json
│
├── schema/
│ └── skill.schema.json # JSON Schema (draft-07) for SKILL.json validation
│
├── scripts/
│ └── smoke_macos_hook.py # macOS AX hook permission helper
Recording won't start
- Verify
VIDEODB_API_KEYis set in.envand is valid - Run
request_capture_permissions_tooland approve any permission prompts - On macOS, check Screen Recording and Accessibility permissions
- Check that no other application is using the accessibility hook
Compilation fails or returns empty steps
- Ensure the recording has meaningful UI interactions (not just idle time)
- Try events-only compilation (
video_id="") if video indexing is slow - The LLM may need another generation attempt — the compiler is configured for up to 2 attempts
Permission prompts not appearing on macOS
# Reset permissions and try again
uv run python scripts/smoke_macos_hook.py --prompt-permissionsIf ready_for_event_recording is false, manually enable the terminal in System Settings > Privacy & Security > Accessibility and Input Monitoring.
Windows: no keyboard events recorded
- Ensure the app being recorded has UI Automation support (most modern apps do)
- Docs: docs.videodb.io
- Issues: GitHub Issues
- Discord: Join the VideoDB community
- API Key: console.videodb.io
Made with ❤️ by the VideoDB team