VisualAgent Command & Result Schema

Command File Format

Command files are JSON objects dropped into the commands directory.

Required Fields

Field	Type	Description
id	string	Unique command identifier (used for result correlation)
type	string	One of: screenshot, tap, swipe, input, start_app
command	string	Alternative to type — some systems use command instead of type for the command type field. Agents should check both: cmd.get('type') or cmd.get('command')
params	object	Type-specific parameters (see below)

Optional Fields

Field	Type	Description
timestamp	string (ISO 8601)	When the command was created
priority	integer	Execution priority (lower = higher priority)
timeout_ms	integer	Maximum time to wait for execution

Command Types & Parameters

screenshot

{
  "id": "cmd_001",
  "type": "screenshot",
  "params": {
    "path": "/sdcard/VisualAgent/results/screenshot_001.png"
  }
}

Param	Type	Required	Default	Description
path	string	No	auto-generated	Where to save the PNG file

tap

{
  "id": "cmd_002",
  "type": "tap",
  "params": {
    "x": 500,
    "y": 800
  }
}

Param	Type	Required	Description
x	integer	Yes	Horizontal pixel coordinate
y	integer	Yes	Vertical pixel coordinate

swipe

{
  "id": "cmd_003",
  "type": "swipe",
  "params": {
    "x1": 500,
    "y1": 1200,
    "x2": 500,
    "y2": 400,
    "duration_ms": 300
  }
}

Param	Type	Required	Default	Description
x1	integer	Yes	—	Start X coordinate
y1	integer	Yes	—	Start Y coordinate
x2	integer	Yes	—	End X coordinate
y2	integer	Yes	—	End Y coordinate
duration_ms	integer	No	300	Gesture duration in milliseconds

input

{
  "id": "cmd_004",
  "type": "input",
  "params": {
    "text": "Hello, World!"
  }
}

Param	Type	Required	Description
text	string	Yes	Text to type into the focused input field

start_app

{
  "id": "cmd_005",
  "type": "start_app",
  "params": {
    "package": "com.google.android.calculator"
  }
}

Param	Type	Required	Description
package	string	Yes	Android package name to launch

Result File Format

Result files are written to the results directory with the naming convention {command_id}_result.json.

Success Result

{
  "id": "cmd_001",
  "type": "screenshot",
  "status": "success",
  "timestamp": "2026-06-25T17:16:00Z",
  "result": {
    "path": "/sdcard/VisualAgent/results/screenshot_001.png",
    "size_bytes": 340658
  },
  "error": null
}

Error Result

{
  "id": "cmd_002",
  "type": "tap",
  "status": "error",
  "timestamp": "2026-06-25T17:16:00Z",
  "result": null,
  "error": {
    "code": "SCREEN_LOCKED",
    "message": "Cannot tap: screen is locked",
    "details": "Screen wakefulness=Asleep, locked=true"
  }
}

Result Fields

Field	Type	Description
id	string	Command ID (matches the command file)
type	string	Command type
status	string	"success" or "error"
timestamp	string (ISO 8601)	When the result was produced
result	object or null	Type-specific result data
error	object or null	Error details if status is "error"

Error Object

Field	Type	Description
code	string	Machine-readable error code
message	string	Human-readable error message
details	string	Additional context (optional)

Common Error Codes

Code	Description
SCREEN_LOCKED	Screen is off or locked, UI operations unavailable
INVALID_COMMAND	Unknown command type or malformed JSON
MISSING_PARAM	Required parameter not provided
PERMISSION_DENIED	Insufficient permissions for operation
TIMEOUT	Command exceeded timeout limit
APP_NOT_FOUND	Package name not found for start_app
COORDINATE_OUT_OF_BOUNDS	Tap/swipe coordinates exceed screen dimensions

Directory Structure

/sdcard/VisualAgent/
├── commands/
│   ├── cmd_001.json
│   ├── cmd_002.json
│   └── archive/
│       ├── cmd_001.json (moved after processing)
│       └── cmd_002.json
└── results/
    ├── cmd_001_result.json
    └── cmd_002_result.json