# VisualAgent Command & Result Schema

## Command File Format

Command files are JSON objects dropped into the commands directory.

### Required Fields

| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Unique command identifier (used for result correlation) |
| `type` | string | One of: `screenshot`, `tap`, `swipe`, `input`, `start_app` |
| `command` | string | Alternative to `type` — some systems use `command` instead of `type` for the command type field. Agents should check both: `cmd.get('type') or cmd.get('command')` |
| `params` | object | Type-specific parameters (see below) |

### Optional Fields

| Field | Type | Description |
|-------|------|-------------|
| `timestamp` | string (ISO 8601) | When the command was created |
| `priority` | integer | Execution priority (lower = higher priority) |
| `timeout_ms` | integer | Maximum time to wait for execution |

## Command Types & Parameters

### screenshot

```json
{
  "id": "cmd_001",
  "type": "screenshot",
  "params": {
    "path": "/sdcard/VisualAgent/results/screenshot_001.png"
  }
}
```

| Param | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `path` | string | No | auto-generated | Where to save the PNG file |

### tap

```json
{
  "id": "cmd_002",
  "type": "tap",
  "params": {
    "x": 500,
    "y": 800
  }
}
```

| Param | Type | Required | Description |
|-------|------|----------|-------------|
| `x` | integer | Yes | Horizontal pixel coordinate |
| `y` | integer | Yes | Vertical pixel coordinate |

### swipe

```json
{
  "id": "cmd_003",
  "type": "swipe",
  "params": {
    "x1": 500,
    "y1": 1200,
    "x2": 500,
    "y2": 400,
    "duration_ms": 300
  }
}
```

| Param | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `x1` | integer | Yes | — | Start X coordinate |
| `y1` | integer | Yes | — | Start Y coordinate |
| `x2` | integer | Yes | — | End X coordinate |
| `y2` | integer | Yes | — | End Y coordinate |
| `duration_ms` | integer | No | 300 | Gesture duration in milliseconds |

### input

```json
{
  "id": "cmd_004",
  "type": "input",
  "params": {
    "text": "Hello, World!"
  }
}
```

| Param | Type | Required | Description |
|-------|------|----------|-------------|
| `text` | string | Yes | Text to type into the focused input field |

### start_app

```json
{
  "id": "cmd_005",
  "type": "start_app",
  "params": {
    "package": "com.google.android.calculator"
  }
}
```

| Param | Type | Required | Description |
|-------|------|----------|-------------|
| `package` | string | Yes | Android package name to launch |

## Result File Format

Result files are written to the results directory with the naming convention `{command_id}_result.json`.

### Success Result

```json
{
  "id": "cmd_001",
  "type": "screenshot",
  "status": "success",
  "timestamp": "2026-06-25T17:16:00Z",
  "result": {
    "path": "/sdcard/VisualAgent/results/screenshot_001.png",
    "size_bytes": 340658
  },
  "error": null
}
```

### Error Result

```json
{
  "id": "cmd_002",
  "type": "tap",
  "status": "error",
  "timestamp": "2026-06-25T17:16:00Z",
  "result": null,
  "error": {
    "code": "SCREEN_LOCKED",
    "message": "Cannot tap: screen is locked",
    "details": "Screen wakefulness=Asleep, locked=true"
  }
}
```

### Result Fields

| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Command ID (matches the command file) |
| `type` | string | Command type |
| `status` | string | `"success"` or `"error"` |
| `timestamp` | string (ISO 8601) | When the result was produced |
| `result` | object or null | Type-specific result data |
| `error` | object or null | Error details if status is `"error"` |

### Error Object

| Field | Type | Description |
|-------|------|-------------|
| `code` | string | Machine-readable error code |
| `message` | string | Human-readable error message |
| `details` | string | Additional context (optional) |

## Common Error Codes

| Code | Description |
|------|-------------|
| `SCREEN_LOCKED` | Screen is off or locked, UI operations unavailable |
| `INVALID_COMMAND` | Unknown command type or malformed JSON |
| `MISSING_PARAM` | Required parameter not provided |
| `PERMISSION_DENIED` | Insufficient permissions for operation |
| `TIMEOUT` | Command exceeded timeout limit |
| `APP_NOT_FOUND` | Package name not found for start_app |
| `COORDINATE_OUT_OF_BOUNDS` | Tap/swipe coordinates exceed screen dimensions |

## Directory Structure

```
/sdcard/VisualAgent/
├── commands/
│   ├── cmd_001.json
│   ├── cmd_002.json
│   └── archive/
│       ├── cmd_001.json (moved after processing)
│       └── cmd_002.json
└── results/
    ├── cmd_001_result.json
    └── cmd_002_result.json
```