visual-agent-service-patterns.md 18.8 KB

VisualAgent Service Patterns

Background

This reference captures the full VisualAgent service architecture and common patterns discovered while building and debugging Android UI automation backends. It complements the main android-ui-automation SKILL.md with session-specific details and condensed knowledge about the VisualAgent pattern.

VisualAgent Directory Structure

/sdcard/VisualAgent/
├── commands/          # Drop .json command files here
│   └── cmd_001.json
├── results/           # Agent writes results here
│   └── cmd_001_result.json
├── processed/         # Archive processed commands (preferred over deletion)
│   └── cmd_001.json
├── .processed_files   # Newline-separated list of processed filenames
└── visualagent.log    # Service log with timestamped entries

The .processed_files Tracking File

This is a simple text file, one filename per line:

test_screenshot.json
test_tap.json

Note: The .processed_files list and the processed/ directory are separate mechanisms. The processed/ directory contains the actual moved command files, while .processed_files is a lightweight index. Some implementations use only one, some use both. When checking for unprocessed commands, compare against whichever tracking mechanism exists.

Command File Format Variations

Standard Format (type-based)

{
  "type": "screenshot",
  "path": "/storage/emulated/0/VisualAgent/screenshot_test.png"
}

Alternative Format (command-based)

Some systems use command instead of type:

{
  "command": "tap",
  "x": 500,
  "y": 1000
}

Robust Parsing

When reading command files, check both fields:

# Python one-liner approach
python3 -c "
import json, sys
cmd = json.load(sys.stdin)
cmd_type = cmd.get('type') or cmd.get('command')
print(cmd_type)
"

Result File Format Variations

Standard Result (with result wrapper)

{
  "command": {"type": "screenshot", "path": "..."},
  "status": "success",
  "result": {"path": "...", "size_bytes": 583926},
  "timestamp": "2026-06-25T19:45:00Z"
}

Simple Result (direct output)

{
  "status": "success",
  "path": "/sdcard/VisualAgent/screenshot_test.png",
  "message": "Screenshot saved successfully"
}

Processing Workflow (Shell-Based, Cron-Safe)

Since execute_code is blocked in cron jobs, use this shell-based workflow:

Step 1: Check Directory and List Files

# Single command to check everything
su -c 'ls -la /sdcard/VisualAgent/commands/ 2>/dev/null || echo "DIR_NOT_FOUND"'

Step 2: Read Each Command File

# Read with su
su -c 'cat /sdcard/VisualAgent/commands/test_screenshot.json'

Step 3: Execute via MCP Tools

Prefer MCP tools over shell commands for UI operations:

Command Type MCP Tool Shell Fallback
screenshot mcp_android_phone_screenshot su -c 'screencap -p /path.png'
tap mcp_android_phone_tap su -c 'input tap x y'
swipe mcp_android_phone_swipe su -c 'input swipe x1 y1 x2 y2 duration'
input mcp_android_phone_input_text su -c 'input text "text"'
start_app mcp_android_phone_open_app su -c 'monkey -p pkg -c android.intent.category.LAUNCHER 1'

Step 4: Write Result

# Write result with su
su -c 'echo "{\"status\":\"success\",\"path\":\"/sdcard/...\"}" > /sdcard/VisualAgent/results/result.json'

Step 5: Track Processed Files

# Append to processed files tracking
su -c 'echo "cmd_001.json" >> /sdcard/VisualAgent/.processed_files'

Checking for Unprocessed Commands

When both processed/ directory and .processed_files tracking exist, use this robust check:

# List all command files
su -c 'ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null'

# List processed files (from directory)
su -c 'ls -1 /sdcard/VisualAgent/processed/ 2>/dev/null'

# List tracked files (from tracking file)
su -c 'cat /sdcard/VisualAgent/.processed_files 2>/dev/null'

A file is unprocessed if it exists in commands/ but NOT in processed/ AND NOT in .processed_files. In practice, checking one mechanism is usually sufficient — they should be kept in sync by the service.

The Three-Way Check (When Tracking Is Out of Sync)

Some VisualAgent implementations have inconsistent state between commands/, processed/, and .processed_files. Use this robust check:

# Get all three sources of truth in one command
su -c '
  echo "=== COMMANDS ===";
  ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null | xargs -n1 basename;
  echo "=== PROCESSED ===";
  ls -1 /sdcard/VisualAgent/processed/ 2>/dev/null;
  echo "=== TRACKED ===";
  cat /sdcard/VisualAgent/.processed_files 2>/dev/null
'

A file is truly new only if ALL of these are true:

  1. It exists in commands/
  2. It is NOT in processed/
  3. It is NOT in .processed_files

A file is already handled if ANY of these are true:

  1. It does NOT exist in commands/ (was moved to processed/ or deleted)
  2. It IS in processed/ (regardless of .processed_files state)
  3. It IS in .processed_files (regardless of commands/ state)

Common inconsistency: Files are moved to processed/ but .processed_files is not updated, or files are listed in .processed_files but still exist in commands/ (move failed). Always cross-reference all three sources.

Command File Format Variations

Minimal Format (type + params inline)

{"type": "tap", "x": 500, "y": 1000}

Wrapped Format (type in params object)

{"type": "screenshot", "path": "/storage/emulated/0/VisualAgent/screenshot_test.png"}

Start App Format

{"type": "start_app", "package": "com.example.app"}

Input Format

{"type": "input", "text": "Hello World"}

Swipe Format

{"type": "swipe", "x1": 500, "y1": 1000, "x2": 500, "y2": 500, "duration_ms": 300}

Common Pitfalls and Solutions

1. Permission Denied on /sdcard/

Symptom: ls: cannot access '/sdcard/VisualAgent/': Permission denied

Solution: Always use su -c for file operations on /sdcard/:

su -c 'ls -la /sdcard/VisualAgent/'

2. Empty Commands Directory

Symptom: No new commands to process

Solution: In cron jobs, respond with [SILENT] — do not poll repeatedly. A single check is sufficient.

3. File Tools Fail on /sdcard/

Symptom: read_file or write_file tools fail with "Permission denied"

Solution: Use terminal with su -c instead of Hermes file tools for /sdcard/ paths.

4. JSON Parsing in Shell

Symptom: grep/cut parsing breaks on nested quotes or unicode

Solution: Use python3 -c or jq for reliable JSON parsing:

su -c 'python3 -c "import json; print(json.load(open(\"/sdcard/...\")).get(\"type\"))"'

5. Glob Expansion with su

Symptom: for f in /sdcard/.../*.json fails because glob expands before su

Solution: Use ls inside su:

for f in $(su -c 'ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null'); do
    echo "Processing: $f"
done

Service Log Format

VisualAgent services often maintain a log file:

/sdcard/VisualAgent/visualagent.log

Example log entries (from real VisualAgent service):

[2026-06-25 19:45:06] VisualAgent service started
[2026-06-25 19:45:06] Watching: /storage/emulated/0/VisualAgent/commands
[2026-06-25 19:46:18] Processing: test_screenshot.json
[2026-06-25 19:46:19] Result: success - Screenshot saved to /storage/emulated/0/VisualAgent/screenshot_test.png
[2026-06-25 19:46:19] Processing: test_tap.json
[2026-06-25 19:46:20] Result: success - Tapped at coordinates (500, 1000)
[2026-06-25 19:46:20] VisualAgent service completed

Note: The log format uses [YYYY-MM-DD HH:MM:SS] brackets with space-separated fields, not structured [LEVEL] prefixes. The service logs both "started" and "completed" lifecycle events, and uses "Watching: " to indicate the monitored directory.

Integration with Hermes Cron Jobs

When running as a Hermes cron job:

  1. Check once, not repeatedly — Use a single terminal call to list files
  2. Process all found files — Iterate through results in one batch
  3. Report or silence — If files were processed, report results. If empty, respond [SILENT]
  4. Never use execute_code — Use terminal + shell or direct MCP tools

6. All Commands Already Processed

Symptom: commands/ directory is empty, but processed/ has files and .processed_files has entries. The service has already handled everything.

Solution: This is normal. Report the current state (e.g., "0 pending commands, 6 processed") and respond [SILENT] in cron jobs if there's nothing new to do. Do NOT attempt to re-process files from processed/.

7. Service Log Shows Previous Runs

Symptom: The visualagent.log shows multiple "service started/completed" cycles with the same commands.

Solution: This indicates the service is running as a cron job and polling correctly. Each run checks for new commands, processes them if found, and exits. The log timestamps reveal the polling frequency. If you see the same commands being processed multiple times, check that .processed_files or processed/ tracking is working correctly.

8. Steady State: Service Running but No New Commands

Symptom: The commands/ directory has no .json files, processed/ has many historical files, .processed_files has entries, and results/ has many result files. This is a healthy, idle service — not an error condition.

Real-world example:

commands/:   myfile.dat, test.txt        (no .json files)
processed/:  cmd_test.json, cron_test.json, demo_cmd.json, ... (15 files)
results/:    cmd_test_result.json, cron_test_result.json, ... (20 files)
.processed_files: test_screenshot.json, test_tap.json, ...

Incorrect response: Performing a health check screenshot "just to be sure", then writing a verbose status report listing all historical files, directory sizes, and a "service is healthy" conclusion. This wastes tool calls and produces unnecessary delivery noise.

Correct response: After confirming the three-way check (commands/ has no .json, processed/ has history, .processed_files has entries), respond [SILENT] immediately. The service is clearly operational — it has processed commands before and produced results. There is simply no new work.

Decision tree:

commands/ has .json files? 
  → YES → Process them → Report results
  → NO → Check processed/ and .processed_files:
    → Has history (files in processed/, entries in .processed_files)
      → [SILENT] (steady state, service is healthy and idle)
    → NO history (fresh install, no processed files)
      → Optional: one-time health check screenshot → [SILENT]
    → Inconsistent state (files in commands/ but also in processed/)
      → Investigate → Report findings (not [SILENT])

Key insight: The presence of historical data in processed/ and results/ is itself evidence of a working service. A health check screenshot adds no new information in this case. Reserve health checks for when there are ERROR indicators (stale PID files, permission errors, empty directories on a service that should have history).

Example: Complete Cron Job Response

## VisualAgent Service Status Report

### Commands Processed
1. **test_screenshot.json**
   - Type: screenshot
   - Status: ✅ Success
   - Result: /storage/emulated/0/VisualAgent/screenshot_test.png (583,926 bytes)

2. **test_tap.json**
   - Type: tap
   - Status: ✅ Success
   - Result: Tapped at (500, 1000)

### Directory Status
- commands/: 0 pending files
- results/: 2 result files
- All commands processed successfully

VisualAgent as a Cron Job (No User Present)

When VisualAgent runs as a Hermes scheduled cron job with no user present, the workflow differs from interactive sessions. The cron job is a single-shot execution that checks for commands, processes any found, and exits. Key constraints and patterns:

Cron Job Constraints

  • Cannot use execute_code — Python scripts with subprocess calls are blocked for safety. Use terminal with shell commands or direct MCP tools instead.
  • Cannot ask clarifying questions — Make reasonable decisions autonomously. If a command is ambiguous, skip it and log an error result.
  • Must respond [SILENT] when idle — If the command directory is empty or all commands are already processed, respond with exactly [SILENT] and nothing else. This suppresses unnecessary delivery.
  • Single-shot, not a daemon — The cron job runs once per schedule, checks once, processes, and exits. Do NOT implement polling loops or sleep/retry logic inside the job.

Cron Job Workflow

1. Check for pre-existing VisualAgent infrastructure:
   a. Look for scripts: ls -la ~/ | grep visual_agent
   b. Check for README: cat ~/VisualAgent_README.md
   c. Check directories: su -c 'ls -la /sdcard/VisualAgent/'
2. If infrastructure exists, decide: use existing executor, start daemon, or use MCP tools
3. Check /sdcard/VisualAgent/commands/ for ALL files (not just .json — may be .cmd, .txt, or no extension)
4. Cross-reference with .processed_files and processed/ directory
5. If no new commands → respond [SILENT] and exit
6. If new commands found:
   a. Read each command file with su -c 'cat ...'
   b. Parse JSON (use python3 -c or jq, not grep/cut)
   c. Execute via MCP tools (preferred) or su -c 'input ...'
   d. Write result to results/ with su -c 'echo ... > ...'
   e. Move command to processed/ with su -c 'mv ...'
   f. Append filename to .processed_files with su -c 'echo ... >> ...'
7. Report processed commands in final response

Cron Job Anti-Patterns to Avoid

Anti-Pattern Why It Fails Correct Approach
Polling loop with sleep Hits max iteration limit, wastes tool calls Single check, process all found, exit
Using execute_code for JSON parsing Blocked in cron mode, produces error Use terminal with python3 -c or jq
Re-processing files from processed/ Already handled, may cause duplicate actions Check .processed_files and processed/ first
Reporting "0 commands found" as content Generates unnecessary delivery noise Use [SILENT] instead
Using rm to delete processed commands May trigger approval blocks, loses history Use mv to processed/ directory
Using Hermes read_file/write_file on /sdcard/ Permission denied in Termux context Use terminal with su -c
Ignoring pre-existing infrastructure Reimplements what already exists Check for existing scripts first, then decide
Trusting daemon PID file without verification PID file may be stale, daemon actually dead Verify with ps -p <pid> before relying on daemon
Using shell grep/cut to parse JSON Breaks on nested quotes, unicode, whitespace variations Always use python3 -c or jq for JSON parsing
Re-processing same command repeatedly File not archived after execution, found again next run Always mv to processed/ after execution; see references/command-file-extension-flexibility.md
Using for f in /sdcard/.../*.json with su Glob expands in parent shell before su runs Use for f in $(su -c 'ls -1 ...') instead

Example: Minimal Cron Job Implementation

#!/system/bin/sh
# visualagent-cron.sh — Minimal VisualAgent cron job

COMMANDS_DIR="/sdcard/VisualAgent/commands"
RESULTS_DIR="/sdcard/VisualAgent/results"
PROCESSED_DIR="/sdcard/VisualAgent/processed"
TRACKING_FILE="/sdcard/VisualAgent/.processed_files"

# Ensure directories exist
su -c "mkdir -p $COMMANDS_DIR $RESULTS_DIR $PROCESSED_DIR"

# List ALL files (not just .json — may be .cmd, .txt, etc.)
ALL_FILES=$(su -c "ls -1 $COMMANDS_DIR/ 2>/dev/null")

if [ -z "$ALL_FILES" ]; then
    # No commands — silent exit
    exit 0
fi

# Filter to valid command files (JSON with type/command field)
COMMANDS=""
for f in $ALL_FILES; do
    # Skip directories
    if su -c "test -d $COMMANDS_DIR/$f"; then
        continue
    fi
    # Validate JSON content
    cmd_type=$(su -c "cat $COMMANDS_DIR/$f" | python3 -c "
import json, sys
try:
    d = json.load(sys.stdin)
    print(d.get('type') or d.get('command', ''))
except:
    print('')
" 2>/dev/null)
    if [ -n "$cmd_type" ]; then
        COMMANDS="$COMMANDS $COMMANDS_DIR/$f"
    fi
done

if [ -z "$COMMANDS" ]; then
    # No valid commands — silent exit
    exit 0
fi

# Process each command
for cmd_file in $COMMANDS; do
    basename=$(basename "$cmd_file")

    # Check if already processed
    if su -c "grep -q '^$basename$' $TRACKING_FILE 2>/dev/null"; then
        continue
    fi

    # Read and parse command
    cmd_json=$(su -c "cat $cmd_file")
    cmd_type=$(echo "$cmd_json" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('type') or d.get('command',''))")

    # Execute (shell example — prefer MCP tools in practice)
    case "$cmd_type" in
        screenshot)
            path=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('path','/sdcard/VisualAgent/screenshot.png'))")
            su -c "screencap -p $path"
            result="{\"status\":\"success\",\"path\":\"$path\"}"
            ;;
        tap)
            x=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('x',0))")
            y=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('y',0))")
            su -c "input tap $x $y"
            result="{\"status\":\"success\",\"x\":$x,\"y\":$y}"
            ;;
        *)
            result="{\"status\":\"error\",\"error\":\"Unknown command type: $cmd_type\"}"
            ;;
    esac

    # Write result
    result_file="$RESULTS_DIR/${basename%.json}_result.json"
    su -c "echo '$result' > $result_file"

    # Move to processed and track
    su -c "mv $cmd_file $PROCESSED_DIR/"
    su -c "echo $basename >> $TRACKING_FILE"
done

Note: This shell script is for illustration. In practice, a Hermes cron job uses the terminal tool to run shell snippets and MCP tools for UI operations, not a standalone shell script.

  • cron-execute-code-blocked.md — Why execute_code is blocked and workaround patterns
  • termux-sdcard-permission-patterns.md — Detailed su-wrapping patterns for file operations
  • cron-polling-anti-patterns.md — Correct polling behavior in cron jobs
  • sdcard-path-discovery.md — Discovering correct external storage paths
  • pre-existing-visualagent-infrastructure.md — Discovery checklist for existing VisualAgent installations
  • visual-agent-daemon-lifecycle.md — Daemon health check and restart procedures
  • command-file-extension-flexibility.md — Handling non-.json command file extensions and preventing infinite re-processing loops