# VisualAgent Service Patterns ## Background This reference captures the full VisualAgent service architecture and common patterns discovered while building and debugging Android UI automation backends. It complements the main `android-ui-automation` SKILL.md with session-specific details and condensed knowledge about the VisualAgent pattern. ## VisualAgent Directory Structure ``` /sdcard/VisualAgent/ ├── commands/ # Drop .json command files here │ └── cmd_001.json ├── results/ # Agent writes results here │ └── cmd_001_result.json ├── processed/ # Archive processed commands (preferred over deletion) │ └── cmd_001.json ├── .processed_files # Newline-separated list of processed filenames └── visualagent.log # Service log with timestamped entries ``` ### The `.processed_files` Tracking File This is a simple text file, one filename per line: ``` test_screenshot.json test_tap.json ``` **Note**: The `.processed_files` list and the `processed/` directory are separate mechanisms. The `processed/` directory contains the actual moved command files, while `.processed_files` is a lightweight index. Some implementations use only one, some use both. When checking for unprocessed commands, compare against whichever tracking mechanism exists. ## Command File Format Variations ### Standard Format (type-based) ```json { "type": "screenshot", "path": "/storage/emulated/0/VisualAgent/screenshot_test.png" } ``` ### Alternative Format (command-based) Some systems use `command` instead of `type`: ```json { "command": "tap", "x": 500, "y": 1000 } ``` ### Robust Parsing When reading command files, check both fields: ```bash # Python one-liner approach python3 -c " import json, sys cmd = json.load(sys.stdin) cmd_type = cmd.get('type') or cmd.get('command') print(cmd_type) " ``` ## Result File Format Variations ### Standard Result (with result wrapper) ```json { "command": {"type": "screenshot", "path": "..."}, "status": "success", "result": {"path": "...", "size_bytes": 583926}, "timestamp": "2026-06-25T19:45:00Z" } ``` ### Simple Result (direct output) ```json { "status": "success", "path": "/sdcard/VisualAgent/screenshot_test.png", "message": "Screenshot saved successfully" } ``` ## Processing Workflow (Shell-Based, Cron-Safe) Since `execute_code` is blocked in cron jobs, use this shell-based workflow: ### Step 1: Check Directory and List Files ```bash # Single command to check everything su -c 'ls -la /sdcard/VisualAgent/commands/ 2>/dev/null || echo "DIR_NOT_FOUND"' ``` ### Step 2: Read Each Command File ```bash # Read with su su -c 'cat /sdcard/VisualAgent/commands/test_screenshot.json' ``` ### Step 3: Execute via MCP Tools Prefer MCP tools over shell commands for UI operations: | Command Type | MCP Tool | Shell Fallback | |-------------|----------|----------------| | screenshot | `mcp_android_phone_screenshot` | `su -c 'screencap -p /path.png'` | | tap | `mcp_android_phone_tap` | `su -c 'input tap x y'` | | swipe | `mcp_android_phone_swipe` | `su -c 'input swipe x1 y1 x2 y2 duration'` | | input | `mcp_android_phone_input_text` | `su -c 'input text "text"'` | | start_app | `mcp_android_phone_open_app` | `su -c 'monkey -p pkg -c android.intent.category.LAUNCHER 1'` | ### Step 4: Write Result ```bash # Write result with su su -c 'echo "{\"status\":\"success\",\"path\":\"/sdcard/...\"}" > /sdcard/VisualAgent/results/result.json' ``` ### Step 5: Track Processed Files ```bash # Append to processed files tracking su -c 'echo "cmd_001.json" >> /sdcard/VisualAgent/.processed_files' ``` ## Checking for Unprocessed Commands When both `processed/` directory and `.processed_files` tracking exist, use this robust check: ```bash # List all command files su -c 'ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null' # List processed files (from directory) su -c 'ls -1 /sdcard/VisualAgent/processed/ 2>/dev/null' # List tracked files (from tracking file) su -c 'cat /sdcard/VisualAgent/.processed_files 2>/dev/null' ``` A file is unprocessed if it exists in `commands/` but NOT in `processed/` AND NOT in `.processed_files`. In practice, checking one mechanism is usually sufficient — they should be kept in sync by the service. ### The Three-Way Check (When Tracking Is Out of Sync) Some VisualAgent implementations have inconsistent state between `commands/`, `processed/`, and `.processed_files`. Use this robust check: ```bash # Get all three sources of truth in one command su -c ' echo "=== COMMANDS ==="; ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null | xargs -n1 basename; echo "=== PROCESSED ==="; ls -1 /sdcard/VisualAgent/processed/ 2>/dev/null; echo "=== TRACKED ==="; cat /sdcard/VisualAgent/.processed_files 2>/dev/null ' ``` A file is **truly new** only if ALL of these are true: 1. It exists in `commands/` 2. It is NOT in `processed/` 3. It is NOT in `.processed_files` A file is **already handled** if ANY of these are true: 1. It does NOT exist in `commands/` (was moved to `processed/` or deleted) 2. It IS in `processed/` (regardless of `.processed_files` state) 3. It IS in `.processed_files` (regardless of `commands/` state) **Common inconsistency**: Files are moved to `processed/` but `.processed_files` is not updated, or files are listed in `.processed_files` but still exist in `commands/` (move failed). Always cross-reference all three sources. ## Command File Format Variations ### Minimal Format (type + params inline) ```json {"type": "tap", "x": 500, "y": 1000} ``` ### Wrapped Format (type in params object) ```json {"type": "screenshot", "path": "/storage/emulated/0/VisualAgent/screenshot_test.png"} ``` ### Start App Format ```json {"type": "start_app", "package": "com.example.app"} ``` ### Input Format ```json {"type": "input", "text": "Hello World"} ``` ### Swipe Format ```json {"type": "swipe", "x1": 500, "y1": 1000, "x2": 500, "y2": 500, "duration_ms": 300} ``` ## Common Pitfalls and Solutions ### 1. Permission Denied on /sdcard/ **Symptom**: `ls: cannot access '/sdcard/VisualAgent/': Permission denied` **Solution**: Always use `su -c` for file operations on `/sdcard/`: ```bash su -c 'ls -la /sdcard/VisualAgent/' ``` ### 2. Empty Commands Directory **Symptom**: No new commands to process **Solution**: In cron jobs, respond with `[SILENT]` — do not poll repeatedly. A single check is sufficient. ### 3. File Tools Fail on /sdcard/ **Symptom**: `read_file` or `write_file` tools fail with "Permission denied" **Solution**: Use `terminal` with `su -c` instead of Hermes file tools for `/sdcard/` paths. ### 4. JSON Parsing in Shell **Symptom**: `grep`/`cut` parsing breaks on nested quotes or unicode **Solution**: Use `python3 -c` or `jq` for reliable JSON parsing: ```bash su -c 'python3 -c "import json; print(json.load(open(\"/sdcard/...\")).get(\"type\"))"' ``` ### 5. Glob Expansion with su **Symptom**: `for f in /sdcard/.../*.json` fails because glob expands before su **Solution**: Use `ls` inside su: ```bash for f in $(su -c 'ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null'); do echo "Processing: $f" done ``` ## Service Log Format VisualAgent services often maintain a log file: ``` /sdcard/VisualAgent/visualagent.log ``` Example log entries (from real VisualAgent service): ``` [2026-06-25 19:45:06] VisualAgent service started [2026-06-25 19:45:06] Watching: /storage/emulated/0/VisualAgent/commands [2026-06-25 19:46:18] Processing: test_screenshot.json [2026-06-25 19:46:19] Result: success - Screenshot saved to /storage/emulated/0/VisualAgent/screenshot_test.png [2026-06-25 19:46:19] Processing: test_tap.json [2026-06-25 19:46:20] Result: success - Tapped at coordinates (500, 1000) [2026-06-25 19:46:20] VisualAgent service completed ``` **Note**: The log format uses `[YYYY-MM-DD HH:MM:SS]` brackets with space-separated fields, not structured `[LEVEL]` prefixes. The service logs both "started" and "completed" lifecycle events, and uses "Watching: " to indicate the monitored directory. ## Integration with Hermes Cron Jobs When running as a Hermes cron job: 1. **Check once, not repeatedly** — Use a single `terminal` call to list files 2. **Process all found files** — Iterate through results in one batch 3. **Report or silence** — If files were processed, report results. If empty, respond `[SILENT]` 4. **Never use execute_code** — Use terminal + shell or direct MCP tools ### 6. All Commands Already Processed **Symptom**: `commands/` directory is empty, but `processed/` has files and `.processed_files` has entries. The service has already handled everything. **Solution**: This is normal. Report the current state (e.g., "0 pending commands, 6 processed") and respond `[SILENT]` in cron jobs if there's nothing new to do. Do NOT attempt to re-process files from `processed/`. ### 7. Service Log Shows Previous Runs **Symptom**: The `visualagent.log` shows multiple "service started/completed" cycles with the same commands. **Solution**: This indicates the service is running as a cron job and polling correctly. Each run checks for new commands, processes them if found, and exits. The log timestamps reveal the polling frequency. If you see the same commands being processed multiple times, check that `.processed_files` or `processed/` tracking is working correctly. ### 8. Steady State: Service Running but No New Commands **Symptom**: The `commands/` directory has no `.json` files, `processed/` has many historical files, `.processed_files` has entries, and `results/` has many result files. This is a healthy, idle service — not an error condition. **Real-world example**: ``` commands/: myfile.dat, test.txt (no .json files) processed/: cmd_test.json, cron_test.json, demo_cmd.json, ... (15 files) results/: cmd_test_result.json, cron_test_result.json, ... (20 files) .processed_files: test_screenshot.json, test_tap.json, ... ``` **Incorrect response**: Performing a health check screenshot "just to be sure", then writing a verbose status report listing all historical files, directory sizes, and a "service is healthy" conclusion. This wastes tool calls and produces unnecessary delivery noise. **Correct response**: After confirming the three-way check (commands/ has no .json, processed/ has history, .processed_files has entries), respond `[SILENT]` immediately. The service is clearly operational — it has processed commands before and produced results. There is simply no new work. **Decision tree**: ``` commands/ has .json files? → YES → Process them → Report results → NO → Check processed/ and .processed_files: → Has history (files in processed/, entries in .processed_files) → [SILENT] (steady state, service is healthy and idle) → NO history (fresh install, no processed files) → Optional: one-time health check screenshot → [SILENT] → Inconsistent state (files in commands/ but also in processed/) → Investigate → Report findings (not [SILENT]) ``` **Key insight**: The presence of historical data in `processed/` and `results/` is itself evidence of a working service. A health check screenshot adds no new information in this case. Reserve health checks for when there are ERROR indicators (stale PID files, permission errors, empty directories on a service that should have history). ## Example: Complete Cron Job Response ``` ## VisualAgent Service Status Report ### Commands Processed 1. **test_screenshot.json** - Type: screenshot - Status: ✅ Success - Result: /storage/emulated/0/VisualAgent/screenshot_test.png (583,926 bytes) 2. **test_tap.json** - Type: tap - Status: ✅ Success - Result: Tapped at (500, 1000) ### Directory Status - commands/: 0 pending files - results/: 2 result files - All commands processed successfully ``` ## VisualAgent as a Cron Job (No User Present) When VisualAgent runs as a **Hermes scheduled cron job** with no user present, the workflow differs from interactive sessions. The cron job is a single-shot execution that checks for commands, processes any found, and exits. Key constraints and patterns: ### Cron Job Constraints - **Cannot use `execute_code`** — Python scripts with subprocess calls are blocked for safety. Use `terminal` with shell commands or direct MCP tools instead. - **Cannot ask clarifying questions** — Make reasonable decisions autonomously. If a command is ambiguous, skip it and log an error result. - **Must respond `[SILENT]` when idle** — If the command directory is empty or all commands are already processed, respond with exactly `[SILENT]` and nothing else. This suppresses unnecessary delivery. - **Single-shot, not a daemon** — The cron job runs once per schedule, checks once, processes, and exits. Do NOT implement polling loops or sleep/retry logic inside the job. ### Cron Job Workflow ``` 1. Check for pre-existing VisualAgent infrastructure: a. Look for scripts: ls -la ~/ | grep visual_agent b. Check for README: cat ~/VisualAgent_README.md c. Check directories: su -c 'ls -la /sdcard/VisualAgent/' 2. If infrastructure exists, decide: use existing executor, start daemon, or use MCP tools 3. Check /sdcard/VisualAgent/commands/ for ALL files (not just .json — may be .cmd, .txt, or no extension) 4. Cross-reference with .processed_files and processed/ directory 5. If no new commands → respond [SILENT] and exit 6. If new commands found: a. Read each command file with su -c 'cat ...' b. Parse JSON (use python3 -c or jq, not grep/cut) c. Execute via MCP tools (preferred) or su -c 'input ...' d. Write result to results/ with su -c 'echo ... > ...' e. Move command to processed/ with su -c 'mv ...' f. Append filename to .processed_files with su -c 'echo ... >> ...' 7. Report processed commands in final response ``` ### Cron Job Anti-Patterns to Avoid | Anti-Pattern | Why It Fails | Correct Approach | |-------------|--------------|----------------| | Polling loop with `sleep` | Hits max iteration limit, wastes tool calls | Single check, process all found, exit | | Using `execute_code` for JSON parsing | Blocked in cron mode, produces error | Use `terminal` with `python3 -c` or `jq` | | Re-processing files from `processed/` | Already handled, may cause duplicate actions | Check `.processed_files` and `processed/` first | | Reporting "0 commands found" as content | Generates unnecessary delivery noise | Use `[SILENT]` instead | | Using `rm` to delete processed commands | May trigger approval blocks, loses history | Use `mv` to `processed/` directory | | Using Hermes `read_file`/`write_file` on `/sdcard/` | Permission denied in Termux context | Use `terminal` with `su -c` | | Ignoring pre-existing infrastructure | Reimplements what already exists | Check for existing scripts first, then decide | | Trusting daemon PID file without verification | PID file may be stale, daemon actually dead | Verify with `ps -p ` before relying on daemon | | Using shell `grep`/`cut` to parse JSON | Breaks on nested quotes, unicode, whitespace variations | Always use `python3 -c` or `jq` for JSON parsing | | **Re-processing same command repeatedly** | **File not archived after execution, found again next run** | **Always `mv` to `processed/` after execution; see `references/command-file-extension-flexibility.md`** | | Using `for f in /sdcard/.../*.json` with `su` | Glob expands in parent shell before `su` runs | Use `for f in $(su -c 'ls -1 ...')` instead | ### Example: Minimal Cron Job Implementation ```bash #!/system/bin/sh # visualagent-cron.sh — Minimal VisualAgent cron job COMMANDS_DIR="/sdcard/VisualAgent/commands" RESULTS_DIR="/sdcard/VisualAgent/results" PROCESSED_DIR="/sdcard/VisualAgent/processed" TRACKING_FILE="/sdcard/VisualAgent/.processed_files" # Ensure directories exist su -c "mkdir -p $COMMANDS_DIR $RESULTS_DIR $PROCESSED_DIR" # List ALL files (not just .json — may be .cmd, .txt, etc.) ALL_FILES=$(su -c "ls -1 $COMMANDS_DIR/ 2>/dev/null") if [ -z "$ALL_FILES" ]; then # No commands — silent exit exit 0 fi # Filter to valid command files (JSON with type/command field) COMMANDS="" for f in $ALL_FILES; do # Skip directories if su -c "test -d $COMMANDS_DIR/$f"; then continue fi # Validate JSON content cmd_type=$(su -c "cat $COMMANDS_DIR/$f" | python3 -c " import json, sys try: d = json.load(sys.stdin) print(d.get('type') or d.get('command', '')) except: print('') " 2>/dev/null) if [ -n "$cmd_type" ]; then COMMANDS="$COMMANDS $COMMANDS_DIR/$f" fi done if [ -z "$COMMANDS" ]; then # No valid commands — silent exit exit 0 fi # Process each command for cmd_file in $COMMANDS; do basename=$(basename "$cmd_file") # Check if already processed if su -c "grep -q '^$basename$' $TRACKING_FILE 2>/dev/null"; then continue fi # Read and parse command cmd_json=$(su -c "cat $cmd_file") cmd_type=$(echo "$cmd_json" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('type') or d.get('command',''))") # Execute (shell example — prefer MCP tools in practice) case "$cmd_type" in screenshot) path=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('path','/sdcard/VisualAgent/screenshot.png'))") su -c "screencap -p $path" result="{\"status\":\"success\",\"path\":\"$path\"}" ;; tap) x=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('x',0))") y=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('y',0))") su -c "input tap $x $y" result="{\"status\":\"success\",\"x\":$x,\"y\":$y}" ;; *) result="{\"status\":\"error\",\"error\":\"Unknown command type: $cmd_type\"}" ;; esac # Write result result_file="$RESULTS_DIR/${basename%.json}_result.json" su -c "echo '$result' > $result_file" # Move to processed and track su -c "mv $cmd_file $PROCESSED_DIR/" su -c "echo $basename >> $TRACKING_FILE" done ``` **Note**: This shell script is for illustration. In practice, a Hermes cron job uses the `terminal` tool to run shell snippets and MCP tools for UI operations, not a standalone shell script. ## Related References - `cron-execute-code-blocked.md` — Why execute_code is blocked and workaround patterns - `termux-sdcard-permission-patterns.md` — Detailed su-wrapping patterns for file operations - `cron-polling-anti-patterns.md` — Correct polling behavior in cron jobs - `sdcard-path-discovery.md` — Discovering correct external storage paths - `pre-existing-visualagent-infrastructure.md` — Discovery checklist for existing VisualAgent installations - `visual-agent-daemon-lifecycle.md` — Daemon health check and restart procedures - `command-file-extension-flexibility.md` — Handling non-`.json` command file extensions and preventing infinite re-processing loops