VisualAgent Service Patterns
Background
This reference captures the full VisualAgent service architecture and common patterns discovered while building and debugging Android UI automation backends. It complements the main android-ui-automation SKILL.md with session-specific details and condensed knowledge about the VisualAgent pattern.
VisualAgent Directory Structure
/sdcard/VisualAgent/
├── commands/ # Drop .json command files here
│ └── cmd_001.json
├── results/ # Agent writes results here
│ └── cmd_001_result.json
├── processed/ # Archive processed commands (preferred over deletion)
│ └── cmd_001.json
├── .processed_files # Newline-separated list of processed filenames
└── visualagent.log # Service log with timestamped entries
The .processed_files Tracking File
This is a simple text file, one filename per line:
test_screenshot.json
test_tap.json
Note: The .processed_files list and the processed/ directory are separate mechanisms. The processed/ directory contains the actual moved command files, while .processed_files is a lightweight index. Some implementations use only one, some use both. When checking for unprocessed commands, compare against whichever tracking mechanism exists.
Command File Format Variations
Standard Format (type-based)
{
"type": "screenshot",
"path": "/storage/emulated/0/VisualAgent/screenshot_test.png"
}
Alternative Format (command-based)
Some systems use command instead of type:
{
"command": "tap",
"x": 500,
"y": 1000
}
Robust Parsing
When reading command files, check both fields:
# Python one-liner approach
python3 -c "
import json, sys
cmd = json.load(sys.stdin)
cmd_type = cmd.get('type') or cmd.get('command')
print(cmd_type)
"
Result File Format Variations
Standard Result (with result wrapper)
{
"command": {"type": "screenshot", "path": "..."},
"status": "success",
"result": {"path": "...", "size_bytes": 583926},
"timestamp": "2026-06-25T19:45:00Z"
}
Simple Result (direct output)
{
"status": "success",
"path": "/sdcard/VisualAgent/screenshot_test.png",
"message": "Screenshot saved successfully"
}
Processing Workflow (Shell-Based, Cron-Safe)
Since execute_code is blocked in cron jobs, use this shell-based workflow:
Step 1: Check Directory and List Files
# Single command to check everything
su -c 'ls -la /sdcard/VisualAgent/commands/ 2>/dev/null || echo "DIR_NOT_FOUND"'
Step 2: Read Each Command File
# Read with su
su -c 'cat /sdcard/VisualAgent/commands/test_screenshot.json'
Step 3: Execute via MCP Tools
Prefer MCP tools over shell commands for UI operations:
| Command Type | MCP Tool | Shell Fallback |
|---|---|---|
| screenshot | mcp_android_phone_screenshot |
su -c 'screencap -p /path.png' |
| tap | mcp_android_phone_tap |
su -c 'input tap x y' |
| swipe | mcp_android_phone_swipe |
su -c 'input swipe x1 y1 x2 y2 duration' |
| input | mcp_android_phone_input_text |
su -c 'input text "text"' |
| start_app | mcp_android_phone_open_app |
su -c 'monkey -p pkg -c android.intent.category.LAUNCHER 1' |
Step 4: Write Result
# Write result with su
su -c 'echo "{\"status\":\"success\",\"path\":\"/sdcard/...\"}" > /sdcard/VisualAgent/results/result.json'
Step 5: Track Processed Files
# Append to processed files tracking
su -c 'echo "cmd_001.json" >> /sdcard/VisualAgent/.processed_files'
Checking for Unprocessed Commands
When both processed/ directory and .processed_files tracking exist, use this robust check:
# List all command files
su -c 'ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null'
# List processed files (from directory)
su -c 'ls -1 /sdcard/VisualAgent/processed/ 2>/dev/null'
# List tracked files (from tracking file)
su -c 'cat /sdcard/VisualAgent/.processed_files 2>/dev/null'
A file is unprocessed if it exists in commands/ but NOT in processed/ AND NOT in .processed_files. In practice, checking one mechanism is usually sufficient — they should be kept in sync by the service.
The Three-Way Check (When Tracking Is Out of Sync)
Some VisualAgent implementations have inconsistent state between commands/, processed/, and .processed_files. Use this robust check:
# Get all three sources of truth in one command
su -c '
echo "=== COMMANDS ===";
ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null | xargs -n1 basename;
echo "=== PROCESSED ===";
ls -1 /sdcard/VisualAgent/processed/ 2>/dev/null;
echo "=== TRACKED ===";
cat /sdcard/VisualAgent/.processed_files 2>/dev/null
'
A file is truly new only if ALL of these are true:
- It exists in
commands/ - It is NOT in
processed/ - It is NOT in
.processed_files
A file is already handled if ANY of these are true:
- It does NOT exist in
commands/(was moved toprocessed/or deleted) - It IS in
processed/(regardless of.processed_filesstate) - It IS in
.processed_files(regardless ofcommands/state)
Common inconsistency: Files are moved to processed/ but .processed_files is not updated, or files are listed in .processed_files but still exist in commands/ (move failed). Always cross-reference all three sources.
Command File Format Variations
Minimal Format (type + params inline)
{"type": "tap", "x": 500, "y": 1000}
Wrapped Format (type in params object)
{"type": "screenshot", "path": "/storage/emulated/0/VisualAgent/screenshot_test.png"}
Start App Format
{"type": "start_app", "package": "com.example.app"}
Input Format
{"type": "input", "text": "Hello World"}
Swipe Format
{"type": "swipe", "x1": 500, "y1": 1000, "x2": 500, "y2": 500, "duration_ms": 300}
Common Pitfalls and Solutions
1. Permission Denied on /sdcard/
Symptom: ls: cannot access '/sdcard/VisualAgent/': Permission denied
Solution: Always use su -c for file operations on /sdcard/:
su -c 'ls -la /sdcard/VisualAgent/'
2. Empty Commands Directory
Symptom: No new commands to process
Solution: In cron jobs, respond with [SILENT] — do not poll repeatedly. A single check is sufficient.
3. File Tools Fail on /sdcard/
Symptom: read_file or write_file tools fail with "Permission denied"
Solution: Use terminal with su -c instead of Hermes file tools for /sdcard/ paths.
4. JSON Parsing in Shell
Symptom: grep/cut parsing breaks on nested quotes or unicode
Solution: Use python3 -c or jq for reliable JSON parsing:
su -c 'python3 -c "import json; print(json.load(open(\"/sdcard/...\")).get(\"type\"))"'
5. Glob Expansion with su
Symptom: for f in /sdcard/.../*.json fails because glob expands before su
Solution: Use ls inside su:
for f in $(su -c 'ls -1 /sdcard/VisualAgent/commands/*.json 2>/dev/null'); do
echo "Processing: $f"
done
Service Log Format
VisualAgent services often maintain a log file:
/sdcard/VisualAgent/visualagent.log
Example log entries (from real VisualAgent service):
[2026-06-25 19:45:06] VisualAgent service started
[2026-06-25 19:45:06] Watching: /storage/emulated/0/VisualAgent/commands
[2026-06-25 19:46:18] Processing: test_screenshot.json
[2026-06-25 19:46:19] Result: success - Screenshot saved to /storage/emulated/0/VisualAgent/screenshot_test.png
[2026-06-25 19:46:19] Processing: test_tap.json
[2026-06-25 19:46:20] Result: success - Tapped at coordinates (500, 1000)
[2026-06-25 19:46:20] VisualAgent service completed
Note: The log format uses [YYYY-MM-DD HH:MM:SS] brackets with space-separated fields, not structured [LEVEL] prefixes. The service logs both "started" and "completed" lifecycle events, and uses "Watching: " to indicate the monitored directory.
Integration with Hermes Cron Jobs
When running as a Hermes cron job:
- Check once, not repeatedly — Use a single
terminalcall to list files - Process all found files — Iterate through results in one batch
- Report or silence — If files were processed, report results. If empty, respond
[SILENT] - Never use execute_code — Use terminal + shell or direct MCP tools
6. All Commands Already Processed
Symptom: commands/ directory is empty, but processed/ has files and .processed_files has entries. The service has already handled everything.
Solution: This is normal. Report the current state (e.g., "0 pending commands, 6 processed") and respond [SILENT] in cron jobs if there's nothing new to do. Do NOT attempt to re-process files from processed/.
7. Service Log Shows Previous Runs
Symptom: The visualagent.log shows multiple "service started/completed" cycles with the same commands.
Solution: This indicates the service is running as a cron job and polling correctly. Each run checks for new commands, processes them if found, and exits. The log timestamps reveal the polling frequency. If you see the same commands being processed multiple times, check that .processed_files or processed/ tracking is working correctly.
8. Steady State: Service Running but No New Commands
Symptom: The commands/ directory has no .json files, processed/ has many historical files, .processed_files has entries, and results/ has many result files. This is a healthy, idle service — not an error condition.
Real-world example:
commands/: myfile.dat, test.txt (no .json files)
processed/: cmd_test.json, cron_test.json, demo_cmd.json, ... (15 files)
results/: cmd_test_result.json, cron_test_result.json, ... (20 files)
.processed_files: test_screenshot.json, test_tap.json, ...
Incorrect response: Performing a health check screenshot "just to be sure", then writing a verbose status report listing all historical files, directory sizes, and a "service is healthy" conclusion. This wastes tool calls and produces unnecessary delivery noise.
Correct response: After confirming the three-way check (commands/ has no .json, processed/ has history, .processed_files has entries), respond [SILENT] immediately. The service is clearly operational — it has processed commands before and produced results. There is simply no new work.
Decision tree:
commands/ has .json files?
→ YES → Process them → Report results
→ NO → Check processed/ and .processed_files:
→ Has history (files in processed/, entries in .processed_files)
→ [SILENT] (steady state, service is healthy and idle)
→ NO history (fresh install, no processed files)
→ Optional: one-time health check screenshot → [SILENT]
→ Inconsistent state (files in commands/ but also in processed/)
→ Investigate → Report findings (not [SILENT])
Key insight: The presence of historical data in processed/ and results/ is itself evidence of a working service. A health check screenshot adds no new information in this case. Reserve health checks for when there are ERROR indicators (stale PID files, permission errors, empty directories on a service that should have history).
Example: Complete Cron Job Response
## VisualAgent Service Status Report
### Commands Processed
1. **test_screenshot.json**
- Type: screenshot
- Status: ✅ Success
- Result: /storage/emulated/0/VisualAgent/screenshot_test.png (583,926 bytes)
2. **test_tap.json**
- Type: tap
- Status: ✅ Success
- Result: Tapped at (500, 1000)
### Directory Status
- commands/: 0 pending files
- results/: 2 result files
- All commands processed successfully
VisualAgent as a Cron Job (No User Present)
When VisualAgent runs as a Hermes scheduled cron job with no user present, the workflow differs from interactive sessions. The cron job is a single-shot execution that checks for commands, processes any found, and exits. Key constraints and patterns:
Cron Job Constraints
- Cannot use
execute_code— Python scripts with subprocess calls are blocked for safety. Useterminalwith shell commands or direct MCP tools instead. - Cannot ask clarifying questions — Make reasonable decisions autonomously. If a command is ambiguous, skip it and log an error result.
- Must respond
[SILENT]when idle — If the command directory is empty or all commands are already processed, respond with exactly[SILENT]and nothing else. This suppresses unnecessary delivery. - Single-shot, not a daemon — The cron job runs once per schedule, checks once, processes, and exits. Do NOT implement polling loops or sleep/retry logic inside the job.
Cron Job Workflow
1. Check for pre-existing VisualAgent infrastructure:
a. Look for scripts: ls -la ~/ | grep visual_agent
b. Check for README: cat ~/VisualAgent_README.md
c. Check directories: su -c 'ls -la /sdcard/VisualAgent/'
2. If infrastructure exists, decide: use existing executor, start daemon, or use MCP tools
3. Check /sdcard/VisualAgent/commands/ for ALL files (not just .json — may be .cmd, .txt, or no extension)
4. Cross-reference with .processed_files and processed/ directory
5. If no new commands → respond [SILENT] and exit
6. If new commands found:
a. Read each command file with su -c 'cat ...'
b. Parse JSON (use python3 -c or jq, not grep/cut)
c. Execute via MCP tools (preferred) or su -c 'input ...'
d. Write result to results/ with su -c 'echo ... > ...'
e. Move command to processed/ with su -c 'mv ...'
f. Append filename to .processed_files with su -c 'echo ... >> ...'
7. Report processed commands in final response
Cron Job Anti-Patterns to Avoid
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
Polling loop with sleep |
Hits max iteration limit, wastes tool calls | Single check, process all found, exit |
Using execute_code for JSON parsing |
Blocked in cron mode, produces error | Use terminal with python3 -c or jq |
Re-processing files from processed/ |
Already handled, may cause duplicate actions | Check .processed_files and processed/ first |
| Reporting "0 commands found" as content | Generates unnecessary delivery noise | Use [SILENT] instead |
Using rm to delete processed commands |
May trigger approval blocks, loses history | Use mv to processed/ directory |
Using Hermes read_file/write_file on /sdcard/ |
Permission denied in Termux context | Use terminal with su -c |
| Ignoring pre-existing infrastructure | Reimplements what already exists | Check for existing scripts first, then decide |
| Trusting daemon PID file without verification | PID file may be stale, daemon actually dead | Verify with ps -p <pid> before relying on daemon |
Using shell grep/cut to parse JSON |
Breaks on nested quotes, unicode, whitespace variations | Always use python3 -c or jq for JSON parsing |
| Re-processing same command repeatedly | File not archived after execution, found again next run | Always mv to processed/ after execution; see references/command-file-extension-flexibility.md |
Using for f in /sdcard/.../*.json with su |
Glob expands in parent shell before su runs |
Use for f in $(su -c 'ls -1 ...') instead |
Example: Minimal Cron Job Implementation
#!/system/bin/sh
# visualagent-cron.sh — Minimal VisualAgent cron job
COMMANDS_DIR="/sdcard/VisualAgent/commands"
RESULTS_DIR="/sdcard/VisualAgent/results"
PROCESSED_DIR="/sdcard/VisualAgent/processed"
TRACKING_FILE="/sdcard/VisualAgent/.processed_files"
# Ensure directories exist
su -c "mkdir -p $COMMANDS_DIR $RESULTS_DIR $PROCESSED_DIR"
# List ALL files (not just .json — may be .cmd, .txt, etc.)
ALL_FILES=$(su -c "ls -1 $COMMANDS_DIR/ 2>/dev/null")
if [ -z "$ALL_FILES" ]; then
# No commands — silent exit
exit 0
fi
# Filter to valid command files (JSON with type/command field)
COMMANDS=""
for f in $ALL_FILES; do
# Skip directories
if su -c "test -d $COMMANDS_DIR/$f"; then
continue
fi
# Validate JSON content
cmd_type=$(su -c "cat $COMMANDS_DIR/$f" | python3 -c "
import json, sys
try:
d = json.load(sys.stdin)
print(d.get('type') or d.get('command', ''))
except:
print('')
" 2>/dev/null)
if [ -n "$cmd_type" ]; then
COMMANDS="$COMMANDS $COMMANDS_DIR/$f"
fi
done
if [ -z "$COMMANDS" ]; then
# No valid commands — silent exit
exit 0
fi
# Process each command
for cmd_file in $COMMANDS; do
basename=$(basename "$cmd_file")
# Check if already processed
if su -c "grep -q '^$basename$' $TRACKING_FILE 2>/dev/null"; then
continue
fi
# Read and parse command
cmd_json=$(su -c "cat $cmd_file")
cmd_type=$(echo "$cmd_json" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('type') or d.get('command',''))")
# Execute (shell example — prefer MCP tools in practice)
case "$cmd_type" in
screenshot)
path=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('path','/sdcard/VisualAgent/screenshot.png'))")
su -c "screencap -p $path"
result="{\"status\":\"success\",\"path\":\"$path\"}"
;;
tap)
x=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('x',0))")
y=$(echo "$cmd_json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('y',0))")
su -c "input tap $x $y"
result="{\"status\":\"success\",\"x\":$x,\"y\":$y}"
;;
*)
result="{\"status\":\"error\",\"error\":\"Unknown command type: $cmd_type\"}"
;;
esac
# Write result
result_file="$RESULTS_DIR/${basename%.json}_result.json"
su -c "echo '$result' > $result_file"
# Move to processed and track
su -c "mv $cmd_file $PROCESSED_DIR/"
su -c "echo $basename >> $TRACKING_FILE"
done
Note: This shell script is for illustration. In practice, a Hermes cron job uses the terminal tool to run shell snippets and MCP tools for UI operations, not a standalone shell script.
Related References
cron-execute-code-blocked.md— Why execute_code is blocked and workaround patternstermux-sdcard-permission-patterns.md— Detailed su-wrapping patterns for file operationscron-polling-anti-patterns.md— Correct polling behavior in cron jobssdcard-path-discovery.md— Discovering correct external storage pathspre-existing-visualagent-infrastructure.md— Discovery checklist for existing VisualAgent installationsvisual-agent-daemon-lifecycle.md— Daemon health check and restart procedurescommand-file-extension-flexibility.md— Handling non-.jsoncommand file extensions and preventing infinite re-processing loops