pre-existing-visualagent-infrastructure.md 8.56 KB

Pre-existing VisualAgent Infrastructure Discovery

Background

On Android devices that have been used for UI automation before, a VisualAgent backend may already be installed and configured. Rather than building a new system from scratch, it's more efficient to discover, understand, and leverage the existing infrastructure.

Discovery Checklist

When asked to set up or run a VisualAgent service, check these locations first:

1. Termux Home Directory Scripts

ls -la /data/data/com.termux/files/home/ | grep visual_agent

Common files found:

  • visual_agent.py — Python-based executor (better error handling)
  • visual_agent.sh — Shell wrapper
  • visual_agent_executor.sh — Main executor script (the one found in this session)
  • visual_agent_daemon.sh — Daemon management script (start/stop/status)
  • visual_agent_service.sh — Service management script
  • visualagent_service.py — Canonical Python implementation (uses /data/local/tmp/, reliable JSON parsing)
  • visualagent_daemon.sh — Bash daemon with start/stop/once/file modes
  • visualagent_processor.sh — Simple shell processor (fragile grep/cut parsing)
  • visualagent_cron.sh — Cron wrapper script
  • VisualAgent_README.md — Documentation

2. SDCard Directory Structure

su -c "ls -la /sdcard/VisualAgent/ 2>/dev/null || echo 'NOT_FOUND'"

Expected structure if already initialized:

/sdcard/VisualAgent/
├── commands/          # Command files directory
├── results/           # Result files directory
├── processed/         # Archive of processed commands
├── .processed_files   # Tracking file
├── visualagent.log    # Service log
├── daemon.log         # Daemon log (if daemon was run)
├── daemon.pid         # Daemon PID file
└── *.png              # Screenshots from previous runs

Note: The canonical implementation uses /data/local/tmp/VisualAgent/ instead of /sdcard/VisualAgent/ to avoid FUSE filesystem issues. Check both locations.

3. Check for Running Daemon

# Check PID file
su -c "cat /sdcard/VisualAgent/daemon.pid 2>/dev/null || echo 'NO_PID'"

# Verify process is actually running
ps | grep <pid> | grep -v grep

# Or search for visual_agent processes
ps aux | grep visual_agent | grep -v grep

Integration Patterns

If visual_agent_executor.sh exists, run it directly:

sh /data/data/com.termux/files/home/visual_agent_executor.sh

This processes all pending commands in a single batch and exits.

If visual_agent_daemon.sh exists, manage it via the script:

# Check status
su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh status"

# Start daemon
su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh start"

# Stop daemon
su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh stop"

Note: The daemon may have reliability issues. See references/visual-agent-daemon-lifecycle.md for health check and restart procedures.

If visualagent_service.py exists, use it as the canonical implementation:

# Single scan (for cron)
python3 /data/data/com.termux/files/home/visualagent_service.py

This is the preferred implementation because:

  1. Uses Python json.loads() - reliable parsing, no grep/cut fragility
  2. Uses /data/local/tmp/ - avoids FUSE filesystem visibility issues
  3. Handles both command and type fields
  4. Base64-encodes results for safe writing via su
  5. Produces structured JSON results with success, output, saved_to fields

Pattern D: Hybrid — Existing Infrastructure + Hermes MCP Tools

When the existing infrastructure exists but you want to use Hermes MCP tools for execution (better error handling, structured responses):

  1. Check for existing infrastructure (scripts, directories)
  2. Use existing directories (/sdcard/VisualAgent/commands/, etc.)
  3. Read command files via terminal with su -c 'cat ...'
  4. Execute via MCP tools (mcp_android_phone_screenshot, mcp_android_phone_tap, etc.)
  5. Write results via terminal with su -c 'echo ... > ...'
  6. Move processed files via terminal with su -c 'mv ...'

This approach leverages the existing directory structure and conventions while using the more reliable MCP tools for actual UI operations.

What to Do When Infrastructure Is Found

Step 1: Read the README

cat /data/data/com.termux/files/home/VisualAgent_README.md

The README typically contains:

  • Available command types and their JSON formats
  • Directory structure
  • Usage examples
  • Troubleshooting tips

Step 2: Check Directory State

su -c '
  echo "=== Commands ===";
  ls -la /sdcard/VisualAgent/commands/ 2>/dev/null || echo "EMPTY/NOT_FOUND";
  echo "=== Processed ===";
  ls -la /sdcard/VisualAgent/processed/ 2>/dev/null || echo "EMPTY/NOT_FOUND";
  echo "=== Results ===";
  ls -la /sdcard/VisualAgent/results/ 2>/dev/null || echo "EMPTY/NOT_FOUND";
  echo "=== Log ===";
  tail -20 /sdcard/VisualAgent/visualagent.log 2>/dev/null || echo "NO_LOG"
'

Step 3: Decide on Approach

Scenario Recommended Action
visualagent_service.py exists Use it (canonical implementation)
Scripts exist, daemon not running Start daemon or run executor script
Scripts exist, daemon running Verify daemon health, use existing infrastructure
No scripts, directories exist Use MCP tools with existing directories
Nothing exists Create new infrastructure from scratch

Common Pre-existing Configurations

Configuration 1: Shell-Based Executor (visual_agent_executor.sh)

  • Uses /system/bin/sh shebang
  • Processes commands in a for loop
  • Uses grep/cut for JSON parsing (fragile)
  • Supports: screenshot, tap, swipe, input, start_app
  • Results written as <filename>_result.json
  • Processed files moved to processed/ directory

Configuration 2: Python-Based Executor (visual_agent.py)

  • Uses Python's json module for parsing
  • Better error handling than shell scripts
  • May support additional command types
  • Results typically include size_bytes for screenshots

Configuration 3: Canonical Python Service (visualagent_service.py) ⭐

  • Path: /data/data/com.termux/files/home/visualagent_service.py
  • Working directory: /data/local/tmp/VisualAgent/
  • Uses Python's json module for reliable parsing
  • Handles both command and type fields
  • Base64-encodes results for safe writing via su
  • Produces structured JSON with success, output, saved_to fields
  • Preferred over shell implementations when available
  • Single-scan mode (not a daemon) — run via cron for periodic execution

Configuration 4: Bash Daemon (visualagent_daemon.sh)

  • Wraps the executor in a loop with sleep 5
  • Supports start, stop, once, file modes
  • Uses python3 -c for JSON parsing (more reliable than grep/cut)
  • Writes PID to /sdcard/VisualAgent/daemon.pid
  • Logs to /sdcard/VisualAgent/daemon.log
  • May silently exit due to shell loop behavior

Pitfalls

  1. Don't assume the infrastructure is working — even if files exist, the daemon may be dead, the scripts may be outdated, or the directories may have incorrect permissions
  2. Check the README first — it contains the specific command format and conventions for this installation
  3. Verify daemon health — a PID file doesn't guarantee the process is running
  4. Don't re-create directories if they exist — use mkdir -p which is idempotent
  5. Be careful with JSON quoting — the existing scripts may have specific quoting requirements
  6. Multiple implementations may coexist — check which one is the canonical one (usually visualagent_service.py)
  7. Stale infrastructure produces {"status":"error"} results — if you see many error results, the shell implementation is likely running and failing due to fragile parsing

Example: Discovery and Integration Workflow

1. Check for existing scripts: ls -la ~/ | grep visual_agent
2. Read README: cat ~/VisualAgent_README.md
3. Check directory state: su -c 'ls -la /sdcard/VisualAgent/'
4. Check daemon status: su -c 'sh ~/visual_agent_daemon.sh status'
5. If daemon dead → restart: su -c 'sh ~/visual_agent_daemon.sh start'
6. If no daemon → run executor: sh ~/visual_agent_executor.sh
7. Check for pending commands: su -c 'ls -la /sdcard/VisualAgent/commands/'
8. Process commands using existing infrastructure or MCP tools
9. Report results