Pre-existing VisualAgent Infrastructure Discovery
Background
On Android devices that have been used for UI automation before, a VisualAgent backend may already be installed and configured. Rather than building a new system from scratch, it's more efficient to discover, understand, and leverage the existing infrastructure.
Discovery Checklist
When asked to set up or run a VisualAgent service, check these locations first:
1. Termux Home Directory Scripts
ls -la /data/data/com.termux/files/home/ | grep visual_agent
Common files found:
visual_agent.py— Python-based executor (better error handling)visual_agent.sh— Shell wrappervisual_agent_executor.sh— Main executor script (the one found in this session)visual_agent_daemon.sh— Daemon management script (start/stop/status)visual_agent_service.sh— Service management scriptvisualagent_service.py— Canonical Python implementation (uses/data/local/tmp/, reliable JSON parsing)visualagent_daemon.sh— Bash daemon with start/stop/once/file modesvisualagent_processor.sh— Simple shell processor (fragile grep/cut parsing)visualagent_cron.sh— Cron wrapper scriptVisualAgent_README.md— Documentation
2. SDCard Directory Structure
su -c "ls -la /sdcard/VisualAgent/ 2>/dev/null || echo 'NOT_FOUND'"
Expected structure if already initialized:
/sdcard/VisualAgent/
├── commands/ # Command files directory
├── results/ # Result files directory
├── processed/ # Archive of processed commands
├── .processed_files # Tracking file
├── visualagent.log # Service log
├── daemon.log # Daemon log (if daemon was run)
├── daemon.pid # Daemon PID file
└── *.png # Screenshots from previous runs
Note: The canonical implementation uses /data/local/tmp/VisualAgent/ instead of /sdcard/VisualAgent/ to avoid FUSE filesystem issues. Check both locations.
3. Check for Running Daemon
# Check PID file
su -c "cat /sdcard/VisualAgent/daemon.pid 2>/dev/null || echo 'NO_PID'"
# Verify process is actually running
ps | grep <pid> | grep -v grep
# Or search for visual_agent processes
ps aux | grep visual_agent | grep -v grep
Integration Patterns
Pattern A: Use Existing Executor Script (Recommended for One-Offs)
If visual_agent_executor.sh exists, run it directly:
sh /data/data/com.termux/files/home/visual_agent_executor.sh
This processes all pending commands in a single batch and exits.
Pattern B: Use Existing Daemon (Recommended for Continuous Monitoring)
If visual_agent_daemon.sh exists, manage it via the script:
# Check status
su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh status"
# Start daemon
su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh start"
# Stop daemon
su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh stop"
Note: The daemon may have reliability issues. See references/visual-agent-daemon-lifecycle.md for health check and restart procedures.
Pattern C: Use Canonical Python Service (Recommended)
If visualagent_service.py exists, use it as the canonical implementation:
# Single scan (for cron)
python3 /data/data/com.termux/files/home/visualagent_service.py
This is the preferred implementation because:
- Uses Python
json.loads()- reliable parsing, nogrep/cutfragility - Uses
/data/local/tmp/- avoids FUSE filesystem visibility issues - Handles both
commandandtypefields - Base64-encodes results for safe writing via
su - Produces structured JSON results with
success,output,saved_tofields
Pattern D: Hybrid — Existing Infrastructure + Hermes MCP Tools
When the existing infrastructure exists but you want to use Hermes MCP tools for execution (better error handling, structured responses):
- Check for existing infrastructure (scripts, directories)
- Use existing directories (
/sdcard/VisualAgent/commands/, etc.) - Read command files via
terminalwithsu -c 'cat ...' - Execute via MCP tools (
mcp_android_phone_screenshot,mcp_android_phone_tap, etc.) - Write results via
terminalwithsu -c 'echo ... > ...' - Move processed files via
terminalwithsu -c 'mv ...'
This approach leverages the existing directory structure and conventions while using the more reliable MCP tools for actual UI operations.
What to Do When Infrastructure Is Found
Step 1: Read the README
cat /data/data/com.termux/files/home/VisualAgent_README.md
The README typically contains:
- Available command types and their JSON formats
- Directory structure
- Usage examples
- Troubleshooting tips
Step 2: Check Directory State
su -c '
echo "=== Commands ===";
ls -la /sdcard/VisualAgent/commands/ 2>/dev/null || echo "EMPTY/NOT_FOUND";
echo "=== Processed ===";
ls -la /sdcard/VisualAgent/processed/ 2>/dev/null || echo "EMPTY/NOT_FOUND";
echo "=== Results ===";
ls -la /sdcard/VisualAgent/results/ 2>/dev/null || echo "EMPTY/NOT_FOUND";
echo "=== Log ===";
tail -20 /sdcard/VisualAgent/visualagent.log 2>/dev/null || echo "NO_LOG"
'
Step 3: Decide on Approach
| Scenario | Recommended Action |
|---|---|
visualagent_service.py exists |
Use it (canonical implementation) |
| Scripts exist, daemon not running | Start daemon or run executor script |
| Scripts exist, daemon running | Verify daemon health, use existing infrastructure |
| No scripts, directories exist | Use MCP tools with existing directories |
| Nothing exists | Create new infrastructure from scratch |
Common Pre-existing Configurations
Configuration 1: Shell-Based Executor (visual_agent_executor.sh)
- Uses
/system/bin/shshebang - Processes commands in a
forloop - Uses
grep/cutfor JSON parsing (fragile) - Supports: screenshot, tap, swipe, input, start_app
- Results written as
<filename>_result.json - Processed files moved to
processed/directory
Configuration 2: Python-Based Executor (visual_agent.py)
- Uses Python's
jsonmodule for parsing - Better error handling than shell scripts
- May support additional command types
- Results typically include
size_bytesfor screenshots
Configuration 3: Canonical Python Service (visualagent_service.py) ⭐
- Path:
/data/data/com.termux/files/home/visualagent_service.py - Working directory:
/data/local/tmp/VisualAgent/ - Uses Python's
jsonmodule for reliable parsing - Handles both
commandandtypefields - Base64-encodes results for safe writing via
su - Produces structured JSON with
success,output,saved_tofields - Preferred over shell implementations when available
- Single-scan mode (not a daemon) — run via cron for periodic execution
Configuration 4: Bash Daemon (visualagent_daemon.sh)
- Wraps the executor in a loop with
sleep 5 - Supports
start,stop,once,filemodes - Uses
python3 -cfor JSON parsing (more reliable than grep/cut) - Writes PID to
/sdcard/VisualAgent/daemon.pid - Logs to
/sdcard/VisualAgent/daemon.log - May silently exit due to shell loop behavior
Pitfalls
- Don't assume the infrastructure is working — even if files exist, the daemon may be dead, the scripts may be outdated, or the directories may have incorrect permissions
- Check the README first — it contains the specific command format and conventions for this installation
- Verify daemon health — a PID file doesn't guarantee the process is running
- Don't re-create directories if they exist — use
mkdir -pwhich is idempotent - Be careful with JSON quoting — the existing scripts may have specific quoting requirements
- Multiple implementations may coexist — check which one is the canonical one (usually
visualagent_service.py) - Stale infrastructure produces
{"status":"error"}results — if you see many error results, the shell implementation is likely running and failing due to fragile parsing
Example: Discovery and Integration Workflow
1. Check for existing scripts: ls -la ~/ | grep visual_agent
2. Read README: cat ~/VisualAgent_README.md
3. Check directory state: su -c 'ls -la /sdcard/VisualAgent/'
4. Check daemon status: su -c 'sh ~/visual_agent_daemon.sh status'
5. If daemon dead → restart: su -c 'sh ~/visual_agent_daemon.sh start'
6. If no daemon → run executor: sh ~/visual_agent_executor.sh
7. Check for pending commands: su -c 'ls -la /sdcard/VisualAgent/commands/'
8. Process commands using existing infrastructure or MCP tools
9. Report results