# Pre-existing VisualAgent Infrastructure Discovery ## Background On Android devices that have been used for UI automation before, a VisualAgent backend may already be installed and configured. Rather than building a new system from scratch, it's more efficient to discover, understand, and leverage the existing infrastructure. ## Discovery Checklist When asked to set up or run a VisualAgent service, check these locations first: ### 1. Termux Home Directory Scripts ```bash ls -la /data/data/com.termux/files/home/ | grep visual_agent ``` Common files found: - `visual_agent.py` — Python-based executor (better error handling) - `visual_agent.sh` — Shell wrapper - `visual_agent_executor.sh` — Main executor script (the one found in this session) - `visual_agent_daemon.sh` — Daemon management script (start/stop/status) - `visual_agent_service.sh` — Service management script - `visualagent_service.py` — **Canonical Python implementation** (uses `/data/local/tmp/`, reliable JSON parsing) - `visualagent_daemon.sh` — Bash daemon with start/stop/once/file modes - `visualagent_processor.sh` — Simple shell processor (fragile grep/cut parsing) - `visualagent_cron.sh` — Cron wrapper script - `VisualAgent_README.md` — Documentation ### 2. SDCard Directory Structure ```bash su -c "ls -la /sdcard/VisualAgent/ 2>/dev/null || echo 'NOT_FOUND'" ``` Expected structure if already initialized: ``` /sdcard/VisualAgent/ ├── commands/ # Command files directory ├── results/ # Result files directory ├── processed/ # Archive of processed commands ├── .processed_files # Tracking file ├── visualagent.log # Service log ├── daemon.log # Daemon log (if daemon was run) ├── daemon.pid # Daemon PID file └── *.png # Screenshots from previous runs ``` **Note**: The canonical implementation uses `/data/local/tmp/VisualAgent/` instead of `/sdcard/VisualAgent/` to avoid FUSE filesystem issues. Check both locations. ### 3. Check for Running Daemon ```bash # Check PID file su -c "cat /sdcard/VisualAgent/daemon.pid 2>/dev/null || echo 'NO_PID'" # Verify process is actually running ps | grep | grep -v grep # Or search for visual_agent processes ps aux | grep visual_agent | grep -v grep ``` ## Integration Patterns ### Pattern A: Use Existing Executor Script (Recommended for One-Offs) If `visual_agent_executor.sh` exists, run it directly: ```bash sh /data/data/com.termux/files/home/visual_agent_executor.sh ``` This processes all pending commands in a single batch and exits. ### Pattern B: Use Existing Daemon (Recommended for Continuous Monitoring) If `visual_agent_daemon.sh` exists, manage it via the script: ```bash # Check status su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh status" # Start daemon su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh start" # Stop daemon su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh stop" ``` **Note**: The daemon may have reliability issues. See `references/visual-agent-daemon-lifecycle.md` for health check and restart procedures. ### Pattern C: Use Canonical Python Service (Recommended) If `visualagent_service.py` exists, use it as the canonical implementation: ```bash # Single scan (for cron) python3 /data/data/com.termux/files/home/visualagent_service.py ``` This is the preferred implementation because: 1. Uses Python `json.loads()` - reliable parsing, no `grep`/`cut` fragility 2. Uses `/data/local/tmp/` - avoids FUSE filesystem visibility issues 3. Handles both `command` and `type` fields 4. Base64-encodes results for safe writing via `su` 5. Produces structured JSON results with `success`, `output`, `saved_to` fields ### Pattern D: Hybrid — Existing Infrastructure + Hermes MCP Tools When the existing infrastructure exists but you want to use Hermes MCP tools for execution (better error handling, structured responses): 1. Check for existing infrastructure (scripts, directories) 2. Use existing directories (`/sdcard/VisualAgent/commands/`, etc.) 3. Read command files via `terminal` with `su -c 'cat ...'` 4. Execute via MCP tools (`mcp_android_phone_screenshot`, `mcp_android_phone_tap`, etc.) 5. Write results via `terminal` with `su -c 'echo ... > ...'` 6. Move processed files via `terminal` with `su -c 'mv ...'` This approach leverages the existing directory structure and conventions while using the more reliable MCP tools for actual UI operations. ## What to Do When Infrastructure Is Found ### Step 1: Read the README ```bash cat /data/data/com.termux/files/home/VisualAgent_README.md ``` The README typically contains: - Available command types and their JSON formats - Directory structure - Usage examples - Troubleshooting tips ### Step 2: Check Directory State ```bash su -c ' echo "=== Commands ==="; ls -la /sdcard/VisualAgent/commands/ 2>/dev/null || echo "EMPTY/NOT_FOUND"; echo "=== Processed ==="; ls -la /sdcard/VisualAgent/processed/ 2>/dev/null || echo "EMPTY/NOT_FOUND"; echo "=== Results ==="; ls -la /sdcard/VisualAgent/results/ 2>/dev/null || echo "EMPTY/NOT_FOUND"; echo "=== Log ==="; tail -20 /sdcard/VisualAgent/visualagent.log 2>/dev/null || echo "NO_LOG" ' ``` ### Step 3: Decide on Approach | Scenario | Recommended Action | |----------|-------------------| | `visualagent_service.py` exists | **Use it** (canonical implementation) | | Scripts exist, daemon not running | Start daemon or run executor script | | Scripts exist, daemon running | Verify daemon health, use existing infrastructure | | No scripts, directories exist | Use MCP tools with existing directories | | Nothing exists | Create new infrastructure from scratch | ## Common Pre-existing Configurations ### Configuration 1: Shell-Based Executor (visual_agent_executor.sh) - Uses `/system/bin/sh` shebang - Processes commands in a `for` loop - Uses `grep`/`cut` for JSON parsing (fragile) - Supports: screenshot, tap, swipe, input, start_app - Results written as `_result.json` - Processed files moved to `processed/` directory ### Configuration 2: Python-Based Executor (visual_agent.py) - Uses Python's `json` module for parsing - Better error handling than shell scripts - May support additional command types - Results typically include `size_bytes` for screenshots ### Configuration 3: Canonical Python Service (visualagent_service.py) ⭐ - **Path**: `/data/data/com.termux/files/home/visualagent_service.py` - **Working directory**: `/data/local/tmp/VisualAgent/` - Uses Python's `json` module for reliable parsing - Handles both `command` and `type` fields - Base64-encodes results for safe writing via `su` - Produces structured JSON with `success`, `output`, `saved_to` fields - **Preferred over shell implementations** when available - Single-scan mode (not a daemon) — run via cron for periodic execution ### Configuration 4: Bash Daemon (visualagent_daemon.sh) - Wraps the executor in a loop with `sleep 5` - Supports `start`, `stop`, `once`, `file` modes - Uses `python3 -c` for JSON parsing (more reliable than grep/cut) - Writes PID to `/sdcard/VisualAgent/daemon.pid` - Logs to `/sdcard/VisualAgent/daemon.log` - May silently exit due to shell loop behavior ## Pitfalls 1. **Don't assume the infrastructure is working** — even if files exist, the daemon may be dead, the scripts may be outdated, or the directories may have incorrect permissions 2. **Check the README first** — it contains the specific command format and conventions for this installation 3. **Verify daemon health** — a PID file doesn't guarantee the process is running 4. **Don't re-create directories if they exist** — use `mkdir -p` which is idempotent 5. **Be careful with JSON quoting** — the existing scripts may have specific quoting requirements 6. **Multiple implementations may coexist** — check which one is the canonical one (usually `visualagent_service.py`) 7. **Stale infrastructure produces `{"status":"error"}` results** — if you see many error results, the shell implementation is likely running and failing due to fragile parsing ## Example: Discovery and Integration Workflow ``` 1. Check for existing scripts: ls -la ~/ | grep visual_agent 2. Read README: cat ~/VisualAgent_README.md 3. Check directory state: su -c 'ls -la /sdcard/VisualAgent/' 4. Check daemon status: su -c 'sh ~/visual_agent_daemon.sh status' 5. If daemon dead → restart: su -c 'sh ~/visual_agent_daemon.sh start' 6. If no daemon → run executor: sh ~/visual_agent_executor.sh 7. Check for pending commands: su -c 'ls -la /sdcard/VisualAgent/commands/' 8. Process commands using existing infrastructure or MCP tools 9. Report results ```