VisualAgent Executor Bugs and Workarounds

Background

This reference documents specific bugs found in existing VisualAgent executor implementations (both shell and Python versions) and provides workarounds. These bugs were discovered during real-world debugging sessions on Android devices with Termux.

Bug 1: Shell Executor's Fragile JSON Parsing (visual_agent_executor.sh)

The Problem

The shell executor uses grep/cut to parse JSON command files:

cmd_type=$(su -c "cat '$cmd_file'" | grep -o '"type"[[:space:]]*:[[:space:]]*"[^"]*"' | cut -d'"' -f4)

This pattern breaks when:

• JSON contains nested quotes (e.g., "text": "Say \"hello\"")
• Values contain colons (e.g., "path": "/sdcard/VisualAgent/screenshot_10:00.png")
• Whitespace varies (e.g., "type" : "screenshot" vs "type":"screenshot")
• Unicode characters are present in values

Real-World Impact

In this session, a test_tap.json file containing {"type": "tap", ...} was processed but the shell executor reported:

{"status":"error","message":"未知命令类型"}

The grep/cut pattern failed to extract the type value correctly, causing the command to be rejected as "unknown type".

Workaround

Never use grep/cut for JSON parsing. Use Python or jq instead:

# Correct: Use Python for reliable JSON parsing
cmd_type=$(su -c "cat '$cmd_file'" | python3 -c "import json,sys; print(json.load(sys.stdin).get('type',''))")

# Alternative: Use jq if available
cmd_type=$(su -c "cat '$cmd_file'" | jq -r '.type // .command // empty')

Fix for Shell Executor

Replace the fragile grep/cut patterns with Python one-liners:

# Before (fragile):
cmd_type=$(su -c "cat '$cmd_file'" | grep -o '"type"[[:space:]]*:[[:space:]]*"[^"]*"' | cut -d'"' -f4)

# After (robust):
cmd_type=$(su -c "cat '$cmd_file'" | python3 -c "import json,sys; print(json.load(sys.stdin).get('type',''))")

Similarly for coordinate extraction:

# Before (fragile):
x=$(su -c "cat '$cmd_file'" | grep -o '"x"[[:space:]]*:[[:space:]]*[0-9]*' | grep -o '[0-9]*$')

# After (robust):
x=$(su -c "cat '$cmd_file'" | python3 -c "import json,sys; print(json.load(sys.stdin).get('x',0))")

Bug 2: Python Executor's Broken File Listing (visual_agent.py)

The Problem

The Python executor's get_pending_commands() function has a bug:

def get_pending_commands():
    success, stdout, stderr = run_su(f"ls -1 '{CMD_DIR}'/*.json 2>/dev/null")
    if not success or not stdout.strip():
        return []
    return [line.strip() for line in stdout.strip().split('\n') if line.strip()]

The run_su() function wraps commands with su -c '...', but the glob pattern *.json is expanded by the shell BEFORE su runs. If the parent shell (running as the Termux user) doesn't have permission to read the directory, the glob fails even though su would have permission.

Real-World Impact

In this session, command files existed in /sdcard/VisualAgent/commands/ (visible via su -c "ls -la ..."), but the Python executor reported "没有待执行的命令" (no pending commands) because the glob expansion failed.

Workaround

Use find instead of shell glob, or ensure the glob is expanded inside the su context:

# Correct: Use find (no glob expansion issues)
su -c "find /sdcard/VisualAgent/commands/ -maxdepth 1 -name '*.json' -type f"

# Alternative: Use ls without glob, then filter
su -c "ls -1 /sdcard/VisualAgent/commands/" | grep '\.json$'

Fix for Python Executor

Replace the glob-based listing with find:

def get_pending_commands():
    # Use find instead of glob to avoid expansion issues
    success, stdout, stderr = run_su(f"find '{CMD_DIR}' -maxdepth 1 -name '*.json' -type f 2>/dev/null")
    if not success or not stdout.strip():
        return []
    return [line.strip() for line in stdout.strip().split('\n') if line.strip()]

Bug 3: Shell Executor's Duplicate Code

The Problem

The shell executor (visual_agent_executor.sh) has two nearly identical code blocks:

1. One for non-daemon mode (lines 11-103)
2. One for daemon mode (lines 112-203)

The daemon mode block is a copy-paste of the non-daemon block with only the sleep 5 added. This means:

• Any bug fix must be applied in two places
• The file is twice as long as necessary
• Maintenance is error-prone

Real-World Impact

In this session, the duplicate code made it harder to understand the executor's behavior and increased the risk of inconsistencies between daemon and non-daemon modes.

Fix

Refactor to use a single processing function:

#!/system/bin/sh

CMD_DIR="/sdcard/VisualAgent/commands"
RESULT_DIR="/sdcard/VisualAgent/results"
PROCESSED_DIR="/sdcard/VisualAgent/processed"

process_commands() {
    for cmd_file in $(su -c "ls -1 '$CMD_DIR'/*.json 2>/dev/null"); do
        [ -e "$cmd_file" ] || continue

        filename=$(basename "$cmd_file")
        result_file="$RESULT_DIR/${filename%.json}_result.json"

        echo "[$(date)] 处理命令: $filename"

        # Use Python for reliable JSON parsing
        cmd_type=$(su -c "cat '$cmd_file'" | python3 -c "import json,sys; print(json.load(sys.stdin).get('type',''))")

        # Execute command (same logic for both modes)
        case "$cmd_type" in
            "screenshot")
                # ... screenshot logic
                ;;
            "tap")
                # ... tap logic
                ;;
            # ... other cases
        esac

        # Save result and move processed file
        su -c "echo '$result' > '$result_file'"
        su -c "mv '$cmd_file' '$PROCESSED_DIR/$filename'"
    done
}

# Non-daemon mode
if [ "$1" != "--daemon" ]; then
    su -c "mkdir -p '$RESULT_DIR' '$PROCESSED_DIR'"
    process_commands
    echo "[$(date)] 处理完成"
    exit 0
fi

# Daemon mode
echo "[$(date)] 守护模式运行，每 5 秒检查一次..."
while true; do
    sleep 5
    process_commands
done

Bug 4: Shell Executor's type vs command Field Name Mismatch

The Problem

The shell executor only checks for "type" field:

cmd_type=$(su -c "cat '$cmd_file'" | grep -o '"type"[[:space:]]*:[[:space:]]*"[^"]*"' | cut -d'"' -f4)

But some command generators use "command" instead of "type":

{
  "command": "tap",
  "x": 500,
  "y": 1000
}

Real-World Impact

Commands using "command" instead of "type" are rejected as "unknown type" even though they are valid commands.

Fix

Check both field names:

# Use Python to check both fields
cmd_type=$(su -c "cat '$cmd_file'" | python3 -c "
import json,sys
cmd = json.load(sys.stdin)
print(cmd.get('type') or cmd.get('command', ''))
")

Bug 5: Shell Executor's input text Quoting Issues

The Problem

The shell executor passes text directly to input text:

su -c "input text '$text'"

This breaks when:

• Text contains single quotes (e.g., It's working)
• Text contains special shell characters (e.g., ;, &, |)
• Text contains spaces (may be split incorrectly)

Fix

Use printf with proper escaping, or use ADBKeyboard for reliable text input:

# Safer approach using printf
safe_text=$(printf '%s' "$text" | sed "s/'/'\\''/g")
su -c "input text '$safe_text'"

Or use ADBKeyboard (if installed):

su -c "am broadcast -a ADB_INPUT_TEXT --es msg '$text' -n com.android.adbkeyboard/.AdbIME"

Summary of Recommended Fixes

Bug	Location	Fix
Fragile JSON parsing	visual_agent_executor.sh	Replace grep/cut with python3 -c
Broken file listing	visual_agent.py	Replace glob with find
Duplicate code	visual_agent_executor.sh	Extract process_commands() function
Field name mismatch	Both executors	Check both type and command fields
Text input quoting	visual_agent_executor.sh	Use printf escaping or ADBKeyboard

Testing After Fixes

After applying fixes, verify with these test commands:

# Test 1: Screenshot with path containing spaces
su -c "printf '%s\n' '{\"type\":\"screenshot\",\"path\":\"/sdcard/VisualAgent/test screenshot.png\"}' > /sdcard/VisualAgent/commands/test1.json"

# Test 2: Tap with unicode text
su -c "printf '%s\n' '{\"type\":\"input\",\"text\":\"Hello 世界\"}' > /sdcard/VisualAgent/commands/test2.json"

# Test 3: Command using "command" field instead of "type"
su -c "printf '%s\n' '{\"command\":\"tap\",\"x\":500,\"y\":1000}' > /sdcard/VisualAgent/commands/test3.json"

# Run executor and check results
sh /data/data/com.termux/files/home/visual_agent_executor.sh

Related References

• visual-agent-service-patterns.md — General VisualAgent service patterns and cron job workflows
• visual-agent-daemon-lifecycle.md — Daemon health check and restart procedures
• pre-existing-visualagent-infrastructure.md — Discovery checklist for existing VisualAgent installations
• termux-sdcard-permission-patterns.md — Detailed patterns for su-wrapping file operations