visualagent-service-enhancement.md 5.87 KB

VisualAgent Service Enhancement: Multi-Strategy App Launch

Session Context

  • Date: 2026-06-26
  • Device: Android 16 (Pixel Fold)
  • Task: Build a VisualAgent backend service that monitors /sdcard/VisualAgent/commands/ and executes JSON command files

Key Discovery: am start -n package/.MainActivity is unreliable

The naive approach to launching apps via am start -n package/.MainActivity frequently fails:

$ su -c "am start -n com.android.settings/.MainActivity"
Starting: Intent { cmp=com.android.settings/.MainActivity }
Error type 3
Error: Activity class {com.android.settings/com.android.settings.MainActivity} does not exist.

This is because:

  1. The .MainActivity class may not exist in the manifest
  2. The actual launcher Activity may have a different name
  3. Some apps use aliases or exported activities that don't match the simple naming convention

Solution: Multi-Strategy App Launch

A robust VisualAgent backend must implement three fallback strategies:

Strategy 1: Monkey Tool (Most Reliable)

su -c "monkey -p $package -c android.intent.category.LAUNCHER 1"

Pros:

  • Works for almost all apps with a launcher icon
  • Handles Activity name resolution automatically
  • No need to know the actual Activity class name

Cons:

  • Generates a random touch event (the "1" at the end)
  • May trigger unintended UI interactions
  • Slightly slower than direct am start

Strategy 2: am start with Action/Category

su -c "am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -p $package"

Pros:

  • Standard Android intent resolution
  • Doesn't require knowing Activity names
  • Cleaner than monkey

Cons:

  • May still fail if the app has no MAIN/LAUNCHER activity
  • Less reliable than monkey for some OEM skins

Strategy 3: Iterate Common Activity Names

for activity in ".MainActivity" ".SplashActivity" ".HomeActivity" ".LaunchActivity"; do
    su -c "am start -n $package/$activity"
    if [ $? -eq 0 ]; then
        break
    fi
done

Pros:

  • Direct control over which Activity to launch
  • Fastest when it works

Cons:

  • Requires guessing the Activity name
  • May fail for all common names

Implementation in Processor Script

The actual implementation from this session uses a cascading approach:

"start_app")
    local package=$(cat "$cmd_file" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('params',{}).get('package','') if 'params' in d else d.get('package',''))")

    if [ -n "$package" ]; then
        local start_success=false

        # Strategy 1: monkey tool
        su -c "monkey -p $package -c android.intent.category.LAUNCHER 1" 2>/dev/null
        if [ $? -eq 0 ]; then
            start_success=true
        fi

        # Strategy 2: am start with action/category
        if [ "$start_success" = false ]; then
            su -c "am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -p $package" 2>/dev/null
            if [ $? -eq 0 ]; then
                start_success=true
            fi
        fi

        # Strategy 3: iterate common Activity names
        if [ "$start_success" = false ]; then
            for activity in ".MainActivity" ".SplashActivity" ".HomeActivity" ".LaunchActivity"; do
                su -c "am start -n $package/$activity" 2>/dev/null
                if [ $? -eq 0 ]; then
                    start_success=true
                    break
                fi
            done
        fi

        if [ "$start_success" = true ]; then
            echo "{\"status\":\"success\",\"type\":\"start_app\",\"result\":{\"package\":\"$package\"}}" > "$RESULTS_DIR/${cmd_id}_result.json"
        else
            echo "{\"status\":\"error\",\"type\":\"start_app\",\"error\":\"Failed to start app $package\"}" > "$RESULTS_DIR/${cmd_id}_result.json"
        fi
    fi
    ;;

Test Results from This Session

Command Strategy Used Result
com.android.settings monkey (after am start failed) ✅ Success
screenshot screencap ✅ Success (15,350 bytes)
tap (500, 1000) input tap ✅ Success
swipe (500,1500→500,500) input swipe ✅ Success
input "hello world" input text ✅ Success

Additional Enhancements from This Session

Dual JSON Format Support

The processor supports both flat and nested parameter formats:

// Flat format (direct fields)
{"command": "tap", "x": 500, "y": 1000}

// Nested format (params object)
{"type": "tap", "params": {"x": 500, "y": 1000}}

Parsed via:

x=$(cat "$cmd_file" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('params',{}).get('x',0) if 'params' in d else d.get('x',0))")

Robust su Fallback

Each command tries su first, then falls back to non-su:

su -c "input tap $x $y" 2>/dev/null
if [ $? -ne 0 ]; then
    input tap "$x" "$y" 2>/dev/null
fi

This ensures the processor works on both rooted and non-rooted devices (though UI commands require root on most devices).

Command Compatibility Matrix

Command su -c Required? Fallback Strategy Notes
screenshot Yes (for /sdcard/) Non-su screencap Works without su on some devices
tap Yes MCP tools input tap may fail with "Failed transaction" on some devices
swipe Yes MCP tools Same as tap
input Yes MCP tools Same as tap
start_app Yes monkey tool am start often fails; monkey is most reliable
  • references/termux-internal-path-fallback.md (in android-command-queue skill) — When /sdcard/ is inaccessible, use /data/data/com.termux/files/home/VisualAgent/
  • references/visual-agent-executor-bugs.md — Other bugs found in VisualAgent executors
  • references/mcp-tools-vs-raw-shell.md — Decision matrix for MCP tools vs shell commands