# VisualAgent Service Enhancement: Multi-Strategy App Launch ## Session Context - **Date**: 2026-06-26 - **Device**: Android 16 (Pixel Fold) - **Task**: Build a VisualAgent backend service that monitors `/sdcard/VisualAgent/commands/` and executes JSON command files ## Key Discovery: am start -n package/.MainActivity is unreliable The naive approach to launching apps via `am start -n package/.MainActivity` frequently fails: ```bash $ su -c "am start -n com.android.settings/.MainActivity" Starting: Intent { cmp=com.android.settings/.MainActivity } Error type 3 Error: Activity class {com.android.settings/com.android.settings.MainActivity} does not exist. ``` This is because: 1. The `.MainActivity` class may not exist in the manifest 2. The actual launcher Activity may have a different name 3. Some apps use aliases or exported activities that don't match the simple naming convention ## Solution: Multi-Strategy App Launch A robust VisualAgent backend must implement three fallback strategies: ### Strategy 1: Monkey Tool (Most Reliable) ```bash su -c "monkey -p $package -c android.intent.category.LAUNCHER 1" ``` **Pros:** - Works for almost all apps with a launcher icon - Handles Activity name resolution automatically - No need to know the actual Activity class name **Cons:** - Generates a random touch event (the "1" at the end) - May trigger unintended UI interactions - Slightly slower than direct am start ### Strategy 2: am start with Action/Category ```bash su -c "am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -p $package" ``` **Pros:** - Standard Android intent resolution - Doesn't require knowing Activity names - Cleaner than monkey **Cons:** - May still fail if the app has no MAIN/LAUNCHER activity - Less reliable than monkey for some OEM skins ### Strategy 3: Iterate Common Activity Names ```bash for activity in ".MainActivity" ".SplashActivity" ".HomeActivity" ".LaunchActivity"; do su -c "am start -n $package/$activity" if [ $? -eq 0 ]; then break fi done ``` **Pros:** - Direct control over which Activity to launch - Fastest when it works **Cons:** - Requires guessing the Activity name - May fail for all common names ## Implementation in Processor Script The actual implementation from this session uses a cascading approach: ```bash "start_app") local package=$(cat "$cmd_file" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('params',{}).get('package','') if 'params' in d else d.get('package',''))") if [ -n "$package" ]; then local start_success=false # Strategy 1: monkey tool su -c "monkey -p $package -c android.intent.category.LAUNCHER 1" 2>/dev/null if [ $? -eq 0 ]; then start_success=true fi # Strategy 2: am start with action/category if [ "$start_success" = false ]; then su -c "am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -p $package" 2>/dev/null if [ $? -eq 0 ]; then start_success=true fi fi # Strategy 3: iterate common Activity names if [ "$start_success" = false ]; then for activity in ".MainActivity" ".SplashActivity" ".HomeActivity" ".LaunchActivity"; do su -c "am start -n $package/$activity" 2>/dev/null if [ $? -eq 0 ]; then start_success=true break fi done fi if [ "$start_success" = true ]; then echo "{\"status\":\"success\",\"type\":\"start_app\",\"result\":{\"package\":\"$package\"}}" > "$RESULTS_DIR/${cmd_id}_result.json" else echo "{\"status\":\"error\",\"type\":\"start_app\",\"error\":\"Failed to start app $package\"}" > "$RESULTS_DIR/${cmd_id}_result.json" fi fi ;; ``` ## Test Results from This Session | Command | Strategy Used | Result | |---------|--------------|--------| | `com.android.settings` | monkey (after am start failed) | ✅ Success | | screenshot | screencap | ✅ Success (15,350 bytes) | | tap (500, 1000) | input tap | ✅ Success | | swipe (500,1500→500,500) | input swipe | ✅ Success | | input "hello world" | input text | ✅ Success | ## Additional Enhancements from This Session ### Dual JSON Format Support The processor supports both flat and nested parameter formats: ```json // Flat format (direct fields) {"command": "tap", "x": 500, "y": 1000} // Nested format (params object) {"type": "tap", "params": {"x": 500, "y": 1000}} ``` Parsed via: ```bash x=$(cat "$cmd_file" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('params',{}).get('x',0) if 'params' in d else d.get('x',0))") ``` ### Robust su Fallback Each command tries `su` first, then falls back to non-su: ```bash su -c "input tap $x $y" 2>/dev/null if [ $? -ne 0 ]; then input tap "$x" "$y" 2>/dev/null fi ``` This ensures the processor works on both rooted and non-rooted devices (though UI commands require root on most devices). ## Command Compatibility Matrix | Command | `su -c` Required? | Fallback Strategy | Notes | |---------|-----------------|-------------------|-------| | screenshot | Yes (for /sdcard/) | Non-su screencap | Works without su on some devices | | tap | Yes | MCP tools | `input tap` may fail with "Failed transaction" on some devices | | swipe | Yes | MCP tools | Same as tap | | input | Yes | MCP tools | Same as tap | | start_app | Yes | monkey tool | am start often fails; monkey is most reliable | ## Related References - `references/termux-internal-path-fallback.md` (in `android-command-queue` skill) — When `/sdcard/` is inaccessible, use `/data/data/com.termux/files/home/VisualAgent/` - `references/visual-agent-executor-bugs.md` — Other bugs found in VisualAgent executors - `references/mcp-tools-vs-raw-shell.md` — Decision matrix for MCP tools vs shell commands