visualagent-service-enhancement.md
5.87 KB
VisualAgent Service Enhancement: Multi-Strategy App Launch
Session Context
- Date: 2026-06-26
- Device: Android 16 (Pixel Fold)
- Task: Build a VisualAgent backend service that monitors
/sdcard/VisualAgent/commands/and executes JSON command files
Key Discovery: am start -n package/.MainActivity is unreliable
The naive approach to launching apps via am start -n package/.MainActivity frequently fails:
$ su -c "am start -n com.android.settings/.MainActivity"
Starting: Intent { cmp=com.android.settings/.MainActivity }
Error type 3
Error: Activity class {com.android.settings/com.android.settings.MainActivity} does not exist.
This is because:
- The
.MainActivityclass may not exist in the manifest - The actual launcher Activity may have a different name
- Some apps use aliases or exported activities that don't match the simple naming convention
Solution: Multi-Strategy App Launch
A robust VisualAgent backend must implement three fallback strategies:
Strategy 1: Monkey Tool (Most Reliable)
su -c "monkey -p $package -c android.intent.category.LAUNCHER 1"
Pros:
- Works for almost all apps with a launcher icon
- Handles Activity name resolution automatically
- No need to know the actual Activity class name
Cons:
- Generates a random touch event (the "1" at the end)
- May trigger unintended UI interactions
- Slightly slower than direct am start
Strategy 2: am start with Action/Category
su -c "am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -p $package"
Pros:
- Standard Android intent resolution
- Doesn't require knowing Activity names
- Cleaner than monkey
Cons:
- May still fail if the app has no MAIN/LAUNCHER activity
- Less reliable than monkey for some OEM skins
Strategy 3: Iterate Common Activity Names
for activity in ".MainActivity" ".SplashActivity" ".HomeActivity" ".LaunchActivity"; do
su -c "am start -n $package/$activity"
if [ $? -eq 0 ]; then
break
fi
done
Pros:
- Direct control over which Activity to launch
- Fastest when it works
Cons:
- Requires guessing the Activity name
- May fail for all common names
Implementation in Processor Script
The actual implementation from this session uses a cascading approach:
"start_app")
local package=$(cat "$cmd_file" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('params',{}).get('package','') if 'params' in d else d.get('package',''))")
if [ -n "$package" ]; then
local start_success=false
# Strategy 1: monkey tool
su -c "monkey -p $package -c android.intent.category.LAUNCHER 1" 2>/dev/null
if [ $? -eq 0 ]; then
start_success=true
fi
# Strategy 2: am start with action/category
if [ "$start_success" = false ]; then
su -c "am start -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -p $package" 2>/dev/null
if [ $? -eq 0 ]; then
start_success=true
fi
fi
# Strategy 3: iterate common Activity names
if [ "$start_success" = false ]; then
for activity in ".MainActivity" ".SplashActivity" ".HomeActivity" ".LaunchActivity"; do
su -c "am start -n $package/$activity" 2>/dev/null
if [ $? -eq 0 ]; then
start_success=true
break
fi
done
fi
if [ "$start_success" = true ]; then
echo "{\"status\":\"success\",\"type\":\"start_app\",\"result\":{\"package\":\"$package\"}}" > "$RESULTS_DIR/${cmd_id}_result.json"
else
echo "{\"status\":\"error\",\"type\":\"start_app\",\"error\":\"Failed to start app $package\"}" > "$RESULTS_DIR/${cmd_id}_result.json"
fi
fi
;;
Test Results from This Session
| Command | Strategy Used | Result |
|---|---|---|
com.android.settings |
monkey (after am start failed) | ✅ Success |
| screenshot | screencap | ✅ Success (15,350 bytes) |
| tap (500, 1000) | input tap | ✅ Success |
| swipe (500,1500→500,500) | input swipe | ✅ Success |
| input "hello world" | input text | ✅ Success |
Additional Enhancements from This Session
Dual JSON Format Support
The processor supports both flat and nested parameter formats:
// Flat format (direct fields)
{"command": "tap", "x": 500, "y": 1000}
// Nested format (params object)
{"type": "tap", "params": {"x": 500, "y": 1000}}
Parsed via:
x=$(cat "$cmd_file" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('params',{}).get('x',0) if 'params' in d else d.get('x',0))")
Robust su Fallback
Each command tries su first, then falls back to non-su:
su -c "input tap $x $y" 2>/dev/null
if [ $? -ne 0 ]; then
input tap "$x" "$y" 2>/dev/null
fi
This ensures the processor works on both rooted and non-rooted devices (though UI commands require root on most devices).
Command Compatibility Matrix
| Command | su -c Required? |
Fallback Strategy | Notes |
|---|---|---|---|
| screenshot | Yes (for /sdcard/) | Non-su screencap | Works without su on some devices |
| tap | Yes | MCP tools | input tap may fail with "Failed transaction" on some devices |
| swipe | Yes | MCP tools | Same as tap |
| input | Yes | MCP tools | Same as tap |
| start_app | Yes | monkey tool | am start often fails; monkey is most reliable |
Related References
references/termux-internal-path-fallback.md(inandroid-command-queueskill) — When/sdcard/is inaccessible, use/data/data/com.termux/files/home/VisualAgent/references/visual-agent-executor-bugs.md— Other bugs found in VisualAgent executorsreferences/mcp-tools-vs-raw-shell.md— Decision matrix for MCP tools vs shell commands