# VisualAgent Daemon Lifecycle and Reliability

## Background

The VisualAgent daemon (`visual_agent_daemon.sh`) is a shell script that runs the executor in a loop, checking for new commands every few seconds. However, it has a known reliability issue: the daemon process can exit silently while the PID file remains, creating a "zombie" state where the system thinks the daemon is running but it's actually dead.

## The Problem

### Symptom: Daemon Appears Running But Isn't

```bash
# PID file shows a running process
su -c "cat /sdcard/VisualAgent/daemon.pid"
# Output: 1093

# But ps shows no such process
su -c "ps | grep 1093 | grep -v grep"
# Output: (nothing)

# Or the process exists but is not the daemon
su -c "ps | grep 'sh' | grep -v grep"
# Output: root 1093 1 ... S sh
# But this is just a shell, not the daemon loop
```

### Root Cause

The daemon script typically uses this pattern:

```bash
nohup sh "$SCRIPT_DIR/visual_agent_executor.sh" --daemon > "$LOG_FILE" 2>&1 &
echo $! > "$PID_FILE"
```

The executor script's `--daemon` mode runs:

```bash
while true; do
    sleep 5
    # process commands...
done
```

Issues that cause silent exit:
1. **Shell exits on error** — if `set -e` is in effect, any failed command kills the loop
2. **Sleep returns non-zero** — in some Android shells, `sleep` can return an error signal
3. **Resource exhaustion** — the loop may accumulate file descriptors or memory
4. **SIGTERM/SIGHUP** — the process may receive a termination signal
5. **The `sh` process is just the parent** — the actual daemon loop may be a child that exited

## Health Check Procedure

### Step 1: Check PID File

```bash
su -c "cat /sdcard/VisualAgent/daemon.pid 2>/dev/null || echo 'NO_PID_FILE'"
```

### Step 2: Verify Process Exists

```bash
PID=$(su -c "cat /sdcard/VisualAgent/daemon.pid 2>/dev/null")
if [ -n "$PID" ]; then
    su -c "ps -p $PID 2>/dev/null | grep -v PID || echo 'PROCESS_NOT_FOUND'"
fi
```

### Step 3: Check Process Command Line

```bash
PID=$(su -c "cat /sdcard/VisualAgent/daemon.pid 2>/dev/null")
if [ -n "$PID" ]; then
    su -c "cat /proc/$PID/cmdline 2>/dev/null | tr '\0' ' ' || echo 'NO_CMDLINE'"
fi
```

A healthy daemon should show something like:
```
sh /data/data/com.termux/files/home/visual_agent_executor.sh --daemon
```

### Step 4: Check Log Activity

```bash
su -c "tail -10 /sdcard/VisualAgent/daemon.log 2>/dev/null || echo 'NO_LOG'"
```

A healthy daemon should show recent timestamps (within the last few seconds if the loop is running).

### Step 5: Check for Command Processing

Create a test command and see if it's processed within 10 seconds:

```bash
# Create test command
su -c "printf '%s\n' '{\"type\":\"screenshot\",\"id\":\"daemon_health_check\"}' > /sdcard/VisualAgent/commands/daemon_health_check.json"

# Wait 10 seconds
sleep 10

# Check if processed
su -c "ls -la /sdcard/VisualAgent/processed/ | grep daemon_health_check || echo 'NOT_PROCESSED'"
```

## Restart Procedure

If the daemon is dead or unhealthy:

### Step 1: Stop (Clean Up PID File)

```bash
su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh stop 2>/dev/null || rm -f /sdcard/VisualAgent/daemon.pid"
```

### Step 2: Start Daemon

```bash
su -c "sh /data/data/com.termux/files/home/visual_agent_daemon.sh start"
```

### Step 3: Verify Health

```bash
sleep 2
PID=$(su -c "cat /sdcard/VisualAgent/daemon.pid 2>/dev/null")
if [ -n "$PID" ]; then
    su -c "ps -p $PID 2>/dev/null | grep -v PID || echo 'STILL_NOT_RUNNING'"
fi
```

## Alternative: Use Cron Instead of Daemon

If the daemon is unreliable, consider using a cron job instead:

```bash
# Edit crontab
crontab -e

# Add lines (every 30 seconds)
* * * * * sh /data/data/com.termux/files/home/visual_agent_executor.sh
* * * * * sleep 30; sh /data/data/com.termux/files/home/visual_agent_executor.sh
```

Or use Hermes scheduled cron jobs to run the executor periodically.

## Alternative: Use Single-Shot Execution

For batch processing, run the executor once and exit:

```bash
sh /data/data/com.termux/files/home/visual_agent_executor.sh
```

This is the most reliable approach for intermittent command processing.

## Daemon Log Analysis

The daemon log (`/sdcard/VisualAgent/daemon.log`) reveals the daemon's state:

### Healthy Daemon Log

```
[Thu Jun 25 20:49:59 CST 2026] 处理完成
[Thu Jun 25 20:49:59 CST 2026] 守护模式运行，每 5 秒检查一次...
[Thu Jun 25 20:52:09 CST 2026] 处理命令: test_cron_screenshot.json
[Thu Jun 25 20:52:10 CST 2026] 结果: {"status":"success",...}
[Thu Jun 25 20:52:11 CST 2026] 已移动到: /sdcard/VisualAgent/processed/test_cron_screenshot.json
```

Note: Timestamps should be recent (within loop interval + processing time).

### Dead Daemon Log

```
[Thu Jun 25 20:49:59 CST 2026] 处理完成
[Thu Jun 25 20:49:59 CST 2026] 守护模式运行，每 5 秒检查一次...
```

Note: No new entries after the "守护模式运行" line — the daemon exited right after logging this.

## Prevention: Robust Daemon Implementation

If implementing a new daemon, use these patterns:

### 1. Trap Signals

```bash
trap 'echo "Daemon exiting"; rm -f "$PID_FILE"; exit 0' TERM INT HUP
```

### 2. Use `exec` for the Loop

```bash
exec sh -c '
    while true; do
        sleep 5
        # process commands...
    done
'
```

### 3. Log Before and After Sleep

```bash
while true; do
    echo "[$(date)] Checking for commands..."
    # process commands...
    echo "[$(date)] Sleeping..."
    sleep 5
done
```

### 4. Use `flock` for PID File

```bash
exec 200>"$PID_FILE"
flock -n 200 || { echo "Daemon already running"; exit 1; }
echo $$ >"$PID_FILE"
```

## Quick Reference: Daemon Commands

| Action | Command |
|--------|---------|
| Check status | `su -c "sh ~/visual_agent_daemon.sh status"` |
| Start daemon | `su -c "sh ~/visual_agent_daemon.sh start"` |
| Stop daemon | `su -c "sh ~/visual_agent_daemon.sh stop"` |
| Check PID | `su -c "cat /sdcard/VisualAgent/daemon.pid"` |
| Verify process | `su -c "ps -p \$(cat /sdcard/VisualAgent/daemon.pid)"` |
| View log | `su -c "tail -20 /sdcard/VisualAgent/daemon.log"` |
| Force restart | `su -c "rm -f /sdcard/VisualAgent/daemon.pid; sh ~/visual_agent_daemon.sh start"` |

## Related References

- `pre-existing-visualagent-infrastructure.md` — Discovery checklist for existing VisualAgent installations
- `cron-polling-anti-patterns.md` — Correct polling behavior when using cron instead of daemon
- `visual-agent-service-patterns.md` — General VisualAgent service patterns