foldable-screen-coordinate-calibration.md 5.75 KB

Foldable Screen Coordinate Calibration

Problem

Foldable devices (e.g., Pixel 10 Pro Fold) have different screen resolutions for inner and outer screens. UI automation coordinates (tap, swipe) are screen-specific and must be recalibrated when switching between screens.

Known Resolutions

Device Screen Resolution Verified
Pixel 10 Pro Fold Outer 1080×2364 ✅ 2026-06-25
Pixel 10 Pro Fold Inner 2076×2152 ✅ 2026-06-25

Calibration Workflow

Step 1: Check Current Resolution

su -c "wm size"
# Output: Physical size: 1080x2364

Step 2: Take Screenshot

# Using MCP tool (recommended)
mcp_android_phone_screenshot(path="/sdcard/Pictures/calibrate.png")

# Or using shell
su -c "screencap -p /sdcard/Pictures/calibrate.png"

Step 3: Run OCR to Identify Elements

Prerequisites:

  • OCR models in ~/pixelphone/event_bus/ocr_models/:
    • PP-OCRv6_det_tiny.onnx (~1.7MB)
    • PP-OCRv6_rec_tiny.onnx (~4.3MB)
    • ppocrv6_tiny_dict.txt (~27KB)
  • Python dependencies: pyclipper, Pillow, numpy, scipy, onnxruntime
python3 ~/pixelphone/event_bus/ocr_tools.py /sdcard/Pictures/calibrate.png

Output:

{
  "ok": true,
  "count": 5,
  "results": [
    {
      "text": "文件传输助手",
      "score": 0.8713,
      "bbox": [383, 189, 679, 263],
      "center": {"x": 531, "y": 226}
    }
  ]
}

Step 4: Extract Coordinates

Key fields from OCR output:

  • center.x, center.y — tap-ready coordinates
  • bbox — bounding box [x0, y0, x1, y1]

Step 5: Update Coordinate Tables

Document verified coordinates in the relevant skill:

### 1080×2364 Screen Verified Coordinates
| Element | Coordinate | Verified |
|---------|-----------|----------|
| 微信主界面-搜索按钮 | (874, 228) | ✅ |
| 微信主界面-"+"按钮 | (982, 227) | ✅ |
| 聊天界面-更多按钮 | (1013, 1601) | ✅ |
| 聊天界面-相册按钮 | (167, 1914) | ✅ |
| 相册界面-发送按钮 | (954, 2230) | ✅ |

Verified WeChat Coordinates (Pixel 10 Pro Fold)

Outer Screen (1080×2364)

微信主界面

Element Coordinate Verified
搜索按钮 (874, 228) ✅
"+" 按钮 (982, 227) ✅
底部-微信 (133, 2268) ✅
底部-通讯录 (402, 2270) ✅
底部-发现 (674, 2268) ✅
底部-我 (943, 2268) ✅

聊天界面

Element Coordinate Verified
返回按钮 " (61, 226) ✅
更多按钮 "+" (1013, 1601) ✅
语音按钮 (76, 2218) ✅

更多菜单(附件菜单)

Element Coordinate Verified
相册 (167, 1914) ✅
拍摄 (413, 1914) ✅
位置 (661, 1912) ✅
语音输入 (913, 1915) ✅
收藏 (168, 2175) ✅
个人名片 (414, 2175) ✅
文件 (661, 2175) ✅
音乐 (910, 2176) ✅

相册选择界面

Element Coordinate Verified
发送 (954, 2230) ✅
预览 (87, 2229) ✅
原图 (569, 2231) ✅

Inner Screen (2076×2152)

System Share 选择聊天界面

Element Coordinate Verified
选择聊天 (1563, 228) ✅
最近转发 (1168, 518) ✅
最近聊天 (1169, 1005) ✅
创建聊天 (1972, 1005) ✅
折叠置顶聊天 (1372, 2010) ✅

Note: System Share (ACTION_SEND) on inner screen causes WeChat to crash after selecting contact. Use direct chat window method instead.

Automation Script

#!/usr/bin/env python3
"""Calibrate coordinates for foldable device screens"""

import json
import subprocess

def get_screen_resolution():
    result = subprocess.run(['su', '-c', 'wm size'], capture_output=True, text=True)
    line = result.stdout.strip()
    if 'Physical size:' in line:
        w, h = line.split(':')[1].strip().split('x')
        return int(w), int(h)
    return None, None

def ocr_screenshot(path):
    result = subprocess.run(
        ['python3', '~/pixelphone/event_bus/ocr_tools.py', path],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

def find_element(text, screenshot_path):
    data = ocr_screenshot(screenshot_path)
    for item in data['results']:
        if text in item['text']:
            return item['center']
    return None

if __name__ == '__main__':
    w, h = get_screen_resolution()
    print(f"Current resolution: {w}x{h}")

    # Take screenshot
    subprocess.run(['su', '-c', 'screencap -p /sdcard/Pictures/calibrate.png'])

    # Find key elements
    elements = ['文件传输助手', '发送', '相册', '取消']
    for elem in elements:
        pos = find_element(elem, '/sdcard/Pictures/calibrate.png')
        if pos:
            print(f"{elem}: ({pos['x']}, {pos['y']})")
        else:
            print(f"{elem}: NOT FOUND")

Pitfalls

  1. Always verify resolution before tapping — wm size returns current screen dimensions
  2. OCR models must be deployed — missing models cause FileNotFoundError
  3. System Share crashes on inner screen — WeChat 8.x + Android 16 + foldable inner screen = crash after contact selection
  4. Coordinates are screen-specific — outer screen coordinates do NOT work on inner screen
  5. Screenshot size indicates UI state — small screenshot ( 500KB) means complex UI or overlay
  • wechat-image-sender — WeChat image sending with coordinate tables
  • wechat-group-digest — Daily digest generation with OCR calibration section
  • android-root-troubleshooting — Root/su permission issues