# Foldable Screen Coordinate Calibration ## Problem Foldable devices (e.g., Pixel 10 Pro Fold) have different screen resolutions for inner and outer screens. UI automation coordinates (tap, swipe) are screen-specific and must be recalibrated when switching between screens. ## Known Resolutions | Device | Screen | Resolution | Verified | |--------|--------|------------|----------| | Pixel 10 Pro Fold | Outer | 1080×2364 | ✅ 2026-06-25 | | Pixel 10 Pro Fold | Inner | 2076×2152 | ✅ 2026-06-25 | ## Calibration Workflow ### Step 1: Check Current Resolution ```bash su -c "wm size" # Output: Physical size: 1080x2364 ``` ### Step 2: Take Screenshot ```bash # Using MCP tool (recommended) mcp_android_phone_screenshot(path="/sdcard/Pictures/calibrate.png") # Or using shell su -c "screencap -p /sdcard/Pictures/calibrate.png" ``` ### Step 3: Run OCR to Identify Elements **Prerequisites**: - OCR models in `~/pixelphone/event_bus/ocr_models/`: - PP-OCRv6_det_tiny.onnx (~1.7MB) - PP-OCRv6_rec_tiny.onnx (~4.3MB) - ppocrv6_tiny_dict.txt (~27KB) - Python dependencies: `pyclipper`, `Pillow`, `numpy`, `scipy`, `onnxruntime` ```bash python3 ~/pixelphone/event_bus/ocr_tools.py /sdcard/Pictures/calibrate.png ``` **Output**: ```json { "ok": true, "count": 5, "results": [ { "text": "文件传输助手", "score": 0.8713, "bbox": [383, 189, 679, 263], "center": {"x": 531, "y": 226} } ] } ``` ### Step 4: Extract Coordinates Key fields from OCR output: - `center.x`, `center.y` — tap-ready coordinates - `bbox` — bounding box [x0, y0, x1, y1] ### Step 5: Update Coordinate Tables Document verified coordinates in the relevant skill: ```markdown ### 1080×2364 Screen Verified Coordinates | Element | Coordinate | Verified | |---------|-----------|----------| | 微信主界面-搜索按钮 | (874, 228) | ✅ | | 微信主界面-"+"按钮 | (982, 227) | ✅ | | 聊天界面-更多按钮 | (1013, 1601) | ✅ | | 聊天界面-相册按钮 | (167, 1914) | ✅ | | 相册界面-发送按钮 | (954, 2230) | ✅ | ``` ## Verified WeChat Coordinates (Pixel 10 Pro Fold) ### Outer Screen (1080×2364) #### 微信主界面 | Element | Coordinate | Verified | |---------|-----------|----------| | 搜索按钮 | (874, 228) | ✅ | | "+" 按钮 | (982, 227) | ✅ | | 底部-微信 | (133, 2268) | ✅ | | 底部-通讯录 | (402, 2270) | ✅ | | 底部-发现 | (674, 2268) | ✅ | | 底部-我 | (943, 2268) | ✅ | #### 聊天界面 | Element | Coordinate | Verified | |---------|-----------|----------| | 返回按钮 "<" | (61, 226) | ✅ | | 更多按钮 "+" | (1013, 1601) | ✅ | | 语音按钮 | (76, 2218) | ✅ | #### 更多菜单(附件菜单) | Element | Coordinate | Verified | |---------|-----------|----------| | 相册 | (167, 1914) | ✅ | | 拍摄 | (413, 1914) | ✅ | | 位置 | (661, 1912) | ✅ | | 语音输入 | (913, 1915) | ✅ | | 收藏 | (168, 2175) | ✅ | | 个人名片 | (414, 2175) | ✅ | | 文件 | (661, 2175) | ✅ | | 音乐 | (910, 2176) | ✅ | #### 相册选择界面 | Element | Coordinate | Verified | |---------|-----------|----------| | 发送 | (954, 2230) | ✅ | | 预览 | (87, 2229) | ✅ | | 原图 | (569, 2231) | ✅ | ### Inner Screen (2076×2152) #### System Share 选择聊天界面 | Element | Coordinate | Verified | |---------|-----------|----------| | 选择聊天 | (1563, 228) | ✅ | | 最近转发 | (1168, 518) | ✅ | | 最近聊天 | (1169, 1005) | ✅ | | 创建聊天 | (1972, 1005) | ✅ | | 折叠置顶聊天 | (1372, 2010) | ✅ | **Note**: System Share (ACTION_SEND) on inner screen causes WeChat to crash after selecting contact. Use direct chat window method instead. ## Automation Script ```python #!/usr/bin/env python3 """Calibrate coordinates for foldable device screens""" import json import subprocess def get_screen_resolution(): result = subprocess.run(['su', '-c', 'wm size'], capture_output=True, text=True) line = result.stdout.strip() if 'Physical size:' in line: w, h = line.split(':')[1].strip().split('x') return int(w), int(h) return None, None def ocr_screenshot(path): result = subprocess.run( ['python3', '~/pixelphone/event_bus/ocr_tools.py', path], capture_output=True, text=True ) return json.loads(result.stdout) def find_element(text, screenshot_path): data = ocr_screenshot(screenshot_path) for item in data['results']: if text in item['text']: return item['center'] return None if __name__ == '__main__': w, h = get_screen_resolution() print(f"Current resolution: {w}x{h}") # Take screenshot subprocess.run(['su', '-c', 'screencap -p /sdcard/Pictures/calibrate.png']) # Find key elements elements = ['文件传输助手', '发送', '相册', '取消'] for elem in elements: pos = find_element(elem, '/sdcard/Pictures/calibrate.png') if pos: print(f"{elem}: ({pos['x']}, {pos['y']})") else: print(f"{elem}: NOT FOUND") ``` ## Pitfalls 1. **Always verify resolution before tapping** — `wm size` returns current screen dimensions 2. **OCR models must be deployed** — missing models cause `FileNotFoundError` 3. **System Share crashes on inner screen** — WeChat 8.x + Android 16 + foldable inner screen = crash after contact selection 4. **Coordinates are screen-specific** — outer screen coordinates do NOT work on inner screen 5. **Screenshot size indicates UI state** — small screenshot (< 100KB) usually means simple UI; large screenshot (> 500KB) means complex UI or overlay ## Related Skills - `wechat-image-sender` — WeChat image sending with coordinate tables - `wechat-group-digest` — Daily digest generation with OCR calibration section - `android-root-troubleshooting` — Root/su permission issues