foldable-screen-coordinate-calibration.md
5.75 KB
Foldable Screen Coordinate Calibration
Problem
Foldable devices (e.g., Pixel 10 Pro Fold) have different screen resolutions for inner and outer screens. UI automation coordinates (tap, swipe) are screen-specific and must be recalibrated when switching between screens.
Known Resolutions
| Device | Screen | Resolution | Verified |
|---|---|---|---|
| Pixel 10 Pro Fold | Outer | 1080×2364 | ✅ 2026-06-25 |
| Pixel 10 Pro Fold | Inner | 2076×2152 | ✅ 2026-06-25 |
Calibration Workflow
Step 1: Check Current Resolution
su -c "wm size"
# Output: Physical size: 1080x2364
Step 2: Take Screenshot
# Using MCP tool (recommended)
mcp_android_phone_screenshot(path="/sdcard/Pictures/calibrate.png")
# Or using shell
su -c "screencap -p /sdcard/Pictures/calibrate.png"
Step 3: Run OCR to Identify Elements
Prerequisites:
- OCR models in
~/pixelphone/event_bus/ocr_models/:- PP-OCRv6_det_tiny.onnx (~1.7MB)
- PP-OCRv6_rec_tiny.onnx (~4.3MB)
- ppocrv6_tiny_dict.txt (~27KB)
- Python dependencies:
pyclipper,Pillow,numpy,scipy,onnxruntime
python3 ~/pixelphone/event_bus/ocr_tools.py /sdcard/Pictures/calibrate.png
Output:
{
"ok": true,
"count": 5,
"results": [
{
"text": "文件传输助手",
"score": 0.8713,
"bbox": [383, 189, 679, 263],
"center": {"x": 531, "y": 226}
}
]
}
Step 4: Extract Coordinates
Key fields from OCR output:
center.x,center.y— tap-ready coordinatesbbox— bounding box [x0, y0, x1, y1]
Step 5: Update Coordinate Tables
Document verified coordinates in the relevant skill:
### 1080×2364 Screen Verified Coordinates
| Element | Coordinate | Verified |
|---------|-----------|----------|
| 微信主界面-搜索按钮 | (874, 228) | ✅ |
| 微信主界面-"+"按钮 | (982, 227) | ✅ |
| 聊天界面-更多按钮 | (1013, 1601) | ✅ |
| 聊天界面-相册按钮 | (167, 1914) | ✅ |
| 相册界面-发送按钮 | (954, 2230) | ✅ |
Verified WeChat Coordinates (Pixel 10 Pro Fold)
Outer Screen (1080×2364)
微信主界面
| Element | Coordinate | Verified |
|---|---|---|
| 搜索按钮 | (874, 228) | ✅ |
| "+" 按钮 | (982, 227) | ✅ |
| 底部-微信 | (133, 2268) | ✅ |
| 底部-通讯录 | (402, 2270) | ✅ |
| 底部-发现 | (674, 2268) | ✅ |
| 底部-我 | (943, 2268) | ✅ |
聊天界面
| Element | Coordinate | Verified |
|---|---|---|
| 返回按钮 " | (61, 226) | ✅ |
| 更多按钮 "+" | (1013, 1601) | ✅ |
| 语音按钮 | (76, 2218) | ✅ |
更多菜单(附件菜单)
| Element | Coordinate | Verified |
|---|---|---|
| 相册 | (167, 1914) | ✅ |
| 拍摄 | (413, 1914) | ✅ |
| 位置 | (661, 1912) | ✅ |
| 语音输入 | (913, 1915) | ✅ |
| 收藏 | (168, 2175) | ✅ |
| 个人名片 | (414, 2175) | ✅ |
| 文件 | (661, 2175) | ✅ |
| 音乐 | (910, 2176) | ✅ |
相册选择界面
| Element | Coordinate | Verified |
|---|---|---|
| 发送 | (954, 2230) | ✅ |
| 预览 | (87, 2229) | ✅ |
| 原图 | (569, 2231) | ✅ |
Inner Screen (2076×2152)
System Share 选择聊天界面
| Element | Coordinate | Verified |
|---|---|---|
| 选择聊天 | (1563, 228) | ✅ |
| 最近转发 | (1168, 518) | ✅ |
| 最近聊天 | (1169, 1005) | ✅ |
| 创建聊天 | (1972, 1005) | ✅ |
| 折叠置顶聊天 | (1372, 2010) | ✅ |
Note: System Share (ACTION_SEND) on inner screen causes WeChat to crash after selecting contact. Use direct chat window method instead.
Automation Script
#!/usr/bin/env python3
"""Calibrate coordinates for foldable device screens"""
import json
import subprocess
def get_screen_resolution():
result = subprocess.run(['su', '-c', 'wm size'], capture_output=True, text=True)
line = result.stdout.strip()
if 'Physical size:' in line:
w, h = line.split(':')[1].strip().split('x')
return int(w), int(h)
return None, None
def ocr_screenshot(path):
result = subprocess.run(
['python3', '~/pixelphone/event_bus/ocr_tools.py', path],
capture_output=True, text=True
)
return json.loads(result.stdout)
def find_element(text, screenshot_path):
data = ocr_screenshot(screenshot_path)
for item in data['results']:
if text in item['text']:
return item['center']
return None
if __name__ == '__main__':
w, h = get_screen_resolution()
print(f"Current resolution: {w}x{h}")
# Take screenshot
subprocess.run(['su', '-c', 'screencap -p /sdcard/Pictures/calibrate.png'])
# Find key elements
elements = ['文件传输助手', '发送', '相册', '取消']
for elem in elements:
pos = find_element(elem, '/sdcard/Pictures/calibrate.png')
if pos:
print(f"{elem}: ({pos['x']}, {pos['y']})")
else:
print(f"{elem}: NOT FOUND")
Pitfalls
- Always verify resolution before tapping —
wm sizereturns current screen dimensions - OCR models must be deployed — missing models cause
FileNotFoundError - System Share crashes on inner screen — WeChat 8.x + Android 16 + foldable inner screen = crash after contact selection
- Coordinates are screen-specific — outer screen coordinates do NOT work on inner screen
- Screenshot size indicates UI state — small screenshot ( 500KB) means complex UI or overlay
Related Skills
wechat-image-sender— WeChat image sending with coordinate tableswechat-group-digest— Daily digest generation with OCR calibration sectionandroid-root-troubleshooting— Root/su permission issues