screen_mapper.py analyzes the current iOS simulator screen by retrieving and processing the accessibility tree via idb. It provides structured element discovery as the foundational tool for semantic navigation, producing token-efficient summaries of screen state (element type breakdown, interactive button lists, text field status) without requiring screenshots.
This script is specifically designed for AI agents to make navigation decisions by answering "what is currently visible and interactive?" in a compact format.
Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py1-17 ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py55-71
The ScreenMapper class implements the following core capabilities:
idb ui describe-all --json --nested via shared utilities ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py103-109 ios-simulator-skill/skills/ios-simulator-skill/scripts/common/idb_utils.py22-48AXLabel, AXValue, and AXUniqueId fields ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py131-167AXUniqueId for "ViewController" or "Screen" patterns ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py168-171--verbose and --hints flags ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py12-17Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py39-44 ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py111-129
The script relies on idb_utils.py to interface with the simulator's accessibility APIs.
The following diagram bridges the Natural Language Space (User/Agent Intent) to the Code Entity Space (Internal functions and external tools).
Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py103-109 ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py131-175 ios-simulator-skill/skills/ios-simulator-skill/scripts/common/idb_utils.py22-69
screen_mapper.py maps raw accessibility attributes to semantic navigation info:
| IDB Attribute | Mapper Usage | Logic |
|---|---|---|
type | Categorization | Checks against INTERACTIVE_TYPES ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py75-88 |
AXLabel | Primary Label | Used as display name for buttons/elements ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py134 |
AXValue | Secondary Label | Used if AXLabel is missing; also tracks has_value for TextFields ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py135 |
AXUniqueId | Identifier | Used for screen naming and fallback labeling ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py136 |
enabled | Focusability | Increments focusable count if true and interactive ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py165-166 |
Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py131-167
The script is optimized for token efficiency, reducing the context window usage compared to raw JSON or screenshots.
Focuses on counts and primary action labels.
Screen: LoginViewController (45 elements, 7 interactive)format_summary() ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py177-185--hints)Provides actionable element names and their specific types for navigation.
elements_by_type and lists labels ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py221-228--verbose)Outputs the full nested tree structure as retrieved from IDB.
get_accessibility_tree() ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py284-285| Mode | Typical Token Count | Data Reduction |
|---|---|---|
| Default | ~20-40 tokens | ~98% vs Raw JSON |
| Hints | ~100-200 tokens | ~90% vs Raw JSON |
| Raw JSON | ~2000-5000 tokens | 0% |
Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py32-38 ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py207-230
The script maintains a set of INTERACTIVE_TYPES that represent user interaction points.
Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py75-88
The _analyze_recursive method traverses the tree to populate an analysis dictionary. This logic maps the raw hierarchy to the ScreenMapper internal state.
Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py131-175
screen_mapper.py is a consumer of the common package, specifically for device targeting and tree retrieval.
resolve_udid from common to handle --udid or --name arguments ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py52 ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py269get_accessibility_tree from common which handles the idb subprocess execution and JSON parsing ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py103-109Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py52-53 ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py103-109
| Flag | Description |
|---|---|
--udid | Target a specific device UDID. |
--name | Target a device by name (e.g., "iPhone 15"). |
--verbose | Print the full raw accessibility tree JSON. |
--hints | Show detailed interactive element labels and types. |
--json | Output the analysis dictionary as JSON for programmatic use. |
Sources: ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py19-31 ios-simulator-skill/skills/ios-simulator-skill/scripts/screen_mapper.py240-262
Refresh this wiki