Click Target in XR

Find a target object in the scene and click it using a controller, then verify the click registered.

Arguments

$ARGUMENTS should be a description of the target to find (e.g., "the RESTART button", "the scoreboard", "the settings panel").

Workflow

1. Get Scene Hierarchy

Use mcpiwsdk-dev-mcpscenegetruntime_hierarchy to find the target object's UUID.

• Look for objects with entityIndex (these are ECS entities)
• PanelUI elements typically have nested children for their content

2. Get Target Transform

Use mcpiwsdk-dev-mcpscenegetobject_transform with the target UUID.

• Use positionRelativeToXROrigin for all positioning operations
• Note the position for the next steps

3. Position Headset to Look at Target

Use mcpiwsdk-dev-mcpxrlookat with device headset and the target position.

• This orients the headset to face the target

4. Screenshot to Verify Target Visible

Use mcpiwsdk-dev-mcpbrowser_screenshot to verify:

• Target is visible in the view
• Target is not occluded by other objects
• If not visible, use mcpiwsdk-dev-mcpxrlookat with moveToDistance to get closer

5. Position Controller

Use mcpiwsdk-dev-mcpxrgettransform to check controller position.

• If controller is visible in screenshot and not occluding target, proceed
• If controller is far away or not visible:

1. Get headset position with mcpiwsdk-dev-mcpxrgettransform 2. Set controller to headset position with mcpiwsdk-dev-mcpxrsettransform 3. Micro-adjust: offset x by +0.25 (right) or -0.25 (left), y by -0.1, z by -0.3

6. Point Controller at Target

Use mcpiwsdk-dev-mcpxrlookat with the controller device and target position.

• For UI elements on panels, you may need to target a specific child element
• If the target is a button on a panel, the button may be offset from the panel center

7. Click

Use mcpiwsdk-dev-mcpxr_select with the controller device.

8. Verify Click

Use mcpiwsdk-dev-mcpbrowsergetconsole_logs with a pattern to check for expected log messages.

• If no logs match, the click may have missed - adjust target position and retry
• Common adjustments: change x or z by 0.05-0.1 to hit child elements

Tips

• For PanelUI buttons, the button position is often offset from the panel center
• Use mcpiwsdk-dev-mcpscenegetobject_transform on child elements to find exact button positions
• Always verify with console logs rather than assuming from visuals
• The ray visual in screenshots can be misleading - test with actual clicks
• Either controller (left or right) can be used

Example

To click the RESTART button on the pong scoreboard:

1. Find scoreboard entity (entityIndex 6 in the pong game)
2. Scoreboard is at (0, -0.798, -1.25)
3. Button is at approximately (0.05, -0.798, -1.109) - offset toward player
4. Point controller at button position and click
5. Verify "[BUTTON CLICKED]" appears in console logs