Using Hand to interact with ItemMonitor

The Hand node can interact with 2D content in a natural way using the index finger to point, click, and slide.

The 2D content can be either of the following:

• An XR 2D library based interface.

• A connection to an application supporting XR Remote Desktop API.

The functionality is based on four elements:

• One or multiple Hand instances to interact with an ItemMonitor.

• An ItemMonitor.

• A Texture2D to be displayed inside the ItemMonitor.

• An XR 2D library based user interface to be rendered on the texture.

Configuring the Hands

There is no specific setting to apply to Hand node instances to enable them to interact with the ItemMonitor content.

However, you should always keep in mind that interaction is happening with the hand index finger. To be more correct, interaction is based on the index finger last bone direction vector.

A good practice is to have the hand showing a proper posture, such as the point preset, when interacting with an ItemMonitor. This to emphasize the fact that the interaction is with the finger and to avoid user confusion.

Configuring the ItemMonitor

The first step is configuring the ItemMonitor to be able to properly interact with the hands and to render the Texture2D.

<ItemMonitor name="aMonitor" templateInfo="Monitor|aMonitor" useLocalTransform="true" position="2 0 0" orientation="0 -1.570796 0" scale="1.66 1 1" displayName="aMonitor" displayValue="@aMonitor.value@" itemPosition="2 1.569582E-11 -5E-05"

images="internal:anotherTexture" textureInteractionEnabled="true" textureInteractionMode="hand" textureInteractionHands="test_hand_r" />

The relevant fields to edit are:

• textureInteractionEnabled: Set to true to enable interaction with texture content.

• textureInteractionMode: Set to hand to enable hand interaction.

• textureInteractionHands: Must contain the list of hand nodes that are enabled to interact with the monitor. Depending on project type, interaction can be supported on single or multiple hands (even remotely driven ones).

• images: Set to the Texture2D hosting the 2D interface with the syntax: internal:textureNodeName

Other relevant fields are: handOverDistance, handTouchDistance and handTouchBeyondTolerance whose values can be changed from default to improve user index interaction experience.

Configuring the Texture2D

The Texture2D needs to be configured to properly support the hand interaction.

<Texture2D name="aTexture" templateInfo="Texture2D|aTexture" size="1920 1080" eventsType="custom" />

The most relevant parameter to edit is eventsType. It must be set to custom to properly support hand index finger interaction. Basically, with this setting the index finger operates like a virtual mouse.

Another relevant parameter to consider is the size parameter. It must be set to properly display the 2D user interface and have the same form factor as the ItemMonitor mesh to avoid distortions.

Configuring the Contents

Use XR Studio Graphic Context Editor to assemble a 2D user interface.

When the interface is ready, you can move it into the Texture2D by assigning it to the renderToTexture2D field of all the 2D elements.

Note that the position of the 2D nodes will refer to texture coordinates.

Remote connection

To set up an XR Remote Desktop connection, the user interface will consist of a single Rect2D with the images field configured to point to the remote service.