<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ken Pfeuffer on Medium]]></title>
        <description><![CDATA[Stories by Ken Pfeuffer on Medium]]></description>
        <link>https://medium.com/@ken.pfeuffer?source=rss-efc9299f702e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*HyGBnBO2lbDEEHAV</url>
            <title>Stories by Ken Pfeuffer on Medium</title>
            <link>https://medium.com/@ken.pfeuffer?source=rss-efc9299f702e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:45:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ken.pfeuffer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Blending Direct and Indirect Interaction: A Concept for Seamless Computer Interfaces]]></title>
            <link>https://medium.com/@ken.pfeuffer/blending-direct-and-indirect-interaction-a-future-of-computer-interfaces-30952c9dc785?source=rss-efc9299f702e------2</link>
            <guid isPermaLink="false">https://medium.com/p/30952c9dc785</guid>
            <category><![CDATA[direct-indirect]]></category>
            <category><![CDATA[touchscreen]]></category>
            <category><![CDATA[xr]]></category>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[hci]]></category>
            <dc:creator><![CDATA[Ken Pfeuffer]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 07:28:36 GMT</pubDate>
            <atom:updated>2024-12-27T11:30:01.627Z</atom:updated>
            <content:encoded><![CDATA[<p>Direct and indirect input modes are foundational concepts in modern operating systems (OS), and their integration is redefining how we interact with technology. Devices like the <a href="https://developer.apple.com/design/human-interface-guidelines/gestures#visionOS">Apple Vision Pro</a> are starting to integrate the power of combining these modes, making user interaction more flexible while simplifying the UI. This innovation represents a first step forward since the concept’s introduction in a <a href="https://dl.acm.org/doi/10.1145/2807442.2807460">2015 research paper</a>. In the future, we may move away from the traditional dichotomy of direct versus indirect input, embracing hybrid interaction modes.</p><p>In this article, I’ll explore the scientific background, the core concepts, and the exciting possibilities for hybrid input systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*XLA7b4LZbOtx1kskACANWg.gif" /><figcaption>Direct and indirect (gaze based) touch on a tablet for one- and two-handed tablet usage</figcaption></figure><h3>What Are Direct and Indirect Input Modes?</h3><ul><li><strong>Indirect input devices </strong>(e.g., mice, trackpads) involve a spatial separation between input and display, allowing distant interaction, minimal effort, and less occlusion of content.</li><li><strong>Direct input</strong> <strong>devices </strong>(e.g., touchscreens, hand gestures) combine input and display areas, enabling immediate, intuitive interaction based on real-world metaphors.</li></ul><p>More info? Check out Hinckley and Wigdor’s <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/11/Input-Technologies-and-Techniques-HCI-Handbook-3rd-Edition.pdf">article on input technology</a>.</p><h3>Eye-Tracking and Hybrid Interaction</h3><p>In direct input systems, your gaze typically aligns with where you touch or interact. Conversely, indirect systems decouple gaze and hand movements, as your eyes focus on the cursor rather than your hand. This natural distinction forms the basis for hybrid interaction techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lbQo6tgeNMTkBzvK3xfmpg.jpeg" /><figcaption>Distinct eye-hand coordinations for direct and indirect input devices (adapted from Pfeuffer’s <a href="https://eprints.lancs.ac.uk/id/eprint/89076/">PhD thesis</a>)</figcaption></figure><p>The concept of <a href="https://dl.acm.org/doi/10.1145/2807442.2807460"><strong>Gaze-Shifting</strong></a> was introduced at the UIST 2015 conference to enable seamless transitions between direct and indirect input within the same interface. Here’s how it works:</p><ul><li><strong>Indirect Input</strong>: When you look at an object and perform a gesture from a separate location, the action is redirected to the object you’re focusing on (an example technique is “<a href="https://medium.com/antaeus-ar/design-principles-issues-for-gaze-and-pinch-interaction-a95e251169ae">Gaze + Pinch</a>”).</li><li><strong>Direct Input</strong>: When your hand and gaze align with the same object, the interaction occurs directly at that location.</li></ul><p>This approach mirrors our natural eye-hand coordination patterns, making interactions more expressive for various existing direct input devices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5sUHyuoQNW-LByRj9beACg.jpeg" /><figcaption>Direct and indirect input across three input devices (adapted from <a href="https://eprints.lancs.ac.uk/id/eprint/89076/">P</a>feuffer’s <a href="https://eprints.lancs.ac.uk/id/eprint/89076/">PhD thesis</a>).</figcaption></figure><h3>Application examples</h3><h4>Multimodal Interaction with Tablets</h4><p>For devices like tablets that exploit <a href="https://dl.acm.org/doi/10.1145/1866029.1866036">simultaneous pen-and-touch</a>, gaze-shifting can enhance workflows by integrating direct and indirect input. For instance:</p><ul><li>Drawing with a stylus while looking at the pen tip creates new vertices (direct input).</li><li>Moving the stylus away from the gaze transforms it into a cursor for precision editing (indirect input).</li><li>Pinch-to-zoom gestures allow quick navigation focused on your gaze.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*FlxbYUl1EqUMArWrBTPvjw.gif" /><figcaption>An example where the user engages in direct pen, indirect pen, and indirect touch input in one UI.</figcaption></figure><h3>Eye and Hand Tracking in the Apple Vision Pro</h3><p>The Apple Vision Pro is a recent example of hybrid input in action. Eye and hand tracking allow users to interact with objects directly by touching or indirectly by gazing and pinching. For example, Oleg Frolov’s recent demo shows how users can press a timer button directly or activate it by looking and pinching from a distance.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/Volorf/status/1813177399078514884%3Ft%3DxOOg_0oWqg9spxcRFYWnHw&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/feac0532dd3c9ffd01b41c55bf48ea63/href">https://medium.com/media/feac0532dd3c9ffd01b41c55bf48ea63/href</a></iframe><h3>Transitions between Direct and indirect</h3><p>Hybrid input requires carefully designed transitions to maintain a fluid user experience. Here are four key transitions:</p><ol><li><strong>Direct to Direct</strong>: Switching from one direct input action to another.</li><li><strong>Direct to Indirect</strong>: Moving from a direct input mode to an indirect one.</li><li><strong>Indirect to Indirect</strong>: Transitioning between two indirect input modes.</li><li><strong>Indirect to Direct</strong>: Shifting from an indirect input mode to a direct one.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhiepGLcR92NGRjBMSooUw.jpeg" /><figcaption>Input transitions with direct and indirect input modes</figcaption></figure><h3>Advanced Use Cases</h3><h4>1. Two-Handed Interaction</h4><p>Two-handed interactions open new possibilities for hybrid systems. Inspired by <a href="https://pubmed.ncbi.nlm.nih.gov/15136274/">Guiard’s</a> principles of bimanual asymmetry applied to <a href="https://www.billbuxton.com/input11.2H.pdf">HCI</a>, these techniques divide tasks between the non-dominant hand (NDH) and dominant hand (DH):</p><ul><li><strong>Direct + Indirect</strong>: NDH holds an object while DH manipulates it. This setup is ideal for assembling 3D models.</li><li><strong>Indirect + Direct</strong>: NDH indirectly adjusts the object’s position while DH interacts directly, enabling precise work.</li><li><strong>Indirect + Indirect</strong>: Both hands perform indirect actions, minimizing physical effort for tasks like switching between objects.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/KenPfeuffer/status/1864616222127534346&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fdf1848bbf47311fc8f4f46f4e8961a/href">https://medium.com/media/7fdf1848bbf47311fc8f4f46f4e8961a/href</a></iframe><h4>2. 2D Design with a Stylus and Tablet System</h4><p>Direct to Indirect input transitions can occur in two distinct ways while using a stylus and tablet for 2D design. Initially, you draw with the stylus while your gaze aligns with the stylus tip (Direct input). Then:</p><ol><li>You look toward the menu while keeping your hand in its original position. The stylus switches to indirect input, allowing you to control the menu (Video-left).</li><li>Alternatively, you keep your eyes on the drawing but move your hand away. The stylus toggles to an “editing” mode where a cursor enables precision refinement of strokes (Video-right).</li></ol><h3>Ken Pfeuffer on X (formerly Twitter): &quot;An example of the two atomic transitions between #direct and #indirect input through #eyetracking:Eyes Off: Shift your #gaze to instantly engage with a new #UI element.Hands Off: Move your hand away to switch modes on the same element.From https://t.co/SZWn4A6KmL pic.twitter.com/J7xkSodkxp / X&quot;</h3><p>An example of the two atomic transitions between #direct and #indirect input through #eyetracking:Eyes Off: Shift your #gaze to instantly engage with a new #UI element.Hands Off: Move your hand away to switch modes on the same element.From https://t.co/SZWn4A6KmL pic.twitter.com/J7xkSodkxp</p><h4>3. Gesture Modulation</h4><p>Hybrid input can also refine gesture-based interactions. For example, in a VR interface, users can (X video below, left):</p><ul><li>Start a gesture with direct input (e.g., pinching to grab an object).</li><li>Shift to indirect input during the movement (e.g., repositioning the object with gaze).</li><li>End the gesture with direct input (e.g., placing the object).</li></ul><p>Conversely, you can start with Indirect and end with Direct gestures (Video-right).</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/KenPfeuffer/status/1706754929438654798&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2720709e589e03d7970a60eb34095df/href">https://medium.com/media/e2720709e589e03d7970a60eb34095df/href</a></iframe><h3>Implementation considerations</h3><p>To implement gaze-shifting effectively, developers can use one of two approaches:</p><ol><li><strong>Designated UI Elements</strong>: Specific elements react to the user’s gaze for indirect input, while all other interactions default to direct input.</li><li><strong>Dynamic Alignment Monitoring</strong>: Continuously track the alignment between gaze and hand to classify gestures as direct or indirect in real time.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PkTw1mwVfsNoJdPXZxIufA.jpeg" /><figcaption>Dynamic Gaze-Shifting approach (adapted from <a href="https://dl.acm.org/doi/10.1145/2807442.2807460">UIST’15</a>)</figcaption></figure><p>Indirect input methods offer a variety of models for interacting with objects, unlike direct input, which uses a fixed 1:1 mapping between input position and object manipulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8_StQQD_sG9DJ0mLwpntQ.jpeg" /><figcaption>Several input mappings, image adapted from <a href="https://dl.acm.org/doi/10.1145/2807442.2807460">UIST’15</a></figcaption></figure><h3>Conclusion</h3><p>The integration of direct and indirect input modes is transforming human-computer interaction. By strategically blending these modes, we can:</p><ul><li>Enhance user flexibility and efficiency.</li><li>Create more natural and intuitive interfaces.</li><li>Open new possibilities for creative and professional tools.</li></ul><p>As seen in systems like the Apple Vision Pro, the potential of direct &amp; indirect is just beginning to be realized. Designers and developers have a unique opportunity to shape the future of interaction. See the following diagram for an overview of direct/indirect input states for one- and two-handed interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GqiTNo8G_QJS62KZDKHwbA.jpeg" /><figcaption>Design space for one and two handed control with gaze-modulated direct and indirect input</figcaption></figure><p>What do you think about hybrid input modes? Share your thoughts in the comments below!</p><h3>More information and resources</h3><p>Background information on this topic:</p><ul><li>Direct/indirect input: Hinckley’s chapter on <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/11/Input-Technologies-and-Techniques-HCI-Handbook-3rd-Edition.pdf">Input Technologies and Techniques</a> elaborates on this and further input-related topics.</li><li>Eye-tracking + direct/indirect input: the original article on <a href="https://kenpfeuffer.com/2015/10/31/uist15-gaze-shifting-direct-indirect-input-with-pen-and-touch-modulated-by-gaze/">Gaze-Shifting</a>, the extended discussion in my <a href="https://eprints.lancs.ac.uk/id/eprint/89076/">PhD thesis</a>, and a recent <a href="https://youtu.be/5nkBzeYnprQ?si=myFWRgJu2ozIJnRK">talk</a> on Gaze + Hand Tracking design.</li><li>Apple’s <a href="https://developer.apple.com/design/human-interface-guidelines/gestures#visionOS">human interface guidelines for direct/indirect gestures of the Vision Pro.</a></li></ul><p>This article is part of the eye-hand UI blog series:</p><ul><li><a href="https://medium.com/antaeus-ar/design-principles-issues-for-gaze-and-pinch-interaction-a95e251169ae">Design Principles &amp; Issues for Gaze and Pinch Interaction</a></li><li><a href="https://medium.com/@ken.pfeuffer/eyes-hands-in-ar-a-sci-fi-inspired-mobile-ui-research-7f75b69b15f2">Eyes &amp; hands in AR: A sci-fi-inspired mobile UI research</a></li><li><a href="https://medium.com/@ken.pfeuffer/history-of-eyes-and-hands-for-computer-control-fd3a62b56aa1">History of Eyes and Hands for Computer Control</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30952c9dc785" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[History of Eyes and Hands for Computer Control]]></title>
            <link>https://medium.com/@ken.pfeuffer/history-of-eyes-and-hands-for-computer-control-fd3a62b56aa1?source=rss-efc9299f702e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd3a62b56aa1</guid>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[hci]]></category>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[user-interface]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <dc:creator><![CDATA[Ken Pfeuffer]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 12:20:59 GMT</pubDate>
            <atom:updated>2024-03-28T23:04:18.118Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*cW0OcOQ7YYVo2gA3hzrd5w.jpeg" /><figcaption>Descartes’ 1644 woodcut in “Principles of Philosophy” depicts his vision theory. It illustrates the seamless interplay of the eyes and hands, a fundamental principle that affords a powerful class of computer interfaces.</figcaption></figure><p>Apple’s Vision Pro provides the first operating system fully designed for our eyes and hands. Perhaps the first-ever consumer device. A game-changer for XR like the first IPhone’s “multi-touch” effect that transformed the industry.</p><p>In 1644, René Descartes, a seminal figure in the emergence of science, depicted the eye-hand coordination as a fundamental bridge between human consciousness and the external world, indicating its potential for transformative applications. Fast-forward to 1981, a time where the computer started to gain traction, the first scientific paper on a user interface (UI) that exploits the unity of the eyes and hands was released.</p><p>Since then, many eye-hand techniques have popped up in the Human-Computer Interaction (HCI) literature. But, how much was science ahead, how innovative is Apple’s UI, and what can we learn for our future? See for yourself!</p><p>Here’s a rundown of research on interacting with the computer using both eyes and hands.</p><p><strong>Quick navigation</strong>:</p><ul><li>First papers for <a href="#6681">PC</a>, <a href="#d1d1">Large Displays</a>,<a href="#099d"> Touchscreens</a>, <a href="#84ff">Virtual Reality</a></li><li>All videos in a <a href="https://www.youtube.com/playlist?list=PLzHVWVTeYCWLDPPENhSbIlIKwCVqRrMdW">youtube playlist</a></li></ul><h3>Eyes &amp; Hands for PC</h3><h4>1981: Gaze-orchestrated dynamic windows (Bolt, <a href="https://dl.acm.org/doi/10.1145/965161.806796">SIGGRAPH’81</a>)</h4><p>Display: Large projection<br>Hand input: Gaze + joystick<br>Summary: The system showed a set of running TV episodes on a big projection. It used eye-tracking glasses to see which one you were looking at and mute the others. If you focused on one, it zoomed in on that episode. The user can provide explicit zoom in/out commands by using a joystick.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRVMf_SP7iTc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRVMf_SP7iTc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FRVMf_SP7iTc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb70d24502af357a64a194bd1c31fcf7/href">https://medium.com/media/fb70d24502af357a64a194bd1c31fcf7/href</a></iframe><h4>1986: An Evaluation of an Eye Tracker as a Device for Computer Input (Ware, Mikaelian; <a href="https://doi.org/10.1145/332040.332445">CHI’87</a>)</h4><p>Display: Monitor<br>Hand input: Gaze + keyboard button<br>Summary: The user looks at a target on the screen, and then presses a physical button to select. This basic mechanism was evaluated and found as faster but less precise than full-hands control.</p><h4>1990: What you look at is what you get: eye movement-based interaction techniques (Jacob, <a href="https://dl.acm.org/doi/10.1145/97243.97246">CHI’90</a>)</h4><p>Display: PC monitor<br>Input: Gaze + mouse and keyboard<br>Summary: Integrated gaze and mouse/key techniques for GUIs. Looking and clicking for button selection, moving the mouse to drag objects, and using gaze to open and browse menus. Coined the “Midas Touch” problem: <em>“Ideally, the interface should act on the user’s eye input when he wants it to and let him just look around when that’s what he wants, but the two cases are impossible to distinguish”.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*bvIlO09N9Ft_RDjVoBrRGQ.png" /><figcaption>The “What you look at is what you get” work by Jacob explored gaze+mouse interaction for buttons, menus, and object dragging in 1990.</figcaption></figure><h4>1999: Manual and gaze input cascaded (MAGIC) pointing (Zhai, Morimoto, Ihde, <a href="https://dl.acm.org/doi/10.1145/302979.303053">CHI’99</a>)</h4><p>Display: PC monitor<br>Input: Gaze + mouse<br>Summary: Here, the mouse cursor is warped to the gaze point, after which the user continues with using the mouse. This represents a first, more subtle way of using gaze — to enhance pointing, while the rest is done by the hands only. The underlying idea is that gaze is used sparingly, to avoid overloading the eyes with motor control tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*qE_gy6JFB46gSTTfKySfEA.png" /><figcaption>A depiction of the MAGIC selection algorithm, where a mouse cursor is warped from a previous to the gaze position to lower cursor dragging effort. Picture from <a href="https://dl.acm.org/doi/10.1145/302979.303053">CHI’99</a></figcaption></figure><h4>2000: Intelligent Gaze-Added Interfaces (Salvucci, Anderson; <a href="https://dl.acm.org/doi/10.1145/332040.332444">CHI’00</a>)</h4><p>Display: PC monitor<br>Input: Gaze + keyboard<br>Summary: The GUI’s standard mouse and keyboard interface has been enhanced with a ‘gaze button’, enabling users to perform mouse actions like clicking and dragging using their gaze as the pointer. This integration offers users the flexibility to utilize both the traditional UI controls and the added gaze-based control mechanism simultaneously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*LJpkfxGIBzSTw64Th7apZg.png" /><figcaption>Salvucci and Andersen propose a system where a special gaze button on the keyboard allows to interact with the eyes, in addition to default mouse control. Gaze pointing targets are highlighted in yellow.</figcaption></figure><h4>2005: EyeWindows: Evaluation of Eye-Controlled Zooming Windows for Focus Selection (Fono, Vertegaal; <a href="https://dl.acm.org/doi/10.1145/1054972.1054994">CHI’05</a>)</h4><p>Display: PC monitor<br>Input: Gaze + keyboard<br>Summary: The paper proposes to use gaze to switch between different windows of a GUI. By directing their gaze to a window and pressing a key, it becomes focused. Once chosen, size of the window adjust automatically to a preset scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*M5gplWIP_Ax2gxz1JHsbzw.png" /><figcaption>In EyeWindows, users can rapidly select a window by gaze and a button press. Picture from <a href="https://dl.acm.org/doi/10.1145/1054972.1054994">CHI’05</a></figcaption></figure><h4>2007: EyePoint: practical pointing and selection using gaze and keyboard (Kumar, Paepcke, Winograd; <a href="https://dl.acm.org/doi/10.1145/1240624.1240692">CHI’07</a>)</h4><p>Display: PC monitor<br>Input: Gaze + keyboard<br>Summary: Mouse pointing by gaze is inaccurate. The authors propose the “look-press-look-release” technique for enhanced precision. After looking at an area and pressing down, the area is magnified. Then, the user looks at the desired object in the magnified area and releases the button to finalize selection.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmOfA9Q5oc3M%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmOfA9Q5oc3M&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmOfA9Q5oc3M%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5d40ba24712448e8b1faa43ee547a8d/href">https://medium.com/media/d5d40ba24712448e8b1faa43ee547a8d/href</a></iframe><h4>2008: Improving the accuracy of gaze input for interaction (Kumar, Klingner, Puranik, Winograd, Paepcke; <a href="https://dl.acm.org/doi/10.1145/1344471.1344488">ETRA’08</a>)</h4><p>Display: PC monitor<br>Input: Gaze + keypress<br>Summary: Investigated several methods to improve accuracy of gaze and keypress inputs. For example, synchronisation errors, as users may click before they look at the target or already have looked away from the target when pressing the button, which can be mitigated through smoothing algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*SFdVfv6Ih47Or80ImfEeCw.png" /><figcaption>Kumar et al. study error mitigation of gaze and keypress inputs, such as early (a) and late (d) triggers.</figcaption></figure><h4>2009: The MAGIC Touch: Combining MAGIC-Pointing with a Touch-Sensitive Mouse (Drewes, Schmidt; <a href="https://link.springer.com/chapter/10.1007/978-3-642-03658-3_46">INTERACT’09</a>)</h4><p>Display: PC monitor<br>Input: Gaze + touch-sensitive mouse<br>Summary: Extends the MAGIC principle by allowing a new way to warp the mouse cursor to gaze. When users touch the mouse button, before pressing it, the cursor automatically warps to gaze. This avoids the original MAGIC’s problem of when the cursor will warp.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*4xZL4LiXAqSFRx1Y0EyXLg.png" /><figcaption>The MAGIC Touch technique uses a touch-sensitive button. When touched, the mouse cursor warps to the gaze position. From then, the normal mouse actions take over. Picture from <a href="https://link.springer.com/chapter/10.1007/978-3-642-03658-3_46">paper</a></figcaption></figure><h3>Eyes &amp; Hands for Large Displays</h3><h4>2010: 3D User Interface Combining Gaze and Hand Gestures for Large-Scale Display (Yoo, Han, Choi, Yi, Suh, Park, Kim; <a href="https://dl.acm.org/doi/10.1145/1753846.1754043">CHI EA’10</a>)</h4><p>Display: Large display<br>Input: Gaze + hand gesture<br>Summary: The paper proposes control of a large screen by using eye-tracking and hand gestures. The user can change a cursor position by gaze, and zoom by a push-pull gesture of the hands. Several combinations of two hands are used to provide a command set to manipulate an image gallery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*0EParq8GZci05X_kBlmxUA.png" /><figcaption>This work considered gaze pointing with hand tracking, where different hand movements support a set of commands. Picture from <a href="https://dl.acm.org/doi/10.1145/1753846.1754043">CHI EA’10</a></figcaption></figure><h4>2011: Designing gaze-supported multimodal interactions for the exploration of large image collections (Stellmach, Stober, Nurnberger, Dachselt; <a href="https://dl.acm.org/doi/10.1145/1983302.1983303">NGCA’11</a>)</h4><p>Display: PC Monitor<br>Input: Gaze + keyboard + touch<br>Summary: The work combines gaze support with a fisheye lens, a keyboard, and a tilt-sensitive mobile multitouch device for interaction with a remote display. The user can look at an area, and activate a fisheye lens through keypress or a touch sliding gesture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*VErrjwYlfYiCLOKfqT_8xg.png" /><figcaption>Users can look at an image area, and touch gestures to modulate a fisheye lens. Picture from <a href="https://dl.acm.org/doi/10.1145/1983302.1983303">NGCA’11</a></figcaption></figure><h4>2012: Look &amp; touch: gaze-supported target acquisition (Stellmach, Dachselt; <a href="https://dl.acm.org/doi/10.1145/2207676.2208709">CHI’12</a>)</h4><p>Display: PC monitor<br>Input: Gaze + smartphone touch<br>Summary: This work considered mobile-device touch gestures with gaze for interacting with content on a remote display. Users utilize eye movements to position a cursor while touch-dragging gestures enable precise adjustments to the cursor’s location. This functionality is facilitated by holding a handheld touchscreen device in one hand, where thumb touches control the mouse cursor on the screen.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcTM_AKk1Lrw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcTM_AKk1Lrw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcTM_AKk1Lrw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/06af8a00dac1206ab1bb9726ef2609c2/href">https://medium.com/media/06af8a00dac1206ab1bb9726ef2609c2/href</a></iframe><h4>2012: Gaze and gesture based object manipulation in virtual worlds (Slambekova, Bailey, Geigel; <a href="https://dl.acm.org/doi/10.1145/2407336.2407380">VRST ‘12</a>)</h4><p>Display: PC monitor<br>Input: Gaze + hand gesture<br>Summary: Investigates gaze pointing to a virtual element on-screen and issuing 3D grabbing gestures in mid-air for manipulation. Includes two-handed 3D manipulations for dragging, rotation and scaling of objects.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FzyPWC4yYSMU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DzyPWC4yYSMU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzyPWC4yYSMU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a9eaa71a3360cc8b7bfbb3210559d1f/href">https://medium.com/media/6a9eaa71a3360cc8b7bfbb3210559d1f/href</a></iframe><h4>2013: Still looking: investigating seamless gaze-supported selection, positioning, and manipulation of distant targets (Stellmach, Dachselt; <a href="https://dl.acm.org/doi/10.1145/2470654.2470695">CHI’13</a>)</h4><p>Display: Distant large projector<br>Input: Gaze + touch<br>Summary: Expands on the Look &amp; Touch technique, by supporting dragging, scaling and rotation. The user can look at a target, use touches to refine the cursor position. As well, during drag &amp; drop, the user can use their gaze to move a selected object to a new position.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTUt2yTE_o30%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTUt2yTE_o30&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTUt2yTE_o30%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6708c8cd5c09f68b432b81a01b453ec3/href">https://medium.com/media/6708c8cd5c09f68b432b81a01b453ec3/href</a></iframe><h4>2013: Interacting with Objects in the Environment by Gaze and Hand Gestures (Hales, Mardanbeigi, Rozado; <a href="https://pure.itu.dk/en/publications/interacting-with-objects-in-the-environment-by-gaze-and-hand-gest">ECEM’13</a>)</h4><p>Display: PC monitor, IoT objects in environment<br>Input: Gaze + hand gesture<br>Summary: Presents techniques to operate physical appliances in the environment. The technique involves 1–5 finger poses and their directional motion, which was mapped to window control on a screen and robot movements in the environment.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSGqF1Mi6JGI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSGqF1Mi6JGI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSGqF1Mi6JGI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdcff879326045997fad62a590116479/href">https://medium.com/media/bdcff879326045997fad62a590116479/href</a></iframe><h4>2013: Eye Pull, Eye Push: Moving Objects between Large Screens and Personal Devices with Gaze and Touch (Turner, Alexander, Bulling, Schmidt, Gellersen; <a href="https://link.springer.com/chapter/10.1007/978-3-642-40480-1_11">INTERACT’13</a>)</h4><p>Display: Large display + handheld tablet<br>Input: Gaze + touch (tablet)<br>Summary: Presents techniques to move object between large and personal device. The user can look at an object, and swipe down to move it down to their personal device. A similar touch gesture will return the object to the public display.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*y2G2Zh8ry7YXD_3yUR8RRQ.png" /><figcaption>Example use cases for Eye Push, Eye Pull: users can pull an object by simply looking at that and performing a swipe gesture. Picture from <a href="https://link.springer.com/chapter/10.1007/978-3-642-40480-1_11">INTERACT’13</a></figcaption></figure><h4>2014: Cross-device gaze-supported point-to-point content transfer (Turner, Alexander, Bulling, Gellersen; <a href="https://dl.acm.org/doi/abs/10.1145/2578153.2578155">ETRA’14</a>)</h4><p>Display: Large display + handheld tablet<br>Input: Gaze + mouse touch (tablet)<br>Summary: Investigates the use of gaze to transfer content between large and handheld displays. For this, the user points their eyes at the object, performs a touch (or mouse) action, then looks down to the tablet, and performs another touch (or mouse) action to finish transfer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*atmv6YrkAq9ossHVRur_lA.png" /><figcaption>Setup to study how gaze allows to transfer target from large (1) to small (2) display, supported by touch (left) and mouse (right) actions. Picture from <a href="https://dl.acm.org/doi/abs/10.1145/2578153.2578155">ETRA’14</a></figcaption></figure><h4>2014: GaFinC: Gaze and Finger Control interface for 3D model manipulation in CAD application (Song, Choa, Baek, Lee, Bang; <a href="https://www.sciencedirect.com/science/article/pii/S0010448513001796">CAD’14</a>)</h4><p>Display: PC monitor<br>Input: Gaze + hand gesture<br>Summary: This work focused on 3D CAD application with hand gesture controls, and employed gaze to specify a zooming location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*27-fX6-gl3qV-MtJ.jpg" /><figcaption>The GaFinC system proposed a gesture interface that included gaze-based zooming. Image from <a href="https://www.sciencedirect.com/science/article/pii/S0010448513001796">CAD’14</a></figcaption></figure><h3>Eyes &amp; Hands for Touchscreens</h3><h4>2014: Gaze-touch: combining gaze with multi-touch for interaction on the same surface (Pfeuffer, Alexander, Chong, Gellersen, <a href="https://dl.acm.org/doi/10.1145/2642918.2647397">UIST’14</a>)</h4><p>Display: Touchscreen<br>Input: Gaze + Touch<br>Summary: This work also explored gaze and multi-touch gestures together, using the principle division of labour “gaze selects, touch manipulates” for simplicity. This offers users the familiar gesture set, simply redirected to the object defined by gaze. The simplicity afforded a variety of applications as demonstrated below, and supports neat transitions between direct and indirect gestures.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FA31OTYRKZXg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DA31OTYRKZXg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FA31OTYRKZXg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/87f97b4e029fbc83c14cb17689412326/href">https://medium.com/media/87f97b4e029fbc83c14cb17689412326/href</a></iframe><h4>2015: An Empirical Investigation of Gaze Selection in Mid-Air Gestural 3D Manipulation (Velloso, Turner, Alexander, Bulling, Gellersen; <a href="https://link.springer.com/chapter/10.1007/978-3-319-22668-2_25">INTERACT’15</a>)</h4><p>Display: PC monitor<br>Input: Gaze + hand gesture<br>Summary: The technique involves users looking at an object, and using a pinch gesture to move the object. The study showed that the acquisition time was significantly reduced over hands-only methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/0*6YsNMseJyxVPnn-o" /><figcaption>Velloso et al. studied a system where gaze and pinch manipulates 3D objects on a laptop. Image from <a href="https://link.springer.com/chapter/10.1007/978-3-319-22668-2_25">INTERACT’15</a></figcaption></figure><h4>2015: Gaze+RST: Integrating Gaze and Multitouch for Remote Rotate-Scale-Translate Tasks (Turner, Alexander, Bulling, Gellersen; <a href="https://dl.acm.org/doi/10.1145/2702123.2702355">CHI’15</a>)</h4><p>Display: Large projector<br>Input: Gaze + touch (tablet)<br>Summary: Given gaze to select an object, how can gaze be used to then move an object? The work investigates several variations between touch and gaze based object translation, and provides insights into integrality and separability of the multimodal inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLFdfU7jiJJ-fkixqkd0BQ.png" /><figcaption>A user interacting with eye-gaze and multi-touch gestures on a wall-sized projection screen. Image from <a href="https://dl.acm.org/doi/10.1145/2702123.2702355">CHI’15</a></figcaption></figure><h4>2015: Gaze+Gesture: Expressive, Precise and Targeted Free-Space Interactions (Chatterjee, Xiao, Harrison; <a href="https://dl.acm.org/doi/10.1145/2818346.2820752">ICMI’15</a>)</h4><p>Display: PC monitor<br>Input: Gaze + hand gestures<br>Summary: The paper proposes to combine various hand gestures with gaze pointing for desktop PCs. For precise mouse pointing, a MAGIC-like approach is employed where the eyes point, and dragging gestures refine the cursor point. As well, several other gestures are supported to extend the command set of the user.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FIjV2D69Gpio%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DIjV2D69Gpio&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FIjV2D69Gpio%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d968ac0573788fc984073b3566903554/href">https://medium.com/media/d968ac0573788fc984073b3566903554/href</a></iframe><h4>2015: The Costs and Benefits of Combining Gaze and Hand Gestures for Remote Interaction (Zhang, Stellmach, Sellen, Blake; <a href="https://link.springer.com/chapter/10.1007/978-3-319-22698-9_39">INTERACT’15</a>)</h4><p>Display: Large display<br>Input: Gaze + hand gesture (grab and flick)<br>Summary: Presents a study of an interaction concept for a photo sorting task. The user gazes at an element on-screen, and uses hand grab and flick gestures to manipulate the object. The study showed benefits over an gesture-only approach, as users preferred the method, but it also led to a few errors when transitioning from gaze-hover to hand-selection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wrBRJU0hCL-WhbDJkmRc4A.png" /><figcaption>Setup and interaction concept of Zhang et al., <a href="https://link.springer.com/chapter/10.1007/978-3-319-22698-9_39">INTERACT’15</a></figcaption></figure><h4>2015: Gaze-Shifting: Direct-Indirect Input with Pen and Touch Modulated by Gaze (Pfeuffer, Alexander, Chong, Zhang, Gellersen; <a href="https://dl.acm.org/doi/10.1145/2807442.2807460">UIST’15</a>)</h4><p>Display: Pen + touch display<br>Input: Gaze + pen + touch<br>Summary: Investigated the coupling between gaze and a stylus for design tasks, such as to allow rapid access to peripheral menus. As well, explored three-modal interactions with gaze, pen, and touch. The central focus is the Gaze-shifting concept, allowing users to seamlessly shift between direct and indirect modes of touch (and pen) gestures.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ff8_dtdYSDMg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Df8_dtdYSDMg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ff8_dtdYSDMg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7e2a966e12871fbe71850dc7446ac93/href">https://medium.com/media/d7e2a966e12871fbe71850dc7446ac93/href</a></iframe><h4>2016: Three-Point Interaction: Combining Bi-Manual Direct Touch with Gaze (Simeone, Bulling, Alexander, Gellersen; <a href="https://dl.acm.org/doi/10.1145/2909132.2909251">AVI’16</a>)</h4><p>Display: Large touchscreen<br>Input: Gaze + touch<br>Summary: The technique allows interaction with advanced tasks where users manipulate 3 points in parallel. The points are each defined by the gaze, the right hand’s index finger, and the left hand’s index finger. For example, this can be useful for specifying a cube in all three dimensions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*sx6iJdfvZAwdm9rjgS6Gng.png" /><figcaption>Three-Point Interaction interprets both hands’ touches and the gaze point for parallel 3 point manipulations. Image from <a href="https://dl.acm.org/doi/10.1145/2909132.2909251">AVI’16</a>.</figcaption></figure><h4>2016: Partially-indirect Bimanual Input with Gaze, Pen, and Touch for Pan, Zoom, and Ink Interaction (Pfeuffer, Alexander, Gellersen; <a href="https://dl.acm.org/doi/10.1145/2858036.2858201">CHI’16</a>)</h4><p>Display: Large touchscreen<br>Input: Gaze + touch + pen<br>Summary: This study explores asymmetric bimanual interaction using pen and touch, focusing on utilizing gaze to select the pivot for indirect touch-based pinch-to-zoom with the non-dominant hand. Combining direct pen and indirect touch avoids physical interference when both hands manipulate directly. Additionally, continuous gaze at the point of interest enhances zoom accuracy, reducing the need for frequent panning actions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQD2iaokmc9U%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQD2iaokmc9U&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQD2iaokmc9U%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a355774d46d0339744f835af923dc47/href">https://medium.com/media/7a355774d46d0339744f835af923dc47/href</a></iframe><h4>2016: Gaze and Touch Interaction on Tablets (Pfeuffer, Gellersen; <a href="https://dl.acm.org/doi/10.1145/2984511.2984514">UIST’16</a>)</h4><p>Display: Tablet<br>Input: Gaze + touch<br>Summary: This work explored gaze and touch interaction on a mobile touchscreen device in form of a tablet. This is focusing on a more specific but realistic context, when holding the device with one hand. Users mainly interact with brief thumb presses of the gripping hand, with which users can interact with the whole screen through their gaze. The work also proposes Cursor-Shift, a method that allows switching between direct manipulation and precise cursor control, e.g., for browser applications.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBuvgizcmQuk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBuvgizcmQuk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBuvgizcmQuk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e37ca6f811edc4bcc3a8ec9227f31491/href">https://medium.com/media/e37ca6f811edc4bcc3a8ec9227f31491/href</a></iframe><h4>2016: GazeArchers: playing with individual and shared attention in a two-player look&amp;shoot tabletop game (Pfeuffer, Alexander, Gellersen; <a href="https://dl.acm.org/doi/10.1145/3012709.3012717">MUM’16</a>)</h4><p>Display: Large touchscreen<br>Input: Gaze + Touch<br>Summary: A towerdefense game is presented that uses the eyes and touch inputs for a look&amp;shoot game. Various enemies are running up the tabletop-screen, and have to be defeated by a look and a tap. Some game units have shields that are directed toward a looking user, requiring teamwork to accomplish the game.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmbJ2eZRX-0g%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmbJ2eZRX-0g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmbJ2eZRX-0g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f58cba410441ee60709bf8fd20ffbe4/href">https://medium.com/media/8f58cba410441ee60709bf8fd20ffbe4/href</a></iframe><h3>Eyes &amp; Hands for Virtual Reality</h3><h4>2017: Gaze + Pinch Interaction in Virtual Reality (Pfeuffer, Mayer, Mardanbegi, Gellersen; <a href="https://dl.acm.org/doi/10.1145/3131277.3132180">SUI’17</a>)</h4><p>Display: 3D / VR<br>Input: Gaze + pinch gesture<br>Summary: This work proposed the gaze + pinch paradigm for VR, where the eyes select targets and the hands specifically perform pinch gestures. At its core lies a principle division of labour of <em>gaze selects, hands manipulate</em>, which simplifies the interaction through familiar gestures that can be triggered over distance. The work also offers a blueprint for an operating system for buttons, menus and applications. Numerous example prototypes demonstrate the power of the interaction paradigm, that resembles Apple Vision Pro’s UI.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FNzLrZSF8aDM&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DNzLrZSF8aDM&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FNzLrZSF8aDM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/58ea5e2b3e5794a3199e7e7b6183795c/href">https://medium.com/media/58ea5e2b3e5794a3199e7e7b6183795c/href</a></iframe><h4>2017: Understanding the impact of multimodal interaction using gaze informed mid-air gesture control in 3D virtual objects manipulation (Deng, Jiang, Chang, Guo, Zhang; <a href="https://www.sciencedirect.com/science/article/pii/S1071581917300629">IJHCS’17</a>)</h4><p>Display: PC monitor<br>Input: Gaze + hand gesture<br>Summary: The paper tackes the misperception problem for manipulating gaze-selected objects by hand gesture. I.e., the hand can get out of range when performing indirect dragging gestures. The authors study the problem with several variations and provide insights into their usability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-K8ngsooR8cUg2DVztHVw.png" /><figcaption>This paper investigates the problem of an offset between hand and gaze point of indirect gestures.</figcaption></figure><h4>2018: Pinpointing: Precise Head- and Eye-Based Target Selection for Augmented Reality (Kyto, Ens, Piumsomboon, Lee, Billinghurst; <a href="https://dl.acm.org/doi/10.1145/3173574.3173655">CHI’18</a>)</h4><p>Display: 3D / AR<br>Input: Gaze + controller, gesture<br>Summary: This work proposes a set of precision selection techniques for systems that require precise eye input. Users can select a button by looking at it, and then using a freehand pinch or controller movement to refine the initial gaze cursor.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fq9cbAfxKAPI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dq9cbAfxKAPI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fq9cbAfxKAPI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4c1e5127fe0863f0f61cff4ab7d15f3/href">https://medium.com/media/f4c1e5127fe0863f0f61cff4ab7d15f3/href</a></iframe><h4>2019: GazeButton: enhancing buttons with eye gaze interactions (Rivu, Abdradou, Mayer, Pfeuffer, Alt; <a href="https://dl.acm.org/doi/10.1145/3317956.3318154">COGAIN@ETRA’19</a>)</h4><p>Display: Touchscreen<br>Input: Gaze + touch<br>Summary: Studied how general (touch) buttons can be extended with eye-tracking. Buttons can be enhanced from 2 states (click on button vs click somewhere else) to 4, as each state can distinguish further if the user looked at the button or somewhere else. Demonstrated the concepts in a GazeButton for a keyboard to allow various functionality.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKgg1XLAj-QY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKgg1XLAj-QY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKgg1XLAj-QY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/aac5df7e2ca4cd99c93f6af3ef812dd4/href">https://medium.com/media/aac5df7e2ca4cd99c93f6af3ef812dd4/href</a></iframe><h4>2019: <strong>EyeSeeThrough: Unifying Tool Selection and Application in Virtual Environments (</strong>Mardanbegi, Mayer, Pfeuffer, Jalaliniya, Gellersen, Perzl; <a href="https://ieeexplore.ieee.org/document/8797988">IEEE VR’19</a>)</h4><p>Display: 3D / VR<br>Input: Gaze + controller<br>Summary: EyeSeeThrough is a user interface technique that uses the eyes and controller input in concert to apply modes to objects in the 3D scene, such as to specify a new color to a graphical object. By aligning the hand menu with the target object in the user’s line of sight and performing a button click, EyeSeeThrough enables users to intuitively apply modes by “seeing through” the mode toward the target, without the need for a sequential two-step process of mode selection and application. A button click or dwell-time confirms the action.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FjRSh6qOtFOE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DjRSh6qOtFOE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FjRSh6qOtFOE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b89dc8e47de559a19d842a9b98394e23/href">https://medium.com/media/b89dc8e47de559a19d842a9b98394e23/href</a></iframe><h4>2020: Empirical Evaluation of Gaze-enhanced Menus in Virtual Reality (Pfeuffer, Mecke, Delgado Rodriguez, Hassib, Maier, Alt; <a href="https://dl.acm.org/doi/10.1145/3385956.3418962">VRST’20</a>)</h4><p>Display: 3D / VR<br>Input: Gaze + controller<br>Summary: This work explores gaze in conjunction with a controller for compound drawing and menu tasks. Focus is on asymmetric bimanual interaction: the dominant hand draws with direct manipulation, and the non-dominant hand holds a color palette menu. Gaze is used to implicitly change menu modes when looking at them. In their study, direct manipulation by the controller led to the highest performance, and the gaze techniques are comparable to pointer-based menu selection, with less physical effort.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FDyEWX4SF0QE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DDyEWX4SF0QE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FDyEWX4SF0QE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9d01deacc5d2579ffc00a5d2a02e1c6/href">https://medium.com/media/e9d01deacc5d2579ffc00a5d2a02e1c6/href</a></iframe><h4>2021: Gaze-Supported 3D Object Manipulation in Virtual Reality (Yu, Lu, Shi, Liang, Dingler, Velloso, Goncalves; <a href="https://dl.acm.org/doi/10.1145/3411764.3445343">CHI’21</a>)</h4><p>Display: 3D / VR<br>Input: Gaze + controller<br>Summary: Here, the user can look at objects and use the controller to select and drag the object in space. Several translation methods are studied that mix eye and controller inputs to rapidly move objects. No significant differences were revealed in user performance between controller and controller+gaze.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmjhxPR1Q9lU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmjhxPR1Q9lU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmjhxPR1Q9lU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9845056622d1497618eb9dcd4bf9b7f8/href">https://medium.com/media/9845056622d1497618eb9dcd4bf9b7f8/href</a></iframe><h4>2021: Pinch, Click, or Dwell: Comparing Different Selection Techniques for Eye-Gaze-Based Pointing in Virtual Reality (Mutasim, Batmaz, Stuerzlinger; <a href="https://dl.acm.org/doi/10.1145/3448018.3457998">ETRA’21</a>)</h4><p>Display: 3D / VR<br>Input: Gaze + button, pinch gesture<br>Summary: The work evaluated three ways to confirm a target that has been fixated by the eyes. Results showed that both pinch and button-click confirmation led to higher performance than an eyes-only dwell-time approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x3uq399OyENna43l3wLq8w.png" /><figcaption>The Pinch, Click, or Dwell work investigates different confirmation methods for gaze pointing in VR. Picture from ETRA’21.</figcaption></figure><h4>2022: Gaze-Hand Alignment: Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality (Lystbæk, Rosenberg, Pfeuffer, Grønbæk, Gellersen; <a href="https://dl.acm.org/doi/10.1145/3530886">ETRA’22</a>)</h4><p>Display: 3D / AR<br>Input: Gaze + hand alignment, gaze + pinch<br>Summary: This work proposes a new eye-hand selection mechanism where users spatially align their eye gaze ray with a ray defined by the hand. With Gaze &amp; Finger, users move their index finger in line of sight. With Gaze &amp; Hand, a hand-ray cursor is aligned with the gaze position. An evaluation showed that both methods are on par with gaze + pinch, and all multimodal techniques were superior to a hands only approach (point &amp; pinch).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FU0T_wd_VD9w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DU0T_wd_VD9w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FU0T_wd_VD9w%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ccb97f70cf7d88be51fb4152d5e9a71/href">https://medium.com/media/3ccb97f70cf7d88be51fb4152d5e9a71/href</a></iframe><h4>2022: Exploring Gaze for Assisting Freehand Selection-based Text Entry in AR (Lystbæk, Rosenberg, Pfeuffer, Grønbæk, Gellersen; <a href="https://dl.acm.org/doi/10.1145/3530882">ETRA’22</a>)</h4><p>Display: 3D / AR<br>Input: Gaze + hand alignment<br>Summary: This work investigated the gaze-hand alignment concept for text entry, which is a different problem with selection frequency. The original techniques were extended to make them work for text entry, using specific timers and constraints. A study compared the technique to direct tapping on the keyboard. Results show that it leads to significantly less physical movement, without compromising performance.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FafR63hnvOPo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DafR63hnvOPo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FafR63hnvOPo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba33779774ff2cbe87824a68942de9b0/href">https://medium.com/media/ba33779774ff2cbe87824a68942de9b0/href</a></iframe><h4>2022: Look &amp; Turn: One-handed and Expressive Menu Interaction by Gaze and Arm Turns in VR (Reiter, Pfeuffer, Esteves, Mittermeier, Alt; COGAIN@<a href="https://dl.acm.org/doi/10.1145/3517031.3529233">ETRA’22</a>)</h4><p>Display: 3D / VR<br>Input: Gaze + pinch, hand turn motion<br>Summary: This project centers around enhancing expressive drawing tools alongside a sophisticated hand menu interface. Users can seamlessly control the hand menu using both gaze and hand gestures. For precise input, users simply fix their gaze on a target, pinch, and smoothly adjust the desired value by rotating their arm until reaching the exact setting.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FzNIL0saAGsw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DzNIL0saAGsw&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a832e235f145f119318033036f3a14b0/href">https://medium.com/media/a832e235f145f119318033036f3a14b0/href</a></iframe><h4>2023: A Fitts’ Law Study of Gaze-Hand Alignment for Selection in 3D User Interfaces (Wagner, Lystbæk, Manakhov, Grønbæk, Pfeuffer, Gellersen, <a href="https://dl.acm.org/doi/10.1145/3544548.3581423">CHI’23</a>)</h4><p>Display: 3D / AR<br>Input: Gaze + Pinch, Gaze + Finger, Gaze + Handray<br>Summary: This work provides empirical evidence of gaze + pinch interaction in contrast to other multimodal and hands-only techniques in 3D. Results show that all eye-hand techniques led to superior performance than the manual techniques (hand-ray, headcrusher).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXiCkV3eaHOY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXiCkV3eaHOY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXiCkV3eaHOY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9255cd7955868ae1eeefbd97aaa23f96/href">https://medium.com/media/9255cd7955868ae1eeefbd97aaa23f96/href</a></iframe><h4>2023: PalmGazer: Unimanual Eye-hand Menus in Augmented Reality (Pfeuffer, Obernolte, Dietz, Mäkelä, Sidenmark, Manakhov, Pakanen, Alt; <a href="https://dl.acm.org/doi/abs/10.1145/3607822.3614523">SUI’23</a>)</h4><p>Display: 3D / AR<br>Input: Gaze + pinch gesture<br>Summary: Users can interact with a handheld menu using gaze and pinch. The focus is on mobile interaction, particularly one-handed, akin to holding and interacting with a smartphone. Users initiate the UI with a palm-up gesture and subsequently issue commands through gaze and pinch gestures with the same hand.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ffqx7mrzHE90%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dfqx7mrzHE90&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ffqx7mrzHE90%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5dc1e9634e147d88516134b7cc91eedc/href">https://medium.com/media/5dc1e9634e147d88516134b7cc91eedc/href</a></iframe><h4>2023: Compass+Ring: A Multimodal Menu to Improve Interaction Performance and Comfortability in One-handed Scenarios (Chen, Guo, Feng, Chen, Liu; <a href="https://ieeexplore.ieee.org/abstract/document/10316414">ISMAR’23</a>)</h4><p>Display: 3D / AR<br>Input: Gaze + Pinch, Gaze + Finger, Gaze + Handray<br>Summary: Compass+Ring is a pie menu design based on gaze, voice, and hand rotation for efficient and low-effort interaction. To interact, the technique supports three steps: 1) the user looks at one of the pie menus making it move closer, 2) they confirm their choice by saying “select”, and 3) they rotate the wrist to specify a mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e8vSi2u_B9-AsQRoAldqjw.png" /><figcaption>The Compass+Ring menu allows users to summon and select menu items through multimodal gaze, speech, and hand rotation inputs.</figcaption></figure><h4>2023: GazeHand: A Gaze-Driven Virtual Hand Interface (Jeong, Kim, Yang, Lee, Kim; <a href="https://ieeexplore.ieee.org/document/10332195">ACCESS’23</a>)</h4><p>Display: 3D / AR<br>Input: Gaze + pinch gesture<br>Summary: This work presents a technique where gaze indicates a distant target, at which then the virtual hands are warped closely to the target. The idea is that it will add more precision in the far space, in contrast to a gaze-selection of far targets. The study showed that a head-pointing variant was found as more stable.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FydGWXHLDtP8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DydGWXHLDtP8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FydGWXHLDtP8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c8d37afb74e3f78a0474400ad6bce19/href">https://medium.com/media/4c8d37afb74e3f78a0474400ad6bce19/href</a></iframe><h4>2023: Exploring gaze-assisted and hand-based region selection in augmented reality (Shi, Wei, Qin, Hui, Liang, Hai-Ning; <a href="https://ieeexplore.ieee.org/abstract/document/10332195"><em>Proc. HCI</em></a><em>’23)</em></h4><p>Display: 3D / AR<br>Input: Gaze + finger, pinch<br>Summary: This work inspects region selection, a two-step approach to define two corners to specify a rectangular area. In this specific evaluation, the unimodal techniques outperformed the multimodal ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*2ldD44xjApFrfrMJ0TWJaA.png" /><figcaption>For the task of region selection, this paper studied four different techniques based on eyes &amp; hands.</figcaption></figure><h3>The end is the beginning</h3><p>The quest for eye-hand interaction went on for decades across PC, touch devices, and XR environments, culminating in the inclusion in the Vision Pro. But it may just be the beginning to a new era in human interaction, where our eyes will take a major role in many of the actions that have been reserved for our hands so far.</p><p>Would you like to know more? Check out my recent articles on <a href="https://medium.com/antaeus-ar/design-principles-issues-for-gaze-and-pinch-interaction-a95e251169ae">Design Principles and Issues for Gaze + Pinch</a>, <a href="https://medium.com/@ken.pfeuffer/eyes-hands-in-ar-a-sci-fi-inspired-mobile-ui-research-7f75b69b15f2">the PalmGazer</a> work, and the <a href="https://kenpfeuffer.com/eye-hand-symbiosis-what-guide/">Eye-Hand Symbiosis</a> project.</p><p><em>Note: I do not claim any rights for the videos and images in this article. All videos are on linked to Youtube, and all images have been taken from the respective research paper.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd3a62b56aa1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design Principles & Issues for Gaze and Pinch Interaction]]></title>
            <link>https://medium.com/antaeus-ar/design-principles-issues-for-gaze-and-pinch-interaction-a95e251169ae?source=rss-efc9299f702e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a95e251169ae</guid>
            <category><![CDATA[apple-vision-pro]]></category>
            <category><![CDATA[hci]]></category>
            <category><![CDATA[vr]]></category>
            <category><![CDATA[eye-tracking]]></category>
            <category><![CDATA[ar]]></category>
            <dc:creator><![CDATA[Ken Pfeuffer]]></dc:creator>
            <pubDate>Thu, 18 Jan 2024 16:23:35 GMT</pubDate>
            <atom:updated>2024-07-04T06:18:08.050Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FjsvK3q1OLe4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DjsvK3q1OLe4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FjsvK3q1OLe4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a23802de7e90e5fc5a5bbeb2366db0f/href">https://medium.com/media/9a23802de7e90e5fc5a5bbeb2366db0f/href</a></iframe><p><em>Disclaimer: We have also extended this article as a research paper in Computer Graphics &amp; Applications with more details and background: </em><a href="https://ieeexplore.ieee.org/document/10568496"><em>https://ieeexplore.ieee.org/document/10568496</em></a><em> .</em></p><p>Controllers, hand gestures, eye movements, and voice: many ways to click buttons in virtual reality environments. What about: glance at a UI object with your eyes, then simply pinch with your fingers to activate it. Apple innovates with the first wide adoption of this interaction style for their Vision Pro spatial computer. As well the Hololens 2 and Magic Leap offered similar functionalities. But Apple, renowned for stellar product design, may nail it. Early users are raving about the <a href="https://twitter.com/raywongy/status/1667637329941413895">mind-blowing</a> and <a href="https://www.youtube.com/watch?v=jsvK3q1OLe4&amp;ab_channel=WaveformClips">telepathic</a> technology.</p><p>To shed light on the interaction design, we present 5 design principles and 5 design issues. These are based on human-computer interaction research, mostly the paper “Gaze + Pinch Interaction in Virtual Reality” presented at the 2017 Spatial User Interfaces symposium. We’ll see how much Apple has considered the scientific roots when we get our hands on it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xv_CLEeHX3jwu1WAe8ShXg.jpeg" /><figcaption>Basic operation of Gaze + Pinch in contrast to direct manipulation. Adapted from Pfeuffer et al. SUI’17.</figcaption></figure><p>Principles:</p><ol><li><a href="#c9c9">Division of labor: The eyes select, the hands manipulate</a></li><li><a href="#0b53">Minimalistic multimodal timing</a></li><li><a href="#3e75">Flexibility of gestural commands</a></li><li><a href="#c113">Infallible eyes</a></li><li><a href="#9169">Compatibility with hand-based interfaces</a></li></ol><p>Issues:</p><ol><li><a href="#a3b9">(Un)Learning</a></li><li><a href="#0dfb">Early and late triggers</a></li><li><a href="#296d">Control-display ratio</a></li><li><a href="#594d">Drag &amp; drop sequences</a></li><li><a href="#7930">Continuous eye-selection</a></li></ol><p><a href="#b9c0">Summary</a></p><p>Sources and further material:</p><ol><li><a href="#6470">Author &amp; colleagues’ research</a></li><li><a href="#b3fd">Related research literature</a></li><li><a href="#5388">Design guidelines and tutorials</a></li><li><a href="#8609">Related Medium articles</a></li></ol><h3>Design Principles</h3><h4>1. Division of labor: The eyes select, the hands manipulate</h4><p>Our eyes’ natural role involves indicating points of interest, and we can easily look at any point at will. In contrast, the hands are adept at physical manipulation through the interplay of finger movement and hand posture. Use a clear separation of concerns: the eyes perform selection tasks, the hands do the actual work. This avoids the pitfalls of (i) <a href="https://dl.acm.org/doi/10.1145/302979.303053">overloading the eyes with motor control tasks</a> — here you only actively “use” the eyes to select, and (ii) <a href="https://dl.acm.org/doi/10.1145/2556288.2557130">physical fatigue</a> — gaze pointing minimizing the hands’ physical motion needs.</p><p>The hands, then, make indirect gestures. This is similar to a controller in having the ability to interact at a distance, but now with intuitive pinch gestures. Actually, there are hands-only techniques for selection and manipulation, such as the defaults of the <a href="https://developer.oculus.com/resources/hands-design-interactions/">Meta Quest</a> series and <a href="https://learn.microsoft.com/en-us/windows/mixed-reality/design/point-and-commit">Hololens 2</a>. Yet, assigning both selection and manipulation to the hand can be susceptible to hand jitter issues (the <a href="https://dl.acm.org/doi/abs/10.1145/3313831.3376876">Heisenberg problem</a>), and studies (<a href="https://dl.acm.org/doi/10.1145/3544548.3581423">a</a>, <a href="https://dl.acm.org/doi/10.1145/3530886">b</a>) showed that combining eyes and hands leads to improved performance and comfort.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/KenPfeuffer/status/1674198938650390528%3Fs%3D20&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/23777b309e8ef93038358a76011b91f4/href">https://medium.com/media/23777b309e8ef93038358a76011b91f4/href</a></iframe><h4>2. Minimalistic multimodal timing</h4><p>A single moment in time matters for the eyes—the moment that the index finger and thumb have first contact, one has to fixate on the desired target. Afterward, whether it’s a drag, pan, or zoom gesture, the eyes relinquish control. This affords the freedom to inspect the surroundings independently and avoids accidental actions by eye input (<a href="https://dl.acm.org/doi/10.1145/97243.97246">‘Midas Touch’ problem</a>). For instance, in drag &amp; drop after selection, one can freely look around to locate the destination for the dragged object and follow with the hand via indirect control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ZYRE7mxY5tUA0IQIxj6oA.jpeg" /><figcaption>Eye-tracking as input is only active the moment that a pinch gesture is registered to avoid erratic behaviours.</figcaption></figure><p>The hands alone allow you to point without continuously monitoring the target. Gaze + Pinch inverses the relationship as the eyes must be on the target but the hands can be anywhere. Employing gaze minimally facilitates a quick adaptation of this new relationship. Beginners may find themselves more attentive to ensure their gaze is on the UI element until receiving the right feedback. More experienced users may swiftly execute a Gaze + Pinch command even without having fully perceived the target and its selection feedback.</p><h4>3. Flexible gesture support</h4><p>Hand gestures are in control of virtual objects acquired by the eyes. This can flexibly extend to all atomic classes of the hand-based manipulation — one vs. two-handed interactions, and single vs. multiple target manipulation. Users can seamlessly shift between 1 or two objects and hands by simply re-engaging pinch gestures as desired. This is inherited from the default hand-tracking gestures— but the indirection by gaze renders all those basic hand actions extremely lightweight and flexible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YRmWuji81uXsOoB5" /><figcaption>Fundamental interaction classes with Gaze + Pinch. Taken from Pfeuffer et al. SUI’17.</figcaption></figure><h4>4. Infallible eyes</h4><p>Our hands adhere to a speed-accuracy trade-off: faster means less accurate. In contrast, the eyes can fixate on a target in almost instant time, with constant accuracy given by the eye-tracking sensor. From a user’s perspective, the eyes are infallible: hand pointing can miss or overshoot a target, but the eyes can’t miss as we are either on-target or we look elsewhere. The content design may need to be adjusted, to account for the altered attention of the user during the computer interaction tasks, as well as how UIs support undo and error-recovery measures.</p><p>Large buttons invite wandering around with the eyes, potentially leading to outliers. Drawing the most salient parts to the center of the button will be welcomed by the selection mechanism, and a generous buffer space around targets makes outliers less impactful. If smaller targets are required, hand-refinement and cursor extensions can be integrated with Gaze + Pinch, although departing from the original simplicity.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/KenPfeuffer/status/1747662769984287005%3Fs%3D20&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5444a77b8a9f14596424173eb2dc63c/href">https://medium.com/media/b5444a77b8a9f14596424173eb2dc63c/href</a></iframe><h4>5. Compatibility with hand-based interfaces</h4><p>The motto is to get the best of both worlds. Gaze + Pinch can also work together with the direct gestures in nearspace. This is possible via <a href="https://dl.acm.org/doi/10.1145/2642918.2647397">mode-switching methods</a> that use time and space multiplexing of the inputs. For time-multiplexing, imagine a user opens an app with a menu through Gaze + Pinch, which activates the app UI in front of the user. The user switches to direct touch gestures to scroll the app’s content. In space multiplexing, picture holding a menu with one hand while using Gaze + Pinch commands with the other, enabling direct and indirect inputs at the same time for on-hand virtual menus. Hand menus usually position menus off the hand to prevent hand-tracking interference. Indirect gestures are spatially separated, avoiding hand overlap. It’s one of the intriguing outcomes when UI systems support transitions between complementary modes of interaction.</p><h3>Ken Pfeuffer on Twitter: &quot;3D UI concepts that mix direct&amp;indirect hand actions for spatial design.Right: Gaze&amp;Pinch to get object from a worldmenu, place them on the table using direct gesture.Left: Direct gesture to get object from a hand menu, place them with Gaze&amp;Pinch in the room.#VR #AR #UIUX pic.twitter.com/tnLEbLCL1j / Twitter&quot;</h3><p>3D UI concepts that mix direct&amp;indirect hand actions for spatial design.Right: Gaze&amp;Pinch to get object from a worldmenu, place them on the table using direct gesture.Left: Direct gesture to get object from a hand menu, place them with Gaze&amp;Pinch in the room.#VR #AR #UIUX pic.twitter.com/tnLEbLCL1j</p><h3>Some issues</h3><h4>1. (Un)Learning</h4><p>When we want to get something, we intuitively move toward it. Gaze + Pinch commands don’t need this movement anymore, which can be considered almost counter-intuitive. Hence it is important to balance the advantage of using a new way that requires re-thinking of the action process, over just using the hands without the eyes. In a sense, Gaze + Pinch does not mean learning a new way of interaction — it’s about unlearning the common way: don’t move your hands, just confirm right where you are with your hand.</p><h4>2. Early and late triggers</h4><p>A <a href="https://dl.acm.org/doi/10.1145/1344471.1344488">known problem</a> of multimodal interaction is that the eyes may leave the target before a manual command is registered, or the command is issued just before the gaze lands on the target. It’s possible to have a predictive and generous timing, e.g., using the last fixation (~200–300ms of a stable gaze), rather than the current gaze coordinate (see <a href="https://doi.org/10.3758/s13428-016-0738-9">fixation detection methods</a>). If the early or late trigger frequency is known for a specific UI, the timing can be adapted.</p><h4>3. Control-display ratio</h4><p>With direct manipulation the dragged object translates 1:1 with the hand. With Gaze + Pinch, moving a distant object at a 1:1 mapping makes it feel very slow. What one can do is amplify the speed in the transfer function with the object distance. Or, use visual angle to determine dragging speed as a distance-independent metric: If your hand moves by 5 degrees in your FoV, the remote object corresponds with 5 degree motion. This works well for objects at a distance but may be confusing when targets are near — here the UI can revert to a 1:1 transfer function.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/KenPfeuffer/status/1675851939286855685%3Fs%3D20&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7b69fb84f6fb9e6d5930fd7faf337e8/href">https://medium.com/media/b7b69fb84f6fb9e6d5930fd7faf337e8/href</a></iframe><h4>4. Drag &amp; drop sequences</h4><p>In regular interactions, dropping something means your hand is right there for you to pick it up again. But with Gaze + Pinch, you can look away after dropping and finding it again means you have to look back. This can be a hassle, especially if turning your body is involved. So, when designing interfaces, it’s crucial to think about what type of tasks are supported — ideally, most tasks require only a single action to finish drag &amp; drop, and for sequences consider potential enhancements. Hand tracking systems can make sure not to disengage the target from the control of a pinch gesture if just briefly undetected, to keep dragging robust and avoid object loss.</p><h4>5. Continuous eye-selection</h4><p>With a two-handed pinch-to-zoom gesture, one can use the eyes continuously to refine the zooming pivot. Doing this will put more responsibility on the eyes, which for zooming and related tasks can feel natural. Less panning actions can be the result, if the end of a zooming gesture means that we zoomed closer to where we wanted to than when we used the hands.</p><h3>Ken Pfeuffer on Twitter: &quot;How does eye interaction transform zoom operations in the UI?Left: zoom by 🤏 creates an offset between the target and finger position.Right: With👁️, you can maintain the visual focus on the intended target while zooming.From: https://t.co/N9Frt3zNJo#UIUX #eyetracking pic.twitter.com/dL3h0uL3Kw / Twitter&quot;</h3><p>How does eye interaction transform zoom operations in the UI?Left: zoom by 🤏 creates an offset between the target and finger position.Right: With👁️, you can maintain the visual focus on the intended target while zooming.From: https://t.co/N9Frt3zNJo#UIUX #eyetracking pic.twitter.com/dL3h0uL3Kw</p><h3>Summary</h3><p>The interaction design for the eyes and hands is a novel space that is gradually gaining momentum. Gaze + Pinch or the Vision Pro UI are examples with a mix of underlying design principles and issues, that may just hit the right balance for transforming the ways we have been using our hands after all. It will be exciting to see how the principles will apply in practice, and what else may be ahead in the coming years for XR UX. To read more, check out novel eye-hand concepts such as <a href="https://dl.acm.org/doi/abs/10.1145/3530886">Gaze-Hand Alignment</a>, <a href="https://medium.com/@ken.pfeuffer/eyes-hands-in-ar-a-sci-fi-inspired-mobile-ui-research-7f75b69b15f2">Gaze + Pinch to mobile hand menus</a>, a short video for <a href="https://www.youtube.com/watch?v=OLuiePR5p5A">Gaze + Pinch applications</a>, and more below.</p><h3>Sources and further material</h3><p>The article is primarily based on the author’s experiences, notes, and thoughts, with some knowledge of the human-computer interaction (HCI) research field.</p><h4>Author &amp; colleagues’ research &amp; projects</h4><ul><li>Paper on Gaze + pinch interaction in virtual reality (SUI ‘17). By Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi, and Hans Gellersen. <a href="https://doi.org/10.1145/3131277.3132180">doi</a>, <a href="https://kenpfeuffer.files.wordpress.com/2015/02/p99-pfeuffer.pdf">pdf</a>, <a href="https://youtu.be/NzLrZSF8aDM">video</a>, <a href="https://youtu.be/YdKT42tZdQE?si=iqpt85kzf2-9Lnf5">talk</a></li><li>Eye-Hand Symbiosis project: <a href="https://kenpfeuffer.com/eye-hand-symbiosis-what-guide/">https://kenpfeuffer.com/eye-hand-symbiosis-what-guide/</a></li><li>The XI team at Aarhus University: <a href="https://kenpfeuffer.com/extended-interactionxi-team/">https://kenpfeuffer.com/extended-interactionxi-team/</a></li><li>The GEMINI project studies eye and body motion: <a href="https://gemini-erc.eu/">https://gemini-erc.eu/</a></li></ul><h4>Related research literature</h4><p>Important precursory research has been done, <a href="https://dl.acm.org/doi/10.1145/3491207">among many others</a>, by Sophie Stellmach et al. (<a href="https://dl.acm.org/doi/abs/10.1145/2207676.2208709">CHI’12</a>–<a href="https://dl.acm.org/doi/abs/10.1145/2470654.2470695">13</a>) and Jayson Turner et al. (<a href="https://link.springer.com/chapter/10.1007/978-3-642-40480-1_11">INTERACT’13</a>, MUM’13, <a href="https://dl.acm.org/doi/abs/10.1145/2578153.2578155">ETRA’14</a>, <a href="https://dl.acm.org/doi/abs/10.1145/2702123.2702355">CHI’15</a>), on gaze and touch gesture for large displays. Zhai et al’s <a href="https://dl.acm.org/doi/10.1145/302979.303053">MAGIC</a> (CHI’99), Jacob’s <a href="https://dl.acm.org/doi/10.1145/97243.97246">What You Look At Is What You Get</a> (CHI’90), and Bolt’s <a href="https://dl.acm.org/doi/10.1145/965161.806796">Gaze-orchestrated dynamic windows</a> (SIGGRAPH’81) are early influential entries in the field.</p><h4>Design guidelines and tutorials</h4><ul><li>For Microsoft’s Mixed Reality Toolkit (MRTK): <a href="https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-overview/architecture/interactors">https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-overview/architecture/interactors</a></li><li>For the Meta Quest Pro (by Dilmer Valecillos), for Unity <a href="https://youtu.be/2xWhBKn7Wp0?si=Aq2FVPdkPYVv-tm8">https://youtu.be/2xWhBKn7Wp0?si=Aq2FVPdkPYVv-tm8</a>, for ShapesXR <a href="https://youtu.be/rkKGfp1PZ3c?si=NxmsgdbZ0J7oonwA">https://youtu.be/rkKGfp1PZ3c?si=NxmsgdbZ0J7oonwA</a></li><li>Apple Vision Pro “Designing for visionOS”: <a href="https://developer.apple.com/design/human-interface-guidelines/designing-for-visionos">https://developer.apple.com/design/human-interface-guidelines/designing-for-visionos</a></li></ul><h4>Related Medium articles</h4><ul><li>Eyes &amp; hands in AR: A sci-fi-inspired mobile UI research: <a href="https://medium.com/@ken.pfeuffer/eyes-hands-in-ar-a-sci-fi-inspired-mobile-ui-research-7f75b69b15f2">https://medium.com/@ken.pfeuffer/eyes-hands-in-ar-a-sci-fi-inspired-mobile-ui-research-7f75b69b15f2</a></li><li>Summary of Eye-Hand HCI Research in the Past 10 Years for Apple’s Vision OS: <a href="https://medium.com/@cybrain/summary-of-eye-hand-hci-research-in-the-past-10-years-for-apples-vision-os-6585ba52716c">https://medium.com/@cybrain/summary-of-eye-hand-hci-research-in-the-past-10-years-for-apples-vision-os-6585ba52716c</a></li><li>Fitts’s Law meets Apple’s Vision Pro — Ergonomic design challenges from mobile to spatial computing: <a href="https://medium.com/@jim-ekanem/fittss-law-meets-apple-s-vision-pro-ergonomic-design-challenges-from-mobile-to-spatial-6c8994ecf975">https://medium.com/@jim-ekanem/fittss-law-meets-apple-s-vision-pro-ergonomic-design-challenges-from-mobile-to-spatial-6c8994ecf975</a></li><li>Gaze-Based Interaction: 30 Years in Retrospect and Future Outlook <a href="https://medium.com/@cybrain/gaze-based-interaction-30-years-in-retrospect-and-future-outlook-af923a32a921">https://medium.com/@cybrain/gaze-based-interaction-30-years-in-retrospect-and-future-outlook-af923a32a921</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a95e251169ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/antaeus-ar/design-principles-issues-for-gaze-and-pinch-interaction-a95e251169ae">Design Principles &amp; Issues for Gaze and Pinch Interaction</a> was originally published in <a href="https://medium.com/antaeus-ar">Antaeus AR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Eyes & hands in AR: A sci-fi-inspired mobile UI research]]></title>
            <link>https://medium.com/@ken.pfeuffer/eyes-hands-in-ar-a-sci-fi-inspired-mobile-ui-research-7f75b69b15f2?source=rss-efc9299f702e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f75b69b15f2</guid>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[eye-tracking]]></category>
            <category><![CDATA[human-comuter-interaction]]></category>
            <dc:creator><![CDATA[Ken Pfeuffer]]></dc:creator>
            <pubDate>Mon, 16 Oct 2023 03:25:07 GMT</pubDate>
            <atom:updated>2023-10-21T14:55:59.124Z</atom:updated>
            <content:encoded><![CDATA[<p>One-handed interaction is ubiquitous in our daily lives, primarily through the use of smartphones. Smartglasses and head-mounted displays (HMDs) become increasingly prevalent, with big players like Apple, Microsoft, and Meta taking center stage in the race for the next generation of spatial computing devices and AR/VR. In our research, we aimed to take a step beyond the home and explore how we can design user interfaces (UIs) that are as mobile as smartphones. Interestingly, science fiction (e.g., Star Wars, Marvel) has introduced numerous innovative concepts that offer a “freehand” alternative to a smartphone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5WETe9FUoCb3GLBSyzlvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*P4OTJF7JTYLMzPxw_UlKGg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWEU3iVEquSyLlmjmSYq3A.gif" /><figcaption>Image Credit: Disney, <a href="https://youtu.be/Oed_RffEjvE">youtube</a> (right)</figcaption></figure><p>Partly, it’s possible with current HMDs. For instance, Microsoft’s Mixed Reality Toolkit (MRTK) offers the <a href="https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-uxcomponents/packages/uxcomponents/hand-menu">Hand Menu</a>, a virtual UI element that attaches to and moves with the user’s hand, for anytime access. Earlier, the HoloLens 1 employed a <a href="https://learn.microsoft.com/en-us/dynamics365/mixed-reality/guides/hl1">“bloom” gesture</a> to summon the device’s home menu at once, positioning it in a world-anchored space. Meta Quest devices offer the <a href="https://www.meta.com/help/quest/articles/getting-started/getting-started-with-quest-2/universal-menu-for-quest-2-and-quest/#:~:text=You%20can%20access%20the%20universal,and%20record%20video%20in%20VR.">Universal Menu</a>, which spawns at the hand and is then world-anchored, for one-time actions via pinch dragging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YzByjA928uFCYYGWHHMQqQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*DakJT8W3bzfl2HWhUkOc3A.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*vrIJEWPn176Sbr-5rBkucw.gif" /><figcaption>Hand menu (left), Bloom gesture (middle), Universal Menu (right). Image Credit: Microsoft (left, center)</figcaption></figure><p>Our 2017 work ‘<a href="https://dl.acm.org/doi/10.1145/3131277.3132180"><em>Gaze + Pinch</em></a>’ laid the scientific foundation for integrating eye-tracking to enhance hand gestures, a concept to be found in the <a href="https://www.apple.com/apple-vision-pro/">Apple Vision Pro</a>, as well as partly the <a href="https://learn.microsoft.com/en-us/windows/mixed-reality/design/gaze-and-commit">Holo Lens 2</a>. There are many more ways of designing a spatial UI. Below, you’ll find an overview of how systems support activating the UI and selecting buttons. Most systems were not designed with mobility in mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qw_raKf1O4V_SUD_" /></figure><h3>PalmGazer</h3><p>In our research paper, we proposed PalmGazer as a UI concept for one-handed AR. It blends hand and eye-tracking technology to enable a Hand Menu that is in control of the same hand. Our final UI system went through a lot of design iterations, to eventually culminate in the following three-fold UI concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V57U7tIDnabSw_qJ" /><figcaption>Three operations supported by the PalmGazer UI concept</figcaption></figure><ul><li><strong>ACTIVATION:</strong> The user activates the UI with a palm-open (Bloom) gesture, summons the Hand Menu, and can freely adjust the UI’s position in mid-air with an open palm. The user closes their hand to disengage the UI at any moment of time, facilitating the notion that it is readily available and easily dismissable at will.</li><li><strong>SELECTION:</strong> Instead of using a second hand, the interaction design is customized for a single hand, utilizing pinch gestures with an open palm for menu selection. This combines gaze for target selection and a quick-release pinch gesture for confirmation (<em>Gaze + Pinch</em>).</li><li><strong>NAVIGATION:</strong> Using a pinch-drag gesture to scroll can causes a design conflict because it moves the Hand Menu as well. We solve this by implementing a peephole-inspired UI behavior that maintains content’s spatial relationships in space.</li></ul><p>It establishes a coherent and simple input command language for one hand only, that lends itself to be as expressive as two-handed or a world-anchored UI. The menu is always available and easily dismissable at will, like smartphones, but in mid-air. Here’s an example: the user is having coffee and wants to play a song with their AR music player. In the video, the orange ray indicates the user’s eye direction, for demo purposes only, not in real use.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FC-aN0XsRYqI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DC-aN0XsRYqI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FC-aN0XsRYqI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9113df5ba9ca540244af6f1d510bbb81/href">https://medium.com/media/9113df5ba9ca540244af6f1d510bbb81/href</a></iframe><h3>Where can we place the UI?</h3><p>We experimented with three UI locations after its instantiation, each having its own set of advantages and drawbacks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/0*5IcCKmhbkQwsFMIt" /></figure><p><em>Head-attached</em> is similar to world-anchored UIs with the difference that it moves with the user. The UI appears in front of the user, at a fixed distance relative to the HMD’s position. However, it may block the center of the user’s view. The <em>On-hand</em> frame emulates the positioning of a smartphone, potentially rendering it an intuitive choice. Nonetheless, like smartphones, it necessitates users to either lower their gaze to the hand (potentially causing neck strain) or raise the hand into their line of sight (potentially leading to arm and shoulder fatigue). Lastly,<em> Above-Hand</em> represents a middle ground. Here, the UI is positioned well above the user’s hand, allowing us to maintain the hand at waist level while still retaining adequate control. This presents a trade-off, as it can amplify hand tracking jitter in the UI, but doesn’t occlude the view so much.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F6jHCuslOCkc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6jHCuslOCkc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F6jHCuslOCkc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/717f262aee00a3f4adc3d8cb71896366/href">https://medium.com/media/717f262aee00a3f4adc3d8cb71896366/href</a></iframe><h3>Example apps</h3><p>We built a home menu similar to a smartphone home screen, or the Holo Lens 2 UI. All menu buttons are activated through a Gaze + Pinch command, and offer access to app examples showcased in the video below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3gFUf6ktLck%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3gFUf6ktLck&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3gFUf6ktLck%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/76681d3b2c5404000b23f96360d4883f/href">https://medium.com/media/76681d3b2c5404000b23f96360d4883f/href</a></iframe><p>Scrolling in our “Downloads” file explorer is straightforward for the <em>Head-attached</em> UI. The <em>On-Hand</em> and <em>Above-Hand</em> locations employ a peephole metaphor to resolve the conflict between using hand movement to both move the entire UI <em>and </em>scroll the content. Meaning, the scrolling direction is reversed in contrast to a <em>Head-attached</em>, making it feel like a moving window. Of course, each scroll command means the UI is moving around, which can be distracting, which we hope to resolve in the future through a more adaptive UI stabilization.</p><p>Another interesting avenue we explored is 3D navigation. Our image gallery app involves several hierarchies that users can traverse (Folders — images — one image). Here the eyes will select the folder or image, depending on the current hierarchy level. Then, you can pinch-drag-forward to traverse one level deeper, whereas backward motion returns to the prior level. This makes it possible to do standard actions one after another (below video, left), but also to continuously use one pinch gesture to traverse many levels of the hierarchy to avoid repetitive pinching (right).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FsOCNYGhXw-I&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DsOCNYGhXw-I&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FsOCNYGhXw-I%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e10c336885aa4e66d3150fb63c07efd/href">https://medium.com/media/1e10c336885aa4e66d3150fb63c07efd/href</a></iframe><h3>The limits of one-handed interaction</h3><p>How much can we do with one hand and gaze only? The Map Viewer app is our interpretation of “Google Maps”. Inspired by our prior work on 2D UIs ( <a href="https://dl.acm.org/doi/10.1145/2642918.2647397">Gaze-touch</a>, <a href="https://dl.acm.org/doi/10.1145/2984511.2984514">Gaze on tablets</a>), gaze directly defines the zooming pivot, like one of the demos of the <a href="https://www.uploadvr.com/apple-vision-pro-gesture-controls/">Apple Vision Pro</a>. Differently though, our approach is one-handed — you can zoom in/out by pinch forward/backward, and simultaneously pan by pinch left/right/up/down. As the UI is attached and moves with the hand, this leads to a lot of movement effort with some UI locations.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuKFgQAAI7xA&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuKFgQAAI7xA&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FuKFgQAAI7xA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc8425854ab03c2b7b30837307b81202/href">https://medium.com/media/fc8425854ab03c2b7b30837307b81202/href</a></iframe><p>In a study with 18 participants, we encountered hand and eye tracking issues with our implementation, that somewhat affected our findings. However, our findings offer an initial assessment and suggest future directions. Participants found PalmGazer easy to use after brief training. Adjusting the UI’s distance from the eyes improved eye tracking and served as an alternative to Vision Pro’s automatic UI resizing. Basic actions for selection and navigation worked well, but complex pan and zoom operations proved challenging with one hand and gaze only. See the image for a brief summary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/0*lgrHNTbVil8zolw3" /></figure><p>More information in the paper presented at the <a href="https://sui.acm.org/2023/">ACM SUI’23</a>, where it received a Best Paper Honorable Mention Award. I also regularly add content on spatial UIs by our research group on X (<a href="https://twitter.com/KenPfeuffer">@KenPfeuffer</a>).</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/KenPfeuffer/status/1671932829930864646%3Fs%3D20&amp;image=https%3A//i.embed.ly/1/image%3Furl%3Dhttps%253A%252F%252Fabs.twimg.com%252Ferrors%252Flogo46x38.png%26key%3Da19fcc184b9711e1b4764040d3dc5c07" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/e53359842d24386d0cd2a4ce4e37e981/href">https://medium.com/media/e53359842d24386d0cd2a4ce4e37e981/href</a></iframe><p>Ken Pfeuffer, Jan Obernolte, Felix Dietz, Ville Mäkelä, Ludwig Sidenmark, Pavel Manakhov, Minna Pakanen, and Florian Alt. 2023. <em>PalmGazer: Unimanual Eye-hand Menus in Augmented Reality. </em>In <em>Proceedings of the 2023 ACM Symposium on Spatial User Interaction (SUI ‘23). </em>Association for Computing Machinery, New York, NY, USA, Article 10, 1–12. <a href="https://doi.org/10.1145/3607822.3614523">https://doi.org/10.1145/3607822.3614523</a></p><p><em>Originally published at </em><a href="https://kenpfeuffer.com/2023/10/16/sui23-palmgazer-one-handed-interaction-in-ar-using-eye-and-hand-tracking/"><em>http://kenpfeuffer.com</em></a><em> on October 16, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f75b69b15f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>