<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Young-Ho Kim on Medium]]></title>
        <description><![CDATA[Stories by Young-Ho Kim on Medium]]></description>
        <link>https://medium.com/@yghokim?source=rss-d44e7126ea04------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*a-IIOrJGm4HfcmMI</url>
            <title>Stories by Young-Ho Kim on Medium</title>
            <link>https://medium.com/@yghokim?source=rss-d44e7126ea04------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:56:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yghokim/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data@Hand: Combining Speech and Touch for Data Exploration on Smartphone]]></title>
            <link>https://medium.com/hcil-at-umd/data-hand-combining-speech-and-touch-for-data-exploration-on-smartphone-e2149cff3ff4?source=rss-d44e7126ea04------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2149cff3ff4</guid>
            <category><![CDATA[voice-user-interfaces]]></category>
            <category><![CDATA[human-comuter-interaction]]></category>
            <category><![CDATA[multimodal-interface]]></category>
            <category><![CDATA[hcil-symposium-2021]]></category>
            <category><![CDATA[mobile-health]]></category>
            <dc:creator><![CDATA[Young-Ho Kim]]></dc:creator>
            <pubDate>Mon, 10 May 2021 15:43:54 GMT</pubDate>
            <atom:updated>2021-05-12T22:13:03.016Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data@Hand: Combining Speech and Touch for Data Exploration on Smartphones</h3><h4><em>How can we facilitate visual exploration of personal data on a small smartphone screen?</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HsVyn-EZcNe6IvcbswUVzQ.jpeg" /><figcaption>Executing a global speech command (left) and a touch+speech combined command (right) on Data@Hand.</figcaption></figure><p><em>This article introduces a mobile application named Data@Hand, which leverages speech and touch interaction to foster visual exploration of personal data on mobile devices. This work is a collaborative effort of Young-Ho Kim (University of Maryland), Bongshin Lee (Microsoft Research), Arjun Srinivasan (then at Georgia Tech), and Eun Kyoung Choe (University of Maryland).</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fct2s29n-mJM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dct2s29n-mJM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fct2s29n-mJM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/12f7bbcf3b3d804f4b761ae152235c66/href">https://medium.com/media/12f7bbcf3b3d804f4b761ae152235c66/href</a></iframe><p>Smartphones have become a dominant way to view and access personal data. People track their activities and physiological metrics using smartwatches and wearables, and review the data in the companion apps on smartphones. For example, people can browse their activity data captured by their Fitbit device in the Fitbit mobile app, and the data captured by Apple Watch in the Apple Health app. Mobile health apps commonly incorporate interactive visualizations such as bar and line charts to deliver comprehensive trends in various health metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yAzpsHhPQzPi2k3SqWicXg.jpeg" /><figcaption>Heart rate tracking of Apple Watch and Health app (left) and Sleep tracking with Fitbit band and Fitbit app (right).</figcaption></figure><p>However, smartphones have small displays: they can’t display much information at once and don’t have precise input methods such as a mouse. For these reasons, visualization support in mobile health apps is limited. For example, they usually restrict people to view data by predefined time segments, such as one week, one month, three months, and one year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9cTlU4P6VUUAUpC3ozOqjQ.jpeg" /><figcaption>Time segment selection widgets of Fitbit, Apple Health, and Google Fit.</figcaption></figure><p>As a result, it is difficult to browse through a large amount of activity data on smartphones. People’s questions on their personal data often involve <em>time components</em> — e.g., “How has my step count changed before and after the COVID-19 Pandemic?” or “How does my sleep pattern of April 2021 differ from that of April 2020?” To answer these questions, we need to jump between different time periods, which is tedious to do on a smartphone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*Goz5ZMBgFDvEA1SIEIHZ9Q.gif" /><figcaption>Infinite scroll in the Fitbit app to locate a specific date in the past.</figcaption></figure><h3>Introducing Data@Hand</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*meQoiOUMIeOwZ2XnPEi-vA.gif" /></figure><p>How can we effectively facilitate “personal” data exploration on a small smartphone screen? In this work, we propose <strong>Data@Hand</strong>, a personal data exploration app that leverages both speech and touch modalities to facilitate flexible exploration. Our approach is to use <em>natural language</em> for specifying various interaction parameters such as time and activity types. Natural language is flexible to cover different ways of specifying dates and their ranges (e.g., “<em>October 7th</em>”, “<em>Last Sunday</em>”, “<em>This month</em>”). Specifying time by speech can be much easier than using screen widgets like a calendar (or time) picker. Also, it is straightforward to specify multiple parameters in a short phrase. For example, we can jump directly into a specific activity by uttering, “<em>Show heart rate in March.</em>” This command specifies both the activity type and a period at once, which is likely to take longer to do by graphical widgets.</p><p>Motivated by the recent advancement in <strong><em>multimodal interfaces</em></strong><em> </em>in the areas of data visualization and HCI, we combined speech and touch interaction to leverage a complementary synergy of the two input methods. Speech is flexible and fast for specifying long commands with multiple parameters, whereas touch is faster for pointing on a screen element or executing simple actions mapped with gestures. What if we can use both at the same time? With Data@Hand, we can execute a speech command while holding on a screen element such as chart marks and labels, indicating the context of the command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*EdYmUNekjAJ2_E9xlCHlBg.gif" /><figcaption>Switching the left period using touch+speech action.</figcaption></figure><p>For example, we can change the start date by simply uttering a specific date while holding on a start date label. Also, while comparing two different periods, we can quickly modify one period into another by holding on a chart mark.</p><h3>Using Data@Hand: The Study</h3><p>To understand how people — who are not necessarily experts in data analytics and visualization — use Data@Hand, we observed 13 long-term Fitbit users exploring their Fitbit data using Data@Hand for about 20 minutes. Participants actively mixed speech and touch throughout their exploration. Among all operations participants performed, around a half of the operations included the speech modality. 70% of the speech commands involved time manipulation, implying that time manipulation was a strong facilitator of participants’ visual exploration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LnAHe02W4UMB1dBGFzPwvw.jpeg" /><figcaption>Due to the COVID-19 outbreak, we conducted a think-aloud study on Zoom, using TeamViewer for mobile screen sharing.</figcaption></figure><p>As long-term Fitbit users, participants expressed excitement about viewing their data using multimodal interaction.</p><blockquote>“…it was cool to say like ‘around this year’ or ‘this month’ and it would get what you were talking about, whereas I think it could be hard to do in Fitbit.”</blockquote><p>They liked the flexible time navigation and comparison features Data@Hand offered, describing time manipulation in natural language to be fast and flexible.</p><h3>Toward Better Exploration of Personal Data on Smartphones</h3><p>Our work contributes the first mobile app that leverages the synergy of speech and touch input modalities for personal data exploration. Our study revealed that participants could quickly learn and adopt multimodal interaction toward fast and fluid exploration, which suggests potential opportunities of our design approach. However, research in mobile data visualization is sparse, despite the proliferation of mobile devices as a tool for personal tracking and data exploration. We believe that multimodal interaction leveraging speech and touch is a promising approach to address the inherent limitations of smartphones’ small screen space.</p><h3>Further Information</h3><p>For more information, please check out our full version of the Data@Hand paper, our project website, and source code:</p><ul><li><strong>Paper published in ACM CHI 2021 (Honorable Mention Award) </strong>[<a href="http://younghokim.net/download/publication/12">Link</a>]</li><li><strong>Project website</strong> [<a href="https://data-at-hand.github.io/">Link</a>]</li><li><strong>Source code on Github</strong> [<a href="https://github.com/umdsquare/data-at-hand-mobile">Link</a>]</li></ul><h4><strong>Full citation:</strong></h4><ul><li>Young-Ho Kim, Bongshin Lee, Arjun Srinivasan, and Eun Kyoung Choe. 2021. Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction. In CHI Conference on Human Factors in Computing Systems (CHI ’21), May 8–13, 2021, Yokohama, Japan. ACM, New York, NY, USA, 17 pages. <a href="https://doi.org/10.1145/3411764.%203445421">https://doi.org/10.1145/3411764. 3445421</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2149cff3ff4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hcil-at-umd/data-hand-combining-speech-and-touch-for-data-exploration-on-smartphone-e2149cff3ff4">Data@Hand: Combining Speech and Touch for Data Exploration on Smartphone</a> was originally published in <a href="https://medium.com/hcil-at-umd">Sparks of Innovation: Stories from the HCIL</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boosting Mobile In-Situ Data Collection with OmniTrack for Research]]></title>
            <link>https://medium.com/hcil-at-umd/boosting-mobile-in-situ-data-collection-with-omnitrack-for-research-d485d27f60ba?source=rss-d44e7126ea04------2</link>
            <guid isPermaLink="false">https://medium.com/p/d485d27f60ba</guid>
            <category><![CDATA[hci]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[hcil-symposium-2020]]></category>
            <category><![CDATA[self-tracking]]></category>
            <dc:creator><![CDATA[Young-Ho Kim]]></dc:creator>
            <pubDate>Mon, 25 May 2020 02:51:34 GMT</pubDate>
            <atom:updated>2020-05-27T15:05:53.812Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introducing OmniTrack for Research, an open-source research platform for streamlining mobile-based, in situ data collection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ezEiec1NDZ3cbEik_9YvEA.png" /><figcaption>A plan builder view of OmniTrack for Research and the participant app.</figcaption></figure><p>Collecting <em>in situ</em> data from participants’ mobile devices in their personal context has recently gained tremendous interest among researchers in many fields. <em>In situ</em> data range from low-level sensing data, such as GPS position, to subjective self-report data, such as sleep quality. To collect such data, researchers must distribute their study participants a proper recording method (e.g., mobile app) and find ways to gather the recorded data from participants.</p><p>We designed <strong>OmniTrack for Research (O4R)</strong> to help researchers to implement, conduct, and manage <em>in situ</em> data collection studies without any coding. With O4R, researchers can implement a data collection protocol on a website and deploy an Android reporting app to study participants remotely. The recorded data are synchronized with the server in real-time so that the researcher can monitor the participants’ progress and gain preliminary insights on the present dataset. Currently, O4R supports <em>diary studies</em> and <em>experience sampling</em>, which are two exemplary <em>in situ</em> data collection methods.</p><h3>Implementing a Study with OmniTrack for Research</h3><p>OmniTrack for Research is built on top of <strong>OmniTrack </strong>(<a href="https://omnitrack.github.io/">https://omnitrack.github.io</a>), a flexible self-tracking app. We designed OmniTrack to enable people to build personalized trackers from scratch by combining both manual and automated tracking methods. As OmniTrack can produce diverse trackers ranging from a simple diary to a fully-automated location logger, this project inspired us to envision an ecosystem where researchers deploy a predefined OmniTrack app to study participants.</p><p>On a web-based researcher dashboard, researchers can author a study protocol, designing a tracking plan (a unit corresponding to a data table), reminders, and automated logging triggers. Depending on the configuration of the tracking plan and reminder, the study can become a diary study or experience sampling study. If needed, these configurations can be customized for each participant.</p><p>When authoring a tracking plan, researchers leverage various types of data fields. The field values can be fed by existing tracking services such as Fitbit. For example, Fitbit’s step count can be attached to a number field then the value is automatically loaded from the Fitbit server when the data capture is initiated, by manual inputs or the background logging triggers. By combining these components, researchers can produce various alternatives of the tracking system for capturing the same behavior. Let’s see an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fkqVFjN1nvFf8-yu5GK-DA.png" /></figure><p>The above image shows three different designs of sleep trackers. The leftmost one (a) is a manual sleep diary: It requires participants to insert <em>date</em>, <em>sleep time range</em>, and <em>subjective sleep quality</em>. If a researcher wants to lower the data capture burden, he or she may attach Fitbit to the sleep time field (b). When the participant opens the diary, the sleep time field will automatically fetch the last night’s sleep time from the Fitbit server. If the study aims to capture only the sleep time ranges, the entire tracker recording may be automated by attaching a background logging trigger (c). According to the settings in (c), the sleep log will be automatically recorded every day at midnight.</p><h3>What Can We Do with OmniTrack for Research?</h3><p>We have tested the feasibility of O4R by applying it in real-world studies. Here, we introduce several published projects:</p><ol><li>A <em>diary study</em> (by Young-Ho Kim and colleagues) to understand how knowledge workers conceptualize personal productivity: O4R helped produce rich qualitative diary entries about diverse work- and nonwork-related activities that participants related to their productivity. See <a href="https://medium.com/acm-chi/understanding-personal-productivity-how-knowledge-workers-define-evaluate-and-reflect-on-their-241bc36b60b4">this Medium article</a> for more detail.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_eWGaFHqOm8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_eWGaFHqOm8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_eWGaFHqOm8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/885ee34e5513e6f216139d7188faaa8a/href">https://medium.com/media/885ee34e5513e6f216139d7188faaa8a/href</a></iframe><p>The following projects leveraged the customizability of O4R: The participants built their own customized trackers (diaries) on-site and performed self-tracking.</p><ol><li>A <em>co-design habit tracking study</em> (by Sung-In Kim and colleagues) for supporting autistic adolescents’ habit formation: Autistic children and their parents collaborated to design a customized habit tracker from scratch. They used the OmniTrack app on-site to implement their tracker as well as record the data. See <a href="http://younghokim.net/files/publications/pdfs/10_p169-kim.pdf">the paper</a> for more detail.</li><li>A <em>co-design diary study </em>(by Eunkyung Jo and colleagues) for understanding the role of self-tracking in parenting stress of new parents: Young mothers used OmniTrack to design personalized trackers regarding their parenting. See <a href="https://eunkyungjo.weebly.com/uploads/1/1/7/8/117894411/self-tracking_for_parents_chi2020_cameraready.pdf">the paper</a> for more detail.</li></ol><h3>Learn and Use OmniTrack for Research</h3><p>If you are interested in OmniTrack for Research and want to dig more deeply, please check these resources:</p><ol><li>Official website <a href="https://omnitrack.github.io/research">https://omnitrack.github.io/research</a>.</li><li>Documentation for installing and using O4R <a href="https://github.com/OmniTrack/omnitrack_for_research/wiki">https://github.com/OmniTrack/omnitrack_for_research/wiki</a></li><li>Source code (OmniTrack for Research is completely free open-sourced!) <a href="https://github.com/OmniTrack/omnitrack_for_research">https://github.com/OmniTrack/omnitrack_for_research</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d485d27f60ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hcil-at-umd/boosting-mobile-in-situ-data-collection-with-omnitrack-for-research-d485d27f60ba">Boosting Mobile In-Situ Data Collection with OmniTrack for Research</a> was originally published in <a href="https://medium.com/hcil-at-umd">Sparks of Innovation: Stories from the HCIL</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>