<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arjun Srinivasan on Medium]]></title>
        <description><![CDATA[Stories by Arjun Srinivasan on Medium]]></description>
        <link>https://medium.com/@arjun.srinivasan.10?source=rss-898484b95976------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*aC04vCQqWQPP36LZtQxZXA.jpeg</url>
            <title>Stories by Arjun Srinivasan on Medium</title>
            <link>https://medium.com/@arjun.srinivasan.10?source=rss-898484b95976------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:46:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arjun.srinivasan.10/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Helping People Ask Data Questions]]></title>
            <link>https://medium.com/tableau-engineering-blog/helping-people-ask-data-questions-4fbb2f8ee9d?source=rss-898484b95976------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fbb2f8ee9d</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[hci]]></category>
            <category><![CDATA[user-interface]]></category>
            <dc:creator><![CDATA[Arjun Srinivasan]]></dc:creator>
            <pubDate>Wed, 17 Nov 2021 16:53:15 GMT</pubDate>
            <atom:updated>2023-03-24T16:00:24.476Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR: </strong>Natural language interfaces (NLIs) for visualization have become a prevalent medium for data exploration and analysis. However, the <em>lack of analytic guidance</em> (i.e., knowing <em>what</em> aspects of the data to consider) and <em>discoverability of natural language input</em> (i.e., knowing <em>how</em> to phrase input utterances) make it challenging for people to interact with such NLIs.</p><p>This post summarizes our recent work investigating the use of <em>utterance recommendations</em> to provide analytic guidance by suggesting data features (e.g., attributes, values, trends) while implicitly making users aware of the types of phrasings that an NLI supports.</p><p>This post is based on the <a href="https://uist.acm.org/uist2021/">ACM UIST 2021</a> conference paper <strong>“</strong><a href="https://research.tableau.com/sites/default/files/snowy-uist21.pdf"><strong><em>Snowy: Recommending Utterances for Conversational Visual Analysis</em></strong></a><strong>”</strong> by <a href="https://research.tableau.com/user/arjun-srinivasan"><strong>Arjun Srinivasan</strong></a><strong> and </strong><a href="https://research.tableau.com/user/vidya-setlur"><strong>Vidya Setlur</strong></a>.</p><p>Popular visualization tools like <a href="https://www.tableau.com/solutions/ai-analytics/augmented-analytics">Tableau</a>, Microsoft’s <a href="https://docs.microsoft.com/en-us/power-bi/consumer/end-user-q-and-a">Power BI</a>, and <a href="https://www.qlik.com/us/products/qlik-sense/conversational-analytics">Qlik</a> now offer NLIs for visual data analysis. These NLIs make it possible for people to directly ask questions about their data and specify visualizations without having to learn the nitty-gritty of using an interface. With improvements in natural language processing (NLP) techniques, these NLIs continue to improve at understanding user queries. Furthermore, to assist people in issuing queries and enable a smoother user experience, such systems now also provide smart autocompletion features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/0*Vg0wP7ScMpL0uHsm.gif" /><figcaption>Autocomplete in Tableau Ask Data. Source: <a href="https://help.tableau.com/current/pro/desktop/en-us/ask_data.htm">https://help.tableau.com/current/pro/desktop/en-us/ask_data.htm</a></figcaption></figure><p>Despite the improvements in NL understanding and assistive features like query autocompletion, asking questions during data analysis remains a challenging task for two key reasons. First, formulating questions requires users to know about the data domain along with potential attributes and patterns to query (e.g., choosing which attributes to investigate a correlation between). This can be particularly challenging when starting a new analysis or when exploring a dataset for the first time. Second, practical limitations of a system’s NL understanding capabilities often require users to phrase their utterances in specific ways that the underlying system can best interpret them (for the remainder of this post, we use the term <em>utterance</em> to refer to any NL command, statement, query, question, or instruction that one may issue to an NLI). Without a clear understanding of an NLI’s interpretation capabilities, users often end up “guessing” utterances, thus being more prone to system failures.</p><blockquote>While the lack of analytic guidance (i.e., knowing <em>what to ask)</em> and a lack of discoverability of NL input (i.e., knowing <em>how to phrase utterances</em>) are fundamental challenges on their own, in tandem, these can disrupt the analytic workflow and discourage the use of NLIs for visual analysis altogether.</blockquote><p>To address these challenges, we explored the idea of generating contextual utterance recommendations during visual analysis. We operationalize this idea within a prototype system, <strong><em>Snowy</em></strong>, that recommends utterances at different points during an analytic session.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ccVJtptn3RsTLncpwREsQ.png" /><figcaption>Examples of utterance recommendations in Snowy. (A) To assist with the “cold start” problem during data analysis, Snowy infers potentially interesting patterns from the underlying dataset and suggests analytic inquiries one may want to begin exploring the data with. (B) Upon executing an NL utterance, Snowy suggests follow-up utterances to adjust the current view and drill down into specific data subsets. (C) As marks are selected on the view through direct manipulation, Snowy recommends deictic utterances (i.e., utterances that apply exclusively to the selected points) to perform popular calculations.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*DNi8U1610eJtQ6Ziu8IcYg.png" /><figcaption>*The name Snowy is inspired by the namesake fictional dog in the “<a href="https://en.wikipedia.org/wiki/The_Adventures_of_Tintin"><em>Adventures of Tintin</em></a>” comic series where he addresses his internal monologue to the reader with speech bubbles.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pL04qfktszPVUmrXVPoN4Q.png" /><figcaption>Snapshot of the movies dataset.</figcaption></figure><p>Now let’s walk through a brief illustrative scenario to better understand how Snowy generates utterance recommendations. Imagine Tintin, an analyst at a production company is examining a movie dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oW9qx2kvZruIChbksGrqlw.gif" /><figcaption>Snowy presents utterance recommendations covering a range of analytic intents (e.g., detecting correlations, observing temporal trends) as a starting point and Tintin selects an utterance corresponding to the ‘correlation’ intent.</figcaption></figure><p>As Tintin loads the dataset, Snowy scans the underlying data to identify potentially interesting attribute combinations to explore and presents a list of utterance recommendations that Tintin can use to start his exploration. Specifically, Snowy recommends utterances that cover popular analytic intents (e.g., finding correlations, observing temporal trends) and populates the recommendations with attributes that might highlight salient data patterns (e.g., Production Budget and Worldwide Gross are suggested for the <em>correlation </em>intent since these attributes are highly correlated). Skimming through the recommendations, Tintin is curious to learn more about the relationship between budget and gross, and selects the utterance “<em>Show the relationship between Production Budget and Worldwide Gross.” </em>Snowy interprets this utterance using its NL interpreter and generates a scatterplot in response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xluKXa2ocex1wq_4sf4iSw.gif" /><figcaption>Tintin selects the follow-up utterance recommendation to break down the scatterplot by the Content Rating attribute, resulting in Snowy mapping point colors to the content ratings.</figcaption></figure><p>Next, taking into consideration the fact that Tintin just issued an NL utterance, to promote a fluid analytic experience, Snowy recommends a follow-up utterance “<em>Break down by</em> <em>Content Ratings,</em>” and also updates the initial set of recommendations covering different analytic intents. Notice that the follow-up recommendation is much terser than the other recommendations since this utterance builds upon prior inputs. To investigate the effect of different content ratings on the relationship, Tintin selects this recommendation, resulting in Snowy updating the chart to a colored scatterplot where the colors map to the different content ratings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mKNY2Rnj4704v6gvib00KQ.gif" /><figcaption>As Tintin selects points on the scatterplot, Snowy’s deictic utterance recommendations help him rapidly perform popular computations on the selected points.</figcaption></figure><p>With the modified chart, Snowy again updates its recommendations — now suggesting filters for potentially interesting data subsets (e.g., “<em>Drill down into R</em>” since R-rated movies in the dataset exhibit a strong correlation between budget and gross) as well as utterances mapping to other intents and attributes to start a new line of analytic inquiry. However, instead of using the recommendations, at this point Tintin decides he wants to investigate the active chart further to better understand the values and relationships for high budget movies. To do this, Tintin drags his cursor to select movies with a budget of over $200M. As he makes the selection, Snowy updates its follow-up recommendations to now suggest utterances that can be issued exclusively on the selected points (referred to as <em>deictic utterances</em> because their context is derived from the user’s selection). Specifically, in this case, inferring that the active chart is a scatterplot, Snowy suggests computational utterances for calculating the correlation coefficient and the average attributes values. Tintin notes the averages and also that investing in a high-budget movie does not guarantee a significant return and continues his analysis.</p><h3>So… do utterance recommendations help?</h3><p>We conducted a preliminary user study with 10 participants to gather feedback on the idea of presenting utterance recommendations during conversational visual analysis. Participants were given two datasets (one about movies and one about US colleges) and asked to perform both targeted analysis and open-ended data exploration using Snowy. After the session, we gathered subjective feedback through a semi-structured interview and a series of Likert-scale questions.</p><p>At a high-level, participants said that the recommendations supported their analytic workflows and provided useful guidance during data exploration, without intervening with their self-directed or targeted analysis. Although participants were in agreement about the utility of recommendations to guide their exploration, the feedback on the specific content within the recommendations (i.e., the attributes and values) was more mixed. In particular, participants were unsure about how and why certain recommendations were generated and this lack of clarity resulted in them not feeling confident about the recommended content. However, all participants unanimously agreed the recommendations helped them discover and learn about the system’s NL capabilities including both the situations in which they could issue NL utterances (e.g., to create a new chart, updating an active chart, performing deictic operations) and the types of phrasings they could use (e.g., well structured commands vs. terser keyword-style queries).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fi8UFFmjX61XMP7cRHrCvw.gif" /><figcaption>An example of how Snowy’s recommendations could be used within a data-oriented chat application.</figcaption></figure><p>To summarize, this feedback from the preliminary study suggests further investigating the use of utterance recommendations — particularly delving into aspects of the recommendations’ interpretability (i.e., conveying why the recommendations are shown and how they are generated) and their impact on learnability and use of NL input. Besides these directions, we are also excited about exploring new scenarios that utterance recommendations can facilitate (e.g., data-oriented chat applications, or in accessibility contexts where traditional visualization recommendations cannot be shown).</p><p>For more details about the Snowy system and the user study, please refer to the accompanying <a href="https://research.tableau.com/sites/default/files/snowy-uist21.pdf">research paper</a> and <a href="https://research.tableau.com/paper/snowy-recommending-utterances-conversational-visual-analysis">project page</a>.</p><p>If you find this topic exciting, be sure to <em>continue the conversation </em>and reach out to us with any feedback or questions<em>.</em></p><figure><a href="https://www.salesforce.com/form/other/privacy-request/"><img alt="" src="https://cdn-images-1.medium.com/max/100/1*ImBy4Z_J1PCcv4T0kP9DJg.png" /></a><figcaption>Your Privacy Choices</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fbb2f8ee9d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tableau-engineering-blog/helping-people-ask-data-questions-4fbb2f8ee9d">Helping People Ask Data Questions</a> was originally published in <a href="https://medium.com/tableau-engineering-blog">Tableau Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Talk about Natural Language Interfaces for Data Visualization]]></title>
            <link>https://medium.com/multiple-views-visualization-research-explained/lets-talk-about-natural-language-interfaces-for-data-visualization-b8aaad90c798?source=rss-898484b95976------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8aaad90c798</guid>
            <category><![CDATA[user-interface]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[voice-user-interfaces]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Arjun Srinivasan]]></dc:creator>
            <pubDate>Wed, 27 Mar 2019 03:56:25 GMT</pubDate>
            <atom:updated>2019-03-27T03:56:25.110Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR: </strong>A new suite of natural language-based visualization systems have begun to emerge. These systems allow users to directly ask questions about their data and assist users in answering those questions by suggesting or modifying visualizations. But what are the potential advantages of such systems? What makes them challenging to design and use?</p><p>In this post, I summarize recent research on natural language interfaces for data visualization addressing these questions. I also give my two cents on what may be interesting directions to explore going forward — discussing the potential role of system generated natural language and complementing visual data analysis with question answering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*42aWMJur-weJ_cBZ8JnGDQ.png" /><figcaption>Figure 1: NLIs let people directly ask questions about their data and get visualizations that help answer those questions.</figcaption></figure><p>Given the prevalence of smart speakers like Amazon Echo and voice-based assistants like Siri and Alexa in our everyday lives, interacting with devices using natural language has become quite the norm. With the increasing demand for data visualization tools and the varying levels of expertise of users of these tools, it comes as no surprise that both commercial vendors and academic researchers have begun to explore <strong>natural language interfaces</strong> (NLIs) for data analysis with visualization. While the idea of using language as a way to create visualizations was explored as long as <a href="https://link.springer.com/content/pdf/10.1023/A:1011368926479.pdf">18 years ago</a>, recent advances in natural language processing (NLP) coupled with the maturity of data visualization as a field have led to renewed interest in the topic in the last few years.</p><p>Prominent visualization software vendors including <a href="https://www.ibm.com/watson-analytics">IBM</a>, <a href="https://docs.microsoft.com/en-us/power-bi/consumer/end-user-q-and-a">Microsoft</a>, and most recently, <a href="https://www.tableau.com/about/blog/2018/11/ask-data-simplifying-analytics-natural-language-98655">Tableau</a>, now offer NLIs that allow users to directly communicate their intended questions about the data using natural language. At a high-level, given a user question or command, these systems try to interpret the underlying intent (e.g., comparing trends, applying a filter) and parameters corresponding to those intents (e.g., data field names, data values) and map them to appropriate system actions such as creating new visualizations or updating an existing chart. An example of this can be seen in Figure 2 showing Tableau’s Ask Data system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QbLWFMQGhpv83gLPwOBVdQ.gif" /><figcaption>Figure 2: Tableau’s Ask Data being used to explore the superstore dataset. The user asks “what is the profit for each state?” In response, the system automatically computes the sum of profits for all cities in different states and presents a choropleth map. The image is taken from <a href="https://www.tableau.com/about/blog/2018/11/ask-data-simplifying-analytics-natural-language-98655">this official blog post by Tableau</a>.</figcaption></figure><p>But before we get into the nitty-gritty of how these systems work, let’s take a step back and think about why we might want to use natural language as a way to interact with visualization systems in the first place.</p><h3>Why natural language?</h3><p>Systems like Tableau and Power BI are powerful visualization tools that allow people to interactively pose and answer their analytical questions. However, these tools often present a steep learning curve and require people to translate the questions in their minds into operations or actions supported by the tool. This “translation” of questions into tool-specific actions is something that many users, <a href="https://www.researchgate.net/publication/47544565_How_Information_Visualization_Novices_Construct_Visualizations">particularly novices, struggle with</a>. Natural language can help overcome some of these issues by allowing users to express their questions in their own terms. A <a href="https://research.tableau.com/sites/default/files/uist4832-setlurA_0.pdf">study</a> by Tableau’s research team compared their initial prototype NLI to Tableau desktop and found that participants were faster with the NLI than Tableau for refining given visualizations. Both their study and <a href="https://arjun010.github.io/static/papers/orko-infovis17.pdf">a study we conducted</a> with a speech- and touch-based network visualization tool also showed that participants preferred natural language when the standard interface would require them to perform multiple steps to complete a task, interrupting their analytic flow.</p><p>So, if natural language is an easier way to get started with a visualization tool and is also faster than the current style of input for certain scenarios, what prevents it from being the primary way we interact with visualization tools today?</p><h3>What are the challenges in supporting natural language input? And what are visualization researchers doing to tackle these challenges?</h3><p>Unsurprisingly, implementing NLIs is challenging since we need to build software that first interprets the human language as other humans would and then performs an appropriate set of actions based on its interpretation. Below I list a subset of challenges these systems pose from a design and usability standpoint. I also highlight how researchers are trying to address some of these challenges in the context of NLIs for data visualization.</p><p><strong>Ambiguity and Underspecification: </strong>User questions are often ambiguous and underspecified. For example, imagine you were exploring an Olympic Games dataset with details about all medals won by different countries over several decades. While exploring this dataset with an NLI, you say “show me medals for hockey and skating by country.” This seemingly simple query presents multiple <em>ambiguities</em>. Specifically, the word “medals” can map to either the total number of medals or specific types of medals (e.g. gold). Similarly, “hockey” and “skating” can refer to different sports (e.g., ice hockey vs. field hockey or figure skating vs. speed skating). Even if we assume all of these ambiguities were resolved, there is still the question of which visualization to use (e.g. grouped vs. stacked bar chart).</p><p>One proposed solution to expose and help users resolve these ambiguities has been to use multimodal input (i.e., leverage another form of input such as a mouse). In this approach, given an input query, the system tries to identify ambiguous phrases using a suite of string matching and word association algorithms. These ambiguous phrases are then displayed through interactive widgets (referred to as <em>ambiguity widgets</em>) — allowing users to refine their queries and resolve the ambiguity. This idea was first presented and incorporated in a system called <a href="http://www.zcliu.org/datatone/DataTone-UIST15.pdf">DataTone</a> (Figure 3) by <a href="https://theblog.adobe.com/datatone/">Adobe</a>. An example of how the DataTone system handles the ambiguous query corresponding to the Olympic Games dataset can be seen in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6YXWZK0zDCv18VpvcECpA.png" /><figcaption>Figure 3: A snapshot of the DataTone system highlighting ambiguities through interactive widgets. The image is taken from <a href="http://www.zcliu.org/datatone/DataTone-UIST15.pdf">the DataTone research paper</a>.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CSCnTjXtk8VmUJDlY4eJVg.png" /><figcaption>Figure 4: Tableau’s response to the underspecified query “What’s the correlation of GDP?” The image is based on their <a href="https://research.tableau.com/sites/default/files/IUI_vsetlur.pdf">research paper</a>.</figcaption></figure><p>An associated problem to ambiguity is that of <em>underspecification</em>. While ambiguity arises when the system has multiple options to consider, underspecification refers to cases where the input query lacks enough information (e.g., attributes, keywords that help infer intent) for the system to make a decision. People naturally tend to be imprecise when asking questions, frequently presenting incomplete or underspecified queries. For instance, when exploring a <a href="https://www.gapminder.org/data/">dataset</a> about the population and associated economic metrics of different counties, one might wonder about the gross domestic product (GDP) of different counties and ask the question “What’s the correlation of GDP?” While this question may be clear to the user, from the system’s standpoint, it is both ambiguous (e.g. there might be different GDP columns, perhaps for multiple years) and underspecified (e.g. correlation of GDP against what?)</p><p>Addressing such issues, researchers at Tableau recently devised <a href="https://research.tableau.com/sites/default/files/IUI_vsetlur.pdf">a set of techniques to infer missing details in a query</a> based on the meaning and usage frequency of different data attributes as well as constraints imposed by the system’s supported operations. Using these inference techniques, given the underspecified query “What’s the correlation of GDP?”, the system can generate a scatterplot visualizing GDP per Capita and Life Expectancy since they were the most popular combination of attributes explored for the same dataset (Figure 4).</p><p><strong>Preserving context to support an analytic flow: </strong>To support natural language input, it is not enough if a system allows users to enter one-off commands that result in a visualization. During visual data analysis, users often need to iterate upon their questions and refine existing visualizations — diving deeper into specific aspects of a chart or adding new visualizations to the current view. Supporting such actions implies that the system should be able to support a “conversation” between the user and the data. A key component of supporting a conversation is interpreting the <em>context </em>in which a query is posed. In other words, in addition to interpreting the current query, the system also needs to consider previously issued queries (to identify data attributes and values used) and the active view (e.g., visualizations, colors) so it can answer questions effectively. For instance, in Figure 2, if as a follow-up to the question “What is the profit for each state?” the user asked “What about different cities?”, the system needs to understand that the implicit attribute the user is referring to is “Profit” and accordingly adjust the choropleth map to color cities as opposed to states. As a first step towards supporting such interactions, <a href="https://research.tableau.com/sites/default/files/VAST2017_105.pdf">research</a> <a href="https://arjun010.github.io/static/papers/orko-infovis17.pdf">systems</a> have employed <a href="https://dash.harvard.edu/bitstream/handle/1/2579648/Grosz_AttentionIntentions.pdf?sequence=2">conversational centering techniques</a> from the field of <a href="https://en.wikipedia.org/wiki/Pragmatics"><em>pragmatics</em></a><em> </em>which is a subfield of linguistics focusing on the ways in which context contributes to meaning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4rMm28gY9LT_tv0h7blAtA.png" /><figcaption>Figure 5: An example of pragmatics concepts being used to support a follow-up command. The image is taken from <a href="https://research.tableau.com/sites/default/files/VAST2017_105.pdf">this research paper</a> by Tableau.</figcaption></figure><p>Consider the example in Figure 5 where a user is exploring a Seattle house price dataset. The user first says “houses less than 1M in Ballard” understanding which the system applies two filters for sales price and the neighborhood “Ballard”. Next, with the previous query in mind, the user simply says “townhomes” implying that the system must show townhomes in the Ballard neighborhood that cost less than 1M. To capture this implicit meaning, the system preserves (or technically, <em>retains</em>) the neighborhood and price filters and changes (or technically, <em>shifts</em>)<em> </em>the focus on all houses in the dataset to only townhomes. This combination of the retain and shift operations results in the system updating the chart as per the user’s expectations.</p><p><strong>Discoverability: </strong>A<strong> </strong>key challenge faced by users of NLIs (particularly new users) is that they are unsure about what the system is capable of doing (i.e. which operations it can perform) and if the system expects the user to conform to a specific language structure. This uncertainty in knowing what can be asked and how is commonly referred to as the lack of <em>discoverability</em>. Although the advances in natural language understanding are allowing users to more freely phrase their intended operations, I would argue that discovering “what” can be done remains an open problem. Compared to the other challenges, discoverability has received relatively little attention in current visualization NLIs. The most common approach current systems take to aid discoverability is using an autocomplete feature. An example of this can be seen Microsoft’s Power BI (Figure 6). However, as found in <a href="https://research.tableau.com/sites/default/files/uist4832-setlurA_0.pdf">a study</a>, this approach gives users a false sense of the system’s ability to interpret more complex queries. Furthermore, this approach is limited to text input and does not work well for spoken commands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*KplGHLAJUJSNSX763GU5sw.gif" /><figcaption>Figure 6: Autocomplete feature in Microsoft Power BI Q&amp;A. The image is taken from this <a href="https://docs.microsoft.com/en-us/power-bi/consumer/end-user-q-and-a">official blog post</a> by Microsoft.</figcaption></figure><p>A more recent approach to aid discoverability of natural language commands is using command suggestions. As shown in Figure 2, Tableau’s Ask Data system tries to help users start interacting with the system by suggesting sample commands. We also recently proposed a general framework to suggest natural language commands as a way to <a href="https://arjun010.github.io/projects/discover-nl.html">enhance the discoverability of natural language input in speech-based multimodal interfaces</a>. While the idea of suggesting commands is simple at a surface-level, deciding <em>which</em> commands to show, and <em>when</em> and <em>how</em> the suggestions must be made are important design considerations that warrant additional research. Furthermore, the effects of such suggestions during visual data analysis (e.g. do they encourage/discourage users from thinking about new questions) are yet to be investigated. Given the increasing popularity of speech-based UIs, devising more ways to address the challenge of discoverability remains an open area for research.</p><h3>Emerging Themes and Future Directions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hvidpnjSAt6P_tlU-pnzsA.gif" /><figcaption>Figure 7: Interactive data facts in <a href="https://arjun010.github.io/projects/voder.html">Voder</a> helping users interpret visualizations and embellish them to highlight key findings.</figcaption></figure><p><strong>Natural language as i̵n̵p̵u̵t̵ output</strong>: Until now, I have only discussed natural language as an input modality (i.e., as a way for users to communicate with the system). An emerging theme in visualization research is to complement visualizations with natural language and use language as an output modality (i.e., as a way for systems to communicate with users). This idea is being explored commercially by companies such as <a href="https://narrativescience.com/products/quill/">Narrative Science</a> and <a href="https://automatedinsights.com/">Automated Insights</a> that offer services to “summarize” visualizations in plain text. We also recently presented <a href="https://arjun010.github.io/projects/voder.html">a system</a> called Voder (see Figure 7) that automatically infers key data facts from a visualization and allows users to interact with these facts to highlight them in the visualization.</p><p>Current work on this topic is largely focusing on exploring the best way to identify key facts and creating natural language generation (NLG) models to present them in a communicable manner. However, with increasing concerns regarding the <a href="https://research.tableau.com/sites/default/files/CHI-CameraReady.pdf">ethical dimensions of visualization research</a> pertaining to automated analysis, I believe an interesting opportunity lies in <em>leveraging NLG to communicate the logic behind the system’s actions so that it can build trust in the users’ mind and lead to more confident decision making. </em>In my opinion,<em> </em>this is not only specific to NLIs but is an area where natural language research can contribute to data analytics and visualization tools in general.</p><p><strong>Complementing Visual Data Analysis with Question Answering: </strong>While this post mostly exemplifies queries that create or modify visualizations, not all queries during data analysis may need a visualization as a response. For instance, looking at the map in Figure 2, one might ask “What were sales in California last year?” or “What are the total sales across regions?” In such cases, all the user wants is the value of sales. Although the system could generate visualizations in response to these questions, it is probably better if the system returned the actual value or answer and not just a chart that the user needs to interpret to get the answer.</p><p>Going forward, I hypothesize that to truly support a cycle of visual analysis through natural language, <em>we need to design systems that can not only render a visualization that “contains” an answer but also provide direct responses when appropriate</em>. Building such tools warrants further research at the intersection of data visualization systems that can identify the best chart in response to a question and <a href="http://&quot;what&/#39;re%20the%20total%20sales%20across%20regions?&quot;">question answering</a> systems, or specifically, <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.47.3954&amp;rep=rep1&amp;type=pdf">NLIs for databases</a> that can directly compute and present values in response to user questions.</p><p>This post only briefly discusses some of the ongoing research on NLIs for data visualization. A more comprehensive review of a subset of the systems described in this post along with additional research opportunities can be found in <a href="https://arjun010.github.io/static/papers/nli-vis-eurovis17.pdf">our paper</a>.</p><p>If you find this topic exciting, let’s keep talking (both to systems and each other). You can learn more about our work on natural language and multimodal interfaces for data analysis with visualization on the <a href="https://www.cc.gatech.edu/gvu/ii/naturalvis/">Information Interfaces Research Group project page</a> or <a href="https://arjun010.github.io/">my personal website</a>.</p><h3>Acknowledgments</h3><p>Thanks to <a href="https://research.tableau.com/user/vidya-setlur">Vidya Setlur</a>, <a href="http://enrico.bertini.io/">Enrico Bertini</a>, <a href="http://users.eecs.northwestern.edu/~jhullman/">Jessica Hullman</a>, and the <a href="http://vis.gatech.edu/">Georgia Tech Visualization Lab</a> for their feedback on this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8aaad90c798" width="1" height="1" alt=""><hr><p><a href="https://medium.com/multiple-views-visualization-research-explained/lets-talk-about-natural-language-interfaces-for-data-visualization-b8aaad90c798">Let’s Talk about Natural Language Interfaces for Data Visualization</a> was originally published in <a href="https://medium.com/multiple-views-visualization-research-explained">Multiple Views: Visualization Research Explained</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>