<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sachit Mishra on Medium]]></title>
        <description><![CDATA[Stories by Sachit Mishra on Medium]]></description>
        <link>https://medium.com/@sachitm?source=rss-85a53664df8b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VSSK4bLwX9zzZd8XkHn6zw.jpeg</url>
            <title>Stories by Sachit Mishra on Medium</title>
            <link>https://medium.com/@sachitm?source=rss-85a53664df8b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:30:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sachitm/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How we built it: the Google I/O ’18 Action for the Google Assistant]]></title>
            <link>https://medium.com/google-developers/how-we-built-it-the-google-i-o-18-action-for-the-google-assistant-7f287ad31b7?source=rss-85a53664df8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f287ad31b7</guid>
            <category><![CDATA[voice-assistant]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[firebase]]></category>
            <category><![CDATA[google-assistant]]></category>
            <category><![CDATA[actions-on-google]]></category>
            <dc:creator><![CDATA[Sachit Mishra]]></dc:creator>
            <pubDate>Thu, 13 Sep 2018 17:31:25 GMT</pubDate>
            <atom:updated>2018-09-13T17:50:50.930Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*84fRDUDuhl7FNgSD" /></figure><p>Welcome to Part 2 of a post about building the Google I/O ’18 Action for the Google Assistant. Check out the code <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs">here</a>.</p><h3>tl;dr</h3><p>Dialogflow</p><ul><li>See our Dialogflow agent <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/tree/master/dialogflow-agent">here</a> (zip up the directory to import into Dialogflow)</li><li>Use follow up intents and contexts for sub-menus like browsing flows. Learn more <a href="https://dialogflow.com/docs/getting-started/state-contexts">here</a>.</li><li>When doing this, be careful of how contexts commingle to create unexpected behavior. To be on the safe side, deactivate extraneous contexts when entering a sub-menu.</li><li>Unzip your Dialogflow agent before checking into version control. This allows for better code review and accountability.</li></ul><p>Webhook</p><ul><li>Create a re-usable data structure for your prompts, and organize them by intents (e.g. <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/tree/master/functions/prompts">our directory</a>, how intents were <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/blob/master/functions/app.js">routed</a>).</li><li>When using Cloud Functions, be aware of <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/blob/master/functions/app.js">global scope caching</a>. It’s useful when you want it, painful otherwise.</li><li>Actions are like any other software. Dream big, but start small and iterate.</li></ul><p>In <a href="https://medium.com/google-developers/how-we-designed-it-the-google-i-o-18-action-for-the-google-assistant-9370ffbaf9b0">Part 1</a> of this post, we walked through the process of designing the Google I/O ’18 Action. This included brainstorming our use cases and persona to produce sample dialogs and early high-level flows like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/0*hYbLdmd3b-ZAD3wX" /></figure><p>So the question became how to get from there to here in Dialogflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/0*bD6Zwhj6uc02GKxb" /></figure><h3>Implementing the Dialogflow agent</h3><p>Our herculean task was to implement the entire conversational design (i.e. all possible turns of dialog and the transitions between them) as a Dialogflow agent. At this point, we had a detailed design specification which included coverage for the <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/design-for-the-long-tail.html">long tail</a> of ways a conversation can deviate from the most common paths as well as handling for <a href="https://designguidelines.withgoogle.com/conversation/conversational-components/errors.html">errors</a> and other unlikely or uncommon scenarios. Note that this required more than a simple flowchart. After all, directional relationships in flowcharts don’t necessarily reflect all the unique paths through a conversation. A flowchart more often represents the turns in a particular instance of a dialog. Though the turns progress in a certain order for that dialog, different users’ conversations may follow other paths that aren’t represented in the flowchart. Moreover, a flowchart does not always reveal how contexts create multiple conversation paths.</p><p>When porting this spec to Dialogflow, it’s important to consider what a Dialogflow “<a href="https://dialogflow.com/docs/intents">intent</a>” actually is. It is a user-triggered state of the dialog that <a href="https://dialogflow.com/docs/intents#user-says">matches a training phrase</a> and returns a response from the agent. Each intent represents a single turn of dialog. The collection of intents in a Dialogflow agent forms a finite state machine, where some collection of states combine to form a complete dialog. From any single state in the machine, the possible subsequent states are any for which the <a href="https://dialogflow.com/docs/contexts">input contexts</a> are active (or which have no input contexts). Out of these possible state transitions, Dialogflow chooses the one whose training phrases most closely match a user’s utterance above a certain threshold.</p><h3>Identify the intents</h3><p>So, what did all this mean for us? First, it meant distilling the entire conversational flow into either top-level or nested (i.e. follow-up) intents. Top-level intents could be matched at any time in a given dialog. For example, the user can technically ask about directions to a session at any point in the dialog. Although our conversational flowchart places this conversational state several levels deep in the dialog, there is actually no additional context required to provide directions to a user at the start, for instance. As it turns out, most intents can be top-level and provide more flexibility to users.</p><p><a href="https://dialogflow.com/docs/intents#follow-up_intents">Follow-up intents</a> were required for more involved sub-dialogs within the larger scheme of the conversation. Take, for example, the use case of finding a session topic to hear more about. This interaction starts with the browse-topics intent, which listens for the user to say things like the following:</p><ul><li>“Find a session”</li><li>“Topics”</li><li>“What are the talks at I/O this year?”</li></ul><p>For any of these phrases, we consider the user to be expressing the same underlying goal (intent). That is, they want to know the categories of sessions at the event. Giving them this list, however, gets complicated. On a speaker device, the list of topics is presented in segments to avoid overwhelming the user. Because of this, we needed an intent for the user to declare that they wanted the next set of options. The user also needs a way to ask to repeat a given set of options, as required in the design. This setup requires at least two follow-up intents to the browse-topics intent (one for requesting the next set of options, and one for repeating the current set of options).</p><h3>Create the browsing flows</h3><p>Given that there are multiple browsing flows in the Action (e.g., browsing for topics, browsing for sessions within a topic, etc), one solution would have been to create a top-level “next option” intent to be shared by any of these flows. We decided against this for a few reasons. Firstly, separating these flows into multiple follow-up intents creates more conceptual separation. Secondly, handling a single “next option” intent would have required extra logic and the use of special contexts in Dialogflow to fetch the correct “next” set of options within a browsing flow. Instead, we duplicated the necessary follow-up intents for each browsing flow, resulting in sets of intents that looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*MBX_z-gszepWHFvd" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*wU3-NKkpvkPbeJzr" /></figure><h3>Handle context conflicts</h3><p>How do follow-up intents work? They define a directional relationship between intents through <a href="https://dialogflow.com/docs/contexts">contexts</a>. The follow-up intents define a context (the input context) which must be active in order for Dialogflow to match that intent. The top-level intent activates this same context (the output context) when it has been matched. In our case, the browse-topics-next and browse-topics-repeat intents require the browse-topics-followup context to be active in order to be matched. The browse-topics intent activates this context when the user says they want to browse the available topics.</p><p>The other key component to recognize is the clearing of the other browsing flow contexts (setting their lifespans to 0). This is a pattern used in each of the top-level browsing intents (browse-sessions and show-schedule). Why is this necessary? Well, let’s imagine those are not there and the user engages in the following dialog (paraphrased):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7b4qtTnmIW_GwAiXyp5ylw.png" /></figure><p>What would happen next? Well, without each top-level intent clearing the contexts of the other browsing flows, it is not guaranteed that the user’s last query would match the browse-sessions-next intent. Instead, it might match the browse-topics-next intent and give the user the next set of topics even after they’ve already chosen a topic!</p><p>This is because both the browse-topics-followup and browse-sessions-followup contexts would be active. We prevent this by clearing the followup contexts of other browsing flows at each top-level browsing intent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n95I0Hp1UGg8vnUCjB3ueA.png" /></figure><p>In developing the agent, we started by building out the top-level intents. These are the easiest to create and test, since there are no contexts to keep track of.</p><h3>Collaborating on the agent (version control)</h3><p>From the start, we could tell that developing this agent would require a heavy amount of collaboration. For that reason, we decided our only option was to include the agent in version control along with our backend logic. Moreover, the agent would live in our Git repo as an unzipped agent directory. Because of this, our code reviews could catch issues in any newly added intents or contexts. This was an important step in creating shared knowledge about the inner workings of the agent and nipping problems in the bud.</p><h3>Implementing the webhook</h3><p>Of course, our webhook powered this entire agent. We chose to use <a href="https://firebase.google.com/docs/functions/">Cloud Functions for Firebase</a>, as we do with all our <a href="https://developers.google.com/actions/samples/github">samples</a>, as it provides a very cost-efficient and straightforward webhook implementation for a service like ours. There’s no need to worry about scaling the service in times of extreme demand or paying for idle time when traffic calms. Moreover it provides first-class integration with other Firebase services, like <a href="https://firebase.google.com/docs/firestore/">Firestore</a> and <a href="https://firebase.google.com/docs/auth/">Auth</a>.</p><h3>Organize the prompts</h3><p>One of the first problems to solve in the logic was having some way to identify the current time during a given dialog. Our design provided a slightly different conversational experience for users depending on the current date (before, during, or after the event). To solve this, one of our first <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/commit/08652898e9ab3496e492c7c4f501f91304101f3c">commits</a> used the <a href="https://github.com/actions-on-google/actions-on-google-nodejs">Actions on Google Node.js Client Library</a> <a href="https://actions-on-google.github.io/actions-on-google-nodejs/interfaces/dialogflow.dialogflowapp.html#middleware">middleware</a> function to pre-process the current user “phase” before every turn of dialog. This phase string value (either “pre”, “during” or “post”) was attached to the global <a href="https://actions-on-google.github.io/actions-on-google-nodejs/classes/dialogflow.dialogflowconversation.html">conv</a> object when fulfilling a given intent. The prompts could then be chosen conditionally based on that value.</p><p>But how would we choose the appropriate prompt for a given phase? Certainly it would not have been wise to use a giant “if/else” within each intent to map the phase to a given prompt. Furthermore, the phase was not the only condition on which to choose a prompt in the design. Other conditions included:</p><ul><li>Whether this was a returning user or a first-time user dialog</li><li>Whether they were on a screen or speaker device</li></ul><p>For each of these conditions, the following had to be chosen and sometimes randomized:</p><ul><li>The <a href="https://developers.google.com/actions/assistant/responses#simple_responses">Simple Response</a>/<a href="https://developers.google.com/actions/assistant/responses#basic_card">Basic Card</a>/<a href="https://developers.google.com/actions/assistant/responses#list">List</a> elements of the response</li><li><a href="https://developers.google.com/actions/assistant/responses#suggestion_chip">Suggestion</a> chips shown after the response</li><li>Fallback responses to use in case the following user utterance is not matched to a Dialogflow intent</li><li>No-input responses to use in the case that the user responds with silence on a speaker device</li></ul><p>To account for all of this variation, it was clear that we needed a dedicated, flexible data structure for prompting. In the <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/tree/master/functions/prompts">prompts</a> directory of our source code, you can find this data structure represented for each of the major use cases of the Action (top-level questions and each browsing flow). The data structure was organized as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*jQaFM35H2AJwQLGxJVqA2A.png" /></figure><p>All prompts were stored in the prompts directory, and the common parsing logic lived in a single JavaScript <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/blob/master/functions/prompts/common/utils.js#L84">file</a>.</p><p>In separating the logic and prompt string data, we also ended up splitting the logic across 3 major flows:</p><ul><li>“Static” questions, mostly corresponding to top-level Dialogflow intents</li><li>Menus for browsing (either topics or sessions within a topic)</li><li>Accessing and browsing one’s schedule</li></ul><p>Each of these became a separate sub-directory of the prompts directory, with a utils.js file encapsulating the application logic unique to each. The driver logic in the root app.js file routes intent handling to any of these utils.js files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/0*7CvTeSwNaayJZ_K_" /></figure><h3>Get conference data: a lesson about Cloud Function caching</h3><p>In order to fulfill any one of the browsing flows, we needed a dedicated module for fetching conference data. The conference data for the event is stored in a JSON file hosted on the cloud. So, what’s the easiest way to handle this? One option was to fetch the JSON data to the Cloud Function once on wake, store it in a global scope variable, pre-process it, and access it from there <a href="https://cloud.google.com/functions/docs/bestpractices/tips#use_global_variables_to_reuse_objects_in_future_invocations">across future invocations</a>. The major problem here is that the JSON file changes content quite a bit (sessions being added during development, livestream links being added during the event, etc), and the Cloud Function may stay awake for quite some time after waking. We needed a real-time solution.</p><p>Our first approach to fixing this was to create a global async function (Promise) that fetched, pre-processed, and returned the JSON data, and then call this function from the intent handling logic. The idea was that by declaring this as a reusable function, we could pull fresh data in every dialog. The interesting problem here was that globally declared functions were run on wake, and as a Promise, the resolved value is cached for future calls. This issue? It created a lot of confusion as to why new data was not being shown during testing!</p><p>The solution, then, was to create a class called <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/blob/master/functions/event/conference.js">ConferenceData</a> that is <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/blob/master/functions/app.js#L80">newly instantiated</a> for each turn of dialog. This class fetches the data when it is first needed, pre-processes it (including clean up, de-duplication, etc), and caches it for the remainder of that function execution. This guaranteed fresh data at all times, and even lowered the response time for the user in cases where a single function execution accessed the data multiple times.</p><p>An astute observer might recognize something wrong here. This solution meant that conference data is potentially fetched and pre-processed at each turn of dialog. This is not an ideal solution given the relative cost of these operations. One easy fix would have been to store the JSON data in <a href="https://developers.google.com/actions/assistant/save-data">conv.data</a> at the start of each dialog, but the size of the data simply did not make sense for this string data.</p><p>The ultimate solution that we considered, but had to drop due to time constraints, was to build a backend system that would fetch, pre-process, and cache the conference data periodically. This would provide near real-time accuracy of the data and significantly lower latency by separating the application logic from the parsing logic. In any future project, I’d consider this a far higher priority task, as it directly affects the user experience. In our case, we had something that worked.</p><h3>Cutting scope</h3><p>So, what else did we leave out? As with any software project, we encountered all kinds of limitations, which resulted in cutting scope. For example, one feature that we cut for time was allowing the user to say whether or not a given set of options sounded interesting to them. The initial design called for dialogs like the following:</p><blockquote>User: “What are all the Android talks?”</blockquote><blockquote>Action: “First there is Android Talk 1, then there is Android Talk 2. Do either of those sound good to you?”</blockquote><blockquote>User: “Yeah”</blockquote><blockquote>Action: “Okay, which one?”</blockquote><blockquote>User: “Let’s hear about…”</blockquote><p>and</p><blockquote>User: “What are all the Cloud talks?”</blockquote><blockquote>Action: “There is only one talk on Cloud. Do you want to hear about it?”</blockquote><blockquote>User: “Yeah”</blockquote><blockquote>Action: “Okay, that one…”</blockquote><p>In the case we present two or more options, the user can either choose a topic or indicate that they are or are not satisfied with those options. In the case we present only one option, the user can simply confirm that they want to hear about it. However, we ended up not supporting this flow. Why? The use of “Yeah” has a collision. In responding to the same prompt, the user might say “Yeah” for hearing about a given single session, or to choose among two options. Because of this, we would have needed to create a single Dialogflow intent that was trained with phrases like “Yeah”, “Yes”, etc, and build in application logic to determine how to respond based on the size of the last set of items presented.</p><p>These kinds of features add up in engineering time (and potential bugs introduced) and had to be dropped in order to build a more complete Action. This required prioritization and estimation, especially in balancing engineering time with the potential UX return.</p><h3>QA and Release</h3><p>After the initial development period, we began to conduct basic QA testing on the Action. This entailed running through comprehensive sets of dialogs and ensuring the experience made sense. Were we asking yes/no questions when it made sense? Did the SSML delivery of the prompts make sense? Was any of the speech too fast or too slow? Many of our changes at this point were tweaks of the prompts.</p><p>We also took this time to add <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/commit/fd74f56b64c154a61661b87b179fc1638a3bba85">implicit invocation intents</a>, which increased discovery of the Action for any Google Assistant user looking to learn more about the event.</p><h3>Bugs encountered</h3><p>This was also a critical time for us to catch many of the bugs we hadn’t encountered during development. One of these was a failure to follow the data structure format in one of our prompts, causing the prompt parsing logic to crash. How did this play out for the user? After being greeted by the Action in one of the first dialog turns, if the user said something unrecognized by the Dialogflow agent, they received a static fallback message repeatedly. The intended behavior is to exit the conversation after 3 turns of unrecognized input, as per our design guidelines. Why was this happening? We forgot to <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs/commit/c045cf2a73fedbd9d5bb43b3e33e59396bee8008#diff-bdf544f1f6170081192c787ec29eb032">wrap RichResponse messages in an array</a> in some of our prompts. This was a great example of the relative brittleness of the prompt data structure and the corresponding parsing logic.</p><p>There was yet another critical bug encountered through a brittle process (and a bit of foolishness). While keeping the unzipped Dialogflow agent in the Git repo was a great idea for code review, it led to some issues with deployment. The deployment process was as follows:</p><ol><li>Fetch latest code and Dialogflow agent from centralized Git repo to local codebase</li><li>Deploy Cloud Functions code</li><li>Zip Dialogflow agent directory</li><li>Import agent .zip file into production Dialogflow console</li></ol><p>Where did this break? In the case where our local development copy of the Dialogflow agent had more intents than the centralized version (introduced earlier in development but deleted since then), the Git fetch did not clear out these unnecessary intents as they were not tracked. This meant my deployments to Dialogflow sometimes included extraneous intents! We started noticing this when seeing too many intents in our production agent, and we fixed it by ensuring we started with a fresh download of the dialogflow-agent directory before every deployment.</p><p>Eventually, even with all of the hiccups and limitations, we produced an Action that we were proud of. Today, the Action has a 4.7 star rating on the <a href="https://assistant.google.com/services/a/uid/000000449633043c?hl=en">Assistant directory</a> with over 300 ratings. There is so much opportunity for all of you developers to build even better Actions on the platform, and we hope these posts serve as guiding lessons in achieving that goal.</p><p>Want to see everything we talked about here and even make some contributions of your own? Check out the code <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f287ad31b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developers/how-we-built-it-the-google-i-o-18-action-for-the-google-assistant-7f287ad31b7">How we built it: the Google I/O ’18 Action for the Google Assistant</a> was originally published in <a href="https://medium.com/google-developers">Google Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we designed it: the Google I/O ’18 Action for the Google Assistant]]></title>
            <link>https://medium.com/google-developers/how-we-designed-it-the-google-i-o-18-action-for-the-google-assistant-9370ffbaf9b0?source=rss-85a53664df8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9370ffbaf9b0</guid>
            <category><![CDATA[actions-on-google]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[voice-assistant]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-assistant]]></category>
            <dc:creator><![CDATA[Sachit Mishra]]></dc:creator>
            <pubDate>Tue, 11 Sep 2018 16:58:21 GMT</pubDate>
            <atom:updated>2018-09-16T19:20:24.272Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3-8UVokoxfIC3fOY" /></figure><p>Welcome to a two-part post about designing and building the <a href="https://assistant.google.com/services/a/uid/000000449633043c">Google I/O ’18 Action</a> for the Google Assistant, co-written with April Pufahl (<a href="https://twitter.com/aprilpufahl">@aprilpufahl</a>) . Check out the code <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs">here</a>.</p><p>Making the most of an event like Google I/O is a challenge, whether you’re attending in person, joining an extended viewing party, or watching from home. We, the Actions on Google team, saw this as an opportunity to provide an assistive experience for event-goers and watchers alike. Like we did last year, the team wanted to build an Action for the Google Assistant that would make Google I/O ’18 easier for everyone. In this post, we highlight our design process for the Google I/O ’18 Action, including some of the key challenges we faced and decisions we made. In a following <a href="https://medium.com/google-developers/how-we-built-it-the-google-i-o-18-action-for-the-google-assistant-7f287ad31b7">post</a>, we’ll detail the technical implementation of the proposed design.</p><h3>Why conversation?</h3><p>Conversation design is a powerful approach, but <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/is-conversation-the-right-fit.html">it’s not right for every </a>use case. For example, dialog works well for the task of finding a restaurant’s business hours, but it feels clunky for browsing a dinner menu. Before starting, we wanted to ensure that conversation would add value for our users through its speed, simplicity, and ubiquity. Here’s why we thought dialog would be <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/is-conversation-the-right-fit.html">a good fit</a>:</p><ul><li><strong>Talking about events is intuitive.</strong> Users already have a mental model for talking to event planners and staff about what events are happening and where they’re located.</li><li><strong>Users can ask questions</strong> about an event. Questions let users shortcut to exactly what they want, whether that’s schedule information, session topics, or directions.</li><li><strong>Users need to multitask,</strong> especially when attending events. Their hands or eyes may be occupied, or they may be on the move.</li><li><strong>Users feel comfortable talking</strong>, or typing, about non-personal information, like asking for details about the event.</li></ul><h3>Nailing down the use cases</h3><p>So, how did we start? As with any software project, we started by <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/gather-requirements.html">gathering requirements</a>. We wanted to focus on use cases that</p><ul><li>cover most users, including attendees and non-attendees</li><li>address users’ most frequently asked questions about I/O</li><li>emphasize speed, particularly for attendees during the event</li><li>respect the different goals users have before, during, and after an event</li></ul><p>Trying to design for all of these goals was a massive challenge. We had to create a single solution for different use cases across multiple user profiles before, during, and after I/O. However, we didn’t want to sacrifice high quality for more features since a secondary goal was to <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs">open-source the Action</a> for devs everywhere. In keeping with the tradition of the Android IOSched app, we hoped our solution could be a tool and model for Actions on Google developers.</p><p>To gain insight into users’ most frequently asked questions, we relied on two sources: 1) data from <a href="https://dialogflow.com/docs/analytics#session_flow">Dialogflow analytics</a> for the I/O ’17 Action, and 2) talking to Googlers who’ve worked at I/O in previous years.</p><p>The I/O ’17 Action answered some basic questions about the keynote, location, etc. and allowed users to find sessions by area of interest. Our usage data shows that the session-finding feature was used quite a bit, with a relatively low fallback rate once users started down that dialog path. Most users asked about the keynote or the swag; they didn’t discover our other canned responses for announcements, I/O extended, etc. Based on this, we wanted to focus on improving the user experience for browsing and refining sessions, and enhancing discoverability of the supported FAQs about the event.</p><p>Our talks with Googlers who worked at I/O revealed four major categories for questions asked by attendees:</p><ul><li>General navigation (e.g., Where’s the bathroom?)</li><li>Personal navigation (e.g., Where’s my next session?)</li><li>Event details (e.g., What time is lunch?)</li><li>Location-specific event details (e.g., What’s the next session in this room?)</li></ul><p>Based on this, we wanted to add support for navigation, making it as easy as possible to get around the event. We also wanted to connect to user’s schedules so they could access that navigation quickly and on the go. Each of these would require some compromise when it came to the actual implementation (as you’ll read about in <a href="https://medium.com/google-developers/how-we-built-it-the-google-i-o-18-action-for-the-google-assistant-7f287ad31b7">part 2</a>).</p><p>Given all this information, we created <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/gather-requirements.html#gather-requirements-identify-your-users">user personas and journeys</a> that would guide our designs.</p><h3>Creating a persona</h3><p>Next, we focused on designing the front end of our Action — i.e., the <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/create-a-persona.html">persona</a> users would interact with. The goal of creating a persona is not to trick the user into thinking they’re talking to a human, but simply to leverage the communication system users learned first and know best: conversation. We focused on the qualities we wanted users to perceive when talking to our Action — that is, someone who is</p><ul><li>practical and straightforward</li><li>techie</li><li>enthusiastic</li><li>an I/O expert</li></ul><p>We decided that a Google Developer Expert (GDE) is a great example of a character who embodies these characteristics. These are curious, inventive, enthusiastic folks the developer community looks to as models of openness and expertise. At this point, we made the mistake of preparing a page-long biography of the persona, including their life story, motivation, and technological preferences. <strong>This was overkill</strong>; a good persona should be simple enough to keep top-of-mind when writing dialog. So, we cut it down to <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/create-a-persona.html#create-a-persona-examples">a short paragraph</a>. We wanted to evoke the sense of a game master by calling our persona the Keeper of I/O-Specific Knowledge. For new users, this name is spoken out as part of the greeting. Notice that the name has an acronym… KIOSK. Pretty punny, huh?</p><p>Next, we had to choose a voice to give life to this persona. Recording voice audio would have been ideal to improve the experience, but, considering the variation in prompt strings, this would have made implementation extremely tedious. Instead, we reviewed the available voices in the Actions on Google catalog. Of the available <a href="https://developers.google.com/actions/localization/languages-locales#en-us_tts_voices">TTS voices for United States English</a>, we ranked “Female 2” highest on the attributes “practical”, “techie” and “straightforward”.</p><h3>Creating sample dialogs</h3><p>Once we had a clear picture of who’s communicating (our users and our persona) and what they’re communicating about (our use cases), we could start <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/write-sample-dialogs.html">writing the dialog</a>. This let us experiment to get a quick, low-fidelity sense of the sound and feel of the interaction, without the technical distractions of code notation, complex flow diagrams, recognition-grammar issues, etc. We started with a spoken dialog in which our user (Anna) is currently at I/O and wants information about the remaining sessions that day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*U52mqGXE6yOXc1GU" /></figure><p>First, we had to write the <a href="https://designguidelines.withgoogle.com/conversation/conversational-components/greetings.html">greeting</a> in a way that 1) welcomed the user, 2) set expectations, and 3) let the user take control. Since the name “Google I/O 18” doesn’t really give a sense of what our Action can do, we described it as a “launchpad”. Critically, we wanted to tailor the user experience for attendees and non-attendees since they have different goals in mind (e.g., getting directions versus watching the livestream). So we asked a narrow-focus <a href="https://designguidelines.withgoogle.com/conversation/conversational-components/questions.html">question</a> (in this case, a yes/no question). At this point, we did a little more to set expectations by leveraging our persona’s title (the Keeper of I/O-Specific Knowledge) as well as providing the user with some <a href="https://designguidelines.withgoogle.com/conversation/conversational-components/suggestions.html">suggestions</a> (manage schedule, find things to do, get directions) before asking a wide-focus <a href="https://designguidelines.withgoogle.com/conversation/conversational-components/questions.html">question</a> (which do you need?). At this point, the user could choose one of the suggestions or shortcut to anything else our Action supports (e.g., keynote or browse sessions).</p><p>In the rest of <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/write-sample-dialogs.html#write-sample-dialogs-sample-dialogs-for-beginners">this sample dialog</a>, we created menus to guide the user to the various use cases we wanted to support. This is helpful because users won’t automatically know what they can ask for when they start talking to our Action. At the same time, we didn’t want to overwhelm the user with too many options, so we tried to place them into intuitive categories. Instead of offering a flat menu of nine options, we grouped them into these 3 categories:</p><p>Learn from the experts</p><ul><li>Keynote</li><li>Sessions</li><li>Office hours</li><li>App reviews</li></ul><p>Demos</p><ul><li>Codelabs</li><li>Sandboxes</li></ul><p>Kick back</p><ul><li>Food</li><li>Swag</li><li>After hours</li></ul><p>We ended up creating 10 more sample dialogs, each of which highlighted a different set of use cases.</p><h3>Flesh out the design</h3><p>Once we had a few sample dialogs, we started summarizing the <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/write-sample-dialogs.html#write-sample-dialogs-high-level-flows-for-beginners">high-level flow</a> and logic of the conversation. In doing this, it became clear that the dialog structure was multidimensional — that is, variation in the dialog was dependent on not just one or two user traits, but all three of the following:</p><ul><li>Whether the user was <strong>new to the Action</strong> or a returning user</li><li>Whether the current <strong>date of the session</strong> was before, during, or after I/O</li><li>Whether the user was <strong>attending I/O</strong> or not</li></ul><p>This left us with something like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/0*wINYs3G-Odjrqmnj" /></figure><p>Fundamentally, we accounted for the following use cases:</p><ul><li>Asking for general information about the event (date, location, keynote, etc.)</li><li>Finding things to do or directions at the event (for attendees)</li><li>Browsing the schedule</li><li>Browsing one’s own schedule</li></ul><p>The general event questions were fairly simple to handle. For each question about the keynote, swag, food, etc., we came up with a set of canned responses based on the date relative to the event. We made sure to listen to each response as rendered in text-to-speech (TTS) and used Speech Synthesis Markup Language (SSML) as needed. Typically, we added silence to make the phrase sound more natural, allowing our persona to speak in “breath units” like a person would.</p><p>For example, if the user asked anything about the clothing to wear at I/O, we designed a response along the following lines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ga0230cka_4SPNbPibms6w.png" /></figure><p>Notice that there were multiple conditions considered for each prompt. Furthermore, when it came time to <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/scale-your-design.html">scale our design</a> from a voice-only to a multimodal conversation, we were able to reuse our spoken prompts as the basis for the display prompts, condensing them whenever possible to optimize for scannability.</p><p>For the schedule-browsing use case, we were able to expand on the flow built for the I/O ’17 Action, allowing users to browse office hours as well. We also challenged ourselves to address the changing user needs before, during, and after I/O by offering a smarter browsing experience. For instance, before the event we might want to present all the options to the user. However, during the event, we might only want to provide the sessions upcoming within the same day, only allowing them to browse the next day’s sessions after they’ve exhausted that first list.</p><p>We also tailored the experience depending on whether the user was interacting with the Assistant on a device with or without a screen. If the user were talking to Google Home, we’d only present six randomized topics (of 17 total) in the spoken prompt to avoid overwhelming the user. If the user were on their phone, we’d leverage the screen to show a list of all 17 topics in alphabetical order. Notice the different prompting and the use of more natural speech like “There’re” instead of “There are”. The design grew in ambition here, and we had to make compromises in implementation, but we had something that made sense for any user in any circumstance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*jSwuiRl4EcIdGXn_" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*qf7_IUmxEsrhoQZM" /></figure><p>The use cases for getting directions and browsing one’s own schedule were largely dependent on implementation, so it was hard to sketch out at this stage. There were dependencies on the Android IOSched app (if we decided to open it for users on Android) and on the scheduling backend. Given the preview state of <a href="https://developers.google.com/actions/identity/google-sign-in">Google Sign-In for Assistant</a>, we couldn’t use it for this Action. We left this section of the dialog open-ended until these factors were more clear.</p><p>Once we felt we had a good skeleton, it was time to <a href="https://designguidelines.withgoogle.com/conversation/conversation-design-process/design-for-the-long-tail.html">design for the long tail</a>. This meant handling as many cases as possible where things can go wrong. As you might imagine, it became pretty complicated. The design holds multiple logical decision points, error and rejection handling, menus within menus, and more. We considered every user response for every prompt, including an unknown response (or fallback), a rejection, an acceptance, a system failure, etc. Furthermore, we had to tailor each dialog path and prompt for each Assistant surface, from smart speakers to phones. We created specific prompts for “reentry” into a given state of dialog, where the user finds themselves at a decision point similar to one they’ve encountered before. Remember, the persona had to be woven throughout all of this.</p><p>We were able to leverage some of the same patterns throughout the Action. For instance, the <a href="https://designguidelines.withgoogle.com/conversation/conversational-components/errors.html">error-handling strategy</a> remained largely the same throughout the dialog. It was usually a variation of the following, adjusted based on the context of the last question that was asked:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*hjCnU-X3z0O3eXfi8Q5hpA.png" /></figure><p>Notice that we spelled out “4 0 4” as three numbers to ensure it would be rendered appropriately in TTS. Even when writing error prompts, we continued to write to our persona — including a techie joke while remaining practical and straightforward.</p><p>At this point, we had an ambitious and comprehensive design. We accounted for all intended use cases and as many edge cases as possible with some space for technical dependencies. Now, we had to actually build it…</p><p>Check out the <a href="https://medium.com/google-developers/how-we-built-it-the-google-i-o-18-action-for-the-google-assistant-7f287ad31b7">second portion</a> of this post, where we discuss the implementation of everything above. Curious to see how it ended up? You can try the Google I/O ’18 Action for the Google Assistant today! Just say “Hey Google, talk to Google I/O”. We’ve also open-sourced the project on Github <a href="https://github.com/actions-on-google/dialogflow-iosched-nodejs">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9370ffbaf9b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developers/how-we-designed-it-the-google-i-o-18-action-for-the-google-assistant-9370ffbaf9b0">How we designed it: the Google I/O ’18 Action for the Google Assistant</a> was originally published in <a href="https://medium.com/google-developers">Google Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>