<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gabriele Randelli on Medium]]></title>
        <description><![CDATA[Stories by Gabriele Randelli on Medium]]></description>
        <link>https://medium.com/@gabrielerandelli?source=rss-8f61f3a4e09c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9al--x_1YX-ZFdmPoWwiMw.jpeg</url>
            <title>Stories by Gabriele Randelli on Medium</title>
            <link>https://medium.com/@gabrielerandelli?source=rss-8f61f3a4e09c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:25:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gabrielerandelli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Designing Data Store Hybrid Agents with Dialogflow CX & Vertex AI Agents]]></title>
            <link>https://medium.com/google-cloud/designing-data-store-hybrid-agents-with-dialogflow-cx-vertex-ai-agents-070082f07cb4?source=rss-8f61f3a4e09c------2</link>
            <guid isPermaLink="false">https://medium.com/p/070082f07cb4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dialogflow-cx]]></category>
            <category><![CDATA[data-stores]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[vertex-ai-agent-builder]]></category>
            <dc:creator><![CDATA[Gabriele Randelli]]></dc:creator>
            <pubDate>Fri, 28 Jun 2024 11:30:12 GMT</pubDate>
            <atom:updated>2024-07-02T04:09:56.506Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U1eo6P48xaeZOFiD" /></figure><h3>Introduction</h3><p>In the first <a href="https://medium.com/p/8d39035b985d">article</a> of this series we’ve designed a purely intent-based agent with deterministic flows.</p><p>Deterministic agents are effective when a repeatable and auditable process is mandatory (e.g. think of an offer subscription). Companies have heavily invested in the past years in these agents, hence they’re in favor of re-using, at least partially, their current assets.</p><p>On the other hand, generative AI unleashes new opportunities for customer service, such as the possibility to create more natural and engaging conversations, answering to a broader range of topics without the need to explicitly structure them as intent-based flows. Such an approach is particularly appreciated in conversational commerce scenarios.</p><p><a href="https://cloud.google.com/dialogflow/cx/docs/basics">Dialogflow CX</a> offers the possibility to take the best out of both the paradigms, combining intent-based and generative flows with built-in <a href="https://cloud.google.com/dialogflow/cx/docs/concept/generative">generative features</a>: <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/agent-apps">agent apps</a>, <a href="https://cloud.google.com/dialogflow/cx/docs/concept/generative/generative-fallback">generative fallbacks</a>, <a href="https://cloud.google.com/dialogflow/cx/docs/concept/generative/generators">generators</a> and integration with <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent">data store agents</a>. Within the scope of this article, we are going to address data store agents and generative fallbacks.</p><p>In particular, our goal is to:</p><ul><li>Extend our car rental chatbot designed in the first article, mixing deterministic and generative flows by integrating data stores, thus designing a <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent-hybrid"><em>data store hybrid agent</em></a>;</li><li>Enable generative fallback, to gently handle those situations where the user’s intent is not properly matched;</li><li>Provide some best practices on hybrid agents, mainly based on empirical experiences.</li></ul><h3>New expectations from conversational agents</h3><p>The agent presented in our first article is a car rental bot conceived to handle car booking requests. This has been implemented with an intent-based paradigm. The main ingredients of a pure deterministic approach are: each flow is a state diagram where the agent orderly transitions from one page to another by detecting intents, collecting parameters and fulfilling the customer’s requests.</p><p>Let’s assume we would also like to support our beloved customers on generic inquiries, such as: what’s a luxury car, what manufacturers are available, what’s the maximum speed of a specific car model, and so on. These questions shift the role of agents from post-sales assistance to presales, a key activity in conversational commerce experiences.</p><p>We can recap these new emerging requirements as follows:</p><ul><li>Questions like these may happen at any time, even in the middle of the booking process — The agent should intercept these conversation digressions, eventually reverting to the current page, without restarting the deterministic process from scratch every time;</li><li>Open questions pose a maintainability challenge in terms of data volumes and variety. Codifying each possible question with an intent-based approach is almost impossible and we need a more scalable approach;</li><li>Generating answers is fine, but controlling the final output is even more important — above all, we want to ground the answer to our company knowledge base, not to the LLM knowledge.</li></ul><p>A purely deterministic approach cannot fulfill these needs with low effort. We need to shift our paradigm.</p><h3>The Role of Data Stores</h3><p>Generative AI fits in very well to address the aforementioned new requirements. In particular, the scope of this article is mainly focused on integrating <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store#:~:text=Data%20stores%20are%20a%20collection,into%20a%20coherent%20agent%20response.">data stores agents</a>. For the sake of terminology, take into account that data store agents are also known as <em>chat apps</em> and at the time of this article they are part of <a href="https://cloud.google.com/dialogflow/vertex/docs">Vertex AI Agents</a>, Google Cloud brand new natural language understanding (NLU) platform built on LLMs to create agents. We’ll overview this platform in the third article of this series.</p><p><a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store">Data stores</a> are collections of indexed websites and documents, each of which reference your enterprise data. Three are the main ingredients when adopting data stores for AI agents:</p><ul><li>Grounding the customers’ answers in your enterprise data;</li><li>Generating a coherent answer with the underlying LLM’s capabilities;</li><li>Considering the past conversation between the user and the agent.</li></ul><p>Data stores in Google Cloud Platform can be classified according to the following dimensions:</p><ul><li><em>Data source</em>: websites, BigQuery, Cloud Storage, 3rd party systems via connectors;</li><li><em>Data format</em>: structured (e.g. a table or a FAQ) <em>vs</em> unstructured data (e.g. a PDF file);</li><li><em>Metadata availability</em></li><li><em>Configuration</em>: parsing, chunking, and so on</li></ul><p>It’s important to highlight a few aspects when leveraging the built-in integration provided by Dialogflow CX:</p><ul><li>Do not confuse agent data stores, addressed in this article, with <a href="https://cloud.google.com/generative-ai-app-builder/docs/create-datastore-ingest">search data stores</a><strong>;</strong></li><li>The only structured data store type supported is FAQ;</li><li>Apps with both chunked and non-chunked data stores are not supported.</li></ul><p>In particular, this latter is important in our scenario. Since we’re going to use both an unstructured and structured data store and chunking is only supported for unstructured data stores, we need to disable chunking for both. Whenever any of the current limitations affect the accuracy of your implementation, my advice is to bypass the built-in integration and to directly call a search data store via the Vertex AI Search API.</p><h3>Hybrid Agent Main Concepts</h3><h4>Design</h4><p>We are going to extend our car rental bot by adding three new components:</p><ul><li>A <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store#structured">structured data store</a>, based on a FAQ CSV file, for specific questions already addressed and stored in your company knowledge base;</li><li>An <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store#with-metadata">unstructured data store</a> with metadata, based on a set of PDF files, for more open questions;</li><li>Last, we are enabling <a href="https://cloud.google.com/dialogflow/cx/docs/concept/generative/generative-fallback">generative fallback</a> across the agent to handle scenarios with <em>no-match</em> intent detection.</li></ul><p>All these features will be activated in the <strong>start page</strong> of the <strong>default start flow</strong>, hence they will be available across the whole agent, to intercept these questions in any page of every deterministic conversation flow.</p><p>Dialogflow CX evaluates the end-user input in the following order of preference:</p><ul><li>Intent match for routes in scope;</li><li>FAQ data store content;</li><li>Unstructured data store content.</li></ul><p>This evaluation order always prioritizes the intent-based fulfillment, when available, since this is the most deterministic. Again, we are not taking down our existing agent; Generative AI is extending the current implementation.</p><h4>Architecture</h4><p>The architecture is almost the same of the predictive bot:</p><ul><li>The chat widget embedded in your site is based on <a href="https://cloud.google.com/dialogflow/cx/docs/concept/integration/dialogflow-messenger">Dialogflow Messenger</a>;</li><li>The agent is implemented using Dialogflow CX;</li><li>The agent data stores are created with <a href="https://cloud.google.com/products/agent-builder?hl=en">Vertex AI Agent Builder</a>;</li><li>The existing Google Cloud Storage bucket also contains the raw data indexed by the data stores.</li></ul><p>Below is a comprehensive architecture diagram. Both the deterministic and generative flows have been reported. However, feel free to ignore some of the components (webhook, cloud function, Gemini API), since they’re related to the agent multimodal capabilities presented in the <a href="https://medium.com/p/8d39035b985d">first article</a> of this series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*S8CpyZnMCdeVgQiT" /></figure><h4>Best Practices</h4><p>Before moving to the actual implementation, which is in scope for the next section, it’s worth pinpointing some best practices with data stores, to take the best out of this technology.</p><p>The following high-level considerations for data store agents are only based on my empirical experience in former projects:</p><ul><li>A single data store is rarely the solution to all your needs — In most of the scenarios, you need a finer grain knowledge decomposition, with different data stores serving different sub-domains;</li><li>Intents can disambiguate the knowledge subdomain and redirect the user to the right data store — I know the best would be to bind a data store at agent level to solve all your needs, but I’m generally more in favor of using intents to disambiguate the user’s real intent;</li><li>If you’re looking for full control and repeatability, go for pure intent-based agents. On the other hand, if you’re looking for information not present in your company’s knowledge base, turn to a webhook directly invoking the LLM. Data stores sit in the middle: when you need to address open questions, but grounded on your own data.</li></ul><p>Turning to our specific scenario, why did we choose to use a structured data store?</p><p>Whenever such a knowledge base is available in a company, retrieving answers from FAQs is generally more accurate and less prone to LLM hallucinations than unstructured data. Again, when modeling an agent, my suggestion is to always analyze whether you already own this kind of knowledge in your enterprise systems, or if you can implement an easy process to acquire it.</p><p>You may argue that most of the FAQ questions can be addressed with an intent-based approach, and it’s true. However, modeling intents is more time consuming than using a FAQ and the nature of these questions is not so critical to require a fully deterministic approach (e.g. a car damage meanwhile driving is a critical problem and a deterministic process would definitely be the best approach).</p><p>Turning to the unstructured data store, can we answer these questions with the LLM’s knowledge? Again, the rationale is to <em>gain more control</em>. By using our own enterprise data, we track which info we want to expose and tailor it to our needs. Moreover, the LLM’s knowledge might be outdated, while we can always provide fresh content.</p><p>You can find additional best practices on Google Cloud’s <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent-performance#improve">documentation</a> or by running a self-service <a href="https://github.com/GoogleCloudPlatform/dfcx-scrapi/tree/main/examples/vertex_ai_conversation">evaluation script</a>.</p><h3>Pimp My Deterministic Agent!</h3><h4>Convert Your Agent</h4><p>Agent data stores, also known as <em>chat apps</em>, are formally part of our new platform <a href="https://cloud.google.com/dialogflow/vertex/docs">Vertex AI Agents</a>. Dialogflow CX can integrate with them through a built-in feature. However, the very first time we add a data store to our legacy Dialogflow CX agent, we need to convert the agent app. Bear in mind that the flow is slightly different if you directly create the data store agent from the Vertex AI Agent Builder console.</p><p>Let’s start from creating a new Dialogflow CX agent and <a href="https://cloud.google.com/dialogflow/cx/docs/concept/agent#export">restore</a> the car rental agent defined in our first article an exported version of this agent is available on <a href="https://github.com/grandelli/dfcx-geminiprovision/tree/main/dfcx-bot">GitHub</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jQ2G1FUBdC2FXV1j" /></figure><p>Next, expand <em>Start Page</em>, select <em>Add State Handler </em>and tick <em>Data Stores</em> to visualize the corresponding group in the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/0*TD9khKq-EYLnhffm" /></figure><p>In the new Data Store group click on <em>+</em> and you’ll notice on the right-side panel the following message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/0*W5-qM-Ku9zwBYUK3" /></figure><p>Click on <em>Create Vertex AI Search and Conversation app</em> (names may change in the future) and you’ll be redirected to the Vertex AI Agent Builder console. Accept the terms, assign a <em>Company Name</em> and continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3-Ah3eqGzXZyLau8" /></figure><p>We now move to the data store creation section, which will be addressed in the next two sections.</p><h4>Structured Data Store</h4><p>In our example, the structured data store is an <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store#structured">FAQ</a> stored in Google Cloud Storage as a CSV file, without any metadata.</p><p>Let’s put in practice what we have so far described. Copy the snippet below, reporting some basic user inquiries, in a CSV file named <em>car_rental_faq.csv</em> and upload it to a GCS bucket, within a sub-folder <em>car-rental-faq</em>.</p><pre>question, answer<br>What kind of economy car do you have?, Economy cars are Mitsubishi Mirage and Nissan Versa.<br>What kind of luxury car do you have?, Luxury cars are Chevy Tahoe and Dodge Charger.<br>What kind of cars do you have?, Economy cars are Mitsubishi Mirage and Nissan Versa. Luxury cars are Chevy Tahoe and Dodge Charger<br>What number can I call to get info?, +1 123 345 6789</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*517Mp7qhIrURIPyD" /></figure><p>Go back to the Vertex AI Agent Builder UI and create the first data store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H0MvEHDCKrb2IjgE" /></figure><p>Click on <em>Cloud Storage</em> and select the last radio button <em>Structured FAQ data</em>. Click on <em>File</em> and select on the right side panel the uploaded file in the GCS bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3PTeQm2SoanYC76R" /></figure><p>Assign a name to the datastore and create it.</p><p>Now you have to wait a few minutes to let the data store propagate to Dialogflow CX (refresh the UI page to see the applied changes).</p><p>Proceed with the following steps:</p><ul><li>Go back to the Dialogflow CX UI and double check that the just created data store is visible and selected in the <em>FAQ documents</em> combo;</li><li>Save the page and close the side panel.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SnjcLRB5Jabl9buA" /></figure><h4>Unstructured Data Store with Metadata</h4><p>The unstructured datastore is composed of a set of PDF files, sort of booklets for each car. The main purpose is to provide generic information about each model, to fulfill open questions (e.g. <em>what’s the fastest car you have?</em>).</p><p>Below is reported a small snippet of these PDF files:</p><pre>The Chevrolet Tahoe is a full-size SUV known for its spacious interior, powerful engine options, and impressive towing capabilities. Here&#39;s a breakdown of its main technical details:<br>Engine Options:<br>● 5.3L V8 engine: This is the standard engine, delivering 355 horsepower and 383 lb-ft of torque. It offers a good balance of power and fuel efficiency.<br>● 6.2L V8 engine: Available on higher trim levels, this engine boasts 420 horsepower and 460 lb-ft of torque, providing exceptional acceleration and towing capacity.<br>● 3.0L Duramax Turbo-Diesel engine: For those prioritizing fuel economy, this diesel engine offers 277 horsepower and 460 lb-ft of torque while delivering impressive fuel efficiency.<br>Transmission: All Tahoe models come equipped with a smooth-shifting 10-speed automatic transmission, ensuring optimal performance and efficiency.<br>Drivetrain:<br>● Rear-wheel drive (RWD): Standard on LS, LT, RST, and Z71 trims.<br>● Four-wheel drive (4WD): Available on all trims and standard on Premier and High Country. Offers enhanced traction and control in various driving conditions.</pre><p>You can find some sample files in my <a href="https://github.com/grandelli/dfcx-datastore-hybrid-agent">GitHub repository</a>. Alternatively, feel free to create your own documentation. Upload the files to the same GCS bucket, this time within a sub-folder <em>car-rental-kb/files</em> and ensure no other file is in this folder.</p><p>Since we are going to use metadata, we need to create a <a href="https://jsonlines.org/">JSON Line</a> file, each line describing the document to import associated to its URI on Cloud Storage. You must follow the format described in the Google Cloud <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store#with-metadata">documentation</a>.</p><p>In our case, we are going to use some metadata later on, and that’s why I’ve explicitly added an additional field <em>car-type</em>. Our JSONL file is below reported.</p><pre>{ &quot;id&quot;: &quot;d001&quot;, &quot;content&quot;: {&quot;mimeType&quot;: &quot;application/pdf&quot;, &quot;uri&quot;: &quot;gs://car-rental-bkt/car-rental-kb/files/chevrolet.pdf&quot;}, &quot;structData&quot;: {&quot;title&quot;: &quot;Chevrolet Tahoe&quot;, &quot;car-type&quot;: &quot;Luxury&quot;} }<br>{ &quot;id&quot;: &quot;d002&quot;, &quot;content&quot;: {&quot;mimeType&quot;: &quot;application/pdf&quot;, &quot;uri&quot;: &quot;gs://car-rental-bkt/car-rental-kb/files/dodge.pdf&quot;}, &quot;structData&quot;: {&quot;title&quot;: &quot;Dodge Charger&quot;, &quot;car-type&quot;: &quot;Luxury&quot;} }<br>{ &quot;id&quot;: &quot;d003&quot;, &quot;content&quot;: {&quot;mimeType&quot;: &quot;application/pdf&quot;, &quot;uri&quot;: &quot;gs://car-rental-bkt/car-rental-kb/files/mitsubishi.pdf&quot;}, &quot;structData&quot;: {&quot;title&quot;: &quot;Mitsubishi Mirage&quot;, &quot;car-type&quot;: &quot;Economy&quot;} }<br>{ &quot;id&quot;: &quot;d004&quot;, &quot;content&quot;: {&quot;mimeType&quot;: &quot;application/pdf&quot;, &quot;uri&quot;: &quot;gs://car-rental-bkt/car-rental-kb/files/nissan.pdf&quot;}, &quot;structData&quot;: {&quot;title&quot;: &quot;Nissan Versa&quot;, &quot;car-type&quot;: &quot;Economy&quot;} }</pre><p>Last, store this JSONL file in the <em>car-rental-kb</em> folder. The final result should be similar to the picture below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VswfI0El7cJe0JLQ" /></figure><p>To create the unstructured data store we need to repeat the process of the previous section, but this time the Dialogflow CX agent has been already converted:</p><ul><li>Enter the Dialogflow CX bot UI, go to <em>Default Start Flow</em>, select <em>Start Page</em> then <em>Edit Data Stores;</em></li><li>Select on the right side panel the combo <em>Unstructured documents</em> and click on <em>+ Create data store</em>. You’ll be redirected to <em>Create Data Store</em> widget within <em>Vertex AI Agent Builder</em> in the GCP Console;</li><li>Click on <em>Cloud Storage;</em></li><li>Select the last radio button <em>Linked unstructured documents</em>, select <em>File</em> and select on the right side panel the JSONL file;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tkSwRN6VXg7ZGMLe" /></figure><ul><li>Assign a name to the datastore, select <em>Digital Parser</em> and ensure that the <em>Enable advanced chunking configuration</em> checkbox is <strong>disabled</strong>, since we cannot use chunking;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3Dz-gDesX5lebJAW" /></figure><ul><li>Wait a few minutes, go back to the Dialogflow CX UI and double check that the just created data store has been selected in the <em>Unstructured documents</em> combo;</li><li>Save the page and close the side panel.</li></ul><p>To recap, what we’ve so far accomplished is:</p><ul><li>Restoring our previous legacy Dialogflow CX agent;</li><li>Converting this agent to a Vertex AI Agents chat app, to integrate with data stores;</li><li>Adding a first structured data store when converting the agent;</li><li>Adding a second unstructured data store with metadata.</li></ul><h4>Generative Fallback</h4><p>Despite intents and data stores, <em>no-math</em> conditions may still happen. Imagine a scenario where the user asks something significantly out of context. Here is where <a href="https://cloud.google.com/dialogflow/cx/docs/concept/generative/generative-fallback">generative fallback</a> comes in action.</p><p>We can configure a text prompt to instruct the LLM how to respond. Dialogflow CX automatically provides a default prompt and can handle basic scenarios. Otherwise, you can provide your own prompt and use different placeholders to get conversation data. If the response generation is unsuccessful, Dialogflow CX will turn to the old static fulfillment text, if any.</p><p>To properly use generative fallback, a best practice is to properly describe flows and intents, since this is contextual info the generative fallback will leverage.</p><p>Again, since we want to activate this feature across the whole agent:</p><ul><li>let’s go to <em>Default Start Flow</em> and select <em>Start Page</em>;</li><li>Under <em>Event Handler</em> click on <em>View all</em> and on the right side panel select <em>No-match 1;</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c0U9GHLfF7wGHuQ-" /></figure><ul><li>Under the <em>agent responses </em>section enable the generative fallback feature.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/0*kDDkGFmb4Drb_e95" /></figure><p>It’s worth highlighting that below the checkbox there’s also the static fulfillment, to be triggered upon generation failure.</p><p>If you want to change the prompt passed to the LLM, you need to act through the agent settings, as below reported.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DZ8iAzJRyHxI8X7h" /></figure><h4>Generative Settings</h4><p>Let’s review the basic settings of our brand new data store agent. You can access them from <strong>Agent Settings</strong> on the right side of the Dialogflow CX UI, then select the <strong>Generative AI</strong> tab.</p><p>Go to the <strong>Generative agent</strong> section and double check that <em>gemini-1.0-pro-001</em> is the selected model. Or, feel free to choose a newer one, whether available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ltsPnJl4XXKS4cRc" /></figure><p>Go to the <strong>Data store</strong> section and start by enabling <em>grounding</em> and setting the confidence score to <em>medium</em>. This enforces the confidence that all the information in the response is supported by information in the data stores. Tick also the <em>apply grounding heuristics filter</em> to suppress likely inaccurate and hallucinated responses. As you can see, there are different ways to control the generative flow and guarantee high quality responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2K3yn9ZyHNmJ0unY" /></figure><p>You can also add contextual information about the bot’s purpose, which can further improve its quality. See the below snippet for example (whatever your agent language, pass this information in English).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R5BbBHLikhdnB0AF" /></figure><p>Next, select the <em>default summarization prompt</em> and the summarization model (at the time of this article <em>gemini-1.5-flash-001</em> is available). If you want to customize the summarization prompt, follow Google Cloud’s <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent-settings#customize-summarization-prompt">documentation</a> and respect all the hints there provided not to lower the overall quality. If unsure, just stay with the default prompt as in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ctadZjmubhYTHQyo" /></figure><p>Last, select both the <em>Fallback link</em> and <em>Enable Generative AI</em> checkboxes. The former allows, when the generated response is suppressed, at least to return the first retrieved data link from the data store.</p><h3>Dialogflow Messenger</h3><p>The web application conceived in the <a href="https://medium.com/p/8d39035b985d">first article</a> can serve our new agent without any change but the new agent ID, thereby I won’t spend too much time on how to embed the chat widget in your HTML page.</p><p>Nonetheless, there are a couple of potentially useful features in Dialogflow Messenger, which can be useful with data store agents.</p><p>The first one is <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent-performance#personalization">personalization</a>, that is, passing relevant information about the end user to the LLM generating the answer. This information must be passed as JSON format. No specific schema is required and no transformation will be applied under the hood.</p><p>Dialogflow Messenger supports personalization through the <em>setContext</em> method. Let’s modify our previous implementation by adding the following snippet within the <em>script</em> tag:</p><pre>// search personalization metadata<br>const metadata = {<br>  &quot;gps-detected-location&quot;: &quot;NYC&quot;,<br>  &quot;preferred-vehicle-category&quot;: &quot;luxury&quot;,<br>  &quot;devices owned&quot;: [<br>    {<br>      model: &quot;Google Pixel 8&quot;,<br>    },<br>    {<br>      model: &quot;Google Pixel Tablet&quot;,<br>    },<br>  ],<br>};<br>dfMessenger.setContext(metadata);</pre><p>In this example we are passing information about the user device, position and car preferences. This latter is typically retrieved from some backend system (e.g. a CRM). If so, the best is to fetch this information with a webhook on the Dialogflow CX agent side.</p><p>The second feature is related to boost and filter <a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent-performance#search-configuration">search configuration</a>, used to boost, bury and filter documents. In particular, in this article we are introducing boost controls to change search result ranking by applying a boost value (greater than zero for higher ranking, less than zero for lower ranking) to specific documents. In this case a specific JSON format is required:</p><pre>&quot;searchConfig&quot;: {<br>  &quot;boostSpecs&quot;: [<br>    {<br>      &quot;dataStores&quot;: [&quot;DATASTORE_ID&quot;],<br>      &quot;spec&quot;: [<br>        {<br>          &quot;conditionBoostSpecs&quot;: {<br>            &quot;condition&quot;: &quot;CONDITION&quot;,<br>            &quot;boost&quot;: &quot;1.0&quot;<br>          }<br>        }<br>      ]<br>    }<br>  ]<br>}</pre><p>DATASTORE_ID is the full name of the data store in the format <em>projects/your_project_id/locations/your_location/collections/your_collection_name/dataStores/your_datastore_name</em> and can be retrieved from the Vertex AI Agent Builder UI, entering the Data Stores section.</p><p>CONDITION must follow the Vertex AI Agent Builder’s <a href="https://cloud.google.com/generative-ai-app-builder/docs/filter-search-metadata#filter-expression-syntax">filter expression syntax</a> and the boost value is a number between -1.0 (for lower ranking) and 1.0 (for higher ranking).</p><p>Again, Dialogflow Messenger supports search configuration through the method <em>setQueryParameters</em> and we can add a new listener within the <em>script</em> tag in our HTML page:</p><p>In this example, our condition is boosting data sources related to luxury cars, and that’s why we adopted an unstructured data store with metadata, to explicitly associate each document to a car category (economy or luxury).</p><pre>dfMessenger.addEventListener(&#39;df-messenger-loaded&#39;, function (event) {<br>  const searchConfig = {<br>    &quot;boostSpecs&quot;: [<br>      {<br>        &quot;dataStores&quot;: [ &quot;projects/your-project-id/locations/your-datastore-region/collections/default_collection/data-stores/your-datastore-id&quot; ],<br>        &quot;spec&quot;: [<br>          {<br>            &quot;conditionBoostSpecs&quot;: {<br>              &quot;condition&quot;: &quot;car-type: ANY(\&quot;Luxury\&quot;)&quot;,<br>              &quot;boost&quot;: &quot;0.7&quot;<br>            }<br>          }<br>        ]<br>      }<br>    ]<br>  }<br>dfMessenger.setQueryParameters(searchConfig);<br>});</pre><h3>Summary</h3><p>In this article we explored the possibility to create a data store hybrid agent by expanding a legacy deterministic intent-based Dialogflow CX implementation with new Generative AI capabilities. In particular, we have introduced three new built-in Generative AI components: structured data stores, unstructured data stores with metadata and generative fallbacks.</p><p>Our main goal was to showcase that generative AI can greatly help in fulfilling open questions grounded to your data, without the need to explicitly model each topic with an intent. This is a scalable approach that allows to position AI agents more into conversational commerce use cases. Furthermore, you don’t need to restart from scratch. Your existing Dialogflow CX agents can be integrated with the new Vertex AI Agents features.</p><p>Below you can find additional resources, and a basic implementation of this article published on my <a href="https://github.com/grandelli/dfcx-datastore-hybrid-agent">GitHub</a> repository. I strongly encourage you to deploy the code and further customize it according to your use case.</p><p>Stay tuned for the last article of this series, where we’ll shift to a pure Generative AI agent app, built with <a href="https://cloud.google.com/dialogflow/vertex/docs">Vertex AI Agents</a>, Google Cloud’s latest technology for agent development!</p><h3>What’s Next?</h3><h4>Documentation</h4><p><a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store">Data Stores</a></p><p><a href="https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent">Data Store Agents</a></p><p><a href="https://cloud.google.com/dialogflow/cx/docs/concept/generative/generative-fallback">Dialogflow CX Generative Fallback</a></p><p><a href="https://cloud.google.com/products/agent-builder?hl=en">Vertex AI Agent Builder</a></p><p><a href="https://cloud.google.com/dialogflow/vertex/docs">Vertex AI Agents</a></p><p><a href="https://github.com/GoogleCloudPlatform/dfcx-scrapi/tree/main/examples/vertex_ai_conversation">dfcx-scrapi</a></p><h4>Github Sources</h4><p><a href="https://github.com/grandelli/dfcx-datastore-hybrid-agent">https://github.com/grandelli/dfcx-datastore-hybrid-agent</a> (the main GitHub repository for this project — contains the modified DFCX agent, the data stores, the revised Dialogflow Messenger client and the existing components coming from the deterministic agent)</p><p>Feel free to leave your comments here or connect with me on<a href="https://www.linkedin.com/in/gabrielerandelli/"> LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=070082f07cb4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/designing-data-store-hybrid-agents-with-dialogflow-cx-vertex-ai-agents-070082f07cb4">Designing Data Store Hybrid Agents with Dialogflow CX &amp; Vertex AI Agents</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create multimodal conversational experiences with Google Cloud Dialogflow CX and Gemini Vision]]></title>
            <link>https://gabrielerandelli.medium.com/create-multimodal-conversational-experiences-with-google-cloud-dialogflow-cx-and-gemini-vision-8d39035b985d?source=rss-8f61f3a4e09c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d39035b985d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[gemini-pro-vision]]></category>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[dialogflow-cx]]></category>
            <dc:creator><![CDATA[Gabriele Randelli]]></dc:creator>
            <pubDate>Fri, 05 Apr 2024 17:09:40 GMT</pubDate>
            <atom:updated>2024-07-02T10:04:27.899Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Boosting your digital assistant by analyzing images with Gemini Vision</p><figure><img alt="Create multimodal conversational experiences with Google Cloud Dialogflow CX and Gemini Vision" src="https://cdn-images-1.medium.com/max/960/0*4MLWReIuU3ApWDX-" /></figure><h3>Introduction</h3><p>Digital assistants are increasingly prevalent in our lives. Almost every company has deployed vocal or chat bots to improve customer service and reduce their operational costs.</p><p>While speech-to-text technologies are getting better over time and leading to more natural conversations with bots, people often prefer non-verbal communications. As an example, there are specific scenarios where a photo is worth hundreds of words.</p><p><a href="https://cloud.google.com/dialogflow/cx/docs/basics">Dialogflow CX</a> is Google Cloud’s technology to implement omnichannel virtual agents. Alongside the possibility to create intent-based deterministic flows, Dialogflow has recently evolved thanks to the Generative AI foundation models released by Google, namely<a href="https://blog.google/technology/ai/google-palm-2-ai-large-language-model/"> PALM 2</a> and<a href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini"> Gemini</a>.</p><p>While Dialogflow CX can natively integrate with Google’s text-to-text Large Language Models through <a href="https://cloud.google.com/dialogflow/cx/docs/concept/generative/generators#escalation_to_a_human_agent">generators</a>, the main scope of this article is to showcase how <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/overview">Gemini 1.0 Pro Vision</a> can extend your agent with powerful multimodal capabilities. Customers submit an image and delegate the bot for extracting the required information, rather than manually input data one-by-one through manifold conversational rounds. Applications are manifold: banking to extract PII data, insurance to validate car damanes, and so on.</p><p>The rest of this article provides a primer to implement such an extension, sketching out the different components and providing an example applied to a car rental bot.</p><h3>Customer User Journey</h3><p>Before digging into technical aspects, it’s worth clarifying the expected customer user journey:</p><ul><li>At a specific conversational round, the chat bot asks the customer for uploading the image to be analyzed;</li><li>The customer selects a file from her own device, which is in turn transmitted from the chat widget to the bot on the server-side;</li><li>The bot invokes a back-end component, whose role is to interact with the large language model;</li><li>The back-end component submits the image to the multimodal language model, alongside with a prompt to extract the desired information;</li><li>The output of the LLM is passed back to the back-end, which validates the driving license and return a response code to the bot;</li><li>The bot fulfills the customer’s request by transmitting a verbose response to the chat widget;</li><li>The chat widget displays the extracted details and asks the customer for a confirmation.</li></ul><figure><img alt="Flowchart of the customer user journey with Dialogflow CX and Gemini" src="https://cdn-images-1.medium.com/max/960/0*ztxlXPzStwh7St0o" /><figcaption>Flowchart of the customer user journey with Dialogflow CX and Gemini</figcaption></figure><h3>Architecture</h3><p>The architecture to implement our customer user journey consists of five main technologies:</p><ul><li>The chat widget embedded in your site is based on <a href="https://cloud.google.com/dialogflow/cx/docs/concept/integration/dialogflow-messenger">Dialogflow Messenger</a>;</li><li>The agent itself is implemented using Dialogflow CX;</li><li>The back-end business logic consists of a <a href="https://cloud.google.com/functions/?utm_source=google&amp;utm_medium=cpc&amp;utm_campaign=emea-it-all-en-dr-bkws-all-all-trial-e-gcp-1707574&amp;utm_content=text-ad-none-any-DEV_c-CRE_554879513995-ADGP_Hybrid+%7C+BKWS+-+EXA+%7C+Txt+-+Serverless+Computing+-+Cloud+Functions+-+v2-KWID_43700053286402889-kwd-289154592193-userloc_9181234&amp;utm_term=KW_google+cloud+function-NET_g-PLAC_&amp;&amp;gad_source=1&amp;gclid=Cj0KCQjwqdqvBhCPARIsANrmZhPKpvbYk6U25FXdZxLaqIREeIuZo7hf-3NpUBh4ZLS9D7t288IINuwaAunvEALw_wcB&amp;gclsrc=aw.ds&amp;hl=en">cloud function</a>, triggered by a Dialogflow’s <a href="https://cloud.google.com/dialogflow/cx/docs/concept/webhook">webhook</a>;</li><li>The image analysis is executed by invoking the <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/overview#gemini-api">Gemini 1.0 Pro Vision</a> API;</li><li>A staging <a href="https://cloud.google.com/storage?hl=en">Google Cloud Storage</a> bucket temporarily stores the uploaded images.</li></ul><p>Below is reported an architecture diagram:</p><figure><img alt="High-level architecture diagram with the main GCP components" src="https://cdn-images-1.medium.com/max/940/0*cDhKstGBgzgTughh" /><figcaption>High-level architecture diagram with the main GCP components</figcaption></figure><p>Let’s start by <a href="https://cloud.google.com/storage/docs/creating-buckets">creating</a> the staging GCS bucket with the following characteristics:</p><ul><li>Make the bucket <a href="https://cloud.google.com/storage/docs/access-control/making-data-public#buckets">public</a> by assigning <em>allUsers;</em></li><li>Add a 5-minute <a href="https://cloud.google.com/storage/docs/using-bucket-lock">bucket retention policy</a> to remove the uploaded files as soon as possible.</li></ul><p>Assign the bucket whatever name you prefer. Pick up your preferred region (just ensure that you’ll be using the same region for the rest of this article). You’re going to re-use the bucket later on.</p><h3>Dialogflow Messenger</h3><p>Dialogflow Messenger is a lightweight web client for Dialogflow agents. It enables a seamless integration with DFCX bots and it also supports the most recent Generative AI features available.</p><p>Let’s start importing the Dialogflow Messenger JS library:</p><pre>&lt;script src=&quot;https://www.gstatic.com/dialogflow-console/fast/df-messenger/prod/v1/df-messenger.js&quot;&gt;<br>&lt;/script&gt;</pre><p>The first step is to instantiate the messenger widget in the <em>body</em> section of your HTML page and to enable the file upload feature:</p><pre>&lt;df-messenger<br>  project-id=&quot;[your-project-id]&quot;<br>  agent-id=&quot;[your-agent-id]&quot;<br>  language-code=&quot;en&quot;<br>  allow-feedback=&quot;all&quot;&gt;<br>  &lt;df-messenger-chat-bubble<br>    chat-title=&quot;Car Rental&quot;<br>    enable-file-upload&gt;<br>  &lt;/df-messenger-chat-bubble&gt;<br>&lt;/df-messenger&gt;<br>&lt;style&gt;<br>  df-messenger {<br>    --df-messenger-input-inner-padding: 0 48px;<br>  }<br>&lt;/style&gt;</pre><p><em>Heads-up: the padding in the style tag is required to make room for the upload button and avoid rendering issues.</em></p><p>Next, we need to specify the desired GCS upload bucket, using the following javascript segment:</p><pre>globalThis.dfInstallUtils({<br>  &#39;gcs-bucket-upload&#39;: {bucketName: &#39;[your-bucket-name]&#39;},<br>});</pre><p>Write down the previously created bucket name in the <em>bucketName</em> attribute.</p><p>Let me quickly go through the execution flow:</p><ul><li>Once the user selects a file, a <em>df-upload-file-selected</em> event is fired;</li><li>Without any custom listener to intercept the event, the default behavior is to upload the file to the GCS bucket via REST API;</li><li>Upon success, an info message pops up in the widget and a new <em>df-file-upload-completed</em> event is fired;</li><li>(Optional) Feel free to intercept this latter with an event listener if you want to run custom post-upload processing.</li></ul><p>The uploaded file path in GCS is stored as a query parameter for the DF agent, with the following format:</p><pre>parameters: {<br>  files: [<br>    &#39;bucket-id/session-id/random-id_file-name&#39;,<br>  ]<br>}</pre><p><em>Heads-up:</em> <em>the query parameters are not automatically sent to the agent. They will be included in the next request.</em></p><p>That’s why the last step that you need to implement to send out the query parameter is to push a hidden intent. Let’s implement this step with an event handler:</p><pre>dfMessenger.addEventListener(&#39;df-file-upload-completed&#39;, function (event) {<br>  dfMessenger.sendRequest(&#39;query&#39;,&#39;file uploaded&#39;);<br>});</pre><p>With this request we’re pushing a <em>“file uploaded”</em> utterance, without any user action, which will be later on detected by the agent (see next section).</p><h3>Gemini 1.0 Pro Vision</h3><p>Gemini is Google’s family of Large Language Models, released in December 2023. Gemini has been conceived with a <strong>multimodal</strong> approach from scratch. In a nutshell, a multimodal LLM takes as input text, images or videos and outputs text.</p><p>Within the scope of this article, we are going to use a multimodal LLM to pass a driving license photo and ask the model to automatically extract PII data to check the validity of the license and the minimum required age to rent a car (in our example, this latter must be 21).</p><p>You are going to interact with the Gemini Pro Vision API via a <a href="https://cloud.google.com/functions/?utm_source=google&amp;utm_medium=cpc&amp;utm_campaign=emea-it-all-en-dr-bkws-all-all-trial-e-gcp-1707574&amp;utm_content=text-ad-none-any-DEV_c-CRE_554879513995-ADGP_Hybrid+%7C+BKWS+-+EXA+%7C+Txt+-+Serverless+Computing+-+Cloud+Functions+-+v2-KWID_43700053286402889-kwd-289154592193-userloc_9181234&amp;utm_term=KW_google+cloud+function-NET_g-PLAC_&amp;&amp;gad_source=1&amp;gclid=Cj0KCQjwwYSwBhDcARIsAOyL0fjPX6cgvyjdvsx-9BR3la2J8YX3PVlVPWATNgoEAeeTuws1Tui4JBkaAiouEALw_wcB&amp;gclsrc=aw.ds&amp;hl=en">Cloud Function</a>, Google’s Function-as-a-Service (FaaS) technology. However, our ultimate goal is to call this function from Dialogflow via a webhook. Thereby, we need to follow the expected Dialogflow request and response format.</p><p>Let’s start from creating the cloud function with the following settings:</p><ul><li>Name: <em>driving-license-webhook</em></li><li>Require HTTPS</li><li>Timeout: 60 seconds (do not decrease this value or you may experience timeouts due to Gemini)</li><li>AppEngine default service account</li><li>Allow all traffic</li></ul><p>Leave the rest of the settings to their default values. Select <em>Python 3.12</em> as the runtime environment.</p><p>Fill in the <em>requirements.txt</em> file with the following dependencies:</p><pre># Function dependencies, for example:<br># package&gt;=version<br>google-cloud-aiplatform&gt;=1.38</pre><p>Fill in the <em>main.py</em> file with the following code (<em>heads up: you need to edit some parts within the code block</em>):</p><pre>from datetime import datetime<br>from dateutil.relativedelta import relativedelta<br>import vertexai<br>from vertexai.preview.generative_models import GenerativeModel, Part<br><br>def submit_multimodal_prompt(project_id: str, location: str, image_path: str, prompt: str) -&gt; str:<br>    # Initialize Vertex AI<br>    vertexai.init(project=project_id, location=location)<br>    # Load the model<br>    multimodal_model = GenerativeModel(&quot;gemini-pro-vision&quot;)<br>    # Query the model<br>    response = multimodal_model.generate_content(<br>        [<br>            # Add an example image<br>            Part.from_uri(image_path, mime_type=&quot;image/jpeg&quot;<br>            ),<br>            # Add an example query<br>            prompt,<br>        ]<br>    )<br><br>    print(&quot;Gemini Output: &quot; + response.text)<br>    return response.text<br><br>def check_expiration_date(project_id: str, region: str, file_path: str):<br>    #  run multimodal prompt<br>    prompt = &quot;extract the expiration date from the driving license&quot;<br>    exp_date_str = submit_multimodal_prompt(project_id,region,file_path,prompt)<br><br>    # check whether the driving license is still valid based on the expiration date<br>    exp_date = datetime.strptime(exp_date_str.replace(&quot; &quot;, &quot;&quot;), &#39;%m/%d/%Y&#39;)<br>    now_date = datetime.now()<br><br>    if exp_date &gt;= now_date:<br>        return True<br><br>    return False<br><br>def check_driver_age(project_id: str, region: str, file_path: str):<br>    #  run multimodal prompt<br>    prompt = &quot;extract the date of birth from the driving license&quot;<br>    dob_str = submit_multimodal_prompt(project_id,region,file_path,prompt)<br><br>    # check whether the driver is &gt; 21 years, as requested from some car rental companies<br>    dob_date = datetime.strptime(dob_str.replace(&quot; &quot;, &quot;&quot;), &#39;%m/%d/%Y&#39;)<br>    now_date = datetime.now()<br><br>    difference_in_years = relativedelta(now_date, dob_date).years<br><br>    if difference_in_years &gt;= 21:<br>        return True<br><br>    return False<br><br>def validate_driving_license(request):<br>    &quot;&quot;&quot;Responds to any HTTP request.<br>    Args:<br>        request (flask.Request): HTTP request object.<br>    Returns:<br>        The response text or any set of values that can be turned into a<br>        Response object using<br>        `make_response &lt;http://flask.pocoo.org/docs/1.0/api/#flask.Flask.make_response&gt;`.<br>    &quot;&quot;&quot;<br>    req = request.get_json()<br>    file_path = &#39;gs://&#39; + str(req[&quot;sessionInfo&quot;][&quot;parameters&quot;][&quot;files&quot;][0])<br><br>    # CHANGE THESE PARAMS<br>    project_id = &#39;your-project-id&#39;<br>    region = &#39;your-region&#39;<br>    <br>    # check expiration date<br>    is_exp_date_valid = check_expiration_date(project_id, region, file_path)<br><br>    # check minimum driver age<br>    is_driver_age_valid = check_driver_age(project_id, region, file_path) <br><br>    is_valid = &quot;KO&quot;<br>    if is_exp_date_valid is True and is_driver_age_valid is True:<br>        is_valid = &quot;OK&quot;<br>    elif is_exp_date_valid is False:<br>        is_valid = &quot;KO_EXP&quot;<br>    else:<br>        is_valid = &quot;KO_AGE&quot;<br>    <br>    res = {&quot;sessionInfo&quot;: {&quot;parameters&quot;: {&quot;response&quot;: is_valid}} }<br>    return res</pre><p>Let’s analyze the code step by step:</p><ul><li>The cloud function’s entry point is <em>validate_driving_license(request)</em>. Dialogflow passes the input parameter request via the webhook with a predefined format. We can extract from this payload the relevant session parameters, such as the path to the image;</li><li>There are two validation steps: the driver age and the driving license validity;</li><li>Both these steps leverage the <em>submit_multimodal_prompt()</em> method, where the Gemini Vision API is invoked and the real magic happens;</li><li>According to the check results, the corresponding status code is stored in the <em>response</em> session parameter and returned via the webhook to the bot.</li></ul><h3>Dialogflow CX Car Rental Prebuilt Agent</h3><p>Let’s now turn to defining the agent and integrating the various components.</p><p>For simplicity, let’s import one of the prebuilt agents available in Dialogflow CX, a <a href="https://cloud.google.com/dialogflow/cx/docs/concept/agents-prebuilt#travel-car-rental">car rental</a> bot. Later on, you are going to extend the agent itself with a small set of pages and routes.</p><p>You can deploy the initial prebuilt agent by following this <a href="https://cloud.google.com/dialogflow/cx/docs/concept/agents-prebuilt#import">guide</a>. Select <strong>Travel: car rental agent</strong> from the list of available agents.</p><figure><img alt="The prebuilt agents menu in Dialogflow CX" src="https://cdn-images-1.medium.com/max/1024/0*NCyJSy0A32BQ2emS" /><figcaption>The prebuilt agents menu in Dialogflow CX</figcaption></figure><p>Name the agent, select <strong>global</strong> location and <strong>CX flow</strong> as default resource. The import agent is below reported.</p><figure><img alt="The imported pre-build Dialogflow CX car rental agent" src="https://cdn-images-1.medium.com/max/1024/0*nl4CIIXEWwHH2kvW" /><figcaption>The imported prebuilt Dialogflow CX car rental agent</figcaption></figure><h4>Agent Extension</h4><p>You are going to extend the imported agent with a couple of new pages to add the back-end logic so far developed. The final expected result is below highlighted with a red bounding box.</p><figure><img alt="The extended version of the previously imported agent" src="https://cdn-images-1.medium.com/max/1024/0*yDx5SlTBulDcPB7J" /><figcaption>The extended version of the previously imported agent</figcaption></figure><p>The first page, <em>Acquire Driving License</em>, is in charge of collecting the customer’s driving license and triggering its validation. This page complements what we’ve described in the Dialogflow Messenger section.</p><p>This page contains an <a href="https://cloud.google.com/dialogflow/cx/docs/concept/handler#route">intent route</a> based on the <em>file.upload</em> intent, which maps the utterance we’ve just pushed from the chat widget (<em>“file uploaded”)</em>.</p><p>When Dialogflow detects this intent, both the image file and the image path are available and stored respectively in the GCS bucket and in the DF query parameter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/0*LdHiyIB6V4rDPinV" /><figcaption>Intent route in the Acquire Driving License page</figcaption></figure><p>As a result, the <em>driving-license-webhook</em> (described in the next section) is invoked. Upon receiving the webhook’s response, there is an automatic transition to the next page: <em>Validate Driving License</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/0*96bOOx3SDPCpbvAg" /><figcaption>Transition from first page to second page</figcaption></figure><p>The second page, <em>Validate Driving License</em>, is in charge of parsing the webhook’s response and triggering the corresponding behavior. This is accomplished by setting two <a href="https://cloud.google.com/dialogflow/cx/docs/concept/handler#route">conditional routes</a>.</p><figure><img alt="Conditional routes in the Validate Driving License page" src="https://cdn-images-1.medium.com/max/456/0*HDjvYlqTsPoT67tt" /><figcaption>Conditional routes in the Validate Driving License page</figcaption></figure><p>Both the conditions rely on the webhook’s response, which can be retrieved using the <em>$session.params.response</em> session parameter.</p><p>If the webhook returns “OK”, the page will just report a fulfillment and move to the next page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*oRWX5HXOLzaZfMzL" /><figcaption>Fulfillment when the driving license is valid</figcaption></figure><p>If not, since the webhook also returns a specific error code, we apply a <a href="https://cloud.google.com/dialogflow/cx/docs/concept/fulfillment#cond">conditional response</a> to describe the precise problem. Given the very specific problem, we handover its resolution to a live agent.</p><pre>if $session.params.response = &quot;KO_AGE&quot;<br>  Unfortunately our company requires the minimum age for car rental to be 21. Let me transfer you to an agent.<br>elif $session.params.response = &quot;KO_EXP&quot;<br>  Unfortunately your driving license is expired. Let me transfer you to an agent.<br>else<br>  Unfortunately your driving license is not valid. Let me transfer you to an agent.<br>endif</pre><h4>Webhook</h4><p>A <a href="https://cloud.google.com/dialogflow/cx/docs/concept/webhook">webhook</a> is a service that hosts your business logic or calls other services. It’s the suggested approach to leverage back-end services. In our case, the webhook triggers the <a href="https://cloud.google.com/functions?hl=en">cloud function</a> previously defined, which in turn invokes the Gemini Vision API. We could even directly call this latter from the webhook itself, but relying on a cloud function fosters better decoupling between the bot and the back-end and allows us to implement the driving license validation.</p><p>Let’s create the <em>driving-license-webhook </em>that will invoke the corresponding cloud function by going to <em>Manage</em>, <em>Webhooks</em> and click on <em>Create</em>.</p><p>Fill in the following settings:</p><ul><li>Name: <em>driving-license-webhook</em></li><li>Timeout: 30 seconds</li><li>Type: Generic web service</li><li>Webhook URL: <a href="https://your-project-id.cloudfunctions.net/driving-license-webhook">https://<em>your-project-id</em>.cloudfunctions.net/driving-license-webhook</a></li><li>Subtype: Standard</li><li>Service Agent Auth: ID Token</li></ul><p>Click on <em>Save</em>.</p><p>Last, let’s bind the webhook to the conversational page. Click on <em>Build</em> and go back to the bot design UI. Open the <em>Acquire Driving License</em> page, select the <em>file.upload</em> route and from the right panel select <em>Webhook settings</em>, then <em>Enable webhook</em>. Select <em>driving-license-webhook</em> from the combo box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*0KWDgx0kQgErjDqz" /><figcaption>The webhook triggered from the Acquire Driving License page</figcaption></figure><p>This is the last step of our integration and the whole customer journey is ready to be tested.</p><h3>Summary</h3><p>Generative AI is influencing manifold technologies, digital assistants included. Dialogflow CX already integrates LLMs with built-in features (e.g. generative fallbacks, generators, playbooks).</p><p>The scope of this article is to further extend this integration by adopting multimodal LLMs, such as Gemini 1.0 Pro Vision, to create an even more natural interaction between customers and bots. With multimodal customer experience, you can upload images and automatically extract info, without any human intervention in boring conversational rounds.</p><p>Below you can find additional resources, and a basic implementation of this article published on my <a href="https://github.com/grandelli/dfcx-geminiprovision">GitHub</a> repository. I strongly encourage you to deploy the code and further customize it according to your use case.</p><p>Stay tuned for more material around conversational AI!</p><h3>What’s Next?</h3><h4>Documentation</h4><p><a href="https://cloud.google.com/vertex-ai/generative-ai/docs/learn/overview">Overview of Generative AI on Vertex AI</a></p><p><a href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/overview">Overview of Multimodal Models</a></p><p><a href="https://cloud.google.com/dialogflow/cx/docs/basics">Dialogflow CX Basics</a></p><p><a href="https://cloud.google.com/dialogflow/cx/docs/concept/webhook">Dialogflow CX Webhooks</a></p><h4>Github Sources</h4><p><a href="https://github.com/grandelli/dfcx-geminiprovision">https://github.com/grandelli/dfcx-geminiprovision</a> (the main GitHub repository for this project — contains the modified DFCX agent, the cloud function and the Dialogflow Messenger client)</p><h3>Acknowledgements</h3><p>Special thanks to Vojin Katic for reviewing this article,<a href="https://www.linkedin.com/in/giorgio-conte-bb4292105/"> Giorgio Conte</a> for our joint collaboration on a similar project and<a href="https://www.linkedin.com/in/matteo-consoli/"> Matteo Consoli</a> for unconsciously influencing me to get back to writing.</p><p>Feel free to leave your comments here or connect with me on<a href="https://www.linkedin.com/in/gabrielerandelli/"> LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d39035b985d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>