<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mark Watson on Medium]]></title>
        <description><![CDATA[Stories by Mark Watson on Medium]]></description>
        <link>https://medium.com/@markwatsonatx?source=rss-f1b6565b0972------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gFCOO3y5brrSJH43H5BQUw.jpeg</url>
            <title>Stories by Mark Watson on Medium</title>
            <link>https://medium.com/@markwatsonatx?source=rss-f1b6565b0972------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:52:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@markwatsonatx/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Keeping Your Machine Learning Models Up-To-Date]]></title>
            <link>https://medium.com/codait/keeping-your-machine-learning-models-up-to-date-f1ead546591b?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1ead546591b</guid>
            <category><![CDATA[ibm-watson]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[continuous-learning]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Thu, 01 Mar 2018 10:01:01 GMT</pubDate>
            <atom:updated>2018-03-08T22:07:09.936Z</atom:updated>
            <content:encoded><![CDATA[<h4>Continuous learning with IBM Watson Machine Learning (part 1)</h4><p>If you scour the web you will find many tutorials that show you how to train and score machine learning models. You may even find tutorials that teach you how to deploy those models and run predictions from your end-user applications.</p><p>It’s more difficult, however, to find resources that show you how to maintain machine learning systems. How do you ensure your predictions continue to be accurate? How do you keep your models up-to-date with new training data?</p><blockquote>Shameless plug! You can read my previous tutorials on training and scoring a simple ML model with the IBM Watson Data Platform here:</blockquote><ul><li><a href="https://medium.com/ibm-watson-data-lab/building-your-first-machine-learning-system-b3d9401927b7">Building Your First Machine Learning System</a></li><li><a href="https://medium.com/ibm-watson-data-lab/deploying-a-simple-machine-learning-app-with-node-js-and-watson-ml-96afdb6f7819">Deploying a Simple Machine Learning App with Node.js and Watson ML</a></li></ul><p>As I have stated in previous posts, the data is the most crucial component of a successful ML system. If you’ve found a data set that provides you with accurate predictions that’s a great start, but how long will that data continue to provide accurate predictions?</p><p>Let’s take the example of predicting house prices. House prices change all the time. The data you used to train a machine learning model that predicts house prices six months ago could provide terrible predictions today. For house prices, it’s imperative that you have up-to-date information to train your models.</p><p>When designing a machine learning system it is important to understand how your data is going to change over time. A well-architected system should take this into account, and a plan should be put in place for keeping your models updated.</p><h4><strong>Manual retraining</strong></h4><p>One way to maintain models with fresh data is to train and deploy your models using the same process you used to build your models in the first place. As you can imagine this process can be time-consuming. How often do you retrain your models? Weekly? Daily?</p><p>On the other hand, as you are manually retraining your models you may discover a new algorithm or a different set of features that provide improved accuracy.</p><h4><strong>Continuous learning</strong></h4><p>Another way to keep your models up-to-date is to have an automated system to continuously evaluate and retrain your models. This type of system is often referred to as continuous learning, and may look something like this:</p><ol><li>Save new training data as you receive it. For example, if you are receiving updated prices of houses on the market, save that information to a database.</li><li>When you have enough new data, test its accuracy against your machine learning model.</li><li>If you see the accuracy of your model degrading over time, use the new data, or a combination of the new data and old training data to build and deploy a new model.</li></ol><p>The benefit to a continuous learning system is that it can be completely automated.</p><blockquote>It’s probably a still a good idea to review your process on a regular basis. As I mentioned before, you may find a different algorithm or a new set of features that improves your predictions, and this isn’t necessarily something a continuous learning system is good at.</blockquote><h4><strong>Continuous learning with Watson ML</strong></h4><p>If you are using the IBM Watson Data Platform or the Watson Machine Learning service, you will be happy to know that there is built-in support for continuous learning.</p><p><a href="https://medium.com/ibm-data-science-experience/continuous-learning-on-watson-data-platform-cc39f3fd5042">You can use the Model Builder to train your model and enable continuous learning without writing a single line of code</a>, but if you want more control over the life cycle of your machine learning models (like I do) you can use the Watson Machine Learning libraries and APIs.</p><p>To run continuous learning with Watson ML you will need to have instances of the following services:</p><ol><li><a href="https://console.bluemix.net/catalog/services/machine-learning">Watson Machine Learning</a> manages your models and the continuous learning process.</li><li><a href="https://console.bluemix.net/catalog/services/db2-warehouse">Db2 Warehouse on Cloud</a> stores your training and feedback data.</li><li><a href="https://console.bluemix.net/catalog/services/apache-spark">Apache Spark</a> is used by Watson ML to test and train your models.</li></ol><p>The following diagram shows how the continuous learning process works when you build your models in Jupyter notebooks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*0myH4V8nxm4l2-t0MIprPQ.png" /><figcaption>The ML continuous learning process on the IBM Cloud.</figcaption></figure><p>Let’s break this down:</p><p>1. You start by storing your training data in a table in Db2 Warehouse on Cloud. When you are ready to train your model, pull your training data into a Jupyter Notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*AnU6UwlfzVaQfLbm1zxybA.png" /><figcaption>1. Load training data.</figcaption></figure><p>2. In your notebook, build your model using Spark ML. Spark ML is currently the only machine learning library that Watson ML supports for continuous learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*ZzeLO1QdjZaq03bnrj7HQg.png" /><figcaption>2. Build model.</figcaption></figure><p>3. Deploy your model to Watson ML. When deploying your model tell Watson ML where to find the training data (a table in the data warehouse) as well as where it will find feedback data (another table in the warehouse) later for evaluation.</p><p>It’s also important to tell Watson ML how to measure model accuracy and what value represents the current model’s accuracy.</p><p>Finally, instruct Watson ML at what accuracy threshold a model should be retrained. This means if your model is performing below a certain threshold (i.e., it is not performing well) then Watson ML will retrain your model and determine if it can improve predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*XlsT0o-9ZfFhW8c009kwzw.png" /><figcaption>3. Deploy model and learning configuration.</figcaption></figure><p>4. For continuous learning to be effective you need to have some type of automated process for consuming new data. This could be a REST API, a script that downloads data nightly, or any other process that gathers new data. This is referred to as feedback data. When new feedback data is received, send it to Watson ML, and Watson ML will store it in the warehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*JqUsqR0pYLGU-8BF30QVRg.png" /><figcaption>4. Provide feedback for future model training.</figcaption></figure><p>5. When you determine you have amassed enough feedback data to test, instruct Watson ML, via a REST API, to start feedback evaluation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*94HPyWD3AwNWIO2IRUY4sg.png" /><figcaption>5. Start feedback evaluation.</figcaption></figure><p>6. Watson ML pulls any new feedback data and runs predictions against the current model. (Watson ML will not include previously evaluated feedback data.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*IufFfJiwLEOPaWMpTGvnfg.png" /><figcaption>6. Get feedback data.</figcaption></figure><p>7. After feedback evaluation completes, the accuracy of the data is measured against the accuracy threshold. If the accuracy is below that threshold then retraining is triggered.</p><p>8. Watson ML then pulls in <strong><em>all</em></strong> training data and <strong><em>all</em></strong> feedback data to build a new model and measures its accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*wbjrHmjkc_SV1lZOM_a41w.png" /><figcaption>7–8. Evaluate feedback data. Train all feedback data and training data.</figcaption></figure><p>9. If the new model’s accuracy exceeds the original model’s accuracy, then the new model is automatically deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/1*aejrCvZwCSMSI9eiSu4iIw.png" /><figcaption>9. Deploy new model.</figcaption></figure><p>That’s the typical flow one might follow with the Watson ML continuous learning system. As you design your own continuous learning system, you will find that many of these steps are configurable, and enable you to implement variations of this flow. For example, you can choose to redeploy your model every time, whether or not accuracy is improved. Alternatively, you could choose to never automatically redeploy your model.</p><h4>The learnings continue</h4><p>Stay tuned for an upcoming blog post where I will walk through an end-to-end continuous learning example. We’ll start with a notebook where we take a data set, clean it, and then upload it to the warehouse. In a second notebook, we’ll pull the data from the warehouse, train our model, and deploy it to Watson ML. Finally, we’ll upload feedback data to the warehouse, kick off feedback evaluation, and watch the continuous learning process unfold.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1ead546591b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codait/keeping-your-machine-learning-models-up-to-date-f1ead546591b">Keeping Your Machine Learning Models Up-To-Date</a> was originally published in <a href="https://medium.com/codait">Center for Open Source Data and AI Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Simple Machine Learning App with Node.js and Watson ML]]></title>
            <link>https://medium.com/codait/deploying-a-simple-machine-learning-app-with-node-js-and-watson-ml-96afdb6f7819?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/96afdb6f7819</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ibm-cloud]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Thu, 08 Feb 2018 15:37:33 GMT</pubDate>
            <atom:updated>2018-03-10T07:22:28.680Z</atom:updated>
            <content:encoded><![CDATA[<h4>Give users real-time predictions; Watson ML for developers (part 3)</h4><p>This article is a continuation of a series of posts introducing developers to machine learning and Watson ML.</p><p><a href="https://medium.com/ibm-watson-data-lab/watson-machine-learning-for-developers-b98cefb3f890">In Part 1</a> I gave you an overview of machine learning and some of the tools you can use to build ML systems. <a href="https://medium.com/ibm-watson-data-lab/building-your-first-machine-learning-system-b3d9401927b7">In Part 2</a> I showed you how to build an end-to-end machine learning system using the IBM Data Science Experience, Spark ML, and Watson ML. Part 2 also showed you how to deploy your models to Watson ML and make predictions in real-time using a RESTful API.</p><p>In this blog post I’ll close the loop and show you how you can make real-time machine learning predictions from your end-user applications. I’ll walk through a sample Node.js application that predicts house prices based on the models we trained and deployed in parts 1 and 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d82NxD9pqdmuyDt5gSslqA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/aUYmEjU6mvY?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Dawid Zawiła</a> on <a href="https://unsplash.com/search/photos/crystal-ball?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>.</figcaption></figure><h4>Sample application</h4><p>The source code and instructions for the sample Node.js application can be found at <a href="https://github.com/ibm-watson-data-lab/watson-ml-scoring-demo">https://github.com/ibm-watson-data-lab/watson-ml-scoring-demo</a>. In order to run the application you will need a Watson Machine Learning instance provisioned in IBM Cloud and a deployed model based on the example provided in <a href="https://medium.com/ibm-watson-data-lab/building-your-first-machine-learning-system-b3d9401927b7">part 2</a>.</p><blockquote>If you haven’t read part 2 I highly recommend you start there. Part 2 shows you how to provision an instance of Watson Machine Learning, train a model in a Jupyter notebook using Spark ML, and deploy your model to Watson ML.</blockquote><p>The sample application is a single web page with a Node.js backend. The GitHub repo provides step-by-step instructions for running the sample application locally, or deploying the application to the IBM Cloud.</p><p>The sample application provides a simple interface for predicting the price of a house where a user can enter the square footage, number of bedrooms, and color of a house:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*FbjWUhE7H0cSb2mGKZ-WIg.png" /><figcaption>You can try out the sample application at <a href="https://watson-ml-scoring-demo.mybluemix.net/">https://watson-ml-scoring-demo.mybluemix.net</a>.</figcaption></figure><p>Clicking the submit button calls a JavaScript function in the browser, which in turn makes a RESTful call to the Node.js application. The Node.js application then calls your Watson ML scoring endpoint to predict the price of the house.</p><h4>Watson ML scoring endpoints</h4><p>Watson ML makes it easy to make real-time predictions against your deployed models through scoring endpoints. Scoring endpoints are RESTful APIs that can be called from any programming language.</p><p>No matter what programming language you use, the flow for calling your scoring endpoints is as follows:</p><ol><li>Request an access token from Watson ML using your Watson ML username and password.</li><li>Build your payload using the features that you defined when building your model (i.e., “SquareFeet” and “Bedrooms”), and the values of the features you want to use for the prediction. Here is an example payload expected by the Watson ML scoring endpoint:</li></ol><pre>{<br>  &quot;fields&quot;: [&quot;SquareFeet&quot;, &quot;Bedrooms&quot;],<br>  &quot;values&quot;: [squareFeet, numBedrooms]<br>}</pre><p>3. Post the payload to your Watson ML model deployment using the access token you retrieved earlier in the Authorization header:</p><pre>Authorization: Bearer TOKEN_GOES_HERE</pre><p>4. Finally, parse the response from Watson ML. Here is a sample response:</p><pre>{<br>  &quot;fields&quot;: [&quot;SquareFeet&quot;, &quot;Bedrooms&quot;, &quot;features&quot;, &quot;prediction&quot;],<br>  &quot;values&quot;: [[2400, 4, [2400.0, 4.0], 137499.99999999968]]<br>}</pre><p>The Watson ML user interface provides sample source code for calling your scoring endpoints from a few different programming languages as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*tObaRUQKOxx3c5FkRRUs8g.png" /></figure><h4>Simplified scoring with Node.js</h4><p>While working on the sample Node.js application I built a <a href="https://www.npmjs.com/package/watson-ml-model-utils">Node.js helper library that makes it even easier to call your scoring endpoints</a>. You can see the helper library in action in the sample application, or integrate it with your own Node.js application.</p><p>Start by installing the library in your Node.js project:</p><pre>npm install watson-ml-model-utils</pre><p>Define your environment variables in a .env file:</p><pre>WML_SERVICE_PATH=https://ibm-watson-ml.mybluemix.net<br>WML_USERNAME=<br>WML_PASSWORD=<br>WML_INSTANCE_ID=<br>WML_MODEL_ID=<br>WML_DEPLOYMENT_ID=</pre><blockquote>The GitHub repo README for the sample application shows you where you can find the values for your environment variables.</blockquote><p>Import the WatsonMLScoringEndpoint class:</p><pre>const { WatsonMLScoringEndpoint } <strong>=</strong> require(&#39;watson-ml-model-utils&#39;);</pre><p>Create an instance of WatsonMLScoringEndpoint with the features used to train your model:</p><pre>let endpoint = new WatsonMLScoringEndpoint([&#39;SquareFeet&#39;, &#39;Bedrooms&#39;]);</pre><p>Call the score method on the endpoint to make a prediction:</p><pre>endpoint.score([2400, 4])<br>    .then(response =&gt; console.log(response.prediction))<br>    .catch(err =&gt; console.log(err));</pre><p>The score method will return a promise, that when resolved will contain a response with the prediction returned from Watson ML:</p><pre>137499.99999999968</pre><p>You can also make multiple predictions in a single call to the scoring endpoint by calling the scoreMulti method:</p><pre>endpoint.scoreMulti([[2400, 4], [2000, 3], [2600, 6]])<br>    .then(response =&gt; console.log(response.predictions))<br>    .catch(err =&gt; console.log(err));</pre><p>In this case an array is returned with the three predictions:</p><pre>[ 137499.99999999968, 87500.00000000276, 162500.00000000527 ]</pre><p>And if you need the full response from Watson ML you can access that in the data property of the response.</p><h4>Try it yourself!</h4><p>The Node.js helper library will work with any deployed model in Watson ML. Simply pass in your features, and the values you are using for your prediction, and you are good to go.</p><p>If you followed parts 1 and 2 and deployed the sample Node.js application, then you are well on your way to building end-to-end machine learning systems and making real-time predictions from your end-user applications. In future articles we’ll slowly dig into other machine learning problems, deployment considerations, and much more. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96afdb6f7819" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codait/deploying-a-simple-machine-learning-app-with-node-js-and-watson-ml-96afdb6f7819">Deploying a Simple Machine Learning App with Node.js and Watson ML</a> was originally published in <a href="https://medium.com/codait">Center for Open Source Data and AI Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[deprecated] Building Your First Machine Learning System]]></title>
            <link>https://medium.com/codait/building-your-first-machine-learning-system-b3d9401927b7?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3d9401927b7</guid>
            <category><![CDATA[cognitive-computing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pixiedust]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Wed, 18 Oct 2017 18:47:58 GMT</pubDate>
            <atom:updated>2018-05-18T14:32:55.103Z</atom:updated>
            <content:encoded><![CDATA[<h4>Train your model and deploy it, Watson ML for developers (part 2)</h4><blockquote><strong><em>Editor’s note:</em></strong> This article has been deprecated. The IBM DSX service has been upgraded to the new <a href="https://datascience.ibm.com/">IBM Watson Studio platform</a>, which provides additional APIs for more advanced ML and AI applications. Find more recent tutorials <a href="https://www.ibm.com/cloud/watson-studio">here</a>.</blockquote><p><a href="https://medium.com/ibm-watson-data-lab/watson-machine-learning-for-developers-b98cefb3f890">In Part 1</a> I gave you an overview of machine learning, discussed some of the tools you can use to build end-to-end ML systems, and the path I like to follow when building them.</p><p>In this post we are going to follow this path to train a machine learning model, deploy it to Watson ML, and run predictions against it in real time.</p><blockquote>Look Ahead: In Part 3 we’ll create a small web application and backend to demonstrate how you can integrate Watson ML and make machine learning predictions in an end-user application.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*DS3iR99ajcb1zQlbbGlEyw.jpeg" /><figcaption>The Model Cafe in the Allston neighborhood of Boston. Image: <a href="https://fineartamerica.com/featured/brighton-ave-packards-corner-allston-ma-model-cafe-toby-mcguire.html">Toby McGuire</a>.</figcaption></figure><p>We are going to use our small data set from Part 1 because the point of this post is to get something up and running quickly — not to actually build an accurate system for making predictions. Here’s the data set:</p><pre>Square Feet       # Bedrooms       Color         Price<br>-----------       ----------       -----         -----        <br>2,100             3                White         $100,000<br>2,300             4                White         $125,000<br>2,500             4                Brown         $150,000</pre><p>In Part 1 I talked about the tools I use to build machine learning systems. Before we start building our ML system, let’s setup our tools.</p><h4>Tool setup</h4><p><strong>Bluemix/DSX:</strong> You’ll need a Bluemix and Data Science Experience account. If you don’t have one, go to <a href="https://datascience.ibm.com/">https://datascience.ibm.com</a> to sign up. This will create a single account where you can access Bluemix and DSX.</p><p><strong>Watson Machine Learning: </strong>You’ll need an instance of Watson Machine Learning. You can provision a new instance <a href="https://console.bluemix.net/catalog/services/machine-learning">here</a>.</p><p><strong>Apache Spark™:</strong> You’ll need a Spark instance, but if you don’t have one now you can create one later.</p><p>Now that you’re all set up, let’s follow the process I outlined in Part 1.</p><h3>Step 1: Identify what you want to predict and the source of your data</h3><p>We’ve identified that we want to predict house prices, and the data set we want to use to drive those predictions. I have made the data set available on GitHub:</p><p><a href="https://raw.githubusercontent.com/markwatsonatx/watson-ml-for-developers/master/data/house-prices.csv">https://raw.githubusercontent.com/markwatsonatx/watson-ml-for-developers/master/data/house-prices.csv</a></p><p>This URL is important because we’ll need to pull this data into our Jupyter Notebook in the next step.</p><h3>Step 2: Create a Jupyter Notebook — import, clean, and analyze the data</h3><h4>Create a Jupyter Notebook</h4><p>We’re going to analyze our data in a Jupyter Notebook in the IBM Data Science Experience. Jupyter Notebooks are documents that run in a web browser and are composed of cells. Cells can contain markup or executable code. We’ll be coding in Python. I’ll show you how we can import and analyze our data with just three lines of code.</p><p>Download the following notebook to your computer:</p><p><a href="https://dataplatform.ibm.com/analytics/notebooks/3e83ffa1-f52a-4b76-bbb5-498b6b7f9505/view?access_token=a7dfdd01dbc24c53a5ac9688fbdd32da1b59156117d721fe10d12660f18dd591">https://dataplatform.ibm.com/analytics/notebooks/3e83ffa1-f52a-4b76-bbb5-498b6b7f9505/view?access_token=a7dfdd01dbc24c53a5ac9688fbdd32da1b59156117d721fe10d12660f18dd591</a></p><p>Open DSX and create a new project called “Watson ML for Developers”. From here, create a new Spark instance for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tUQN81UGfqbXEILoHuKQgg.png" /></figure><p>In the project navigate to <strong><em>Analytic assets</em></strong> and click <strong><em>New notebook</em></strong>. Choose <strong><em>From file</em></strong>. Specify a name, like “House Prices”, and choose the notebook you downloaded above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AyipHZEW9DS-ZUbE4yETSw.png" /></figure><p>Finally, click <strong><em>Create Notebook</em></strong>. You should be taken directly to edit the notebook.</p><p>If this is your first time using Jupyter notebooks here are a few tips that you may find helpful (if you are already familiar with Jupyter notebooks, feel free to skip ahead):</p><p>1. Always make sure your kernel is running. You should see the status of your kernel in the top right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*xcCQZqrUT-hOXLSgx9I83g.png" /></figure><p>2. If your kernel is not running, you can restart it from the <strong><em>Kernel</em></strong> menu. From here you can also interrupt your kernel, or change your kernel (if you want to use a different version of Python or Apache Spark).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*aXnfAUMsJKMTQRv5nH8suQ.png" /></figure><p>3. A notebook is made of up markup and code cells. You can walk through the notebook and execute the code cells by clicking the run button in the toolbar or from the <strong><em>Cell</em></strong> menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/68/1*AEFSxZCIdre86IhOsjJbqw.png" /></figure><h4>Import, clean, and analyze the data</h4><p>Let’s look at the first three code cells in the notebook where we will load and analyze our data. Here’s the first code cell:</p><pre>import pixiedust</pre><p>This cell just imports a Python library called <a href="https://ibm-watson-data-lab.github.io/pixiedust/">PixieDust</a>. PixieDust is an open source helper library that works as an add-on to Jupyter Notebooks that makes it easy to import and visualize data.</p><p>In the second cell we load our sample data:</p><pre>df = pixiedust.sampleData(&quot;https://raw.githubusercontent.com/markwatsonatx/watson-ml-for-developers/master/data/house-prices.csv&quot;)</pre><p>This will generate a Spark <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">DataFrame</a> called “df”. A DataFrame is a data set organized into named columns. You can think of it as a spreadsheet, or a relational database table. The Spark ML API uses DataFrames to train and test ML models.</p><p>Finally, we’ll call the display function in PixieDust to display our data:</p><pre>display(df)</pre><p>It should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9_yaIrWgKJyUMJSXtzJuTg.png" /></figure><p>In this case we are displaying a simple table, but PixieDust also provides graphs and charts for helping you understand and analyze your data without writing any code.</p><p>In just three lines of code we have imported and analyzed our data set. Now it’s time to do some machine learning!</p><h3>Step 3: Use Apache Spark ML to build and test a machine learning model</h3><h4>Build a machine learning model</h4><p>We’re going to build our first ML model in just a handful of cells. To start we need to import the <a href="https://spark.apache.org/mllib/">Spark ML</a> libraries that we’ll be using:</p><pre>from pyspark.ml import Pipeline<br>from pyspark.ml.regression import LinearRegression<br>from pyspark.ml.feature import VectorAssembler</pre><p>This is a <strong>regression</strong> problem (we’re trying to predict a real number), so we are going to use the LinearRegression algorithm in <em>pyspark.ml.regression</em>. There are other regression algorithms, but those are outside of the scope of this post.</p><p>We are going to build our ML model in just four lines of code. These four lines are in a single cell in our notebook, like so:</p><pre>assembler = VectorAssembler(<br>   inputCols=[&#39;SquareFeet&#39;,&#39;Bedrooms&#39;],<br>   outputCol=&quot;features&quot;<br>)</pre><pre>lr = LinearRegression(labelCol=&#39;Price&#39;, featuresCol=&#39;features&#39;)</pre><pre>pipeline = Pipeline(stages=[assembler, lr])</pre><pre>model = pipeline.fit(df)</pre><p>Let’s break this down, line by line.</p><p>First of all we need to specify our <strong>features</strong>. In the previous post we decided that we would use <em>Square Feet</em> and <em># Bedrooms</em> as our features. Our ML algorithm expects a single vector of feature columns, so here we use a <em>VectorAssembler</em> to tell our ML pipeline (we’ll talk about pipelines in a minute) that we want <em>SquareFeet</em> and <em>Bedrooms</em> as our features:</p><pre>assembler = VectorAssembler(<br>   inputCols=[&#39;SquareFeet&#39;,&#39;Bedrooms&#39;],<br>   outputCol=&quot;features&quot;<br>)</pre><p>Next, we create an instance of <em>LinearRegression</em>, the ML algorithm we are going to use. At a minimum, you must specify the features and the labels. There are other parameters you can provide to tweak the algorithm, but they’re not going to do us much good when working with three data points :)</p><pre>lr = LinearRegression(labelCol=&#39;Price&#39;, featuresCol=&#39;features&#39;)</pre><p>Next, we create our <em>pipeline</em>. A <em>Pipeline</em> allows us to specify the steps that should be performed when training an ML model. In this case, we first want to assemble our two feature columns into a single vector — that’s the assembler. Then we want to run it through our LinearRegression algorithm. In upcoming posts I’ll discuss other operations that you’ll run through the pipeline — like converting non-numeric data to numeric data.</p><pre>pipeline = Pipeline(stages=[assembler, lr])</pre><p>Finally, we pass our DataFrame to the <em>fit</em> method on the pipeline to create our ML model.</p><pre>model = pipeline.fit(df)</pre><p>Congratulations, you now have a machine learning model that you can use to predict house prices!</p><h4>Test the model</h4><p>It’s time to test our model. In our example we are going to run a single prediction. In future posts I’ll discuss how you can analyze the accuracy of your model by running a large number of predictions based on your original data set.</p><p>Here we create a Python function to get our prediction:</p><pre>def get_prediction(square_feet, num_bedrooms):<br>    <br>   request_df = spark.createDataFrame(<br>      [(square_feet, num_bedrooms)],<br>      [&#39;SquareFeet&#39;,&#39;Bedrooms&#39;]<br>   )<br>    <br>   response_df = model.transform(request_df)<br>   <br>   return response_df</pre><p>Let’s break this cell down. First of all, in order to generate a prediction against an ML model generated using Spark ML, we need to pass it a DataFrame with the data we want to use in our prediction (i.e., the square footage and # bedrooms for the house price we want to predict). This line of code creates the DataFrame we’ll pass to our model:</p><pre>request_df = spark.createDataFrame(<br>   [(square_feet, num_bedrooms)],<br>   [&#39;SquareFeet&#39;,&#39;Bedrooms&#39;]<br>)</pre><p>Then we’ll call transform on the model, passing in the request DataFrame. This returns another DataFrame:</p><pre>response_df = model.transform(request_df)</pre><p>Let’s run a prediction for a house that is 2,400 square feet and has 4 bedrooms:</p><pre>response = get_prediction(2400, 4)<br>response.show()</pre><p>The result is a DataFrame that looks like this:</p><pre>+----------+--------+------------+------------------+<br>|SquareFeet|Bedrooms|    features|        prediction|<br>+----------+--------+------------+------------------+<br>|      2400|       4|[2400.0,4.0]|137499.99999999968|<br>+----------+--------+------------+------------------+</pre><blockquote>Tip: You can use PixieDust to visualize any DataFrame, including this one. If you’ve imported PixieDust and you have a DataFrame, display() is your friend :)</blockquote><p>Our ML model returned back our features along with a prediction. In this case, it predicted that a house that is 2,400 square feet and has 4 bedrooms should have a price of about $137,500, which is directly in between our 2,300 square foot house and our 2,500 square foot house.</p><h3>Step 4: Deploy and test the model with Watson ML</h3><h4>Deploy the model</h4><p>We’ve trained and tested our machine learning model, but if we want to predict house prices from a web or mobile app it’s not going to do us much good in this notebook. That’s where Watson ML comes in.</p><p>In the same notebook, we’re going to deploy this model to Watson ML and create a “scoring endpoint”, or a REST API for making predictions.</p><p>The first thing you’ll need to do is specify your Watson ML credentials. You can find your credentials by going to the Watson ML in Bluemix and clicking <strong><em>Service Credentials</em></strong> on the left (<a href="https://console.bluemix.net/catalog/services/machine-learning">head to the catalog to deploy it</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQgwk1JRcVKQ05Ify6d5sg.png" /></figure><p>Fill in the following cell with your credentials:</p><pre>service_path = &#39;https://ibm-watson-ml.mybluemix.net&#39;<br>username = &#39;YOUR_WML_USER_NAME&#39;<br>password = &#39;YOUR_WML_PASSWORD&#39;<br>instance_id = &#39;YOUR_WML_INSTANCE_ID&#39;<br>model_name = &#39;House Prices Model&#39;<br>deployment_name = &#39;House Prices Deployment&#39;</pre><p>The next cell initializes some libraries for connecting to Watson ML. These libraries are built into DSX:</p><pre>from repository.mlrepositoryclient import MLRepositoryClient<br>from repository.mlrepositoryartifact import MLRepositoryArtifact<br>ml_repository_client = MLRepositoryClient(service_path)<br>ml_repository_client.authorize(username, password)</pre><p>Next, we’ll use the same libraries to save our model to Watson ML. We pass the trained model, our data set, and a name for the model — in this case we’re calling it “House Prices Model”:</p><pre>model_artifact = MLRepositoryArtifact(<br>   model,<br>   training_data=df,<br>   name=model_name<br>)<br>saved_model = ml_repository_client.models.save(model_artifact)<br>model_id = saved_model.uid</pre><p>The call to save the model returns an object that we store in our <em>saved_model </em>variable from which we extract the unique ID for the model. This is important as it will be used later to create a deployment for the model.</p><p>We now have a trained machine learning model that we have deployed to Watson ML, but we still don’t have a way to access it. The next few cells will do just that.</p><p>We are going to create a <em>Deployment</em> for our ML model. To do this, we are going to use the <a href="https://watson-ml-api.mybluemix.net/">Watson ML Rest API</a>. The Watson ML Rest API uses token-based authentication, so our first step is to generate a token using our Watson ML credentials:</p><pre>headers = urllib3.util.make_headers(<br>   basic_auth=&#39;{}:{}&#39;.format(username, password)<br>)<br>url = &#39;{}/v3/identity/token&#39;.format(service_path)<br>response = requests.get(url, headers=headers)<br>ml_token = &#39;Bearer &#39; + json.loads(response.text).get(&#39;token&#39;)</pre><p>Now we can create our deployment. Here we make an HTTP POST to the <em>published_models/deployments</em> endpoint — passing in our Watson ML <em>instance_id</em> and the <em>model_id</em> of our newly saved model.</p><pre>deployment_url = service_path<br>   + &quot;/v3/wml_instances/&quot; + instance_id<br>   + &quot;/published_models/&quot; + model_id<br>   + &quot;/deployments/&quot;<br>deployment_header = {<br>   &#39;Content-Type&#39;: &#39;application/json&#39;,<br>   &#39;Authorization&#39;: ml_token<br>}<br>deployment_payload = {<br>   &quot;type&quot;: &quot;online&quot;,<br>   &quot;name&quot;: deployment_name<br>}<br>deployment_response = requests.post(<br>   deployment_url,<br>   json=deployment_payload,<br>   headers=deployment_header<br>)<br>scoring_url = json.loads(deployment_response.text)<br>   .get(&#39;entity&#39;)<br>   .get(&#39;scoring_url&#39;)<br>print scoring_url</pre><p>The last line above prints the <em>scoring_url</em> parsed from the response received from Watson ML. This is an HTTP endpoint that we can use to make predictions. You now have a deployed machine learning model that you can use to predict house prices from anywhere! You can call it from a front-end application, your middleware, or from a notebook — we’ll do just that next :)</p><h4>Test the model</h4><p>For now, we’re going to test our Watson ML deployment from our notebook, but the real value of deploying your ML models to Watson ML is that you can run predictions from anywhere.</p><p>In the notebook I created a new function called <em>get_prediction_from_watson_ml</em>. Just like the last function, this one takes the square footage and the number of bedrooms for the house price you would like to predict.</p><p>Rather than calling the Spark ML APIs, you can see that this function performs an HTTP POST to the <em>scoring_url </em>we received earlier.</p><pre>def get_prediction_from_watson_ml(square_feet, num_bedrooms):<br>   scoring_header = {<br>      &#39;Content-Type&#39;: &#39;application/json&#39;,<br>      &#39;Authorization&#39;: ml_token<br>   }<br>   scoring_payload = {<br>      &#39;fields&#39;: [&#39;SquareFeet&#39;,&#39;Bedrooms&#39;],<br>      &#39;values&#39;: [[square_feet, num_bedrooms]]<br>   }<br>   scoring_response = requests.post(<br>      scoring_url,<br>      json=scoring_payload,<br>      headers=scoring_header<br>   )<br>   return scoring_response.text</pre><p>Let’s run the same prediction we ran earlier — a house that is 2,400 square feet and has 4 bedrooms:</p><pre>response = get_prediction_from_watson_ml(2400, 4)<br>print response</pre><p>The call to our Watson ML REST API returned our features along with the same prediction we received when we ran our test using Spark ML and the local ML model that we generated.</p><pre>{<br>  &quot;fields&quot;: [&quot;SquareFeet&quot;, &quot;Bedrooms&quot;, &quot;features&quot;, &quot;prediction&quot;],<br>  &quot;values&quot;: [[2400, 4, [2400.0, 4.0], 137499.99999999968]]<br>}</pre><h3>Next steps</h3><p>In this post we built an end-to-end machine learning system using the IBM Data Science Experience, Spark ML, and Watson ML. In just a few lines of code, we imported and visualized a data set, built an ML pipeline and trained an ML model, and made that model available to make predictions from software running anywhere. Although we barely scratched the surface of machine learning, I hope this article gave you a basic understanding of how to build an ML system.</p><p>In the next post, I will show you how to consume the Watson ML scoring endpoint from an end-user application. In future posts, I will slowly venture deeper into machine learning with working examples for common ML problems: supervised and unsupervised, binary and multiclass classification, clustering, and more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3d9401927b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codait/building-your-first-machine-learning-system-b3d9401927b7">[deprecated] Building Your First Machine Learning System</a> was originally published in <a href="https://medium.com/codait">Center for Open Source Data and AI Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Watson Machine Learning for Developers]]></title>
            <link>https://medium.com/codait/watson-machine-learning-for-developers-b98cefb3f890?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/b98cefb3f890</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ibm-watson]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ibm-cloud]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Wed, 18 Oct 2017 18:47:18 GMT</pubDate>
            <atom:updated>2018-02-27T21:27:26.053Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding the basic problems and workflow (part 1)</h4><p><a href="https://medium.com/ibm-watson-data-lab/i-am-not-a-data-scientist-efe7ca6ceba2">I am not a Data Scientist</a>, but I am a developer interested in data science and machine learning. I hope you are here because you are as well!</p><p>This is the first installment in a series of posts aimed at introducing developers like me and you to the basic machine learning concepts and tools required to get an ML system up and running. I will not be spending a lot of time talking about how to clean and analyze data, or the finer points of how machine learning works, but I will introduce you to fundamental concepts that you will need to get your first system up and running.</p><p>Let’s start by understanding when and why you would use machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BdcXM7Qk8fmXBIMTEpaugg.png" /><figcaption>We’ll eventually use the Watson ML service to deploy our model, but the problems and workflow I describe here apply broadly to machine learning.</figcaption></figure><h3>Predictions</h3><p>The ultimate goal of a machine learning system is to make a prediction. Here are some examples you may be familiar with:</p><ol><li>Predict whether an image is a cat or dog</li><li>Predict the value of a home</li><li>Predict which products to recommend to a user</li><li>Predict which users share the same interests</li><li>Predict when to turn, accelerate, or apply the brakes in a self-driving car</li></ol><p>Machine learning is all about predictions. If you have a use case where you need to make predictions (and a lot of data), machine learning may be a good fit. How do ML systems make predictions?</p><p>It all starts with the data. ML libraries and platforms can make predictions by analyzing massive amounts of data and finding patterns or mathematical formulas that “explain” the data. The data is the most crucial component to a successful ML system. You need to have a lot of it, and it has to be good. Bad data in = bad predictions out.</p><p>Let’s go a tad deeper to get a better understanding of how machine learning works.</p><h3>Data</h3><p>It can’t be said enough. It all starts with the data, and it has to be good data.</p><p>Let’s start with a simple, well-known machine learning example: predicting house prices. Let’s say we have a data set of known houses and their associated prices:</p><pre>Square Feet       # Bedrooms       Color         Price<br>-----------       ----------       -----         -----        <br>2,100             3                White         $100,000<br>2,300             4                White         $125,000<br>2,500             4                Brown         $150,000</pre><p>Obviously this is not a lot of data, and not good data, but ignore that for now. Our goal is to build a machine learning system to predict house prices using this data set.</p><p>Predicting the price of a house is a <strong>supervised</strong> machine learning problem — that means we know the outcome for a subset of use cases (i.e., we know what the prices are for the houses listed above), and we can use those outcomes to train a ML system to predict outcomes for new use cases (i.e., predict the price for a house that is not in the list). An unsupervised ML problem is one where the system learns from the data, rather than being trained by the data. We’ll cover unsupervised learning in a future post.</p><p>Specifically, this is a <strong>regression</strong> problem. A regression problem is one in which you want to predict a real number, like the price of a house. We will also cover binary and multiclass classification (when you want to predict a class or category from a predefined list of values) and clustering (when you want to group data that is similar).</p><p>When we build a <strong>supervised</strong> ML model, we need to specify which variables we want to use to make our predictions. These variables are referred to as <strong>features</strong>. We know that when a house is 2,100 square feet, has 3 bedrooms, and is the color white, then the price is $100,000. In this example, color is not important to predicting the price of a home, but you could reason that both square footage and the number of bedrooms are. So, it makes sense that we choose <em>Square Feet</em> and <em># Bedrooms</em> as our features.</p><p>The value we want to predict is the <em>Price</em>. This is referred to as our <strong>label</strong>.</p><p>We’ll use the <strong>features</strong> in our data set to build a model that can predict the <strong>label </strong>(Price). That process looks a little like this:</p><ol><li>Choose a ML algorithm. We’ll cover some of the common algorithms used in machine learning.</li><li>Instruct our ML algorithm to use Square Feet and # Bedrooms as our <strong>features</strong> and Price as our <strong>label</strong> (the value we want to predict).</li><li>Feed the data set to our ML algorithm to train an ML model that can make predictions. The algorithm will use the data set that you feed it to come up with a mathematical formula for predicting new outcomes.</li><li>To predict a price, we feed our ML model a set of features (square footage and number of bedrooms) and in response receive a predicted price.</li></ol><p>Now that we have data, and I’ve outlined the general steps from getting from the data to a prediction, let’s see what tools can help us get there.</p><h3>Tools</h3><p>We’ll focus on the tools provided by the <a href="https://datascience.ibm.com/">IBM Data Science Experience</a> (DSX). Many of the tools are open source and can be run locally or on other platforms, and the general concepts should apply to other hosted machine learning offerings.</p><p><a href="http://jupyter.org/"><strong>Jupyter Notebooks</strong></a><strong>: </strong>Notebooks are used by data scientists to clean, visualize, and understand data. DSX uses Jupyter Notebooks, but notebooks come in different flavors. In DSX you code your notebooks in Python or Scala.</p><p><a href="https://spark.apache.org/"><strong>Apache Spark</strong></a><strong>™:</strong> Spark is a cluster computing platform for analyzing massive amounts of data in-memory. For machine learning to be effective, you need lots of data, so it only makes sense that you have a platform like Spark to help.</p><p><a href="https://spark.apache.org/docs/1.2.2/ml-guide.html"><strong>Apache Spark ML</strong></a><strong>: </strong>Spark ML is a library for building ML pipelines on top of Apache Spark. Spark ML includes algorithms and APIs for supervised and unsupervised machine learning problems.</p><p><a href="https://www.ibm.com/analytics/us/en/watson-data-platform/watson-machine-learning/"><strong>IBM Watson ML</strong></a><strong>:</strong> Watson ML is a service for deploying ML models and making predictions at runtime. Watson ML provides a REST API to your ML models which can be called directly from your application or your middleware.</p><p>Let’s see how all these tools work together.</p><h3>Workflow</h3><p>Once I have identified a prediction I want to make, and a data set to help make it, I will typically take the following path to build and host my machine learning model:</p><ol><li>Create a Jupyter Notebook and import, clean, and analyze the data.</li><li>Use Apache Spark ML to build and test a machine learning model.</li><li>Deploy the model to Watson ML.</li><li>Call the Watson ML scoring endpoint (REST API) to make predictions from a client application or backend service.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*CWbfV-xtta5PEw-Np6lIpA.png" /></figure><p>This path works for supervised and unsupervised machine learning, and I’ll use it to show you how you can solve regression, classification, and clustering ML problems.</p><h3>Next steps</h3><p>In this post, I gave an overview of what you can use machine learning for, a tool chain that you can use to build end-to-end ML systems, and the path I follow to build them. <a href="https://medium.com/ibm-watson-data-lab/building-your-first-machine-learning-system-b3d9401927b7">In part two</a>, we’ll follow this path to build an ML system to predict housing prices. I’ll show you how to get from a raw data set to a REST API with just a few lines of code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b98cefb3f890" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codait/watson-machine-learning-for-developers-b98cefb3f890">Watson Machine Learning for Developers</a> was originally published in <a href="https://medium.com/codait">Center for Open Source Data and AI Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Depth-First Look at Watson Conversation + Gremlin + JanusGraph]]></title>
            <link>https://medium.com/codait/a-depth-first-look-at-watson-conversation-gremlin-janusgraph-42be16335eef?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/42be16335eef</guid>
            <category><![CDATA[cognitive-computing]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[ibm-watson]]></category>
            <category><![CDATA[janusgraph]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Thu, 27 Jul 2017 11:06:01 GMT</pubDate>
            <atom:updated>2017-07-27T11:06:01.746Z</atom:updated>
            <content:encoded><![CDATA[<h4>How I updated my sample chatbot to use the latest in graph databases</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*72BjORgysNl5hsyhW6xyBg.png" /><figcaption>Graph image credit: <a href="https://jeremykun.com/2013/01/22/depth-and-breadth-first-search/">Jeremy Kun</a>.</figcaption></figure><p>In a previous blog post I talked about how you could use a graph database to store conversational data, such as chatbot interactions. I covered using the Apache TinkerPop framework and the Gremlin graph traversal language to store vertices and edges in a graph by tracing a typical chatbot conversation.</p><p><a href="https://medium.freecodecamp.org/have-you-had-the-talk-with-your-chatbot-about-graph-data-structures-3aaf5c3ae52c">Have you had “The Talk” with your chatbot about graph data structures?</a></p><p>Recently <a href="https://www.compose.com/articles/composes-first-graph-database-janusgraph/">Compose announced support for JanusGraph</a>, an open source Graph database that supports the Apache TinkerPop framework. You can learn more about JanusGraph at <a href="http://janusgraph.org/">janusgraph.org</a>.</p><p>In this blog post, I’ll show you how I took the concepts from my previous chatbot articles and added support for JanusGraph to the <a href="https://medium.freecodecamp.org/have-you-had-the-talk-with-your-chatbot-about-graph-data-structures-3aaf5c3ae52c">Recipe Chatbot</a> example app.</p><h3>A quick refresher</h3><p>This article is the third in a series of posts [<a href="https://medium.com/ibm-watson-data-lab/persisting-data-for-a-smarter-chatbot-be599480f7b2">1</a>, <a href="https://medium.freecodecamp.org/have-you-had-the-talk-with-your-chatbot-about-graph-data-structures-3aaf5c3ae52c">2</a>] about a chatbot called the Recipe Bot. The Recipe Bot is a Slack Bot that lets people request recipes based on specified ingredients or cuisines. Although not required, it may be useful to read the previous blogs that describe the application’s higher-level architecture.</p><p>At a minimum, it’s important to understand some of the basic relationships used in Recipe Bot’s graph database. As a conversation with the bot progresses, the application creates vertices and edges and stores them in the database. Here are the basics:</p><ol><li>A person starts a conversation with the chatbot, and the code creates a person vertex.</li><li>The person requests an ingredient and the application creates an ingredient vertex and an edge from the person to that ingredient.</li><li>The chatbot recommends a list of recipes, the person selects a recipe, and the code creates a recipe vertex and an edge from the ingredient to that recipe.</li></ol><p>Here’s a simplified version of the graph (you can see <a href="https://cdn-images-1.medium.com/max/800/1*h6fvJxvuKB_6EVn-w1itTg.png">the real one with directed edges</a> in the blog post cited above):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*fzxYyM3iP9Qc08yu4spt_g.png" /></figure><p>It’s more complicated than the diagram suggests. For example, the bot also creates an edge between the person and the recipe to make it easier to find a person’s favorite recipes. However, these simplified mechanics should give you an idea of how the app relates entities in the conversation to vertices &amp; edges in the persisted graph.</p><p>Before jumping into the new chatbot implementation with JanusGraph, let’s talk a bit about Gremlin.</p><h3>Gremlin at a glance</h3><p>As I mentioned before JanusGraph supports Gremlin.</p><blockquote>“Gremlin is a functional, data-flow language that enables users to succinctly express complex traversals on (or queries of) their application’s property graph.” — <a href="https://tinkerpop.apache.org/gremlin.html">https://tinkerpop.apache.org/gremlin.html</a></blockquote><p>In the previous article I touched on the structure of the vertices and edges associated with our conversational data, but I didn’t go into detail on how to create, update, and query those vertices and edges using Gremlin. So here’s a quick look at how that would work.</p><h4>Creating a vertex</h4><p>The following describes the structure of a person vertex used in the example app. The label identifies this vertex as a person, and this person has a name of U2JBLUPL2 (which is the user’s Slack ID).</p><pre>{<br>  &quot;label&quot;: &quot;person&quot;,<br>  &quot;type&quot;: &quot;vertex&quot;,<br>  &quot;properties&quot;: {<br>    &quot;name&quot;: &quot;U2JBLUPL2&quot;<br>  }<br>}</pre><p>To create this person in a graph database using Gremlin, run the following:</p><pre>graph.addVertex(T.label, &quot;person&quot;, &quot;name&quot;, &quot;U2JBLUPL2&quot;);</pre><p>The response would look something like this:</p><pre>{<br>  &quot;id&quot;: 4224,<br>  &quot;label&quot;: &quot;person&quot;,<br>  &quot;type&quot;: &quot;vertex&quot;,<br>  &quot;properties&quot;: {<br>    &quot;name&quot;: [<br>      {<br>        &quot;id&quot;: &quot;17b-3aw-1l1&quot;,<br>        &quot;value&quot;: &quot;U2JBLUPL2&quot;<br>      }<br>    ]<br>  }<br>}</pre><h4>Creating an edge</h4><p>The following describes the structure of an edge. In this case it represents the edge between a person (with ID 4224) and a recipe that the person has selected (with ID 4320).</p><pre>{<br>  &quot;label&quot;: &quot;selects&quot;,<br>  &quot;type&quot;: &quot;edge&quot;,<br>  &quot;inV&quot;: 4320,<br>  &quot;outV&quot;: 4224,<br>  &quot;properties&quot;: {<br>    &quot;count&quot;: 1<br>  }<br>}</pre><p>To create this edge in a graph database using Gremlin run the following:</p><pre>def g = graph.traversal();<br>def outV = g.V(4224).next();<br>def inV = g.V(4320).next();<br>outV.addEdge(&quot;selects&quot;, inV, &quot;count&quot;, 1);</pre><p>Here, you first needed to find the two vertices: the person (ID 4224) and the recipe (ID 4320). The response looks something like this:</p><pre>{<br>  &quot;id&quot;: &quot;2dl-oe05k-3yt-9ns&quot;,<br>  &quot;label&quot;: &quot;selects&quot;,<br>  &quot;type&quot;: &quot;edge&quot;,<br>  &quot;inVLabel&quot;: &quot;recipe&quot;,<br>  &quot;outVLabel&quot;: &quot;person&quot;,<br>  &quot;inV&quot;: 4320,<br>  &quot;outV&quot;: 4224,<br>  &quot;properties&quot;: {<br>    &quot;count&quot;: 1<br>  }<br>}</pre><h4>Traversing a graph</h4><p>In the previous post I showed you the following Gremlin query for getting a user’s top-five favorite recipes, sorted by count:</p><pre>g.V().hasLabel(&quot;person&quot;).has(&quot;name&quot;,&quot;U2JBLUPL2&quot;)<br>.outE().order().by(&quot;count&quot;, decr)<br>.inV().hasLabel(&quot;recipe&quot;).limit(5)</pre><p>This query uses the edge between a person and the recipes the person has selected, and ultimately returns the recipes themselves (inV). The response is an array of recipes:</p><pre>[<br>  {<br>    &quot;id&quot;: 4320,<br>    &quot;label&quot;: &quot;recipe&quot;,<br>    “type”: “vertex”,<br>    &quot;properties&quot;: {...},<br>  }<br>  {<br>    &quot;id&quot;: 4450,<br>    &quot;label&quot;: &quot;recipe&quot;,<br>    “type”: “vertex”,<br>    &quot;properties&quot;: {...},<br>  }<br>  ...<br>]</pre><p>How about the full graph for the query?</p><pre>g.V().hasLabel(&quot;person&quot;).has(&quot;name&quot;,&quot;U2JBLUPL2&quot;)<br>.outE().order().by(&quot;count&quot;, decr)<br>.inV().hasLabel(&quot;recipe&quot;).limit(5)<strong>.path()</strong></pre><p>In the query above, I simply added .path() to the end. The response includes an array of paths, where each path has an array of labels and objects. Each array of objects includes the vertices and edges traversed in the path. For example, a path would have an array of objects that includes the person vertex, the edge between the person and the recipe, and the recipe vertex. Here is a sample response:</p><pre>[<br>  {<br>    &quot;labels&quot;: [<br>      [],<br>      [],<br>      []<br>    ],<br>    &quot;objects&quot;: [<br>      {<br>        &quot;id&quot;: 4224,<br>        &quot;label&quot;: &quot;person&quot;,<br>        &quot;type&quot;: &quot;vertex&quot;<br>        ...<br>      },<br>      {<br>        &quot;id&quot;: &quot;2dl-oe05k-3yt-9ns&quot;,<br>        &quot;label&quot;: &quot;selects&quot;,<br>        &quot;type&quot;: &quot;edge&quot;<br>        ...<br>      },<br>      {<br>         &quot;id&quot;: 4320,<br>         &quot;label&quot;: &quot;recipe&quot;,<br>         &quot;type&quot;: &quot;vertex&quot;<br>         ...<br>      }<br>    ]<br>  },<br>  {<br>    &quot;labels&quot;: [...],<br>    &quot;objects&quot;: [...]<br>  }<br>  ...<br>]</pre><p>Now that you have a few Gremlin queries under your belt, here’s how to run them on JanusGraph.</p><h3>JanusGraph HTTP API</h3><p>You can execute Gremlin queries on JanusGraph in a couple of ways:</p><ol><li>You can connect to JanusGraph via WebSockets. This method allows you to have long-lived conversations between your application and JanusGraph, where state can be saved across Gremlin queries.</li><li>You can connect to JanusGraph via the HTTP API. In this method, each API call is its own unit of work, and state is not saved across API calls. Sometimes you may need to provide more information to execute an HTTP API call, but connection management is greatly simplified.</li></ol><p>I’ll focus on the HTTP API for the rest of this post. JanusGraph exposes a single HTTP POST endpoint to execute Gremlin queries. The endpoint expects a JSON-formatted document with a single key (gremlin) that has the value of your Gremlin query:</p><pre>{<br> &quot;gremlin&quot;: &quot;YOUR_GREMLIN_QUERY_HERE&quot;<br>}</pre><p>Every Gremlin query discussed above can be submitted to this endpoint, with one minor update — you must prefix every query with the following:</p><pre>def graph=ConfiguredGraphFactory.open(&quot;YOUR_GRAPH_ID&quot;);</pre><p>You should replace YOUR_GRAPH_ID with the ID you specified when creating your graph. But, how do I create a graph? — with the HTTP API of course! To create a graph, send the following Gremlin query. This example creates a graph called &quot;recipebot&quot;:</p><pre>{<br> &quot;gremlin&quot;: &quot;def graph=ConfiguredGraphFactory.create(\&quot;recipebot\&quot;);0;&quot;<br>}</pre><p>Now, every other Gremlin query you want to run against the recipebot graph simply needs to include the prefix I mentioned above. For example, here is what the HTTP POST body looks like for creating a person vertex:</p><pre>{<br>  &quot;gremlin&quot;: &quot;def graph=ConfiguredGraphFactory.open(\&quot;recipebot\&quot;); graph.addVertex(T.label, \&quot;person\&quot;, \&quot;name\&quot;, \&quot;U2JBLUPL2\&quot;);&quot;<br>}</pre><p>The HTTP response to the Gremlin endpoint looks like this:</p><pre>{<br>  &quot;requestId&quot;: &quot;6f0a533e-76ab-412e-a4f9-d73842ea12c2&quot;,<br>  &quot;status&quot;: {<br>    &quot;message&quot;: &quot;&quot;,<br>    &quot;code&quot;: 200,<br>    &quot;attributes&quot;: {}<br>  },<br>  &quot;result&quot;: {<br>    &quot;data&quot;: [<br>      {<br>        &quot;id&quot;: 4224,<br>        &quot;label&quot;: &quot;person&quot;,<br>        &quot;type&quot;: &quot;vertex&quot;<br>        ...<br>      }<br>    ],<br>    &quot;meta&quot;: {}<br>  }<br>}</pre><p>Check out the <a href="https://help.compose.com/v2.0/docs/janusgraph-creating-and-accessing-graphs">JanusGraph documentation</a> for sample queries, curl commands, and more information on how to query JanusGraph with Gremlin.</p><h3>Tying it all together</h3><p>The project uses a Node.js application that communicates with the Recipe Bot and JanusGraph. The architecture looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*GKq_TgoJcDnc4JvI9WmZuw.png" /></figure><p>The Node.js application (labeled in the diagram as “Application”) manages the integration between Slack (for relaying messages to and from users), Watson Conversation (for running the chatbot), and JanusGraph (for storing and querying vertices and edges).</p><p>You can run the sample application by following the instructions in the GitHub repo at <a href="https://github.com/ibm-watson-data-lab/watson-recipe-bot-nodejs-janusgraph">https://github.com/ibm-watson-data-lab/watson-recipe-bot-nodejs-janusgraph</a>. You’ll find the graph-related code in the following files:</p><ul><li><a href="https://github.com/ibm-watson-data-lab/watson-recipe-bot-nodejs-janusgraph/blob/master/JanusGraphRecipeStore.js">JanusGraphRecipeStore.js</a></li><li><a href="https://github.com/ibm-watson-data-lab/watson-recipe-bot-nodejs-janusgraph/blob/master/JanusGraphClient.js">JanusGraphClient.js</a></li></ul><p>JanusGraphRecipeStore.js contains the high-level functions used by the chatbot to save and retrieve entities in JanusGraph. It includes functions like:</p><ul><li>addUser — Adds a user vertex to the graph</li><li>addIngredient — Adds an ingredient vertex to the graph</li><li>recordIngredientRequestForUser — Adds or updates the edge between a user vertex and an ingredient vertex</li></ul><p>These functions, in turn, call the lower-level functions defined in JanusGraphClient.js. That file contains generic functions for working with JanusGraph, including:</p><ul><li>getOrCreateGraph — Creates a graph with the specified ID if it does not already exist</li><li>runGremlinQuery — Executes a Gremlin query on the specified graph</li><li>createVertex — Creates a vertex on the specified graph</li><li>createEdge — Creates an edge on the specified graph</li></ul><p>Remember that the HTTP API for JanusGraph contains a single endpoint — an HTTP POST for executing Gremlin queries. The runGremlinQuery function takes a Gremlin query and executes the HTTP POST. It’s the only function really required to communicate with JanusGraph and is ultimately called by all of the other functions in JanusGraphClient.js.</p><p>The other functions, like createVertex and createEdge, are convenience functions that provide a higher level of abstraction for the developer. For example, createVertex allows you to pass in a vertex object with the following structure:</p><pre>{<br>  &quot;label&quot;: &quot;person&quot;,<br>  &quot;type&quot;: &quot;vertex&quot;,<br>  &quot;properties&quot;: {<br>    &quot;name&quot;: &quot;U2JBLUPL2&quot;<br>  }<br>}</pre><p>This object is then converted into a Gremlin query, like so, and sent to the runGremlinQuery function:</p><pre>graph.addVertex(T.label, &quot;person&quot;, &quot;name&quot;, &quot;U2JBLUPL2&quot;);</pre><p>Feel free to use JanusGraphClient.js in your own project to help you get started with JanusGraph.</p><h3>Start chatting</h3><p><a href="https://hackernoon.com/chatbot-architecture-496f5bf820ed#320b">Conversations are a natural fit for graph data structures</a>. By persisting chat metadata to a graph database, you can now see how it can be a more pleasant model to develop code against—without having to implement lots of special relationship tables as in a relational database. Hopefully, this article gives you an idea of the mechanics of using Gremlin to persist data to JanusGraph.</p><p>You can try out the Recipe Bot with JanusGraph support by following the instructions in the <a href="https://github.com/ibm-watson-data-lab/watson-recipe-bot-nodejs-janusgraph">GitHub repo</a>. The <a href="https://github.com/ibm-watson-data-lab/watson-recipe-bot-nodejs-janusgraph#watson-recipe-bot--janusgraph">README</a> has everything you’ll need to get up and running. You’ll walk through the process of configuring Slack and Spoonacular, creating a Watson Conversation service instance, and provisioning your first JanusGraph deployment.</p><p>Tell your Recipe Bot I said hello!</p><p><em>If you enjoyed this article, please ♡ it to recommend it to other Medium readers.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42be16335eef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codait/a-depth-first-look-at-watson-conversation-gremlin-janusgraph-42be16335eef">A Depth-First Look at Watson Conversation + Gremlin + JanusGraph</a> was originally published in <a href="https://medium.com/codait">Center for Open Source Data and AI Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run Docker Compose in Kubernetes]]></title>
            <link>https://medium.com/@markwatsonatx/introducing-minienv-0-1-for-kubernetes-7c787b88732?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c787b88732</guid>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Sun, 16 Jul 2017 06:35:22 GMT</pubDate>
            <atom:updated>2018-04-08T07:03:01.141Z</atom:updated>
            <content:encoded><![CDATA[<p>Ever since I joined IBM as a Developer Advocate I have been thinking about and working on ways to make sample applications more accessible to developers. That combined with an abnormal obsession with containers has resulted in <a href="http://minienv.com/">minienv</a>.</p><blockquote>Note: minienv is an open source project that I built in my free time and is not affiliated with or endorsed by IBM.</blockquote><p>minienv was created to help make it easier to deliver sample applications and developer tutorials that connect to databases and other backend services. If you want to learn javascript and frontend web development there are a lot of really great web-based tools available, like <a href="https://codepen.io/">CodePen</a> and <a href="https://jsfiddle.net/">JSFiddle</a>, but what if you want to learn how to connect a Node.js application with MongoDB, or a Jupyter notebook with Apache Spark? With minienv users get their own mini Docker Compose environments where they can experiment with just about any technology, as long as it can run in a container.</p><p>Here’s a preview:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPtvYWO_5pFk%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPtvYWO_5pFk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPtvYWO_5pFk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a397ba5735a0495b8da6dc311420f47/href">https://medium.com/media/5a397ba5735a0495b8da6dc311420f47/href</a></iframe><p>minienv is a web application and collection of backend services that allow you to quickly spin up Docker Compose environments in the cloud that you can access from your web browser.</p><p>minienv includes an online editor that allows you to edit source code or data files included in your Docker Compose environments. This helps make minienv a great tool for learning (or teaching) new programming languages, databases, and other technologies right from your browser.</p><p>minenv runs on Kubernetes, but I am also working on support for Docker Swarm.</p><p>You can try minienv at <a href="http://bx.minienv.com:31111/">http://bx.minienv.com:31111</a>. This is running on a free single-node Kubernetes cluster in the <a href="https://www.ibm.com/cloud-computing/bluemix/containers">Bluemix Container Service</a> and can run up to 4 environments (instances of a sample applications). When you run minienv you specify the number of environments you want to provision. In these cases I am running on fairly lightweight hardware, so I’ve only provisioned minienv to run a handful of environments.</p><h3>How It All Started</h3><p>Last fall I discovered that Docker Compose was a really great way to deliver sample applications. In three simple steps developers could install and run sample applications that included databases and other backend services:</p><ol><li>Clone the repo</li><li>cd into the sample application directory</li><li>Run docker-compose up</li></ol><p>That’s it! With those 3 steps you could have a Swift application connecting to CouchDB, or a Scala notebook connecting to Apache Spark.</p><p>I felt so strongly that this was a great way to deliver sample applications that I built a bunch of them in my free time (it was also a great way for me to learn new technologies) and created this ugly website: <a href="http://dockercomposeup.com/">http://dockercomposeup.com</a>. I also created a handful of videos showing how it worked. Here’s one of them:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F6fyKSu1cxGc%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6fyKSu1cxGc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F6fyKSu1cxGc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/55e6ed26e5efbcf37784451ec31a2ec7/href">https://medium.com/media/55e6ed26e5efbcf37784451ec31a2ec7/href</a></iframe><p>I still think Docker Compose is a great way to deliver sample applications, but a few months ago I was inspired to take it a step further.</p><h3>Play With Docker</h3><p>minienv was inspired by a DockerCon presentation on <a href="http://labs.play-with-docker.com/">Play With Docker</a>. Play With Docker is an awesome way to learn Docker right in your web browser. If you’ve never tried it I highly recommend checking it out.</p><p>I first tried to figure out a way to run my Docker Compose sample applications inside Play With Docker, but pretty quickly realized that it wasn’t the right solution for what I wanted to accomplish. I had been playing a lot with Kubernetes in my free time (I don’t have a lot of opportunity to work with containers for my day job, but I did try and shoe horn in some of my existing work into a blog post on Kubernetes <a href="https://medium.com/ibm-watson-data-lab/zero-to-kubernetes-on-the-ibm-bluemix-container-service-fd104fd193c1">here</a>), so that’s the route I took, and before I knew it I found myself knee-deep in containers:</p><h3>Mark Watson on Twitter</h3><p>Learning Go while building a system to run docker-compose in docker-in-docker in a kubernetes pod. I&#39;ll explain later...</p><p>I spent the next 2 1/2 months building minienv, learning the ins and outs of Kubernetes, and learning some Go while I was at it (I always try to learn new programming languages or frameworks on side projects because ultimately most never make it, and at a minimum I want to gain new knowledge and experience).</p><p>So, today I am happy to release version 0.1 of minienv for Kubernetes. There is still a lot of work to be done – mostly on performance, security, and of course documentation, but I’m pretty excited that I was able to get a prototype up and running in just a few months.</p><p>If you are interested in learning more about minienv please hit me up. If you you are interested in how minienv runs in Kubernetes this video covers the basics:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuVQZPdleE1o%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuVQZPdleE1o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuVQZPdleE1o%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/341a85366549098fe552cabefa49a1ab/href">https://medium.com/media/341a85366549098fe552cabefa49a1ab/href</a></iframe><p>Thanks for reading!</p><p>-Mark</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c787b88732" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero to Kubernetes on the IBM Bluemix Container Service]]></title>
            <link>https://medium.com/codait/zero-to-kubernetes-on-the-ibm-bluemix-container-service-fd104fd193c1?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd104fd193c1</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Tue, 18 Apr 2017 14:13:43 GMT</pubDate>
            <atom:updated>2018-01-31T18:44:43.792Z</atom:updated>
            <content:encoded><![CDATA[<h4>Just the basics. Nothing fancy.</h4><p>A few weeks ago I stumbled across this tweet:</p><h3>Jake Kitchener on Twitter</h3><p>Proud to bring you Kubernetes from IBM https://t.co/IXsEPHm3gZ</p><p>I love containers, and I have been playing with Kubernetes for a while now, so I was excited to see <a href="https://www.ibm.com/blogs/bluemix/2017/03/kubernetes-now-available-ibm-bluemix-container-service/">IBM offering Kubernetes as part of the IBM Bluemix Container Service</a> (Note: as of this writing this service is in beta).</p><p>After the <a href="https://medium.com/ibm-watson-data-lab/interconnect-with-us-89539e15b515">InterConnect</a> conference I got a chance to experiment with the Kubernetes offering, and I decided to deploy some of my favorite Bluemix apps to Kubernetes.</p><p>Before I get into all that, I want to talk a little about containers and Kubernetes.</p><h3><strong>Containers</strong></h3><p>For the purpose of this article I am going to assume that you have some understanding of containers. Most likely you have used, or are at least familiar with, Docker. If you are not familiar with Docker, or containers in general, I would recommend <a href="https://medium.freecodecamp.com/a-beginner-friendly-introduction-to-containers-vms-and-docker-79a9e3e119b">this post</a> from our friends at <a href="https://medium.freecodecamp.com/">freeCodeCamp</a>.</p><p>Containers provide developers with a way to package up applications and their dependencies in a lightweight manner. They are an attractive deployment option for some developers as they provide consistency across deployment environments. When you package your application into a container you are guaranteed that the same code you ran in development will run in QA, staging, and production. This works best when you incorporate containers into your entire development cycle.</p><h3>Kubernetes</h3><p>Kubernetes was originally developed by Google as part of the Borg project and open sourced in 2014. In March, 2016 Kubernetes became the first project hosted by the <a href="https://www.cncf.io/">CNCF</a>. From the Kubernetes website:</p><blockquote><a href="https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/">Kubernetes</a> is an open-source system for automating deployment, scaling, and management of containerized applications.</blockquote><p>In a nutshell, Kubernetes helps you run containers in production. You can deploy your containers to Kubernetes, and Kubernetes will do things like:</p><ol><li>Ensure your containers are always running.</li><li>Handle network routing between your containers.</li><li>Auto-scale your containers across multiple nodes.</li></ol><p>Kubernetes also supports Ingress and Load Balancers, Stateful Sets for running databases in containers, Persistent Volumes, and more. Some of these features are provided by the underlying cloud platform and not Kubernetes itself. For example, load balancing is currently not built-in to Kubernetes, but rather implemented by the cloud platform that is running Kubernetes.</p><p>For this example I’m not going to worry about networking, auto-scaling, or any of these other features. I’m going to simply deploy a single pod (I’ll tell you what a pod is later) and a single secret.</p><h3><strong>Kubernetes/Containers vs. PaaS</strong></h3><p>There are many ways to deploy containers to production. Many cloud providers have proprietary container services. Kubernetes, on the other hand, is an open source standard which ensures your Kubernetes deployment will run on any cloud platform that supports Kubernetes. You can even run Kubernetes in a multi-cloud infrastructure.</p><p>Kubernetes is a great option for running containers in production, but are containers right for you?</p><p>PaaS offerings like Cloud Foundry and Bluemix make it super simple for developers to deploy applications to the cloud. If you are running a popular software stack like Node.js or Python, there are very good PaaS offerings available to you as a developer. PaaS offerings are often more mature and have more built-in features like user management, integrated CI/CD, metrics, and logging.</p><p>Sometimes, however, these offerings are not enough. Maybe you want to run a software stack not supported by your PaaS provider, or maybe you want to run a custom version of your software stack. In these cases containers and Kubernetes might be a better option.</p><p>See <a href="http://stackoverflow.com/questions/32047563/kubernetes-vs-cloudfoundry">this Stack Overflow post</a> for a good discussion on the differences between Kubernetes and Cloud Foundry.</p><h3>Let’s run Kubernetes on Bluemix!</h3><p>If this is your first time working with Kubernetes, this isn’t the right place to start. I would recommend <a href="https://kubernetes.io/docs/tutorials/">starting with the Kubernetes tutorials</a>. I also recommend installing and running <a href="https://kubernetes.io/docs/getting-started-guides/minikube/">Minikube</a>, which is a local, single-node Kubernetes cluster that runs inside a VM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A9Y92UZ3QwLGgwIfpdRCEQ.png" /><figcaption><a href="https://www.ibm.com/cloud-computing/bluemix/containers">IBM Bluemix Container Service</a> logo. I can hardly <strong>contain</strong> myself!</figcaption></figure><h4><strong>Prerequisites</strong></h4><p>If this is your first time working with Kubernetes—and you ignored my earlier warning about this not being the best place to start—then you’ll need to <a href="https://kubernetes.io/docs/tasks/kubectl/install/">download the Kubernetes CLI</a>. After you have the Kubernetes CLI installed, you’ll need to <a href="https://console.ng.bluemix.net/docs/cli/reference/bluemix_cli/index.html">set up the Bluemix CLI</a>.</p><p>At this point you should have the <strong>kubectl</strong> and <strong>bx</strong> commands available from your command line interface. The next step is to configure the container service plugin for the Bluemix CLI. Log in using the <strong>bx</strong> command:</p><pre>bx login -a <a href="https://api.ng.bluemix.net/">https://api.ng.bluemix.net</a></pre><p>Run the following command:</p><pre>bx plugin install container-service -r Bluemix</pre><p>Next, run this command to initialize the container service plugin:</p><pre>bx cs init</pre><h4><strong>Creating a cluster</strong></h4><p>You’ll start by creating a free-tier cluster in the Bluemix Container Service:</p><pre>bx cs cluster-create --name my-cluster</pre><p>This will create a cluster with a single worker node with 2 vCPU and 4 GB memory. List your clusters by running the following command:</p><pre>bx cs clusters</pre><p>Next, you’ll need to get your remote cluster config to use a local <strong>kubectl</strong> context from your command line. It may take a few minutes for your cluster to be created. Run the following command every few minutes until the cluster is ready:</p><pre>bx cs cluster-config my-cluster</pre><p>When your cluster is up and running, the result of this command should look something like this:</p><pre>Downloading cluster config for my-cluster<br>OK<br>The configuration for my-cluster was downloaded successfully. Export environment variables to start using Kubernetes.</pre><pre><strong>export KUBECONFIG=/Users/markwatson/.bluemix/plugins/container-service/clusters/my-cluster/kube-config-prod-dal10-my-cluster.yml</strong></pre><p>Copy the export command and run it in a new command line window.</p><blockquote>Note: You will have to run this command every time you open a new command line interface. Alternatively you can configure a new context in your local kube config, typically found at <strong>~/.kube/config</strong>. For example, I created a context called “bx-context,” so if I run the command <strong>kubectl config use-context bx-context</strong>, kubectl will be configured to access my Bluemix cluster globally. See <strong>~/.kube/config</strong>, and your cluster YML file located at <strong>KUBECONFIG</strong> (see snippet above) for more info.</blockquote><h4><strong>Kubernetes dashboard</strong></h4><p>You can access your Bluemix Kubernetes dashboard locally by running:</p><pre>kubectl proxy</pre><p>After running this command open http://127.0.0.1:8001/ui in your web browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZhcSLglUK4q2CRK_DXJc2A.png" /></figure><p>If your dashboard looks similar to the screenshot above, then congrats! You are officially running Kubernetes in the IBM Bluemix Container Service. Next, you’ll get your first pod up and running.</p><h3>Hello, Kubernetes</h3><p>At this point you should have a Kubernetes cluster running in the IBM Bluemix Container Service, but it’s not actually running any containers yet. You’ll soon change that, but first a quick discussion on <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/">Kubernetes Pods</a>.</p><p>A pod, in this context, is a group of one or more containers. In this article, you’re going to set up your pods with a single container. You can deploy a pod (or container) to Kubernetes using <strong>kubectl</strong> and passing it a YAML file that describes the pod:</p><pre><strong>apiVersion: </strong>v1<br><strong>kind: </strong>Pod<br><strong>metadata:<br>  name: </strong>my-nginx-pod<br>  <strong>labels</strong>:<br>    <strong>app</strong>: my-app<br><strong>spec:<br>  containers:<br>    </strong>- <strong>name: </strong>nginx<br>      <strong>image: </strong>nginx:latest</pre><p>This YAML file tells Kubernetes to deploy a <strong>pod</strong> with the name <strong>my-nginx-pod</strong>, which uses the <strong>nginx:latest</strong> container. By default it will download the latest container image from Docker, but you could also use your private Bluemix container registry.</p><p>Let’s deploy it! Copy the YAML above into a file called my-nginx-pod.yaml and run the following command:</p><pre>kubectl create -f my-nginx-pod.yaml</pre><p>To see what pods are running in Kubernetes, click the Pods link in the left nav of the dashboard or run the following command:</p><pre>kubectl get pods</pre><p>If the pod has started you should see something like this:</p><pre>NAME           READY     STATUS    RESTARTS   AGE<br>my-nginx-pod   1/1       Running   0          7s</pre><p>An nginx instance is now running inside Kubernetes, but you can’t access it yet. You’ll need to expose it to the outside world. There are a number of ways to expose ports on your pods. The first and easiest one is by using the expose command:</p><pre>kubectl expose pods my-nginx-pod --type=NodePort --port=80 --name=my-nginx-pod-svc</pre><p>This will create a <a href="https://kubernetes.io/docs/user-guide/services/">Kubernetes service</a>, which will act as a proxy to port 80 in our nginx container. Kubernetes will allocate a port for the service in the range of 30000–32767. To get the port assigned by Kubernetes, run:</p><pre>kubectl get services</pre><p>The output should look similar to the following:</p><pre>NAME               CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE<br>kubernetes         10.10.10.1     &lt;none&gt;        443/TCP        4h<br>my-nginx-pod-svc   10.10.10.252   &lt;nodes&gt;       80:<strong>32398</strong>/TCP   9</pre><p>Here you can see the service that Kubernetes created and that the service is exposed on <strong>port 32398</strong>. (Ignore the cluster-ip value. That’s the internal IP address of the service inside the cluster.) You’ll need to get the IP address of the node within the cluster to access the service. So, get a list of nodes:</p><pre>kubectl get nodes</pre><p>The output should look something like this:</p><pre>NAME             STATUS    AGE<br>169.47.249.158   Ready     4h</pre><p>You can now access your nginx service. In this case, it’s available at http://169.47.249.158:32398:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JBGYJ54o_iLeOMS7vWuqgw.png" /></figure><p>You can use services to expose a port on a single pod like above, or load-balance a group of pods. You can also define a service in a YAML file and deploy it to Kubernetes. The YAML file for this service would look something like this:</p><pre><strong>apiVersion: </strong>v1<br><strong>kind: </strong>Service<br><strong>metadata:<br></strong>  <strong>name: </strong>my-nginx-pod-svc<br><strong>spec:<br>  selector:<br>    app: </strong>my-app<br>  <strong>ports:<br>  </strong><strong>- protocol: TCP<br>    port: 80</strong><br>  <strong>type: </strong>NodePort</pre><p>You now have an nginx instance running inside Kubernetes that you can access from outside the cluster, but this just barely scratches the surface of Kubernetes. I’d like to point out a few other Kubernetes concepts that are extremely important, but outside the scope of this article:</p><h4>1. ReplicaSets</h4><p>You can and should use <a href="https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/">ReplicaSets</a> to ensure your pods are always running. I typically use <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployments</a>, which essentially allow you to define your pods and replica sets in one file along with your desired state. For example, the following deployment tells Kubernetes to ensure three nginx pods are always running:</p><pre><strong>apiVersion: </strong>apps/v1beta1<br><strong>kind: </strong>Deployment<br><strong>metadata:<br>  name: </strong>my-nginx-deployment<br><strong>spec:<br>  replicas: </strong>3<br>  <strong>template:<br>    metadata:<br>      labels:<br>        app: </strong>my-app<br>    <strong>spec:<br>      containers:<br>      </strong>- <strong>name: </strong>nginx<br>        <strong>image: </strong>nginx:latest<br>        <strong>ports:<br>        </strong>- <strong>containerPort: </strong>80</pre><h4>2. LoadBalancers or Ingress controllers</h4><p>You would typically configure your service with type <a href="https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/">LoadBalancer</a>. This will use the underlying cloud provider’s load balancing capabilities, allow you to configure the port (or use standard http/https ports), and in most cases allow you to configure SSL. Alternatively, you can create an <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress</a> to accomplish the same task. I used a NodePort because the free tier of Kubernetes on Bluemix does not support load balancer or ingress options.</p><h3>Deploying the Recipe Chatbot</h3><p>Now that you’ve configured your Kubernetes cluster on Bluemix and I’ve introduced you to some simple Kubernetes concepts, you’re ready to deploy a real application. For this exercise you’ll deploy my <a href="https://github.com/ibm-cds-labs/watson-recipe-bot-nodejs-cloudant">Recipe Chatbot</a>.</p><p>The Recipe Chatbot is a simple Slackbot for finding recipes based on ingredients or cuisines. It uses Watson Conversation to manage the chat, Spoonacular for looking up recipes and instructions, and IBM Watson Data Platform services like Cloudant or IBM Graph to store user behavior. You can read more about it:</p><ul><li><a href="https://medium.com/ibm-watson-data-lab/persisting-data-for-a-smarter-chatbot-be599480f7b2">Persisting Data for a Smarter Chatbot</a></li><li><a href="https://medium.freecodecamp.com/have-you-had-the-talk-with-your-chatbot-about-graph-data-structures-3aaf5c3ae52c">Have you had “The Talk” with your chatbot about graph data structures?</a></li></ul><p>The Recipe Chatbot uses hosted database services and hosted APIs. This is typical of many applications. Even though you are running your applications inside containers, you will still need to access external services. In this case, each of these external services has its own set of credentials. I’ll show you below how to configure them in Kubernetes.</p><p>There is quite a bit of setup required to get the Recipe Chatbot up and running, so I recommend you follow the instructions at the GitHub repo for the Node.js/Cloudant version at <a href="https://github.com/ibm-cds-labs/watson-recipe-bot-nodejs-cloudant">https://github.com/ibm-cds-labs/watson-recipe-bot-nodejs-cloudant</a>.</p><h4><strong>Build the container</strong></h4><p>Before you can create a pod, you’ll need a container.</p><blockquote>Tip: you can skip these steps and go straight to the <strong>Deploy to IBM Bluemix Container Service</strong> section if you don’t want to create the container yourself. I have made the container publicly available from <a href="https://hub.docker.com/r/markwatsonatx/">my Docker Hub account</a>.</blockquote><p>The Recipe Chatbot includes a Dockerfile called bot.Dockerfile in the docker folder. Here are the contents of the Dockerfile:</p><pre>FROM node:latest<br>MAINTAINER Mark Watson &lt;markwatsonatx@gmail.com&gt;<br>RUN mkdir -p /usr/src/bot<br>COPY package.json /usr/src/bot/package.json<br>COPY index.js /usr/src/bot/index.js<br>COPY CloudantRecipeStore.js /usr/src/bot/CloudantRecipeStore.js<br>COPY RecipeClient.js /usr/src/bot/RecipeClient.js<br>COPY SousChef.js /usr/src/bot/SousChef.js<br>WORKDIR /usr/src/bot<br>RUN npm install<br>CMD [&quot;node&quot;,&quot;index.js&quot;]</pre><p>The code uses the latest node (node:latest), copies only the files needed to run the bot and nothing more, and runs npm install when it builds the container. When the container is started it will run node index.js.</p><blockquote>Note: <strong>Do not include your .env file in your container</strong>. It will contain your Cloudant credentials, Watson Conversation credentials, etc. If you <em>do</em> include your .env file in your container, <strong>do not share it publicly</strong>.</blockquote><p>To build the container, cd into the docker folder:</p><pre>cd /dev/github/ibm-cds-labs/watson-recipe-bot-nodejs-cloudant/docker</pre><p>Run the following docker command to build the container (replace markwatsonatx with your own Docker username):</p><pre>docker<em> </em>build -t markwatsonatx/watson-recipe-bot-nodejs-cloudant:latest -f ./bot.Dockerfile ../</pre><p>Upload to <a href="https://hub.docker.com/">Docker Hub</a>:</p><pre>docker push markwatsonatx/watson-recipe-bot-nodejs-cloudant:latest</pre><h4><strong>Deploy to IBM Bluemix Container Service</strong></h4><p>Now that you’ve created your Docker container and uploaded it to Docker Hub, you’re ready to deploy it to Kubernetes. The Recipe Chatbot requires the following environment variables to run properly:</p><pre>SLACK_BOT_TOKEN<br>SLACK_BOT_ID<br>SPOONACULAR_KEY<br>CONVERSATION_USERNAME<br>CONVERSATION_PASSWORD<br>CONVERSATION_WORKSPACE_ID<br>CLOUDANT_URL<br>CLOUDANT_DB_NAME</pre><p>You <em>could</em> include these variables in a .env file in your container, but as I mentioned <strong>I highly discourage this approach</strong>. Instead, you can use <a href="https://kubernetes.io/docs/user-guide/secrets/">Kubernetes Secrets</a>.</p><p>You deploy secrets to Kubernetes just like you deploy pods. Start by creating a YAML file for your secret:</p><pre><strong>apiVersion: </strong>v1<br><strong>kind: </strong>Secret<br><strong>metadata:<br>  name: </strong>bot-secrets<br><strong>data:<br>  slackBotToken: </strong>eGXXX<br>  <strong>slackBotId: </strong>VTXXX<br>  <strong>spoonacularKey: </strong>dnXXX<br>  <strong>conversationUsername: </strong>ZTXXX<br>  <strong>conversationPassword: </strong>N2XXX<br>  <strong>conversationWorkspaceId: </strong>YmXXX<br>  <strong>cloudantUrl: </strong>aHXXX<br>  <strong>cloudantDbName: </strong>d2XXX</pre><p>Obviously I did not share the actual values here, but there is one important note: <strong>these values must be base-64 encoded</strong>. For example, if your slackBotToken is 1234 then the value in your secret file should be MTIzNA==.</p><p>Save the YAML above to a file called bot-secrets.yaml. Base-64 encode each value in your .env file (created when you set up your Recipe Chatbot) and copy it into your secrets.</p><p>Time to deploy your secret to Kubernetes. Run the following command:</p><pre>kubectl create -f bot-secrets.yaml</pre><p>Now you’re ready to take the Recipe Bot container and deploy it as a pod. Create a file called bot.yaml with the following contents (replace markwatsonatx with your Docker username if you deployed it yourself):</p><pre><strong>apiVersion: </strong>v1<br><strong>kind: </strong>Pod<br><strong>metadata:<br>  name: </strong>bot-pod<br><strong>spec:<br>  containers:<br>    </strong>- <strong>name: </strong>watson-recipe-bot<br>      <strong>image: </strong>markwatsonatx/watson-recipe-bot-nodejs-cloudant:latest<br>      <strong>env:<br>        </strong>- <strong>name: </strong>SLACK_BOT_TOKEN<br>          <strong>valueFrom:<br>            secretKeyRef:<br>              name: </strong>bot-secrets<br>              <strong>key: </strong>slackBotToken<br>        - <strong>name: </strong>SLACK_BOT_ID<br>          <strong>valueFrom:<br>            secretKeyRef:<br>              name: </strong>bot-secrets<br>              <strong>key: </strong>slackBotId<br>        - <strong>name: </strong>SPOONACULAR_KEY<br>          <strong>valueFrom:<br>            secretKeyRef:<br>              name: </strong>bot-secrets<br>              <strong>key: </strong>spoonacularKey<br>        - <strong>name: </strong>CONVERSATION_USERNAME<br>          <strong>valueFrom:<br>            secretKeyRef:<br>              name: </strong>bot-secrets<br>              <strong>key: </strong>conversationUsername<br>        - <strong>name: </strong>CONVERSATION_PASSWORD<br>          <strong>valueFrom:<br>            secretKeyRef:<br>              name: </strong>bot-secrets<br>              <strong>key: </strong>conversationPassword<br>        - <strong>name: </strong>CONVERSATION_WORKSPACE_ID<br>          <strong>valueFrom:<br>            secretKeyRef:<br>              name: </strong>bot-secrets<br>              <strong>key: </strong>conversationWorkspaceId<br>        - <strong>name: </strong>CLOUDANT_URL<br>          <strong>valueFrom:<br>            secretKeyRef:<br>              name: </strong>bot-secrets<br>              <strong>key: </strong>cloudantUrl<br>        - <strong>name: </strong>CLOUDANT_DB_NAME<br>          <strong>valueFrom:<br>            secretKeyRef:<br>              name: </strong>bot-secrets<br>              <strong>key: </strong>cloudantDbName</pre><p>You’ll notice above that the configurations map the variables in your bot-secrets to environment variables in your pod. The Node.js application doesn’t care how the environment variables are set, as long as they are there when it starts.</p><p>Time to deploy the pod! Run the following command and cross your fingers:</p><pre>kubectl create -f bot.yaml</pre><p>If all goes well, your Recipe Chatbot should be up and running in a few seconds, and you should be able to chat with your Slackbot. As you can see below I can access my favorite recipes from Cloudant:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1AhUEaSHsIAFti6n5ViSA.png" /></figure><p>If anything goes wrong with your chatbot, you can simply kubectl delete pod bot-pod and then recreate it using kubectl create -f bot.yaml.</p><h3>What just happened?</h3><p>If you followed the exercises in this post, then you did at least some of the following:</p><ul><li>Created a Kubernetes cluster in the IBM Bluemix Container Service.</li><li>Deployed an nginx container as a Kuberntes pod and made it publicly accessible.</li><li>Built a container and pushed it to Docker Hub.</li><li>Deployed the Recipe Chatbot to Kubernetes via a single secret and a single pod.</li></ul><p>If you are currently deploying containers to production; interested in deploying containers to production; or curious about moving your application, solution, or service to containers, then Kubernetes could be a great fit.</p><p>If this post helped you get something up and running in Kubernetes on Bluemix — or if you leave here with just a little bit better understanding of Kubernetes, why it’s important, or where it might fit in your deployment strategy — then it was all worth it. Congratulations!</p><p><em>If you enjoyed this article, please ♡ it to recommend it to other Medium readers.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd104fd193c1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codait/zero-to-kubernetes-on-the-ibm-bluemix-container-service-fd104fd193c1">Zero to Kubernetes on the IBM Bluemix Container Service</a> was originally published in <a href="https://medium.com/codait">Center for Open Source Data and AI Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Most Popular Search Term at SXSW, According to Our Chatbot]]></title>
            <link>https://medium.com/codait/the-most-popular-search-term-at-sxsw-according-to-our-chatbot-9f4a47a33406?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f4a47a33406</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[pixiedust]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Mon, 10 Apr 2017 18:59:34 GMT</pubDate>
            <atom:updated>2017-04-11T16:26:40.992Z</atom:updated>
            <content:encoded><![CDATA[<h4>Analyzing conversation data using Spark, Jupyter, and PixieDust</h4><p>A few weeks ago <a href="https://medium.com/u/b650e7381f14">Raj Singh</a> and I demonstrated a chatbot called Cognitive Event Finder at <a href="https://www.ibm.com/design/sxsw/">IBM’s installation at SXSW</a>. The chatbot allowed users to search for tech sessions, music gigs, or film screenings from our mobile-optimized web app or via SMS using Twilio. We’re <a href="https://cognitive-event-finder.mybluemix.net/">still running the demo app</a>, and we also <a href="https://medium.com/ibm-watson-data-lab/navigating-sxsw-via-cognitive-chatbot-28a2e1f7a456">wrote up its architecture</a>. Of course, it’s <a href="https://github.com/ibm-cds-labs/cognitive-event-finder/">also on GitHub</a>.</p><p>In this article, I’ll look at how we analyzed conversations with our bot. I’ll also provide a Jupyter notebook that Raj and I worked on, which hopefully gives you ideas on how to connect application logs to <a href="http://spark.apache.org/">Apache Spark™</a> for back-end analytics. The notebook takes you through data access, all the way to visualizing results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQx6r2YKg9P3G2YYyfO9xw.png" /><figcaption>The chatbot in action. We had about 500 users, with a bias toward tech as part of SXSW Interactive.</figcaption></figure><h3>Logging conversations</h3><p>We used <a href="https://www.ibm.com/cloudant">Cloudant</a> to log conversations, including every node of the dialog tree traversed in the <a href="https://www.ibm.com/watson/developercloud/conversation.html">Watson Conversation</a> service. We captured this data for several reasons:</p><ol><li>To allow users to recall previous searches. (At any point a user can say “show me my recent searches,” and the chatbot pulls them from Cloudant and displays them to the user.)</li><li>To analyze conversations to see what type of experience users were having with the chatbot. (Answer: mixed, but that’s a story for another day.)</li><li>To compile fun stats!</li></ol><p>I’m here to talk about #3. <a href="http://ibm.biz/sxswappmetrics">Raj and I built a Jupyter notebook</a> that you can access in the IBM Data Science Experience (DSX), which wraps up Spark with notebooks, object storage, and other handy features. Here’s the basic flow of the notebook:</p><ol><li>Access Cloudant data directly from DSX.</li><li>Transform Cloudant JSON into relational table-like structures required to use the data efficiently in Spark.</li><li>Use PySpark operations to shape the data for analysis.</li><li>Visualize application usage patterns via the <a href="https://medium.com/ibm-watson-data-lab/i-am-not-a-data-scientist-efe7ca6ceba2">PixieDust Python helper library for Spark notebooks</a>.</li></ol><p>In my case, the usage pattern I care about the most is popular search terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*Cw1nWHrU9Mf8IzS-9cNkyA.png" /><figcaption>There are no data science notebooks in the movie “The Notebook,” but if there were it would be more exciting.</figcaption></figure><h3>The Notebook (2004)</h3><p>Again, <a href="http://ibm.biz/sxswappmetrics">here’s the notebook on IBM’s Data Science Experience</a>. If you plan on doing anything more than just viewing it, you’ll want to download it via the icon in the menu bar.</p><p>When you walk through its cells, you will see how we import data from Cloudant. Here’s the basic structure of our JSON documents. Each one represents a single conversation between the user and the chatbot:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0c8cf2bb9b324f42aee7a9de79ecc24/href">https://medium.com/media/d0c8cf2bb9b324f42aee7a9de79ecc24/href</a></iframe><p>In the notebook, you’ll see how we flatten this data for analysis. In this case, we transform this one document into two rows — one row for each node in the dialog tree:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2sBx5dbnMHuiPWbEuFbvug.png" /><figcaption>The relational representation of our Cloudant JSON. It’s a Spark SQL DataFrame, visualized by PixieDust as a simple table.</figcaption></figure><p>Ultimately, the notebook shows how we found the most popular tech event search. (It was “AI”.) You’ll also see how we used PixieDust to visualize a range of popular search terms in a single line of code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6apiicspWEN1k2ClP03tJA.png" /><figcaption>A pie chart, courtesy of the display() API in PixieDust.</figcaption></figure><h3>Backing into the end</h3><p>That’s the overview of the back end of the chatbot I built with Raj. We hope it shows how you can quickly design a plan for monitoring and analyzing application usage, all using cloud-based persistence and analytics services.</p><p>So to conclude: feel free to play with the example we set up by uploading the notebook into your own DSX account. What other interesting trends can you find? And if you’ve enjoyed this article, please remember to recommend it on Medium using the ♡ here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f4a47a33406" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codait/the-most-popular-search-term-at-sxsw-according-to-our-chatbot-9f4a47a33406">The Most Popular Search Term at SXSW, According to Our Chatbot</a> was originally published in <a href="https://medium.com/codait">Center for Open Source Data and AI Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Have you had “The Talk” with your chatbot about graph data structures?]]></title>
            <link>https://medium.com/free-code-camp/have-you-had-the-talk-with-your-chatbot-about-graph-data-structures-3aaf5c3ae52c?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/3aaf5c3ae52c</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[ibm-watson]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[graph-database]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Sun, 19 Feb 2017 00:28:53 GMT</pubDate>
            <atom:updated>2017-02-19T00:28:53.414Z</atom:updated>
            <content:encoded><![CDATA[<h4>A coming-of-age story for your database model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AeUDHuLM_pyfNZ3ZP5znyQ.png" /><figcaption>Image credit: <a href="http://www.charlotteparent.com/CLT/Health-Development/More-Than-the-Birds-and-the-Bees-Teaching-Your-Child-About-Healthy-Sexuality/">Charlotte Parent</a></figcaption></figure><p>Graph databases are a great way to store conversational data. A simple <a href="https://hackernoon.com/chatbot-architecture-496f5bf820ed#320b">dialog</a> <a href="https://en.wikipedia.org/wiki/Dialog_tree">tree</a> can add depth to character interactions in a video game. A <a href="https://gigaom.com/2013/05/15/how-google-is-setting-the-new-search-standard-with-voice-and-knowledge-graph/">knowledge</a> <a href="http://www.aclweb.org/anthology/N15-1086">graph</a> can extract more meaning from dialog to better understand how user intent relates to an application’s data.</p><p>In this article, I’ll show you a basic graph model for capturing chatbot interactions and how to persist them using the <a href="https://tinkerpop.apache.org/">Apache TinkerPop</a> framework. I’ll also show you some <a href="http://tinkerpop.apache.org/gremlin.html">Gremlin</a> queries for adding a recommendation feature to the chatbot. The source code and setup instructions for my example “Recipe Bot” are <a href="https://github.com/ibm-cds-labs/watson-recipe-bot-nodejs-graph">on GitHub</a>.</p><h3>Review: Recipe Bot</h3><p>The Recipe Bot is a <a href="https://api.slack.com/bot-users">Slack Bot User</a> that lets people request recipes based on specified ingredients or cuisines. Previously I showed you how to add support for users to request their favorite recipes, like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*SjILWmRnPN3sSdr133pHRg.png" /></figure><p>The graph version of the bot has all of the same features I discussed in my <a href="https://medium.com/ibm-watson-data-lab/persisting-data-for-a-smarter-chatbot-be599480f7b2#.jvmry69xz">previous article on persisting metadata with JSON</a>, but with the graph version you’ll be adding recommendations.</p><h3>How it works with TinkerPop</h3><p>Here is an architecture diagram of how the bot works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*uGXlWtFX65TWPtvpKUkWGg.png" /><figcaption>My Recipe Bot. Hey! The diagram is actually an <a href="https://en.wikipedia.org/wiki/Graph_theory">undirected graph</a>. Who knew?</figcaption></figure><p>You’ll see that I’m using the Watson Conversation service. Watson Conversation lets me describe the flow of the conversation through the use of dialogs, and it helps me extract information and user intent from chat messages. You can code your own dialog tree and perform your own message parsing, or you can use tools like Watson Conversation or Botkit to help. Here is how the dialog tree for the Recipe Bot is modeled:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qaJL5vQGKkhUCShVpiusGg.png" /><figcaption>The Watson Conversation UI. Graphs are everywhere.</figcaption></figure><p>You can follow a conversation through the dialog tree similar to how you can follow vertices and edges in a graph (after all, <a href="https://en.wikipedia.org/wiki/Tree_(graph_theory)">trees are graphs</a> too):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*fzxYyM3iP9Qc08yu4spt_g.png" /><figcaption>It’s not much of a tree, but I’m keeping it simple, y’all.</figcaption></figure><p>In the simplified graph above, the Recipe Bot cares only about the progression between the major entities of the bot:</p><ol><li>People</li><li>Ingredients</li><li>Cuisines</li><li>Recipes</li></ol><h3>Data model &amp; access pattern</h3><p>As the conversation progresses, you store the following vertices and edges using the TinkerPop API:</p><ol><li><strong>Person vertex:</strong> For each person that interacts with the bot, store that person as a vertex in the graph.</li></ol><pre>{<br>  &quot;label&quot;: &quot;person&quot;,<br>  &quot;type&quot;: &quot;vertex&quot;,<br>  &quot;properties&quot;: {<br>    &quot;name&quot;: &quot;U2JBLUPL2&quot;<br>  }<br>}</pre><p><strong>2. Ingredient or cuisine vertex:</strong> When a person requests a specific ingredient or cuisine, you store that ingredient or cuisine — along with the list of recipes retrieved from <a href="https://spoonacular.com/food-api">Spoonacular</a> — as a vertex.</p><pre>{<br>  &quot;label&quot;: &quot;cuisine&quot;,<br>  &quot;type&quot;: &quot;vertex&quot;,<br>  &quot;properties&quot;: {<br>    &quot;name&quot;: &quot;chinese&quot;,<br>    &quot;detail&quot;: &quot;[{\&quot;id\&quot;: 573147, \&quot;title\&quot;: \&quot;Kale Fried Rice\&quot;...&quot;<br>  }<br>}</pre><p><strong>3. Selects edge, person → (ingredient | cuisine):</strong> You create an edge, labelled &quot;selects&quot;, between the person and the ingredient or cuisine (that is, “person selects cuisine”). In addition, store a &quot;count&quot; property on the edge and increment its value each time the user requests the same ingredient or cuisine.</p><pre>{<br>  &quot;label&quot;: &quot;selects&quot;,<br>  &quot;type&quot;: &quot;edge&quot;,<br>  &quot;inV&quot;: 4152,<br>  &quot;outV&quot;: 4224,<br>  &quot;properties&quot;: {<br>    <strong>&quot;count&quot;: 3</strong><br>  }<br>}</pre><p>4. <strong>Recipe vertex: </strong>When a user requests a recipe, store the recipe as a vertex.</p><pre>{<br>  &quot;label&quot;: &quot;recipe&quot;,<br>  &quot;type&quot;: &quot;vertex&quot;,<br>  &quot;properties&quot;: {<br>    &quot;name&quot;: &quot;573147&quot;,<br>    &quot;detail&quot;: &quot;Ok, it takes *45* minutes to make...*&quot;,<br>    &quot;title&quot;: &quot;Kale Fried Rice&quot;<br>  }<br>}</pre><p>5. <strong>Selects edge, (ingredient | cuisine) → recipe:</strong> You create another &quot;selects&quot; edge between the ingredient or cuisine and the recipe (that is, “cuisine selects recipe”). In addition, store a &quot;count&quot; property on the edge and increment it each time the ingredient or cuisine selects the same recipe.</p><pre>{<br>  &quot;label&quot;: &quot;selects&quot;,<br>  &quot;type&quot;: &quot;edge&quot;,<br>  &quot;inV&quot;: 4320,<br>  &quot;outV&quot;: 4152,<br>  &quot;properties&quot;: {<br>    <strong>&quot;count&quot;: 22</strong><br>  }<br>}</pre><p><strong>6. Selects edge, person → recipe:</strong> You create yet another &quot;selects&quot; edge directly between the person and the recipe (that is, “person selects recipe”). Store a &quot;count&quot; property on the edge and increment it each time a person requests the same recipe.</p><pre>{<br>  &quot;label&quot;: &quot;selects&quot;,<br>  &quot;type&quot;: &quot;edge&quot;,<br>  &quot;inV&quot;: 4320,<br>  &quot;outV&quot;: 4224,<br>  &quot;properties&quot;: {<br>    <strong>&quot;count&quot;: 4</strong><br>  }<br>}</pre><p><strong>7. Has edge, recipe → (ingredient | cuisine):</strong> Finally, create an edge, labelled &quot;has&quot;, between the recipe and the ingredient or cuisine (that is, “recipe has cuisine”). This relationship allows you to find all the ingredients and cuisines that a recipe uses. There is no count field on this edge.</p><pre>{<br>  &quot;label&quot;: &quot;has&quot;,<br>  &quot;type&quot;: &quot;edge&quot;,<br>  &quot;inV&quot;: 4152,<br>  &quot;outV&quot;: 4320<br>}</pre><p>The graph for a single user looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*h6fvJxvuKB_6EVn-w1itTg.png" /><figcaption>This graph has it going on. It’s a <a href="https://en.wikipedia.org/wiki/Connectivity_%28graph_theory%29#Definitions_of_components.2C_cuts_and_connectivity">(weakly) connected graph</a>. There are all kinds of graphs.</figcaption></figure><p>So far, by using a graph database, you get the following benefits:</p><ol><li>Reduce third-party API calls by caching entities.</li><li>Provide a more personal experience for users by harnessing metadata on their interactions.</li></ol><p>A “more personal experience” for Recipe Bot means allowing users to request their favorite recipes. To find a user’s favorite recipes, you use the Gremlin <a href="http://tinkerpop.apache.org/docs/current/reference/#traversal">graph traversal</a> language. The following Gremlin query will give you a user’s top-five favorite recipes, sorted by count:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd2ab2e520daca0b22726341140052af/href">https://medium.com/media/bd2ab2e520daca0b22726341140052af/href</a></iframe><h3>Adding recommendations</h3><p>Since you track every user interaction with the bot as a graph, you can find popular ingredients, cuisines, or recipes requested by all users. You can use Gremlin to find popular recipes based on an ingredient or cuisine. Here’s how it works:</p><p>Let’s say, a user is looking for recipes that use onions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*Yf49Io2lbpZeHrsvfyMxLA.png" /></figure><p>You can find popular recipes that use onions by issuing the following query. (I’ll unpack it further below—don’t worry!):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f4ddd512952698aa5f80f84f4fead61/href">https://medium.com/media/2f4ddd512952698aa5f80f84f4fead61/href</a></iframe><p>This query says, “Give me anyone, excluding the calling user, who has requested recipes more than once that have onions.” It breaks down like so:</p><p>1. Start with &quot;onions&quot;:</p><pre>g.V().hasLabel(&quot;ingredient&quot;).has(&quot;name&quot;,&quot;onions&quot;)</pre><p>2. Get the recipes that have &quot;onions&quot;. This API call uses the &quot;has&quot; edge coming from the recipe vertex into the ingredient vertex. Using .in() skips the edge and only returns the recipe vertex. (You don’t need any properties from the edge object, so there’s no reason to return it here.)</p><pre>.in(&quot;has&quot;)</pre><p>3. Get the users that have requested these recipes more than once. This call uses the &quot;selects&quot; edge coming from the person to the recipe:</p><pre>.inE().has(&quot;count&quot;,gt(1)).order().by(&quot;count&quot;, decr)</pre><p>4. Get the users, excluding the current user:</p><pre>.outV().hasLabel(&quot;person&quot;).has(&quot;name&quot;,neq(&quot;CURRENT_USER&quot;))</pre><p>5. Get the full path:</p><pre>.path()</pre><p>This call returns an array of matching paths that looks like this:</p><blockquote>ingredient ← recipe ← edge ← person</blockquote><p>You can access these recommended recipes at index 1.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/822aa1016dd24f13f6b2854aa0690a93/href">https://medium.com/media/822aa1016dd24f13f6b2854aa0690a93/href</a></iframe><p>When you return this recipe list to the user, the app puts the recommended recipes at the top and highlights the number of users who have previously used which recipe:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HS0ft98ZPG4AezihLIHnFA.png" /></figure><h3>What’s Next?</h3><p>Try a deployment for yourself. The <a href="https://github.com/ibm-cds-labs/watson-recipe-bot-nodejs-graph#watson-recipe-bot--ibm-graph">project’s README</a> has step-by-step instructions for completing your first deployment on IBM Bluemix. There’s also a <a href="https://github.com/ibm-cds-labs/watson-recipe-bot-java-graph">Java port</a> of the example app.</p><p>If you’re already using a dialog tree in your applications and want to use a graph database to persist metadata on interactions, I hope the source code in the repo above gives you some ideas on delivering more personalized experiences to your users.</p><p>And if you’ve enjoyed this article, please hit the ol’ ♥ so other Medium users might find it and dig it too. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3aaf5c3ae52c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/free-code-camp/have-you-had-the-talk-with-your-chatbot-about-graph-data-structures-3aaf5c3ae52c">Have you had “The Talk” with your chatbot about graph data structures?</a> was originally published in <a href="https://medium.com/free-code-camp">We’ve moved to freeCodeCamp.org/news</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Persisting Data for a Smarter Chatbot]]></title>
            <link>https://medium.com/codait/persisting-data-for-a-smarter-chatbot-be599480f7b2?source=rss-f1b6565b0972------2</link>
            <guid isPermaLink="false">https://medium.com/p/be599480f7b2</guid>
            <category><![CDATA[cognitive-computing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Mark Watson]]></dc:creator>
            <pubDate>Mon, 12 Dec 2016 20:20:14 GMT</pubDate>
            <atom:updated>2017-02-09T17:06:07.587Z</atom:updated>
            <content:encoded><![CDATA[<h4>With Watson Conversation Service and Cloudant</h4><p>In <a href="https://medium.com/ibm-watson-developer-cloud/how-to-build-a-recipe-slack-bot-using-watson-conversation-and-spoonacular-api-487eacaf01d4#.7mbmd1nmc">a previous blog post</a> Josh Zheng showed us how to build a Slack bot that uses Watson Conversation and a 3rd party API called <a href="https://spoonacular.com/food-api">Spoonacular</a> to provide recipe suggestions and instructions based on ingredients and cuisines.</p><p>In this blog post we’ll show you how we improved the Watson Recipe Chatbot by integrating Cloudant to cache 3rd party API calls, provide a more personal experience to the user, and perform analysis on the conversations users have with the bot.</p><p>The source code and setup instructions for the new version of the Watson Recipe Bot can be found at <a href="https://github.com/ibm-cds-labs/watson-recipe-bot-python-cloudant">https://github.com/ibm-cds-labs/watson-recipe-bot-python-cloudant</a>.</p><h3>In Review: Watson Recipe Bot</h3><p>Before we start, let’s review how the Watson Recipe Bot works.</p><ul><li>First, you start a conversation with the sous-chef bot by sending it a direct message:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*oEldKxh6pJSLpIjW-ih0JQ.png" /></figure><ul><li>Next, you tell the sous-chef what you would like to do. You can say you want to cook something, you want to eat something, etc:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*Y4OgNChtyclvvJ73-qZbJw.png" /></figure><ul><li>Next, you reply to the sous-chef whether or not you would like to use specific ingredients (if you say no sous-chef will ask you to specify a cuisine):</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*C6c-Y9brEkHB6RA0axFNEA.png" /></figure><ul><li>Next, you specify the ingredients or cuisine you would like to cook with, and sous-chef will return a list of recipes:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*dyZj9810BkiNarv7MF92pA.png" /></figure><ul><li>Finally, you select the recipe you’d like to cook, and sous-chef will tell you how:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*d3_d5pxYAS9N5ipl5yN5mw.png" /></figure><p>So, what’s actually going on here?</p><h3>How It Works</h3><p>Here is a high-level architecture diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*lPrSJJV7-zMd6Tb6rVhVbg.png" /></figure><p>Here’s what’s happening behind the scenes:</p><ol><li>sous-chef is a Slack bot that has been registered with a Slack team. Slack users send direct messages to the sous-chef bot.</li><li>Messages are immediately sent from Slack to the application over a WebSocket connection.</li><li>The application forwards the message along with the current state of the conversation for the active user to Watson Conversation.<br> Watson Conversation uses natural language processing to determine the intent of the message. For example, is the user asking to cook something? Did the user specify a cuisine?</li><li>Watson Conversation returns a response to the application which includes the identified intent and any other information related to the current step in the conversation.</li><li>The application uses the response to determine it’s next step. For example, if the user specifies a cuisine the application will query <a href="https://spoonacular.com/food-api">Spoonacular</a> (a 3rd party API) for a list of recipes.</li><li>Spoonacular returns a response to the application.</li><li>The application formulates a message and sends it to Slack.</li><li>Slack delivers the message to the user.</li></ol><h3>Adding Persistence with Cloudant</h3><p>The new version of the application uses Cloudant to store users, ingredients, cuisines, and recipes.</p><p>Here’s how it’s different:</p><ol><li>A document for each user that interacts with the bot is stored in Cloudant.</li><li>A document for each ingredient, cuisine, or recipe is also stored in Cloudant.</li><li>The application looks for ingredients, cuisines, and recipes in Cloudant before querying Spoonacular.<br> If one of these already exists in Cloudant, then it is returned immediately and there is no need for the application to query Spoonacular.</li><li>Every request for an ingredient, cuisine, or recipe made by a user is stored in Cloudant.</li><li>In addition, the number of times the user has requested a particular ingredient, cuisine, or recipe is stored in the user document.</li><li>A new Watson Conversation intent has been created that allows a user to request their favorite recipes.<br> The application queries Cloudant to find the recipes requested most by the user.</li></ol><h3>How It Works Now</h3><p>Here is what the new architecture looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*vaemAjUGjz87LO0bfUMoCw.jpeg" /></figure><p>Most of the application logic has remained the same. We have simply added a Cloudant instance and a little bit of code.</p><p>Let’s rehash some of the benefits of the new implementation:</p><ol><li>Reduce 3rd party API calls by caching entities in Cloudant.</li><li>Provide a more personal experience for users (favorite recipes) using the data in Cloudant.</li><li>Perform analysis on ingredients, cuisines, and recipes.</li></ol><p>Let’s dive into these points in a little more detail…</p><h3>Reduce 3rd Party API Calls</h3><p>Many 3rd Party APIs limit the number of times you can call them, or charge you based on the number of API calls you make. In our example<br> we make the following calls to Spoonacular:</p><ol><li>Request a list of recipes based on an ingredient or comma-separated list of ingredients.</li><li>Request a list of recipes based on a cuisine.</li><li>Request a specific recipe and the steps required to cook it.</li></ol><p>Multiple users may request the same recipe, cuisine, or ingredient from Spoonacular, and if the data we receive from Spoonacular never changes, or changes infrequently, we would be making the same API calls over and over again. By querying Cloudant first, and storing the results of the API calls in Cloudant, we can reduce the number of API calls made to Spoonacular. This could result in cost savings and help us avoid rate limits.</p><p>We could also see performance improvements. Your application performance may suffer if the 3rd party API experiences high load from other users. In many cases you will experience lower latency querying your Cloudant instance vs. a 3rd party API.</p><blockquote><em>Note: If you are using a 3rd party API consult their Terms of Service to make sure you are allowed to cache results. Spoonacular does allow caching as part of their </em><a href="https://market.mashape.com/spoonacular/recipe-food-nutrition"><em>Terms of Service</em></a><em>.</em></blockquote><h3>Provide a More Personal Experience</h3><p>As users request ingredients, cuisines, and recipes we can learn about each user’s habits. Do they cook Chinese food more than Italian food? Do they cook with chicken more than beef? What are their favorite recipes?</p><p>With just a few lines of code we can track every ingredient, cuisine, and recipe requested by a user. We can then use that information to personalize a user’s experience. The application currently stores each ingredient, cuisine, and recipe selected by a user in a &quot;type&quot;: &quot;user&quot; document in Cloudant. Here’s an example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c176ed4e452e25749f753095c43a3b2/href">https://medium.com/media/4c176ed4e452e25749f753095c43a3b2/href</a></iframe><p>Using this information we can give a user their favorite recipes sorted by count. We created a new intent and dialog path in the Watson Conversation workspace to support favorites:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*QczWDIzLG_Gh-lEkt7cn0A.png" /></figure><p>When the application receives the favorite_recipes intent from Watson Conversation, it immediately looks up and returns the recipes stored in Cloudant:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*SjILWmRnPN3sSdr133pHRg.png" /></figure><p>This is a very simple example, but with more data we could make the bot even better. You can see in the user document that this user has requested Chinese food more often than Italian. We also store every request for a cuisine, including the date of the request. It looks something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34be41b4c4cc5dcd0ab2f141d197c7ce/href">https://medium.com/media/34be41b4c4cc5dcd0ab2f141d197c7ce/href</a></iframe><p>We could look for patterns in this data. Maybe the user makes a request for Chinese food every Friday. If the user messages the bot on a Friday it could immediately ask if they want Chinese food.</p><p>By persisting a user’s behavior we can make simple improvements to our application to provide a better, more personal experience for our users.</p><h3>Analyzing the Data</h3><p>Now that we are storing every user request for an ingredient, cuisine, and recipe, we can do some simple analytics on our data. The new application automatically creates views in Cloudant that allow you to find the most popular ingredients, cuisines, and recipes requested by all users. The views are a part of the by_popularity design doc and are named ingredients, cuisines, and recipes.</p><p>For example, you can easily find the most popular cuisines by requesting the cuisines view in the by_popularity design doc. The view looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36bb819165bbc1a680b322e46b351be6/href">https://medium.com/media/36bb819165bbc1a680b322e46b351be6/href</a></iframe><p>You can query the view in your browser at the following URL:</p><pre>https://CLOUDANT-USERNAME.cloudant.com/watson_recipe_bot/_design/by_popularity/_view/cuisines?group=true</pre><p>Here’s an example of the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*F536K7mFgegI-3p2NlVTqQ.png" /></figure><p>Here you can see that your users as a whole are requesting Italian recipes more often than Chinese, unlike the user we looked at earlier.</p><p>How about finding out what days people tend to use the bot more? The application includes a design doc to do that as well. The following recipes view of the by_day_of_week design doc shows you the total number of recipes requested by day:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53775a937a040eeb5d18ec4efe2e2d97/href">https://medium.com/media/53775a937a040eeb5d18ec4efe2e2d97/href</a></iframe><p>Here’s an example of the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*BbI4YzGyoHEV9vipiurKtg.png" /></figure><p>These are fairly simple examples of the analysis you can perform on this type data, but once you start storing the data the sky is the limit. You could correlate weather information with ingredient or cuisines requests. You could determine which cuisines or ingredients result in the most drop off and, hence, the least desirable recipes. You can do this analysis in Cloudant or import your data into other tools like Apache Spark or dashDB for further analysis.</p><p>Performing analysis on your data can help you improve your product, which can lead to better experiences for your users and reduce your user drop off rate.</p><h3>What’s Next?</h3><p>In this blog we showed you how we could make simple, but valuable improvements to the Watson Recipe Bot by adding persistence with Cloudant and a little bit of code. We showed you how we could cut down on 3rd party API calls, personalize the user’s experience, and perform some basic analysis on the data.</p><p>Try a deployment for yourself. The README at <a href="https://github.com/ibm-cds-labs/watson-recipe-bot-python-cloudant">https://github.com/ibm-cds-labs/watson-recipe-bot-python-cloudant</a> has step-by-step instructions for completing your first deployment. Alternatively, if you’re already using Watson Conversation in your applications and want to use Cloudant to persist this data, we hope the source code in the repo above gives you some ideas to improve your apps. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be599480f7b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codait/persisting-data-for-a-smarter-chatbot-be599480f7b2">Persisting Data for a Smarter Chatbot</a> was originally published in <a href="https://medium.com/codait">Center for Open Source Data and AI Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>