<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeff Manville on Medium]]></title>
        <description><![CDATA[Stories by Jeff Manville on Medium]]></description>
        <link>https://medium.com/@jeffmanville?source=rss-1f8656ddca62------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YHGkNXhT0Ad-9FkX-4SWag.jpeg</url>
            <title>Stories by Jeff Manville on Medium</title>
            <link>https://medium.com/@jeffmanville?source=rss-1f8656ddca62------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:21:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jeffmanville/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Setting up a Data Science Environment for ServiceNow With JupyterLab & Docker]]></title>
            <link>https://jeffmanville.medium.com/setting-up-a-data-science-environment-for-servicenow-with-jupyterlab-docker-ebad55b694b8?source=rss-1f8656ddca62------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebad55b694b8</guid>
            <category><![CDATA[servicenow]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Jeff Manville]]></dc:creator>
            <pubDate>Fri, 12 May 2023 11:41:52 GMT</pubDate>
            <atom:updated>2023-05-20T11:45:17.875Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up a Data Science Environment for ServiceNow with JupyterLab &amp; Docker</h3><figure><img alt="A man standing in front of a chart" src="https://cdn-images-1.medium.com/max/926/1*FQtqMlOiONge2-L4IykExw.png" /></figure><p>ServiceNow is the platform for work in Information Technology (IT). For many organizations, it contains a wealth of information about what is going on, from major incidents, changes, customer requests, etc.</p><p>Yet, many executives, directors, and managers are left disappointed or confused by the reporting coming out of their platform and its lack of actionable information.</p><p>With every IT organization evolving at a faster and faster pace to support customer demand, the systems and processes required are becoming more and more difficult measure and understand</p><p>In the wake of this complexity, we need more powerful tools to extract understanding from noise and direction from chaos.</p><h3>What we will be doing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IwI303kZb-HZiQmV_Z4TEA.png" /></figure><p>In this tutorial, we will be using Docker &amp; JupyterLab to create a reliable development environment for data science and combine it with our ServiceNow data. By combining these technologies, we will be able to bring powerful analytical techniques to bear on our data.</p><h3>Requirements:</h3><ul><li>Admin Username &amp; Password for access into a ServiceNow Instance. <a href="http://developer.service-now.com/">Preferably a Personal Developer Instance for this tutorial</a></li><li>A GitHub account</li><li>A Windows(10/11) laptop with the a 64-bit processor and 4GB system RAM with <a href="https://cli.github.com/">GitHub CLI installed &amp; configured</a></li><li>An Internet Connection</li></ul><h3>What is JupyterLab?</h3><blockquote>“JupyterLab is the latest web-based interactive development environment for notebooks, code, and data. Its flexible interface allows users to configure and arrange workflows in data science, scientific computing, computational journalism, and machine learning. A modular design invites extensions to expand and enrich functionality.” — <a href="https://jupyter.org/">Jupyter</a></blockquote><h3>What is ServiceNow?</h3><blockquote>ServiceNow is “a cloud computing platform to help companies manage digital workflows for enterprise operations.” — <a href="https://en.wikipedia.org/wiki/ServiceNow">Wikipedia</a></blockquote><h3>What is a Docker Container?</h3><blockquote>“A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.” — <a href="https://www.docker.com/resources/what-container/">Docker</a></blockquote><h3>Jupyter+Docker vs ServiceNow</h3><p>Out of the box, ServiceNow does have reporting capabilities. Pie charts, timelines, bar charts, heatmaps, etc. However, Almost all of ServiceNow’s reporting capabilities are a type of Descriptive Analytics.</p><p>Descriptive Analytics describe the composition of our data. However, this type of analysis is never actionable. It will not tell you when something has gone wrong (or right), what is driving changes, how it will be in the future, and whats the best course of action. We require more complex techniques to answer those questions.</p><p>Jupyter allows us to leverage advance techniques and tools from the python data science community to transform our ServiceNow data into actionable analyses. Docker will allow us and anyone else to reliably reproduce and iterate on these analyses against their ServiceNow Instances.</p><figure><img alt="Four types of analytics, from least valuable &amp; complex to most: Descriptive, Diagnostic, Predictive, Prescriptive" src="https://cdn-images-1.medium.com/max/1024/1*dw-pYV1gaeSPdMz3sJTCLg.png" /><figcaption>Four types of analytics, from least valuable &amp; complex to most: Descriptive, Diagnostic, Predictive, Prescriptive</figcaption></figure><blockquote><strong>Descriptive </strong>— “What is it?” — pie charts, bar charts, heatmaps, percentages, counts, averages, medians, etc.</blockquote><blockquote><strong>Diagnostic </strong>— “Why is it?” — Root cause analysis, correlation analysis, drilldown, statistical process control, hypothesis testing, etc.</blockquote><blockquote><strong>Predictive </strong>— “What will it be?” — Support Vector Machines, Random forest, k-means, logistic regression, etc.</blockquote><blockquote><strong>Prescriptive </strong>— “What should we do ?” — Simulation, Machine learning, Integer Optimization, recommender systems, etc.</blockquote><h3>Step 1: Install Docker Desktop on your laptop</h3><p><a href="https://docs.docker.com/desktop/install/windows-install/">Docker Documentation for installing on Windows</a></p><figure><img alt="Docker Desktop Download Screenshot" src="https://cdn-images-1.medium.com/max/700/1*utauspvQmdxIqUocqyAmaQ.jpeg" /></figure><p>First, we will want to install Docker Desktop. This will allow us to run any containerized application on our laptop. This will also allow anyone we share our containerized applications with to run them.</p><p><a href="https://www.docker.com/products/docker-desktop/">Go to the docker desktop page and download the appropriate version for your laptop</a></p><p>Run the Docker Desktop installer &amp; make sure the “use WSL2” is selected</p><h3>Step 2: Install Windows Subsytem for Linux</h3><p><a href="https://learn.microsoft.com/en-us/windows/wsl/">Windows Subsystem for Linux Documentation</a></p><p>Windows cannot run a Linux based image by default, so we will need to install the Windows Subsystem for Linux. Fortunately, it is super easy to install.</p><figure><img alt="Searching our laptop for Windows Powershell and Clicking “Run As Administrator”" src="https://cdn-images-1.medium.com/max/686/1*gJ1-DgV-qPQedixAlgHh4Q.png" /></figure><p>Run the following PowerShell command. Make sure it is run as Administrator</p><pre>wsl --install</pre><h3>Step 3: Launch Docker Desktop</h3><p>Go ahead and restart your computer, launch Docker Desktop, and accept the terms of service. You should see something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7sfPL5026lN4XV6SYJwLQ.png" /></figure><h3>Step 4: Clone ServiceNowJupyterDocker</h3><pre>gh repo clone manvillej/ServiceNowJupyterDocker</pre><p>This will copy my GitHub project. Now, lets set it up to access our ServiceNow Instance. Create a .env file in the newly created directory</p><pre># /ServiceNowJupyterDocker/.env<br>SERVICENOW_INSTANCE=yourinstancename<br>SERVICENOW_USERNAME=yourusername<br>SERVICENOW_PASSWORD=yourpassword<br>JUPYTER_TOKEN=yourjupytertoken</pre><p>Replace the values for the first three lines with your ServiceNow instance name, username, and password. These values will be set as environment variables in your docker container. Our python code will then use that to access the data in the instance through ServiceNow’s REST APIs.</p><p>While using an Admin account is the simplest way to avoid any access issues, you could use an account with less permissions depending on the data you are planning on using.</p><h3>Step 5: Running our Docker image</h3><p>With our environment file created, we can now start our JupyterLab container. It may take some time the first time you try, but it will start very quickly anytime after that. Navigate into the newly created directory in your command line and run the following:</p><pre>    docker-compose up -d --build</pre><h3>Step 6: Open JupyterLab</h3><p>Assuming you haven’t changed the value of the JUPYTER_TOKEN, open your browser and go to <a href="http://localhost:8888/?token=jupytertoken">http://localhost:8888/?token=jupytertoken</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_sjWUbiNmcEiSvA2RMk6og.jpeg" /><figcaption>JupyterLab Launch Screen</figcaption></figure><h3>Step 7: A little Data Visualization</h3><p>Now, there isn’t much to see yet, but we can dip our toes into some data visualization. Open up the BaseExample.ipynb in the Jupyter client. Then, click “Run” in the top Menu &amp; “Run All Cells”.</p><figure><img alt="On the Top menu, of the JupyterLab client, Click “Run” then “Run All Cells”" src="https://cdn-images-1.medium.com/max/558/1*sCsdLf08ezn9qIGW_rY7Qw.png" /><figcaption>If someone knows how to create a short cut for Run All Cells on JupyterLab, let me know</figcaption></figure><p>You should now see some logs of your API calls, a little table containing a sample of the data retrieved, a small summary of your data by Priority, and a pie chart of records by Priority.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PN4CG6bNdxN7px1bnE3Ugg.png" /></figure><p>If you see an error, you may have incorrect credentials or you ServiceNow Instance may be asleep if you are using a Personal Developer Instance.</p><h3>Step 8: Tearing down our Docker Container</h3><p>When you are done with running your docker container, use your command line to run the following:</p><pre>docker-compose down -v</pre><p>This will tear down the docker container instance. Your code will still be preserved, but your data will no longer be in the notebook. Your data will still be in your ServiceNow Instance.</p><h3>Conclusion:</h3><p>JupyterLab is a fantastic tool for data analysis &amp; visualization. By combining Jupyter with ServiceNow’s Rest APIs, we can leverage the massive open source Data Science ecosystem against our IT service management data.</p><p>And some good news! I am currently at JupyterCon 2023 &amp; the Github keynote showed us how to leverage <a href="https://github.com/features/codespaces">Github CodeSpaces</a> to run JupyterLab! This would allow anyone with a Github account to run, develop, and share these notebooks without needing to install it on a machine that supports Docker.</p><h3>Next Up:</h3><p>My current plan is to create a Jupyter notebook for an in depth analysis of Major Incident data in ServiceNow.</p><p>I am going to try to set it up for Github CodeSpaces; I think that will make it more accessible for people to run the notebook for themselves. If it takes me too long to figure out, I may split that out int another article.</p><p>Thank you for reading!</p><p>Manville</p><p><a href="https://github.com/manvillej/ServiceNowJupyterDocker">Github Repository</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebad55b694b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[JupyterLab & ServiceNow]]></title>
            <link>https://jeffmanville.medium.com/jupyterlab-servicenow-8549e12bd61b?source=rss-1f8656ddca62------2</link>
            <guid isPermaLink="false">https://medium.com/p/8549e12bd61b</guid>
            <category><![CDATA[servicenow]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <dc:creator><![CDATA[Jeff Manville]]></dc:creator>
            <pubDate>Wed, 11 Jan 2023 22:10:20 GMT</pubDate>
            <atom:updated>2023-01-11T22:10:20.441Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="Jupyter &amp; ServiceNow Logos" src="https://cdn-images-1.medium.com/max/1024/1*_igoJbv4429yM5EqiaFToA.jpeg" /><figcaption>Jupyter &amp; ServiceNow Logos</figcaption></figure><p>More than ever, organizations need to make smart data driven decisions to keep up with the pace of industry and a month old analysis will not cut it. To address this need, organizations have leverage Data Science. Data science is a discipline combining powerful mathematic techniques with computer science to rapidly generate robust analyses.</p><p>There are a wealth of untapped opportunities for the application of data science inside of enterprise platforms like ServiceNow. By combining the rich structured datasets of enterprise platforms with modern data science tools, we can gain greater insights and understandings through advanced statistical analyses, interactive data visualizations, and transforming &amp; merging multiple data sets.</p><h3>What is JupyterLab?</h3><p>JupyterLab is the latest web-based interactive development environment for notebooks, code, and data. Its flexible interface allows users to configure and arrange workflows in data science, scientific computing, computational journalism, and machine learning. A modular design invites extensions to expand and enrich functionality. — <a href="https://jupyter.org/">Jupyter</a></p><h3>What is ServiceNow?</h3><p>Servicenow is “a cloud computing platform to help companies manage digital workflows for enterprise operations.” — <a href="https://en.wikipedia.org/wiki/ServiceNow">Wikipedia</a></p><h3>Before you get started:</h3><p>You will need:</p><ul><li>A Windows laptop with <a href="https://www.anaconda.com/">Anaconda installed for managing our environment</a></li><li>A ServiceNow instance</li></ul><p>You can get a personal developer instance (PDI) at the <a href="http://developer.service-now.com/">ServiceNow Developer site</a>. Take note of your Instance’s URL, your admin username &amp; password. If you are using a PDI, make sure to wake it up before starting the tutorial</p><ul><li>Some familiarity with Python</li></ul><h3>The Setup</h3><p>First, launch the anaconda command prompt and a directory for our project.</p><pre>$ mkdir jupyter_servicenow<br>$ cd jupyter_servicenow</pre><p>If you are using git for version control, you can initialize it here. This is very useful for sharing your notebooks with other data scientists.</p><pre>$ git init</pre><p>Now, we create a Conda environment to manage the python packages we will be using. This will keep our dependencies separate from other projects’ dependencies</p><pre>$ conda create --name jupyterlab python=3<br>$ activate jupyterlab</pre><p>With our environment active, we can start installing some packages. Lets install JupyterLab</p><pre>$ conda install -c conda-forge jupyterlab</pre><p>We will also need the ability to retrieve and analyze our data. For that, we will use httpx for retrieval and Pandas for data analysis</p><pre>$ conda install -c conda-forge httpx <br>$ conda install panda</pre><h3>Picking the right Data Visualization Packages</h3><p>Picking the right package for visualizing data is a tricky subject for the python language. There are many packages for this: matplotlib, seaborn, plotly, bokeh, altair, folium, etc. Some of these packages are equivalent to each other; some have specific purposes. For example, Folium is great for GPS data. We will be diving into Folium in a future tutorial.</p><p>For this project, we will use Bokeh.</p><pre>$ conda install bokeh</pre><h3>Handling Secrets: Environment Variables</h3><p>To access ServiceNow’s data, we will use its Table REST API to retrieve data. However to do so, we will need three pieces of information: Our Instance name, Username, and password.</p><p>In software engineering, sensitive information like this is called a Secret. We REALLY don’t want to accidentally share Secrets.</p><p>We can leverage Environment variables to separate our secrets from our code. We could set environment variables directly, but its better to use a .env file and the python-dotenv package. This will allow us to easily manage our secrets and version control tools like Git will ignore this file when committing code.</p><pre>$ conda install -c conda-forge python-dotenv</pre><p>Next, create a file named <strong><em>.env </em></strong>in your <strong><em>~/jupyter_servicenow/ </em></strong>directory. Make sure to replace the values with your Instance URL, Username, and Password</p><pre>INSTANCE_URL=example.service-now.com<br>INSTANCE_USERNAME=some_username<br>INSTANCE_PASSWORD=some_password</pre><p>We will still need to load this file into our environment, but we will set that up in JupyterLab</p><p>Lets launch JupyterLab. This will open a new window in your browser</p><pre>$ jupyter-lab</pre><figure><img alt="JupyterLab interface" src="https://cdn-images-1.medium.com/max/1024/1*BJUjZTYKN_Ww4PRWJxsuGw.png" /></figure><p>Before we create a notebook, create a python file in the Other section. I like to separate implementation details from my analysis. When sharing a notebook, my audience won’t be distracted by unnecessary details.</p><pre># settings.py <br>import os <br>from os.path import join, dirname <br>from dotenv import load_dotenv <br>def load_env(): <br>    dotenv_path = join(dirname(__file__), &#39;.env&#39;) <br>    load_dotenv(dotenv_path) <br>    root_url = os.environ.get(&quot;INSTANCE_URL&quot;) <br>    username = os.environ.get(&quot;INSTANCE_USERNAME&quot;) <br>    password = os.environ.get(&quot;INSTANCE_PASSWORD&quot;) <br>     <br>    return [root_url, username, password]</pre><p>Now, we are ready to create out first Notebook.</p><h3>Retrieving Data</h3><p>Create a notebook by clicking the grey “+” and then clicking Python 3 (ipykernel) under Notebook. Right click the tab header of your notebook and rename it to “myFirstNotebook”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*0fdqHKu6bR12C5LXQOamHA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*omXkVdP03JyEhVYU7zo_Rg.png" /></figure><p>Next, past the following code inside your first cell. Then, press the play button to run that cell.</p><pre>from settings import load_env <br>import httpx <br>import pandas as pd <br>[root_url, username, password] = load_env() <br>with httpx.Client(base_url=f&#39;https://{root_url}&#39;, auth=(username, password)) as client: <br>    # gets the most recently created 100 incident records <br>    fields_i_want = [&#39;parent&#39;, &#39;sys_updated_on&#39;, &#39;number&#39;, &#39;resolved_by&#39;, &#39;sys_updated_by&#39;, &#39;opened_by&#39;, &#39;sys_created_on&#39;, &#39;state&#39;, &#39;sys_created_by&#39;, &#39;closed_at&#39;, &#39;cmdb_ci&#39;, &#39;impact&#39;, &#39;active&#39;, &#39;priority&#39;, &#39;opened_at&#39;, &#39;business_duration&#39;, &#39;caller_id&#39;, &#39;resolved_at&#39;, &#39;subcategory&#39;, &#39;short_description&#39;, &#39;close_code&#39;, &#39;assignment_group&#39;, &#39;cause&#39;, &#39;description&#39;, &#39;close_notes&#39;, &#39;service_offering&#39;, &#39;closed_by&#39;, &#39;sys_id&#39;, &#39;contact_type&#39;, &#39;urgency&#39;, &#39;assigned_to&#39;, &#39;location&#39;, &#39;category&#39;] <br>    response = client.get(f&#39;/api/now/table/incident?sysparm_query=ORDERBYDESCopened_at&amp;sysparm_display_value=true&amp;sysparm_exclude_reference_link=true&amp;sysparm_fields={&quot;,&quot;.join(fields_i_want)}&#39;) <br>    body = response.json() <br>    incidents = pd.DataFrame.from_dict(body[&#39;result&#39;])</pre><p>This cell is doing a few things:</p><ul><li>Loading our environment variables</li><li>Importing our dependencies</li><li>Requesting the most recent 100 incidents, but only specific list of fields (a brand new PDI will only have 67 incidents)</li><li>Loading that data into pandas dataframe</li></ul><p>In a new cell, add the following code and run it.</p><pre>incident.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AxeG7OqqxA_eYagMnEkfyQ.png" /></figure><h3>Some Basic Visualizations</h3><p>Now, lets get into some visualization. First, We need to add some imports for bokeh to our first cell. Add this code to your import &amp; make sure to rerun the cell. You may get an error about Jquery, you can ignore it.</p><pre>from bokeh.plotting import figure, show <br>from bokeh.io import output_notebook <br>from bokeh.resources import INLINE <br>from bokeh.models import ColumnDataSource <br>from bokeh.palettes import Spectral <br>from bokeh.models import Legend <br>from bokeh.models.tools import HoverTool<br><br># this is import to display your charts in the jupyter notebook instead of opening a new tab<br>output_notebook(resources=INLINE) </pre><h3>Just A Bit Of Calculation</h3><p>Next, lets add a third cell and do a bit of calculation before we visualize it. Make sure to run the cell</p><pre># we&#39;re labeling any empty values with &quot;None&quot;, this way the groupby won&#39;t ignore them.<br>incidents[[&#39;category&#39;]] = incidents[[&#39;category&#39;]].fillna(&#39;None&#39;) <br>category_count = incidents.groupby([&#39;category&#39;])[&#39;category&#39;].aggregate(&#39;count&#39;).sort_values(ascending=False) <br>category_percent = category_count.cumsum()/category_count.sum()*100</pre><p>This cell does a few things. First, we are filling empty category cells with the value “None”. Without this, our empty cells would not be shown when grouping.</p><p>Next, we are getting grouping by the category field, getting a count for each, and then sorting from highest to lowest.</p><p>Finally, we are using category count to calculate the percent share of each category.</p><h3>Finally, Visualizing Our Data</h3><p>Create another new cell at the bottom and add the following</p><pre>#  combining our category data for easy use in our charts<br>source = ColumnDataSource(data=dict( <br>    category=category_count.index.values,  <br>    count=category_count.values,  <br>    percent=category_percent.values, <br>    color=Spectral[category_count.shape[0]]))<br><br><br># Generating the base figure without any data plotted on it<br>p = figure( <br>        x_range=category_count.index.values,  <br>        y_range=(0,category_count.max() + 1), # set the range to be a minimum of 0 and a max of the max value <br>        title=&quot;Incident Category Counts&quot;, <br>        toolbar_location=&quot;below&quot;, <br>        tools=&quot;pan, reset, save&quot;) # the tools I want in the toolbar<br><br># adding a vertical bar chart to our figure<br>p.vbar(x=&#39;category&#39;, top=&#39;count&#39;, width=0.5, color=&#39;color&#39;, legend_field=&quot;category&quot;, source=source)<br><br><br># Just some styling<br>p.xgrid.grid_line_color = None<br>p.title.align = &quot;center&quot; <br>p.title.text_color = &quot;black&quot; <br>p.title.text_font_size = &quot;20px&quot; <br>p.axis.axis_label_text_font_style=&#39;bold&#39; <br>p.yaxis.axis_label_text_font_size = &quot;12pt&quot; <br>p.yaxis.axis_label = &#39;Count&#39; <br>p.add_layout(Legend(), &#39;right&#39;) <br>p.legend.title = &quot;Category&quot;<br><br><br># add a hover tool <br>p.add_tools(HoverTool(tooltips=[(&quot;Category&quot;, &quot;@category&quot;), (&quot;Count&quot;, &quot;@count&quot;),(&quot;Cumulative Percent&quot;,&quot;@percent&quot;)]))<br><br># displaying the chart<br>show(p)</pre><p>If you run your cell you should get this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PISzOiXCtN6Leqrvb0BSww.png" /></figure><h3>Making a Pareto Chart</h3><p>Now, this is nice, but take it an extra step and make it into a pareto chart. Add the following in a new cell and run it.</p><pre>p.extra_y_ranges = {&#39;pct&#39; : Range1d(start=0, end=100)} <br>p.add_layout(LinearAxis(y_range_name=&#39;pct&#39;, axis_label=&#39;Percent&#39;), &#39;left&#39;) <br>p.line(x=&#39;category&#39;, y=&#39;percent&#39;, source=source, y_range_name=&#39;pct&#39;)<br><br>show(p)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*ZrYCDLKSA8dzMLhhhXJvdw.png" /></figure><h3>Using Functions for Charts</h3><p>If you’ve been running and rerunning these cells without restarting your notebook, you may have noticed some oddities with your chart like multiple “percent” Y axes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*_WPOUtobos6jVL7y1hHLTQ.png" /></figure><p>This might be because we’ve been manipulating the same figure object over and over. We can fix this and make our charts much more reusable by using functions to generate our charts. Create a new cell and add the following:</p><pre>def createParetoChart(source, x, top, x_range, y_range, title=&quot;&quot;, legendTitle=&quot;&quot;, axisLabel=&quot;&quot;, hoverTools=False): <br>    plot = figure( <br>            x_range=x_range, # category_count.index.values  <br>            y_range=y_range, # (0, category_count.max() + 1) <br>            title=title, # &quot;Incident Category Counts&quot; <br>            toolbar_location=&quot;below&quot;, <br>            tools=&quot;pan, reset, save&quot;) # the tools I want in the toolbar<br><br>    plot.vbar(x=x, top=top, width=0.5, color=&#39;color&#39;, legend_field=x, source=source)<br><br>    # just some styling<br>    plot.xgrid.grid_line_color = None <br>    plot.title.align = &quot;center&quot; <br>    plot.title.text_color = &quot;black&quot; <br>    plot.title.text_font_size = &quot;20px&quot; <br>    plot.axis.axis_label_text_font_style=&#39;bold&#39; <br>    plot.yaxis.axis_label_text_font_size = &quot;12pt&quot; <br>    plot.add_layout(Legend(), &#39;right&#39;) <br>     <br>    if(hoverTools): <br>        plot.add_tools(hoverTools)<br><br>    plot.legend.title = legendTitle <br>    plot.yaxis.axis_label = axisLabel<br><br>    plot.extra_y_ranges = {&#39;pct&#39; : Range1d(start=0, end=100)} <br>    plot.add_layout(LinearAxis(y_range_name=&#39;pct&#39;, axis_label=&#39;Percent&#39;), &#39;left&#39;) <br>    plot.line(x=x, y=&#39;percent&#39;, source=source, y_range_name=&#39;pct&#39;) <br>    return plot<br><br><br>show(createParetoChart(source, &#39;category&#39;, &#39;count&#39;, category_count.index.values, (0, category_count.max() + 1), &quot;Incident Category Count&quot;, &quot;Category&quot;, &quot;Count&quot;, hoverTools=HoverTool(tooltips=[(&quot;Category&quot;, &quot;@category&quot;), (&quot;Count&quot;, &quot;@count&quot;),(&quot;Percent&quot;,&quot;@percent&quot;)])))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*Hb2I28IKxAyoTI1h2ryHqQ.png" /></figure><p>You can still manipulate the plot after you generate it. You could even break this apart further to have a bar chart function that is called by the pareto chart function</p><p>By setting this our chart up as a function, we can quickly generate a pareto chart for other categories</p><pre>incidents[[&#39;assignment_group&#39;]] = incidents[[&#39;assignment_group&#39;]].fillna(&#39;None&#39;) # we&#39;re labeling any empty values with &quot;None&quot;, this way the group by won&#39;t ignore them. <br>group_count = incidents.groupby([&#39;assignment_group&#39;])[&#39;assignment_group&#39;].aggregate(&#39;count&#39;).sort_values(ascending=False) <br>group_percent = group_count.cumsum()/category_count.sum()*100<br><br>groupSource = ColumnDataSource(data=dict( <br>    group=group_count.index.values,  <br>    count=group_count.values,  <br>    percent=group_percent.values, <br>     color=Spectral[group_count.shape[0]]))<br><br>show(generateParetoChart(groupSource, &#39;group&#39;, &#39;count&#39;, group_count.index.values, (0, group_count.max() + 1), &quot;Incident Group Count&quot;, &quot;Group&quot;, &quot;Count&quot;, hoverTools=HoverTool(tooltips=[(&quot;Group&quot;, &quot;@group&quot;), (&quot;Count&quot;, &quot;@count&quot;),(&quot;Cumulative Percent&quot;,&quot;@percent&quot;)])))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*5Y2nvJvwG7zuL9D5jnAIuA.png" /></figure><h3>Conclusion:</h3><p>While ServiceNow does have a set of tools for analyzing your data, there are a wealth of technologies in the data science space for extracting, manipulating, and displaying your data. By applying these to ServiceNow, we can analyze and visualize data</p><p>In the next two articles, we’ll go beyond basic data visualization and apply complex analyses to our ServiceNow data to derive actionable insights.</p><h3>Things I am working on Next:</h3><ul><li>Jupyter &amp; ServiceNow: Docker for Sharing &amp; Reproducing Jupyter notebooks</li><li>Jupyter &amp; ServiceNow: Catalog Items, Combining Data Sets, and Gannt charts</li><li>Jupyter &amp; ServiceNow: Major Incidents &amp; Quality Control Charts</li></ul><p><a href="https://robertosalazarr.medium.com/">Another cool Python Industrial Engineer</a></p><p><a href="https://developer.servicenow.com/">ServiceNow Developer site</a></p><p><a href="https://developer.servicenow.com/dev.do#!/reference/api/tokyo/rest/c_TableAPI#table-GET?navFilter=TAble">ServiceNow Table Rest API Documentation</a></p><p><a href="https://medium.com/towards-data-science/8-tips-for-creating-data-visualizations-in-python-using-bokeh-d7a01701503d">Tips for Formatting Bokeh Charts</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8549e12bd61b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving Google’s Knight Dialer With Graph Theory: Part 2]]></title>
            <link>https://jeffmanville.medium.com/solving-googles-knight-dialer-with-graph-theory-part-2-f56aca04b576?source=rss-1f8656ddca62------2</link>
            <guid isPermaLink="false">https://medium.com/p/f56aca04b576</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[graph-theory]]></category>
            <dc:creator><![CDATA[Jeff Manville]]></dc:creator>
            <pubDate>Sun, 09 May 2021 13:55:31 GMT</pubDate>
            <atom:updated>2021-05-11T22:03:25.575Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6PpvPH7tHlioBJ9yE5fXHQ.jpeg" /><figcaption>“Knight to E4!…. Yes Hermione, I think this is going to be exactly like wizard’s chess.” — Ronald Weasley</figcaption></figure><h3>Binary Decomposition</h3><p><a href="https://jeffmanville.medium.com/solving-googles-knight-dialer-with-graph-theory-part-1-213120cfc856">In Solving Google’s Knight Dialer With Graph Theory: Part 1</a>, we were essentially taking 1 step forward at a time until we reach the matrix we need. However, we don’t have to take 1 step at a time; we can take many steps forward with a technique called Binary Decomposition.</p><p>The simplest introduction to Binary Decomposition is often the example of multiplication.</p><p>Let’s say we have to solve x=3⁸. We could multiply “3” 8 times and we would get our answer. But we know that 3² =9 and we could use substitution and only need to solve x=9⁴. But we also know 9² =81, we could substitute that and only have to find x=81² which equals 6561.</p><p>81² might be difficult to do for you and me, but its less work for a computer than 3⁸ as there are fewer operations. With Binary Decomposition, instead of 7 separate multiplications, a computer would only need 3. This allows the work to grow logarithmically instead of linearly as we need to multiply larger and larger powers.</p><p>But how would we solve for 3¹⁰? 10 isn’t a power of 2. Well, all integers can be expressed as a series of powers of 2. 10= 2¹ *+ 2³ because 2¹ is 2 and 2³ is 8, 2+ 8=10. So, x =3¹⁰ = 3²*3⁸= 3²*(3²)⁴ = 9*9⁴= 9*(9²)² = 9*81² = 9*6561 = 59049. Again, this is a lot less work for a computer.</p><p>This can all be applied to square matrices, too.</p><h3>Binary Decomposition Applied</h3><p>One way to implement Binary Decomposition is with recursion. There are three situations we need to be able to handle.</p><ol><li>Our exit condition, which will be when the number of turns is equal to one</li><li>Our condition when the number of turns is equal to 2 to some power</li><li>Our condition when the number of turns is not equal to 2 to some power</li></ol><p>Now, this program could be calculated entirely with recursion and it would be effective. At 5000 turns, our final linear solution from the previous article took 1.387 seconds. A recursive solution without memoization took about .02 seconds. With memoization, it took about .014 seconds.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e85b93c53cafc82608809acd0cddb9de/href">https://medium.com/media/e85b93c53cafc82608809acd0cddb9de/href</a></iframe><p>However, as stated in Alex’s article, recursion has some problems. It grows in memory requirements as the inputs get larger and is limited to 1000 recursive calls on Linux &amp; Mac OS and limited to 2000 calls on Windows OS.</p><p>However, I hate recursion. It is a pain to test and maintain. I don’t like how easy it is to create an infinite loop and it feels like a step backwards to use it.</p><p>So, let’s get rid of it.</p><h3>Non-Recursive Knight Solver</h3><p>In our recursive solution, we solved the problem using Binary Decomposition and we are going to do the same thing here. There are two recursive paths that need to be handled: breaking the problem into powers of 2 and calculating the matrix raised to a power of 2</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fd28887e2ffbad4aa0a231499ec4efc/href">https://medium.com/media/8fd28887e2ffbad4aa0a231499ec4efc/href</a></iframe><p>The first part is responsible for calculating M^N where N= 2^<strong>n</strong>. It can be phased out by replacing it with a simple function that takes a starting matrix &amp; “<strong>n”</strong>. Then, the function squares the matrix “<strong>n</strong>” times and returns it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/326d9a8a3540892270e59eb18fda9288/href">https://medium.com/media/326d9a8a3540892270e59eb18fda9288/href</a></iframe><p>So, for 8 turns out, we would pass 3 as the baseTwoTurns because 2 raised to the 3rd power is 8. In this function, the first pass would generate M², the second pass would generate M⁴, and the third and final pass would generate M⁸.</p><p>We can implement this function into our Fully Recursive solution with only a slight modification.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/293f5ec0ede744012eff4be0ec6e5f5a/href">https://medium.com/media/293f5ec0ede744012eff4be0ec6e5f5a/href</a></iframe><p>The second recursive path is responsible for breaking the matrix calculation for N turns into a series of base two powers. So 11 breaks into [8, 2, 1] which can be expressed as [2³, 2¹, 2⁰]; So, the array we need [3, 1, 0].</p><p>We can calculate this by iteratively removing the greatest power of two from our original number until it is equal to zero.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27e43c7b6620eae4e367e621bccdb5d8/href">https://medium.com/media/27e43c7b6620eae4e367e621bccdb5d8/href</a></iframe><p>You may have noticed that with our current <strong><em>getBaseTwoTurnsMatrix </em></strong>function, we will not be able to take advantage of cacheing. In our recursive solution, calling <strong>getMatrix </strong>for 8 turns out will eventually call <strong>getMatrix </strong>for 2 turns out. That is not true for this implementation.</p><p>We can circumvent this limitation by going from smallest square to largest square, storing the result from the last turn, and only squaring that matrix by the difference between the two powers.</p><p>For example, if we were going 24 turns out, 24 breaks down to [2³ ,2⁴]. We take M¹ and square it 3 times to get M⁸. By saving M⁸, we only have to square the M⁸ matrix once to get to M¹⁶. Then, we only have to multiply M⁸ &amp; M¹⁶ to get M²⁴.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1932f0e3f449411060c1723e2167907/href">https://medium.com/media/e1932f0e3f449411060c1723e2167907/href</a></iframe><h3>Out of the Box</h3><p>Trying to come up with everything we have done, in 45 minutes, would be a lot of work; Fortunately, you don’t have to. Everything in <strong>getMatrix</strong> has already been implemented in <strong>Numpy’s </strong><a href="https://github.com/numpy/numpy/blob/master/numpy/linalg/linalg.py#L560"><strong>Linalg.matrix_power</strong></a> function and can be applied to any square matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af5b446d408836d354a8c119764f90a1/href">https://medium.com/media/af5b446d408836d354a8c119764f90a1/href</a></iframe><p>Unfortunately, we are now butting up against the limits of the implementation of the matrix multiplication in NumPy. In fact, due to the sheer size of the numbers we are working with all of our implementations grow exponentially as the number of turns out increase. The difference being how quickly they grow exponentially. There isn’t anything we could do to improve the matrix multiplication, but we could use a better matrix.</p><h3>Same Answer, Different Graph</h3><p>Lets take a look at our original Neighbor Diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*KJrMAQVqjD7vSsLvdf4Zfw.png" /></figure><p>In the Neighbor diagram, a lot of the numbers look very similar. “4” &amp; “6” are always connected to “0” and two “corner” positions (1,3,7,9). Lets “4” &amp; “6” call a “bridge” position. “2” &amp; “8” are very similar, they always connect to two corner positions. Lets call them an “edge” position. And lets call “0” a center position. we can handle “5” as an edge case and use the following mapping:</p><p>A. Corners: (1, 3, 7, 9)<br>B. Edges: (2, 8)<br>C. Bridges: (4,6)<br>D. Center: (0)</p><p>By relabeling the first state diagram with the letters above, we can see the similarity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*-QWU-J5ljVMtzZm22wdRoA.png" /></figure><p>“A” (or a Corner) always leads to 1 “B” (or Edge) and 1 “C” (or a Bridge). “B” always leads to 2 “A”s. “C” always leads to 2 “A”s and 1 “D” (or center). “D” always leads to 2 “C”s.</p><p>A: 1-B, 1-C<br>B: 2-A<br>C: 2-A, 1-D<br>D: 2-C</p><p>Since all the corners behave the same, it doesn’t matter which corner the knight is at; the number of next numbers that can be dialed is the same. The same is true for edges or for bridges.</p><p>We are exploiting the graph’s symmetry to generate a new simpler graph that is still capable of answering our problem. This new graph may not be able to answer all the questions our original graph could, but it can still answer our problem.</p><p>Since we have a mapping from one state ( “A”, “B”, “C”, “D”) to another, it can be drawn as a new state diagram with a corresponding transition matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*wqik7k9wcgdVvC5p1mN88A.png" /><figcaption>Our New Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1a12fa244fba215c0a1df7a207c42c2/href">https://medium.com/media/a1a12fa244fba215c0a1df7a207c42c2/href</a></iframe><p>Why is this a better map to use? Well, the number of multiplication operations to multiply two NxN matrices is equal to the N³.</p><p>So, our original 10x10 matrix requires 1000 multiplication operations each time we multiplied it by itself, while our new 4x4 matrix only requires 64 multiplication operations.</p><p>If we needed to calculate M⁸ we would need to square our matrix 3 times. For our original matrix, that would be 3000 multiplication operations. With our New matrix, it would only require 192 multiplication operations, only 6.4% of the operations the original matrix required. This represents a significant reduction in the cost to calculate each matrix.</p><p>However, this new matrix requires another layer of abstraction to map our the starting position on our original matrix to the positions on our new matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/976784622c9a52cc78d8fb38e41945d5/href">https://medium.com/media/976784622c9a52cc78d8fb38e41945d5/href</a></iframe><p>Implementing only requires a slight modification of our function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/774b95befceea3ca25acf8c3c22acdde/href">https://medium.com/media/774b95befceea3ca25acf8c3c22acdde/href</a></iframe><h3>How Much Better Is It?</h3><p>Well, here is a visualization of the performance for our Linear solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NUBViRqnF9Nlaxyk21tBVw.png" /><figcaption>All data was generated by measuring the time to complete with a starting position of 0</figcaption></figure><p>In this visualization, I measured the time to complete for 250 different turns equally spaced from 0 to 20,000. There is a bit of noise in the data, this may have been due to:</p><ul><li>me, idly hitting the space bar to see if it was still working</li><li>a random background update from windows</li><li>magic smoke in computer decided to be slow</li></ul><p>Here is a visualization of the raw performance data for the solutions we’ve done so far.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g3HjPLoD3M0BTk7TxYScig.png" /><figcaption>Solution 1 = linear solution, Solution 2 = Recursive solution, Solution 3 = Recursive solution + Memoization, Solution 4 = dynamic programing solution without reversing the powers, Solution 5 = dynamic programming solution with reversing the powers, Solution 6 = Nump’s matrix power solution</figcaption></figure><p>There are several interesting things to point out in this graph.</p><ol><li><strong>Every implementation of Binary Decomposition is better than our first solution.</strong><br>Hooray!</li><li><strong>There is some noise, but there are also some patterns for solutions two through six.</strong><br>The sudden drops in run time are due to how binary decomposition works. In general, it easier to calculate <strong>2^N</strong> turns out than <strong>(2^N)-1</strong>. In Binary decomposition, we have to calculate each power of 2, but we then have to combine each of the factors. So, there will be an increasingly large drop in runtime upon reaching a power of 2.</li><li><strong>Solution 3, the recursive solution + memoization, performs really well </strong><br>While benchmarking, the cache was cleared between each run as the point was to was to measure individual performance at a number of turns.<br>It still does really well, but the problems start to show towards the tail end of the graph. If we were to use this solution only once for 400,000 turns out, getMatrix would be storing nineteen 10x10 matrices with huge numbers in memory. Its a significant cost to memory that will significantly degrade performance. In fact, Its pretty easy to see the performance start to degrade significantly towards the end of the graph.</li><li><strong>Solutions 5 &amp; 6 are identical in performance.</strong><br>Solution 5 &amp; 6 are the same algorithm; So, their performances will be the same. However, NumPy’s matrix_power is less verbose than Solution 5’s getMatrix and has more error handling around inputs and handles negative exponents. I’d recommend using it.</li></ol><p>Lets see how our final solution compares.</p><h3>Lucky Number 7= condensed graph solution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eW_No--Oazn8WECiGLo8-g.png" /><figcaption>Fun fact: at 1,750,000 turns, the number would be somewhere between 10⁵²⁶⁸⁰² and 10⁸³⁴⁹⁶³, an absolutely incomprehensibly large number</figcaption></figure><p>The final solution blows the competition out of the water; It is consistently faster than the previous six solutions even for lower number of turns. There is significantly less information to store as there are fewer positions and fewer relationships. This flattens the performance curve, limiting the impact of memory issues resulting from the increasingly large numbers</p><h3>Wait, weren’t these O(log(N))?</h3><p>You may be wondering why these curves don’t seem to be logarithmic. Well, we’re actually running into the performance curve of multiplication. Since, the numbers are beyond astronomically large, multiplication is starting to take up the most amount of time.</p><h3>How much further we can go?</h3><p>We could look into how the multiplication algorithm is implemented and implement a <a href="https://en.wikipedia.org/wiki/Multiplication_algorithm#Fast_multiplication_algorithms_for_large_inputs">faster algorithm</a> like the FASTEST multiplication algorithm ever from <a href="https://hal.archives-ouvertes.fr/hal-02070778">David Harvey and Joris van der Hoeven</a></p><p>If this was implemented into a service, there may be ways to do distributed computing or store commonly calculated matrices. We could also leverage a GPU to do the calculations. We could also switch to a language other than python for something closer to bare metal performance.</p><p>This is an educated guesstimate. Running the calculations on a GPU might do a 10² improvement depending on the parallelization. We could get a 10³ improvement by throwing it at a sufficiently large computing cluster which really isn’t crazy. A C implementation could maybe get a 10² to 10³ improvement from what little research I have done. I’m not sure how much the new multiplication algorithm would do. We could also precalculate and store some of the common multiples of 2-power matrices.</p><p>An ML engineer would know better, but I think all these ideas together could feasibly improve the performance from ~10⁶ steps in &lt;10 seconds to &gt;10¹³ of steps in &lt;10 seconds. Assuming, we don’t run into a different constraint at that point like Skynet deciding this is robot cruelty.</p><p>However, we are at a pretty good stopping point. We have hit the limits of graph theory. If anyone figures out how to implement Dr. Harvey and Dr. van der Hoeven’s multiplication or any of the other improvements, please let me know! I’d love to see it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f56aca04b576" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving Google’s Knight Dialer with Graph Theory: Part 1]]></title>
            <link>https://python.plainenglish.io/solving-googles-knight-dialer-with-graph-theory-part-1-213120cfc856?source=rss-1f8656ddca62------2</link>
            <guid isPermaLink="false">https://medium.com/p/213120cfc856</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Jeff Manville]]></dc:creator>
            <pubDate>Sun, 09 May 2021 13:54:24 GMT</pubDate>
            <atom:updated>2021-05-17T10:27:54.939Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D8I3RViD7ShYb6cqvlfxDA.jpeg" /><figcaption>“Ron, you don’t suppose this is going to be like . . real wizard’s chess, do you?” — Hermione Granger</figcaption></figure><p>A few weeks ago, <a href="https://medium.com/u/fc349a0617c0">Alex Golec</a> wrote an awesome walkthrough to <a href="https://medium.com/@alexgolec/google-interview-questions-deconstructed-the-knights-dialer-f780d516f029">the Knight’s Dialer</a>, a former interview problem at Google:</p><blockquote>Imagine you place a knight chess piece on a phone dial pad. This chess piece moves in an uppercase “L” shape: two steps horizontally followed by one vertically, or one step horizontally then two vertically…</blockquote><blockquote>How many distinct numbers can you dial in N hops from a particular starting position?</blockquote><p>Alex demonstrated some of the many ways the problem can be solved using recursion and dynamic programming. I am going to demonstrate how you can match his performance with some basic Graph Theory and then surpass it in a later tutorial.</p><h3>What is Graph Theory?</h3><p>When I think of graphs, the first image in my mind is usually a bar or pie chart. In graph theory, graphs represent the relationships between objects. Here are a few examples:</p><ul><li>A Family Tree (people: family relations)</li><li>A Social Network (people: friendships)</li><li>A Road Map (locations: roads)</li><li>A Website Map (web pages: URL links)</li></ul><p>So what is the graph in this problem? Well, if we consider each number as an object and where the knight can go in one turn are the relationships, we have a graph.</p><p>If we were to compare the knight’s dialer to the road map example, a number would be a location and the numbers a knight can go to next are the roads between numbers. A number we can dial like “0–4–3–8–1 ends up being just a path locations our knight has visited where the path is four “turns” long (0 to 4, 4 to 3, 3 to 8, and 8 to 1).</p><p><em>“</em><strong><em>How many distinct numbers can you dial in N hops from a particular starting position?</em></strong><em>” </em>becomes how many paths are there that are N turns long from a particular starting position.</p><h3>Defining The Graph</h3><p>In Alex’s solution, the first step identified the “neighbors” for each number and we do the exact same thing here to create our graph. Here is Alex’s original’s Neighbors Map.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6a034c118ee16707520055facb3f020/href">https://medium.com/media/f6a034c118ee16707520055facb3f020/href</a></iframe><p>With that, we can create a graph that represents the possible movements of the knight. So, “0” connects to “4” and “6”; “4” connects to “0”, “3”, and “9”; etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*teEwvY4ZOinXzPTVVrx34Q.png" /><figcaption>A directed graph representing the possible movements of our knight</figcaption></figure><p>This image is known as a Directed Graph. It represents how the knight can move from one number to another in a single turn. Every node can go to a new number except for 5, which is off on its own.</p><p>While this kind of image is a useful way to visualize graphs, we can do a lot more by representing the graph as a matrix.</p><h3>Into The Matrix</h3><p>In Graph Theory, a graph can be represented as a matrix known as a State Transition Matrix, where the number of rows and the number of columns are both equal to the number of nodes in the graph. So for our graph, we have a row and a column for each number. Each cell represents a <strong><em>“possible” </em></strong>relationship between the row’s node and the column’s node. However, only cells with a value other than 0 represent an <strong><em>“actual”</em></strong> relationship</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*_g5imCkttR09YhYb-n22kQ.png" /><figcaption>the matrix in excel</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/682297e76b1065de116d1e520fc65da2/href">https://medium.com/media/682297e76b1065de116d1e520fc65da2/href</a></iframe><p>For this problem, it can be helpful to think of each cell’s value as the number of unique paths from one number to another.</p><p>In this case, the starting position is the row and the destination is the column. From our graph, we know that if our knight is at position “<strong>1</strong>”, there is only one path to position “<strong>6</strong>” and only one path to position“<strong>8</strong>”; Then, row “<strong>1</strong>” should have a value of 1 in the columns “<strong>6</strong>” and “<strong>8</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*8f5B_vUM9K3bb_sAFjhwRw.png" /></figure><p>Notice how all the cells in column 5 or row 5 are 0, indicating nothing can come from it or go to it.</p><h3>The Road N̵o̵t̵ Taken</h3><p>If our matrix represents where we can go in <strong><em>one </em></strong>turn, how can we find out where we can go in <strong><em>two</em></strong> turns? We can do this with Matrix Multiplication.</p><p>Matrix Multiplication is a mathematical operation similar to multiplication that allows us to multiply two matrices together to get a new matrix.</p><p>In matrix multiplication, we multiply a column of one matrix by a row of another matrix, sum the result, and put that value into our new matrix in the cell corresponding to the row &amp; column used. We repeat this for each row of our first matrix and each column of the second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*nG3kAuot3ILNxcjwk7Eb7A.png" /><figcaption>Short hand for Matrix Multiplication</figcaption></figure><p>Now, lets do an example with our matrix. In this case, Matrix A and Matrix B are identical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*HVh3qxeGEDqQPdixbRdq3w.png" /></figure><p>How would we find the value for position 0,0 in our new matrix? Well, first we isolate row 0 of Matrix A and Column 0 of Matrix B.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*9xI0Jg-CPGekfYHqjt25uw.png" /></figure><p>We multiply each corresponding cell.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*bSQVVEc2qaz2D85AoN8j9A.png" /><figcaption>Multiply</figcaption></figure><p>Sum the results</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/79/1*yMEj5oh4-JQvbNwZySBjNA.png" /><figcaption>Sum</figcaption></figure><p>And put that into our new matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*Tc4AuMYcKbYkhJKAgVpw8A.png" /><figcaption>One cell down</figcaption></figure><p>We would then repeat this for each row in Matrix A and each Row in Matrix B.</p><p>This is a very long process and we could write a function to handle this, but NumPy already handles this. All you have to do is multiply two matrices the same way you would multiply two numbers in python.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/186a8d48d7ef53493eadb81f016591ca/href">https://medium.com/media/186a8d48d7ef53493eadb81f016591ca/href</a></iframe><p>By multiplying the matrix that represented where the knight can go in <strong>one turn</strong>, we found a matrix that represents where the knight can go in <strong>two turns</strong>.</p><p>We can repeat this and go as many turns out by simply multiplying our original matrix by itself “N” times. Our new matrix will represent the number of unique paths the knight can take in <strong><em>N </em></strong>turns starting at one node and ending in another.</p><p>This can be implemented fairly easily with NumPy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a701d54ab441b4bd16285bfe6da5a28/href">https://medium.com/media/6a701d54ab441b4bd16285bfe6da5a28/href</a></iframe><h3>The last step</h3><p>Again, for all of our matrices, each cell represents the number of unique paths from one position to another position in “N” turns. What we need is the total of unique paths from a specific position to anywhere in “N” turns.</p><p>How do we do this? We sum together all of the relevant paths.</p><p>By selecting the relevant row, we have a vector that represents the number of unique paths our knight can take from our original position to any position in N turns. By summing all of those values, we have our answer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28613c776c1f4665c5d5daa42d15a750/href">https://medium.com/media/28613c776c1f4665c5d5daa42d15a750/href</a></iframe><h3>Putting it together</h3><p>Now, we have all the parts to put together our first solution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05cb45f872612c4d3c5c6e292f742f3c/href">https://medium.com/media/05cb45f872612c4d3c5c6e292f742f3c/href</a></iframe><p>This solution reaches the same “level” as Alex’s level 4 solution. It requires constant memory, it is non-recursive, and linear in performance (in Big O notation: O(N)).</p><p>When run for 500 hops on my laptop, it completed in .084 seconds or 84 milliseconds. It was also able to go about 20,000 turns out in ~10 seconds.</p><p>It&#39;s important to note that this should not be compared directly to Alex’s solution; rather, it&#39;s a baseline for comparisons to other solutions run on the same machine.</p><p>However, we can do even better.</p><h3>Next up</h3><p>In the next article, we will improve the getMatrix function with a technique called Binary Decomposition, which will change our performance from Linear to Logarithmic (O(N) to O(log(N)). This will allow our solution to calculate ~10⁶ turns out instead of ~10⁴ in ten seconds.</p><h4><a href="https://jeffmanville.medium.com/solving-googles-knight-dialer-with-graph-theory-part-2-f56aca04b576">Solving Google’s Knight Dialer With Graph Theory: Part 2</a></h4><p><em>More content at </em><a href="http://plainenglish.io/"><em>plainenglish.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=213120cfc856" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/solving-googles-knight-dialer-with-graph-theory-part-1-213120cfc856">Solving Google’s Knight Dialer with Graph Theory: Part 1</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ServiceNow: Custom GraphQL APIs]]></title>
            <link>https://jeffmanville.medium.com/servicenow-custom-graphql-apis-part-1-d34be6b1b64d?source=rss-1f8656ddca62------2</link>
            <guid isPermaLink="false">https://medium.com/p/d34be6b1b64d</guid>
            <category><![CDATA[servicenow-training]]></category>
            <category><![CDATA[servicenow]]></category>
            <category><![CDATA[graphql]]></category>
            <dc:creator><![CDATA[Jeff Manville]]></dc:creator>
            <pubDate>Fri, 30 Apr 2021 17:20:44 GMT</pubDate>
            <atom:updated>2024-10-09T16:07:26.165Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WjVr87nTkybfJLmZM8QFfg.png" /></figure><p>With Paris, ServiceNow has introduced custom GraphQL APIs which enables us to create highly flexible, reusable, and cacheable APIs for integrations or front end engineering.</p><p>However, documentation and tutorials for this new technology is sparse for the ServiceNow community. For this tutorial, we will build a GraphQL API for the Automated Test Framework that will be iteratively improved through the tutorial</p><ul><li>Parts of a GraphQL API</li><li>A basic example of returning data from a single record</li><li>how to connect schemas together to return even more data</li><li>how to return a list of related records</li></ul><p>GraphQL has three types of operations: <strong>Query</strong>, <strong>Mutation</strong>, and <strong>Subscription</strong>.</p><p><strong>Query </strong>provides the ability to read data. <strong>Mutation </strong>provides the ability to create, update, and delete data. <strong>Subscriptions </strong>are a provide the ability to subscribe to real time data.</p><p>ServiceNow supports two of the operation types: <strong>Mutation </strong>&amp; <strong>Query</strong>.</p><p>For this tutorial, we’ll be covering just <strong>Query</strong>, which is the operation type used for reading data.</p><h3>Requirements:</h3><ul><li>ServiceNow Instance on at least Paris</li><li>Postman application for testing</li></ul><h3>First step: creating the API</h3><p>On the left hand nav go to<strong> <em>System Web Services &gt; GraphQL &gt; GraphQL APIs</em></strong> and click <strong>New</strong></p><p>This should take you to the GraphQL API (sys_graphql_schema) form.</p><figure><img alt="GraphQL Schema form for a new schema" src="https://cdn-images-1.medium.com/max/1024/1*isnI6knY3edou2qIoc-DRA.png" /></figure><p><strong><em>Name </em></strong>is just a user friendly naming field. <strong><em>Schema namespace</em></strong> is a unique system name for your API and is used to identify each GraphQL APIs.</p><p>The most interesting part of this form is the <strong><em>Schema </em></strong>field. The <strong><em>Schema </em></strong>field is the structure of your API. It does a couple things.</p><ol><li>It decides the available data &amp; “structure” for return results</li><li>It determines the available inputs &amp; maps them to the schema</li></ol><p>We’ll be creating two queries, one for an ATF test &amp; one for an ATF suite. First, We’ll add the ATF test query.</p><h3>ATF Test Query</h3><p>First, lets update the schema</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe9514e35d90c57135defbbdb9b803c5/href">https://medium.com/media/fe9514e35d90c57135defbbdb9b803c5/href</a></iframe><p>In this Schema, we have created an object type called <strong><em>atfTest </em></strong>that has four fields: active, id, name, and description. These fields will be populated by a JSON object from a scripted Resolver.</p><p>We also added an attribute to the Query Object; the attribute is called <strong><em>atfTest</em></strong>. It expects to have an input of type string called <strong>id </strong>and will return an <strong><em>atfTest </em></strong>object.</p><p>This is the skeletal structure of our API. However, it is not enough to return data.</p><p>We will need to add Scripted Resolvers which will be the muscle of our API and Resolver Mappings which will be the tendons that connect the Scripted Resolvers &amp; Schema together. Script includes will help us provide consistency when interacting with each object type in the schema.</p><p><strong>Scripted Resolver: getTestInformation</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b707fae323ce0f0f0044ba2c81dddb6d/href">https://medium.com/media/b707fae323ce0f0f0044ba2c81dddb6d/href</a></iframe><p>Like I said before, the Scripted Resolver is the muscle of our API. It will do the work of processing inputs and generating the output for our API.</p><p>That said, a lot of our logic for the <strong><em>atfTest </em></strong>object will not be stored in the Scripted Resolver, but rather in a Script Include called <strong>ATFGraphQLAPI</strong>. This will allow us to reuse this logic and provide us a single place to update our Logic when we need to update the <strong><em>atfTest </em></strong>object.</p><p>Scripted Resolvers should handle what data gets used for populating a particular path in your Schema. You should have ONE function (for each Object Type) that will generate the structured object that will be grafted onto each Object Type.</p><p><strong>Script Include: ATFGraphQLAPI</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5c6085358f2ec73ed42d7cf0619f283/href">https://medium.com/media/d5c6085358f2ec73ed42d7cf0619f283/href</a></iframe><h3>Mapping values to the Schema</h3><p>Now that we have the the schema, scripted resolver, and the script include; Its time we attach the scripted resolver to the schema and have our first working version of our API. We will use a Resolver Mapping to connect a Path to our scripted resolver.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe9514e35d90c57135defbbdb9b803c5/href">https://medium.com/media/fe9514e35d90c57135defbbdb9b803c5/href</a></iframe><p>The paths available follow the form <strong><em>&lt;object&gt;:&lt;attribute&gt;</em></strong>. For example, <strong><em>Query:atfTest</em></strong> or <strong><em>atfTest:active</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XoZvhIHF_40O3ox3wnqW4Q.png" /><figcaption>GraphQL Resolver Mapping with Path=Query:atfTest and Resolver=getTestInformation</figcaption></figure><p>Resolver mappings allow us to map our scripted resolvers to the paths of our schema. We don’t have to map everything. Since our Scripted Resolver return an object with attributes that mirror the atfTest schema’s attributes, the GraphQL engine will automatically map them for us</p><p>However, You may have noticed the<strong><em> @source(value: “sys_id”)</em></strong> in our atfTest schema. That allows us to tell the GraphQL engine to map an attribute with a different name to the <strong><em>id</em></strong> object. Its not really necessary here, but it allows us to provide a more intuitive name for traditional developers who may not be familiar with sys_ids.</p><h3>First Functional API</h3><p>Now, we can test our API with a REST call. Personally, I like to use Postman over the REST API explorer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nty5RB8sUMM4CRKxLVJNbw.png" /></figure><p>For your own test, you will want to replace the x47399 with your namespace, replace {{test_id}} with your test’s sys_id, and {{api_host}} with your servicenow instance’s URL. Here is the response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzlmwRODdvYFk9Gy0KLahQ.png" /></figure><p>If you have followed along correctly, we have our first functional API! Congratulations!</p><p>You may be thinking “That’s dumb, why do this over just using the Table API” Stick with me, it gets WAY better.</p><h3>Adding Test Results</h3><p>First, lets update our Schema</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90979e37d935bfc755e931e2968a04b1/href">https://medium.com/media/90979e37d935bfc755e931e2968a04b1/href</a></iframe><p>There are quite a few new additions here. We’ve added the atfTestStatus Object Type, the atfTestResultList Object Type, and a new attribute to the atfTest called testHistory.</p><p>The testHistory attribute and atfTestResultList Object Type will be how we return a list of objects. The testHistory attribute is a little more complicated.</p><p>The<strong><em> </em></strong><em>(limit: string)</em> will add an input called limit, which will allow us to control how many test results are going to be returned by the query. The <em>@source(value: “sys_id”) </em>will pass ONLY the sys_id of the Scripted Resolver instead of the entire schema object.</p><p>This is particularly useful functionality if you want to use the same scripted resolver against multiple paths. For example if we had a User Object type &amp; an Incident Object Type, the <em>@source</em> to map multiple user field to the same User Object Scripted Resolver.</p><p>Lets add a new Scripted Resolver &amp; Resolver Mapping and update our Script Include so we can get our test results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e31da4412ab003e2d5848d5481d4bacc/href">https://medium.com/media/e31da4412ab003e2d5848d5481d4bacc/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eaaaf0021f2ed5789d4eef49a2e1befc/href">https://medium.com/media/eaaaf0021f2ed5789d4eef49a2e1befc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A-jasNa1r8vKVQORv4lmfg.png" /><figcaption>GraphQL Resolver Mapping with Path=atfTest:testHistory and Resolver=getTestHistory</figcaption></figure><p>Now, lets check out how new version works!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gR0l5v9Tpufml7b8Vn-NYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qnEPuVm7KRwAhWZ76XHCpw.png" /></figure><p>Awesome! Now, we are able to get the details of our test &amp; our test history in a single API call, but we can take it even further.</p><h3>Adding a Test Suite Query</h3><p>lets add a Test Suite Query by updating our Schema.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c30ccc04f3b954395091ec288847e270/href">https://medium.com/media/c30ccc04f3b954395091ec288847e270/href</a></iframe><p>We’ve added two Object Types, atfTestSuite and testList, and we have added an attribute to the Query Object type. This will allow us to query for a test suite rather than a test.</p><p>Now lets add our Scripted Resolvers:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4494cfd7e2c07466c9b5e5381deed000/href">https://medium.com/media/4494cfd7e2c07466c9b5e5381deed000/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52283a1c9dc3de9830fa190274a15d52/href">https://medium.com/media/52283a1c9dc3de9830fa190274a15d52/href</a></iframe><p>Lets update our Script include</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f54d5215e71ffca031ea174f5d268bd/href">https://medium.com/media/5f54d5215e71ffca031ea174f5d268bd/href</a></iframe><p>And then add our Mappings</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tISRa1wiPfr_y87fO-NhZA.png" /><figcaption>Resolver Mapping with Path=Query:atfTestSuite and Resolver=GetSuiteInformation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aCamtz0KysvlahpRUocOHg.png" /><figcaption>Resolver Mapping with Path=atfTestSuite:tests and Resolver=getTests</figcaption></figure><h3>Opinion on Two ATF Features</h3><p>Before we get into testing our new API, this ATF suite query is a bit opinionated.</p><p>There are two features in ATF that I think add needless complexity; Specifically, child Test Suites and the input_filter for suites. I have purposefully NOT added support for these features in this API.</p><p>Due to how ServiceNow presents suite results, child test suites obfuscates any failing tests and adds a need for recursive logic to check for tests related to a parent suite. Adding tests directly to suites is much simpler and more visible.</p><p>The input_filter feature allows admins/QAs/devs/whoever to specify a query that identifies tests to run. This means a change to the query or a change to a test could change the tests related to a suite.</p><p>This makes identifying suites a test is related to or changes to a suite’s tests invisible or difficult to track. You could have 100 tests in a suite one day and 100 the next day, but be totally unaware that tests have been removed &amp; added.</p><h3>Testing our Suite API</h3><p>Here is our query, notice the change at line 4 which points to the atfTestSuite object type rather than atfTest object type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EXqJbkkeu5xAaYsqH28ajA.png" /></figure><p>Here is our result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u81nXdzXRf6GWr0ePpVd1g.png" /><figcaption>200 result from ATFTestSuite query</figcaption></figure><p>Success!</p><p>You may have noticed that the history array is empty because that test has no result. ServiceNow purges results after a certain amount of time.</p><h3>In Summary</h3><p>Congratulations! We now have two GraphQL API queries for getting our Test Suites, tests, and the test results. You should now have a fundamental grasp on Queries with GraphQL APIs.</p><h3>Next Steps</h3><p>In two follow up articles, I will cover additional GraphQL concepts that are available in ServiceNow: Mutations and Type Resolvers. I will also provide an update set containing a production ready ATF GraphQL API.</p><p>There are also some concepts I won’t cover yet because ServiceNow does not support them: Subscriptions.</p><p>Additionally, I will also cover how GraphQL significantly changes frontend development in an article on Reactjs in ServiceNow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d34be6b1b64d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to integrate ServiceNow & Slack’s Slash Commands]]></title>
            <link>https://jeffmanville.medium.com/how-to-integrate-servicenow-slacks-slash-commands-ecbddaa7a96a?source=rss-1f8656ddca62------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecbddaa7a96a</guid>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[servicenow]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Jeff Manville]]></dc:creator>
            <pubDate>Mon, 19 Jun 2017 15:38:54 GMT</pubDate>
            <atom:updated>2023-02-16T16:52:29.145Z</atom:updated>
            <content:encoded><![CDATA[<p>Slack is a real time messaging platform that makes work entertaining with many ways to integrate third party tools to create powerful and interesting interactions. ServiceNow is a PaaS &amp; SaaS provider for internal service management, automation, &amp; orchestration. Together, they have offer a wide range of opportunities. Today, we will be integrating the two.</p><p>ServiceNow boasts a <a href="https://developer.servicenow.com/app.do#!/rest_api_doc?v=helsinki&amp;id=c_TableAPI">wide range of APIs</a> that can be used to integrate external applications with your instance. Each API is very useful, but they all have their own limitations. So, before you head down the path of a processor, please consider these other APIs to determine whether any of them can fit your needs. It will save you time &amp; resources.</p><p>Processors can have many many uses. They are an extremely powerful tool that can be used to solve a myriad of problems. However, when you have a hammer, everything looks like a nail. Today, we will use it to integrate slash commands with Slack.</p><p><strong>Just to get this out of the way</strong>: <em>“Why should I use a Processor over the Scripted REST API?”</em></p><p><strong>Answer:</strong> Maybe, you shouldn’t. The scripted REST API is a wonderful way to consume external REST calls, but it does have limitations. The Slack slash commands have a content type of “application/x-www-form-urlencoded” while Scripted REST resources only support “application/json”, “application/xml”, and ”text/xml” currently. So, processors are the way to go.</p><p>Processors allow developers to create a custom URL endpoint where we can run server-side scripts and return anything we want. This functionality will allow your instance to consume REST calls in any way we desire. This allows us to build better integrations with 3rd party applications.</p><p><strong>Here is some documentation on the stuff I’ll be touching upon in this tutorial:</strong></p><p><a href="https://slack.com/">Slack</a> <a href="https://api.slack.com/slash-commands">Slash Commands</a> <a href="https://www.servicenow.com/">ServiceNow</a> <a href="https://docs.servicenow.com/bundle/istanbul-servicenow-platform/page/script/processors/concept/c_Processors.html">ServiceNow Processors</a></p><p><strong>Expectations:</strong></p><p>This tutorial is a more advanced tutorial for ServiceNow developers. That said, as long as you are familiar with Javascript &amp; the ServiceNow development environment, you should be all set. I have my own problems with the certification driven craziness within ServiceNow. I do not believe its a good indicator of skill and I prefer a more portfolio-centric approach (says the college graduate with no certifications)</p><p>So as long as you can complete this <a href="http://joeyday.com/2015/03/02/integrating-servicenow-with-slack/">tutorial</a> by <a href="https://medium.com/u/773e869bef6b">Joey Day</a>, You should be all set. In fact, I use stuff built in his tutorial at the end of this tutorial. <strong>I WOULD HIGHLY RECOMMEND YOU COMPLETE HIS TUTORIAL FIRST.</strong></p><p><strong>The Tutorial:</strong></p><p>The result of this tutorial will be an integration that allows you, the developer, to create slash commands in Slack that create records in a custom table in ServiceNow which you can then respond to with simple Business Rules.</p><p><strong>Here are a few things you will need to build this:</strong></p><ul><li>A slack channel — if you don’t have one, you can create one <a href="https://slack.com/create#email">here</a></li><li>A slack developer account — You can create one <a href="https://api.slack.com/">here</a></li><li>A ServiceNow instance — you can get developer instance if you create an account <a href="https://developer.servicenow.com/">here</a></li></ul><h3><strong>Step 1. Create the Processor</strong></h3><p>Go to <em>System Definition &gt; Processors. </em>Click New.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXqx9kF6D9OLgdxyYoDK2w.png" /><figcaption>New Processor</figcaption></figure><p>Set an appropriate <strong>name</strong> for the processor. I called mine Slack Processor. Set the <strong>Type</strong> to <em>script</em>. Write a <strong>description</strong> &amp; add a <strong>Path</strong>. The path will determine the endpoint URL for Slack to POST our slash commands to.</p><p>Then add this code into the script function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7a7debfa88034e9a67e8c37d1d60c01/href">https://medium.com/media/e7a7debfa88034e9a67e8c37d1d60c01/href</a></iframe><p>This code will help us by logging all of the incoming data from Slack which will later help us build the staging table for the application. Write down the <strong>Path</strong> you created. The URL that you will give slack to post to will be like this:</p><blockquote>https//:<strong>&lt;instance name&gt;</strong>.service-now.com/<strong>&lt;Path&gt;</strong>.do</blockquote><blockquote><strong>For example:</strong> MyCompany.service-now/myPath.do</blockquote><p>if you want you can add a parameter. This will make the URL this instead:</p><blockquote>parameter = “hello”</blockquote><blockquote>https//:<strong>&lt;instance name&gt;</strong>.service-now.com/<strong>&lt;Path&gt;</strong>.do?<strong>&lt;parameter&gt;</strong></blockquote><blockquote><strong>For example:</strong> MyCompany.service-now/myPath.do?hello</blockquote><p>If you add a paremeter, you are able to pass in values into using the URL. Do not share this URL. Now, click <em>Submit.</em></p><p>For more information on <a href="https://community.servicenow.com/thread/193515">processors</a></p><h3><strong>Step 2. Make the processor public</strong></h3><p>The processor will currently require credentials to be accessed, but Slack doesn’t have credentials. So, we will have to make it a public page. Type in <em>sys_public.list </em>into the search bar on the left and hit enter. It will bring you to a list of all the public pages for your instance. Click <em>New</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*VC_sjv8DDNLDBKtTqdmLog.png" /><figcaption>New Public Page</figcaption></figure><p>Enter just your <strong>Path</strong> into <strong>Page </strong>section. Do not add your entire URL. Click <em>Submit</em></p><h3><strong>Step 3. The Slash Command</strong></h3><p>Go back to your Slack App window. We are now going to get Slack talking to ServiceNow. Click <em>Slash Commands</em> under <em>Features</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*AhzAAG4puiIjlWkb0PUhDA.png" /></figure><p>You may have to enable Slash Commands. This is just authorizing your application to list for slash commands in your slack channel. Click <em>Create New Command.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iV0-D36iNHqayLbHfnhIZw.png" /><figcaption>Create New Command</figcaption></figure><p>For now, make your command <em>/test</em> and put the URL you created from the end of Step 3 into the <strong>Request URL</strong> field. I would also check the <strong>Escape channels, users, and links sent to your app </strong>box</p><p>Now, click <strong>Save</strong>.</p><h3><strong>Step 4. First Test</strong></h3><p>Go to your slack Channel and enter this command <em>/test data </em>and hit enter. Slack will now send a post to your processor’s URL endpoint, but it won’t show anything happening on the Slack side. There will be logs of all the headers, and parameters as well as the method type in ServiceNow.</p><p>Go to <em>System Logs &gt; System Log &gt; All </em>and search “slack” to find your logs.</p><p>We now have Slack talking to ServiceNow! Congratulations!</p><h3><strong>Step 5. Build the Staging Table</strong></h3><p>Using the logs from our previous test, we are now going to build our staging table.</p><p>Create a new table by going to <em>System Definition &gt; Tables </em>and clicking New. Give it a name like <em>Slash Commands. </em>I also autonumbered mine and created a menu for it. We will later get into the cleaning up the table &amp; menu and adding security.</p><p>Now from my logs, I am going to create dictionary entries for the following:</p><ul><li>response URL</li><li>user_id</li><li>team_id</li><li>text</li><li>channel_name</li><li>token</li><li>command</li><li>channel_id</li><li>user_name</li><li>team_domain</li><li>method</li><li>querystring</li><li>x-forwarded-host</li><li>accept</li><li>x-forwarded-proto</li><li>accept-encoding</li><li>content-type</li><li>host</li><li>x-forwarded-for</li><li>content-length</li><li>user-agent</li></ul><p>I used String or URL as a field type where appropriate. Name the <strong>Column Label </strong>the exact same as the parameter name. ServiceNow will append “u_” to the front of it in the <strong>Column Name </strong>field. It will also replace dashes (“-”) with underscores (“_”). We will have to address this in our processor script when we start to populate the table.</p><h3><strong>Step 6. Replacing the Processor Script</strong></h3><p>Replace all of the code in your processor’s script with this code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6f4cb5325b9a98ac05130806e4a8799/href">https://medium.com/media/f6f4cb5325b9a98ac05130806e4a8799/href</a></iframe><p>Now, all of this code could be put into a script include &amp; I would actually suggest you do that. For now, I won’t.</p><h3><strong>Step 7. Test the slash command again</strong></h3><p>Go back to your channel &amp; run your slash command. Then go back to the list view of the table you created in step 6. You should see a new record. Open up the record &amp; make sure all of the data is populated. You may have misspelled something the first time, just delete the dictionary entry and add a new one with the correct spelling. Rerun your slash command if this happens to confirm it works correctly</p><p>Congratulations! you now have a table which contains all of the information from the REST call.</p><h3><strong>Step 8. Responding/Running Server side actions</strong></h3><p>From this point, you can do anything with this data. Create, update, delete, send emails, start workflows, etc. However, I suspect that you will be likely to want to respond to the command with a slack message from ServiceNow. That is where <a href="http://joeyday.com/2015/03/02/integrating-servicenow-with-slack/">Joey Day’s tutorial</a> comes in handy.</p><p>Create a new business rule for your slack table. Make it an <strong>advanced</strong> business rule &amp; SET THE <strong>When</strong> FIELD TO <em>AFTER,</em> NOT <em>BEFORE</em>. You will get all sorts of weirdness if you run a business rule before insert on a REST call and its a pain to diagnose if you aren’t familiar with it. I would also add a filtered condition that searches for a matching command</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*O5982qIwVTMp1Pyo_OkUZg.png" /></figure><p>I combined/modified two of Joey’s scripts and added a bit of my own code to create this script which will allow us to respond:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b234f38b3088bff41e101f2df1ad43d/href">https://medium.com/media/1b234f38b3088bff41e101f2df1ad43d/href</a></iframe><p>If you were clever, you could create a new script include specifically for slash commands based off of Joey’s script include. <a href="https://api.slack.com/slash-commands">Here</a> would be a great place to start with that.</p><p>Now, rerun the slash command in your instance. If everything was done correctly, you should receive a response! Congratulations!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W3VjiuYncxof2xQEgQdXdw.png" /><figcaption>Response</figcaption></figure><h3><strong>Highly Recommended: Adding Some Security</strong></h3><p>When we created the processor &amp; made it public, we made it available for anyone with the URL, and I mean anyone, to access it. So, we will add a bit of security to this now. As we discovered before, when a slack app sends a slash command post to an endpoint it sends many pieces of data. One of them is the Token. The Token is a code that validates that the command was from your Slack application. This is a very important piece of information and should be kept secret.</p><p>I am not going to pretend that I am an expert in IT security. However, I do have a few suggestions based on Slack’s documentation, experience, &amp; common sense.</p><ol><li>Validate the REST Call using the token field</li></ol><p>Your command’s “token” field is a unique token from your application validating that it is from Slack. So, I would reject it with a 403 status &amp; log the contents for later viewing or have another table for forbidden access like a black hole of data to analyze later.</p><p>2. Reject all REST methods except POST</p><p>For this project, I consider all POSTs to be creating a record of a request.</p><p>3. Validate user_id &amp; team_id</p><p>I would highly suggest storing user_id &amp; team_ids as a field in user table &amp; group table respectively. This makes responses &amp; security easier. You could use these user_ids to identify what users are able or allowed to do.</p><p>4. Turn off web services for your staging table. There is sensitive information in there like your token.</p><p>5. Add ACLs to make sensitive fields only viewable by admin</p><p>6. Add Edge encryption for sensitive fields.</p><p>I truly believe that all data centers have some vulnerability. Its best to limit the possible damage &amp; spread of any infiltration by encrypting any sensitive data</p><p>7. Create records from any slash command, but only run on validated slash commands</p><p>8. Add ACLs to remove the ability for users to create new records</p><p>9. Make all fields populated by the REST call read only</p><p>10. Replace your code with Script Includes.</p><h3><strong>Conclusion</strong></h3><p>Slack is a really cool tool and building an integration with it &amp; ServiceNow opens the doors to many possibilities. This was a really fun project for me and I really hope you find it helpful &amp; useful</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecbddaa7a96a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>