<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Claudio Stamile on Medium]]></title>
        <description><![CDATA[Stories by Claudio Stamile on Medium]]></description>
        <link>https://medium.com/@claudiostamile?source=rss-230e695d0813------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZvQYcfOZ9jT_2miy.</url>
            <title>Stories by Claudio Stamile on Medium</title>
            <link>https://medium.com/@claudiostamile?source=rss-230e695d0813------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:35:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@claudiostamile/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A gentle Introduction to Large Language Models and Graphs]]></title>
            <link>https://medium.com/bip-xtech/a-gentle-introduction-to-large-lange-models-and-graphs-24c50ce2067c?source=rss-230e695d0813------2</link>
            <guid isPermaLink="false">https://medium.com/p/24c50ce2067c</guid>
            <category><![CDATA[graph-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Claudio Stamile]]></dc:creator>
            <pubDate>Thu, 17 Aug 2023 09:05:38 GMT</pubDate>
            <atom:updated>2025-01-09T14:51:31.390Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Gentle Introduction to Large Language Models and Knowledge Graphs</h3><h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFooIT71DD6hOOHseaUvmQ.png" /></figure><p>In the last few months, Large Language Models (LLM) and Generative AI seem to be the only topic the data science community covers. Of course, those models are quite interesting, and a lot of applications can be made in combination with other technologies. As you may understand from the title, in this article we will investigate the application of LLM to Graphs and Graphs Machine Learning (GML).</p><p>A comprehensive survey with all the possible applications of LLM to graphs is provided in the paper <em>Unifying Large Language Models and Knowledge Graphs: A Roadmap</em><strong><em> </em></strong>by Pan et al [1].</p><p>In the paper, the authors describe three main macro-areas where the combination of LLM with Graphs can be beneficial.</p><blockquote>1) KG-enhanced LLMs, which incorporate KGs during the pre-training and inference phases of LLMs, or for the purpose of enhancing understanding of the knowledge learned by LLMs;</blockquote><blockquote>2) LLM-augmented KGs, that leverage LLMs for different KG tasks such as embedding, completion, construction, graph-to-text generation, and question answering;</blockquote><blockquote>3) Synergized LLMs + KGs, in which LLMs and KGs play equal roles and work in a mutually beneficial way to enhance both LLMs and KGs for bidirectional reasoning driven by both data and knowledge</blockquote><p>In this article, we will try to build an example of the third point where, starting from a given graph, we can use LLM to generate all the Cypher queries needed in order to extract data from a Neo4j database. A nice overview of this feature on LangChain is available in a fantastic medium article [2]. Here we are going to understand/debug how this process is performed and how it is possible to generalize it using, eventually, other technologies (both for LLM and graph database side).</p><h3>Tools and techniques</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jz_ttG1k7_pUWWiL9iz8pQ.png" /></figure><p>The goal is then to build a pipeline, using Python, in order to generate Cypher queries capable of actually replying to specific questions using the knowledge base contained in the graph database. In order to do that we do not want to learn any strange Python library, our goal is to let the LLM do the complex stuff.</p><p>In this article, we will use three different tools. Python, of course, Neo4j as a graph database [3], and Google Codey [4] to generate the code from a given input. Before starting with a detailed discussion let’s introduce Google Codey.</p><blockquote>Codey belongs to the family of Google foundational coding models built on PaLM 2. Codey was fine-tuned on a large dataset of high-quality, permissively licensed code from external sources and includes support for 20+ coding languages, including Python, Java, Javascript, Go, and others. The Codey models have been used to enhance various kinds of software development-related tasks across various Google surfaces such as Colab, Android Studio, Google Cloud and Google Search. This has various benefits for developers such as improving coding speed, enhancing code quality and closing the skills gap between novice and expert developers. Some of the tasks that Codey can help with include and enable:</blockquote><blockquote><strong>Code completion:</strong> Codey suggests the next few lines based on the existing context of code.</blockquote><blockquote><strong>Code generation:</strong> Codey generates code based on natural language prompts from a developer.</blockquote><blockquote><strong>Code chat:</strong> Codey lets developers converse with a bot to get help with debugging, documentation, learning new concepts, and other code-related questions.</blockquote><p>Let’s start with the simple operation, create the Neo4j instance we are going to use in order to check and build the code. We used the (free) <a href="https://sandbox.neo4j.com/">Neo4j sandbox</a>. The sandbox allows you to create, in a few clicks, a new Neo4j instance. It is possible to select an empty instance or instances with some pre-loaded datasets. In this article, we will use the instance with the <strong>Twitch </strong>dataset. After a few seconds, the instance is generated and up and running allowing you to connect using Python or other programming languages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XE0dtgR3s0TWGRQ1nHK49g.png" /><figcaption>Example of credentials from the Neo4j sandbox</figcaption></figure><p>The easy part is completed with zero code let’s now deep dive into the code to actually use Google Codey to automatically query the database without any prior knowledge of the Cypher query language.</p><h3>Build the pipeline and write the code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FpR2h2WEQ4uJPw2HRezVKQ.png" /></figure><p>The steps of the process that we want to create are the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iTJ1upq9bk2L1aRtxCdG2g.png" /></figure><h4>Step 1 — Building the prompt for the schema (We are not using the LLM yet)</h4><p>We will start with the first step, generating the “context” prompt that will contain the information about the “structure” of the data graph database. In this process, we want to convert the entities (nodes) and the relationship (edges), along with their property, from a graph format to a “text” version. We can imagine this step as a “graph2text” process where the graph structure is described as text. In order to perform this task we use the class <a href="https://github.com/langchain-ai/langchain/blob/ee3d0513addae0680a30afa384431a032244e76b/langchain/graphs/neo4j_graph.py">Neo4jGraph</a> contained in the LangChain package.</p><pre>from typing import Any, Dict, List<br><br>node_properties_query = &quot;&quot;&quot;<br>CALL apoc.meta.data()<br>YIELD label, other, elementType, type, property<br>WHERE NOT type = &quot;RELATIONSHIP&quot; AND elementType = &quot;node&quot;<br>WITH label AS nodeLabels, collect({property:property, type:type}) AS properties<br>RETURN {labels: nodeLabels, properties: properties} AS output<br><br>&quot;&quot;&quot;<br><br>rel_properties_query = &quot;&quot;&quot;<br>CALL apoc.meta.data()<br>YIELD label, other, elementType, type, property<br>WHERE NOT type = &quot;RELATIONSHIP&quot; AND elementType = &quot;relationship&quot;<br>WITH label AS nodeLabels, collect({property:property, type:type}) AS properties<br>RETURN {type: nodeLabels, properties: properties} AS output<br>&quot;&quot;&quot;<br><br>rel_query = &quot;&quot;&quot;<br>CALL apoc.meta.data()<br>YIELD label, other, elementType, type, property<br>WHERE type = &quot;RELATIONSHIP&quot; AND elementType = &quot;node&quot;<br>RETURN &quot;(:&quot; + label + &quot;)-[:&quot; + property + &quot;]-&gt;(:&quot; + toString(other[0]) + &quot;)&quot; AS output<br>&quot;&quot;&quot;<br><br><br>class Neo4jGraph:<br>    &quot;&quot;&quot;Neo4j wrapper for graph operations.&quot;&quot;&quot;<br><br>    def __init__(<br>        self, url: str, username: str, password: str, database: str = &quot;neo4j&quot;<br>    ) -&gt; None:<br>        &quot;&quot;&quot;Create a new Neo4j graph wrapper instance.&quot;&quot;&quot;<br>        try:<br>            import neo4j<br>        except ImportError:<br>            raise ValueError(<br>                &quot;Could not import neo4j python package. &quot;<br>                &quot;Please install it with `pip install neo4j`.&quot;<br>            )<br><br>        self._driver = neo4j.GraphDatabase.driver(url, auth=(username, password))<br>        self._database = database<br>        self.schema = &quot;&quot;<br>        # Verify connection<br>        try:<br>            self._driver.verify_connectivity()<br>        except neo4j.exceptions.ServiceUnavailable:<br>            raise ValueError(<br>                &quot;Could not connect to Neo4j database. &quot;<br>                &quot;Please ensure that the url is correct&quot;<br>            )<br>        except neo4j.exceptions.AuthError:<br>            raise ValueError(<br>                &quot;Could not connect to Neo4j database. &quot;<br>                &quot;Please ensure that the username and password are correct&quot;<br>            )<br>        # Set schema<br>        try:<br>            self.refresh_schema()<br>        except neo4j.exceptions.ClientError:<br>            raise ValueError(<br>                &quot;Could not use APOC procedures. &quot;<br>                &quot;Please ensure the APOC plugin is installed in Neo4j and that &quot;<br>                &quot;&#39;apoc.meta.data()&#39; is allowed in Neo4j configuration &quot;<br>            )<br><br>    @property<br>    def get_schema(self) -&gt; str:<br>        &quot;&quot;&quot;Returns the schema of the Neo4j database&quot;&quot;&quot;<br>        return self.schema<br><br>    def query(self, query: str, params: dict = {}) -&gt; List[Dict[str, Any]]:<br>        &quot;&quot;&quot;Query Neo4j database.&quot;&quot;&quot;<br>        from neo4j.exceptions import CypherSyntaxError<br><br>        with self._driver.session(database=self._database) as session:<br>            try:<br>                data = session.run(query, params)<br>                return [r.data() for r in data]<br>            except CypherSyntaxError as e:<br>                raise ValueError(&quot;Generated Cypher Statement is not valid\n&quot; f&quot;{e}&quot;)<br><br>    def refresh_schema(self) -&gt; None:<br>        &quot;&quot;&quot;<br>        Refreshes the Neo4j graph schema information.<br>        &quot;&quot;&quot;<br>        node_properties = self.query(node_properties_query)<br>        relationships_properties = self.query(rel_properties_query)<br>        relationships = self.query(rel_query)<br><br>        self.schema = f&quot;&quot;&quot;<br>        Node properties are the following:<br>        {[el[&#39;output&#39;] for el in node_properties]}<br>        Relationship properties are the following:<br>        {[el[&#39;output&#39;] for el in relationships_properties]}<br>        The relationships are the following:<br>        {[el[&#39;output&#39;] for el in relationships]}<br>        &quot;&quot;&quot;</pre><p>This class is really simple. It runs some simple queries on the database in order to extract information about the schema. Running the class and the code, we get.</p><pre>n4jg = Neo4jGraph(&quot;neo4j://xx.xxx.xxx.xxx:7687&quot;,&quot;username&quot;, &quot;password&quot;)<br>n4jg.get_schema</pre><p>Url, username, and password are the ones visible on the screen of the sandbox. As a result, we get.</p><pre>Node properties are the following:<br>[{&#39;properties&#39;: [{&#39;property&#39;: &#39;createdAt&#39;, &#39;type&#39;: &#39;DATE_TIME&#39;}, {&#39;property&#39;: &#39;id&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;description&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;url&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;followers&#39;, &#39;type&#39;: &#39;INTEGER&#39;}, {&#39;property&#39;: &#39;total_view_count&#39;, &#39;type&#39;: &#39;INTEGER&#39;}], &#39;labels&#39;: &#39;Stream&#39;}, {&#39;properties&#39;: [{&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}], &#39;labels&#39;: &#39;Game&#39;}, {&#39;properties&#39;: [{&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}], &#39;labels&#39;: &#39;Language&#39;}, {&#39;properties&#39;: [{&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}], &#39;labels&#39;: &#39;User&#39;}, {&#39;properties&#39;: [{&#39;property&#39;: &#39;createdAt&#39;, &#39;type&#39;: &#39;DATE_TIME&#39;}, {&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;id&#39;, &#39;type&#39;: &#39;STRING&#39;}], &#39;labels&#39;: &#39;Team&#39;}]<br>Relationship properties are the following: []<br>The relationships are the following:<br>[&#39;(:Stream)-[:PLAYS]-&gt;(:Game)&#39;, &#39;(:Stream)-[:HAS_LANGUAGE]-&gt;(:Language)&#39;, &#39;(:Stream)-[:CHATTER]-&gt;(:Stream)&#39;, &#39;(:Stream)-[:HAS_TEAM]-&gt;(:Team)&#39;, &#39;(:Stream)-[:MODERATOR]-&gt;(:Stream)&#39;, &#39;(:Stream)-[:VIP]-&gt;(:Stream)&#39;, &#39;(:User)-[:CHATTER]-&gt;(:Stream)&#39;, &#39;(:User)-[:VIP]-&gt;(:Stream)&#39;, &#39;(:User)-[:PLAYS]-&gt;(:Game)&#39;, &#39;(:User)-[:HAS_LANGUAGE]-&gt;(:Language)&#39;, &#39;(:User)-[:MODERATOR]-&gt;(:Stream)&#39;, &#39;(:User)-[:HAS_TEAM]-&gt;(:Team)&#39;]</pre><p>The string obtained is simply the description of the Twitch Neo4j.</p><h4>Step 2 — Add to the prompt the NLP query and send it to the LLM (Start to use the LLM)</h4><p>The second step is still prompt engineering. In this step, the goal is to build the final prompt that will be “executed” by the LLM (Google Codey). In order to build the prompt we will use the <a href="https://github.com/langchain-ai/langchain/blob/ee3d0513addae0680a30afa384431a032244e76b/langchain/chains/graph_qa/prompts.py">CYPHER_GENERATION_TEMPLATE</a> variable defined in the LangChain source code. We can use this code in order to write the following function.</p><pre>build_qa = lambda schema, question: f&quot;&quot;&quot;Task:Generate Cypher statement to query a graph database.<br>Instructions:<br>Use only the provided relationship types and properties in the schema.<br>Do not use any other relationship types or properties that are not provided.<br>Schema:<br>{schema}<br>Note: Do not include any explanations or apologies in your responses.<br>Do not respond to any questions that might ask anything else than for you to construct a Cypher statement.<br>Do not include any text except the generated Cypher statement.<br><br>The question is:<br>{question}&quot;&quot;&quot;</pre><p>Using this function in combination with the one previously defined (for the schema generation) we get.</p><pre>n4jg = Neo4jGraph(&quot;neo4j://xx.xxx.xxx.xxx:7687&quot;,&quot;username&quot;, &quot;password&quot;)<br>prom = build_qa(n4jg.get_schema, &quot;the users who play the game <br>Call of Duty: Warzone&quot;)</pre><p>The execution of this code will give the prompt that will be sent to the LLM.</p><pre>Task:Generate Cypher statement to query a graph database.<br>Instructions:<br>Use only the provided relationship types and properties in the schema.<br>Do not use any other relationship types or properties that are not provided.<br>Schema:<br><br>        Node properties are the following:<br>        [{&#39;properties&#39;: [{&#39;property&#39;: &#39;createdAt&#39;, &#39;type&#39;: &#39;DATE_TIME&#39;}, {&#39;property&#39;: &#39;id&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;description&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;url&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;followers&#39;, &#39;type&#39;: &#39;INTEGER&#39;}, {&#39;property&#39;: &#39;total_view_count&#39;, &#39;type&#39;: &#39;INTEGER&#39;}], &#39;labels&#39;: &#39;Stream&#39;}, {&#39;properties&#39;: [{&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}], &#39;labels&#39;: &#39;Game&#39;}, {&#39;properties&#39;: [{&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}], &#39;labels&#39;: &#39;Language&#39;}, {&#39;properties&#39;: [{&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}], &#39;labels&#39;: &#39;User&#39;}, {&#39;properties&#39;: [{&#39;property&#39;: &#39;createdAt&#39;, &#39;type&#39;: &#39;DATE_TIME&#39;}, {&#39;property&#39;: &#39;name&#39;, &#39;type&#39;: &#39;STRING&#39;}, {&#39;property&#39;: &#39;id&#39;, &#39;type&#39;: &#39;STRING&#39;}], &#39;labels&#39;: &#39;Team&#39;}]<br>        Relationship properties are the following:<br>        []<br>        The relationships are the following:<br>        [&#39;(:Stream)-[:PLAYS]-&gt;(:Game)&#39;, &#39;(:Stream)-[:HAS_LANGUAGE]-&gt;(:Language)&#39;, &#39;(:Stream)-[:CHATTER]-&gt;(:Stream)&#39;, &#39;(:Stream)-[:HAS_TEAM]-&gt;(:Team)&#39;, &#39;(:Stream)-[:MODERATOR]-&gt;(:Stream)&#39;, &#39;(:Stream)-[:VIP]-&gt;(:Stream)&#39;, &#39;(:User)-[:CHATTER]-&gt;(:Stream)&#39;, &#39;(:User)-[:VIP]-&gt;(:Stream)&#39;, &#39;(:User)-[:MODERATOR]-&gt;(:Stream)&#39;, &#39;(:User)-[:PLAYS]-&gt;(:Game)&#39;, &#39;(:User)-[:HAS_LANGUAGE]-&gt;(:Language)&#39;]<br>        <br>Note: Do not include any explanations or apologies in your responses.<br>Do not respond to any questions that might ask anything else than for you to construct a Cypher statement.<br>Do not include any text except the generated Cypher statement.<br><br>The question is:<br>the users who play the game Call of Duty: Warzone</pre><p>Finally the prompt is generated, let’s now execute. The idea is to build a class capable to: <em>i)</em> Connect to the the Codey environment on the Google Cloud Platform (GCP), <em>ii)</em> Send the generated prompt to the LLM and execute it. The code to perform this action is defined as follow</p><pre>class CodyGraphPrompt:<br><br>  def __init__(self, neo4j_connector: Neo4jGraph, project_name: str, <br>               location: str,<br>               model_name: str = &quot;codechat-bison@001&quot;, parameters:<br>               Dict = {&quot;temperature&quot;: 0.2, &quot;max_output_tokens&quot;: 1024}):<br>    vertexai.init(project=project_name, location=location)<br>    self.chat_model = CodeChatModel.from_pretrained(model_name)<br>    self.parameters = parameters<br>    self.neo4j_connector = neo4j_connector<br><br>  def prompt_builder(self, schema: str, question: str) -&gt; str:<br>    return f&quot;&quot;&quot;Task:Generate Cypher statement to query a graph database.<br>    Instructions:<br>    Use only the provided relationship types and properties in the schema.<br>    Do not use any other relationship types or properties that are not provided.<br>    Schema:<br>    {schema}<br>    Note: Do not include any explanations or apologies in your responses.<br>    Do not respond to any questions that might ask anything else than for you to construct a Cypher statement.<br>    Do not include any text except the generated Cypher statement.<br>    <br>    The question is:<br>    {question}&quot;&quot;&quot;<br><br>  def run_nlp_query(self, prompt: str) -&gt; str:<br>    chat = self.chat_model.start_chat()<br>    response = chat.send_message(<br>               self.prompt_builder(self.neo4j_connector.get_schema, prompt,<br>               **self.parameters)<br>    return response.text.split(&quot;```&quot;)[1]</pre><p>It is possible to use the previous class in the following way:</p><pre>n4jg = Neo4jGraph(&quot;neo4j://xx.xxx.xxx.xxx:7687&quot;,&quot;username&quot;, &quot;password&quot;)<br>asd = CodyGraphPrompt(n4jg, &quot;google_project_name&quot;, &quot;us-central1&quot;)<br>asd.run_nlp_query(&quot;the users who play the game Call of Duty: Warzone&quot;)</pre><p>As results we get the Cypher query created by the the LLM model.</p><pre>\nMATCH (user:User)-[:PLAYS]-&gt;(game:Game)\nWHERE game.name = <br>&quot;Call of Duty: Warzone&quot;\nRETURN user.name;\n</pre><h4>Step 3 — Execute the query to the database and return the results</h4><p>The last, and the easiest step, is to execute the result of the of the model, this step can be easily done by modifying the <em>run_nlp_query </em>function as follows:</p><pre>def run_nlp_query(self, prompt: str) -&gt; str:<br>  chat = self.chat_model.start_chat()<br>  response = chat.send_message(self.prompt_builder(<br>             self.neo4j_connector.get_schema, prompt), <br>             **self.parameters)<br>  return n4jg.query(response.text.split(&quot;```&quot;)[1])</pre><p>by running the new function we get, as a result the list of users that hold the given query.</p><pre>[{&#39;user.name&#39;: &#39;montanablack88&#39;}, {&#39;user.name&#39;: &#39;danucd&#39;}, ....]</pre><h3>Conclusion and Next Article</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ru4fiqoPgYMQFSkeycB1mw.png" /></figure><p>In this first article, we discussed a simple way to build Cypher query using Google Codey. We extensively used and adapted the code of LangChain in order to simplify the understanding of the whole pipeline and the (simple) process under the hood. The notebook with the code discussed in this article is available on <a href="https://drive.google.com/file/d/1w86-470W7fY1jAqZUaAqZvVwMCQQM7iU/view?usp=sharing">Colab</a>.</p><p>In the next series of articles about the topic LLM + Graph, we will show how LLM can boost Graph Machine Learning.</p><h3>References</h3><p>[1] Unifying Large Language Models and Knowledge Graphs: A Roadmap. <a href="https://arxiv.org/pdf/2306.08302.pdf">https://arxiv.org/pdf/2306.08302.pdf</a></p><p>[2] LangChain has added Cypher search. <a href="https://towardsdatascience.com/langchain-has-added-cypher-search-cb9d821120d5">https://towardsdatascience.com/langchain-has-added-cypher-search-cb9d821120d5</a></p><p>[3] Neo4j graph database, <a href="https://neo4j.com/">https://neo4j.com/</a></p><p>[4] Google Codey, <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview">https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24c50ce2067c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bip-xtech/a-gentle-introduction-to-large-lange-models-and-graphs-24c50ce2067c">A gentle Introduction to Large Language Models and Graphs</a> was originally published in <a href="https://medium.com/bip-xtech">BIP xTech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Energy consumption profile clustering using atoti]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/cgnal-tech/energy-consumption-profile-clustering-using-atoti-667ed4a391c?source=rss-230e695d0813------2"><img src="https://cdn-images-1.medium.com/max/2600/1*AoZdd70sphILdmGvO7b4dw.png" width="3629"></a></p><p class="medium-feed-snippet">Use atoti for  energy consumption clustering</p><p class="medium-feed-link"><a href="https://medium.com/cgnal-tech/energy-consumption-profile-clustering-using-atoti-667ed4a391c?source=rss-230e695d0813------2">Continue reading on cgnal-tech »</a></p></div>]]></description>
            <link>https://medium.com/cgnal-tech/energy-consumption-profile-clustering-using-atoti-667ed4a391c?source=rss-230e695d0813------2</link>
            <guid isPermaLink="false">https://medium.com/p/667ed4a391c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[atoti]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Claudio Stamile]]></dc:creator>
            <pubDate>Thu, 25 Feb 2021 10:02:39 GMT</pubDate>
            <atom:updated>2021-02-25T10:02:39.558Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Unsupervised Keyword-based Topic Assignment in Python using Word Embedding]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/cgnal-tech/unsupervised-keyword-based-topic-assignment-in-python-using-word-embedding-801f5c0ae5d1?source=rss-230e695d0813------2"><img src="https://cdn-images-1.medium.com/max/1787/1*TFxYVGYXAQFnjbZnIB0SqQ.png" width="1787"></a></p><p class="medium-feed-snippet">Topic classification is one of the most known problem in natural language processing (NLP). The main goal of this task is to assign to a&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/cgnal-tech/unsupervised-keyword-based-topic-assignment-in-python-using-word-embedding-801f5c0ae5d1?source=rss-230e695d0813------2">Continue reading on cgnal-tech »</a></p></div>]]></description>
            <link>https://medium.com/cgnal-tech/unsupervised-keyword-based-topic-assignment-in-python-using-word-embedding-801f5c0ae5d1?source=rss-230e695d0813------2</link>
            <guid isPermaLink="false">https://medium.com/p/801f5c0ae5d1</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Claudio Stamile]]></dc:creator>
            <pubDate>Thu, 03 Dec 2020 09:46:12 GMT</pubDate>
            <atom:updated>2020-12-03T09:46:12.951Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Tensor Factorization for Graph Analysis in Python]]></title>
            <link>https://medium.com/yadb/tensor-factorization-for-graph-analysis-in-python-590df44c9f6c?source=rss-230e695d0813------2</link>
            <guid isPermaLink="false">https://medium.com/p/590df44c9f6c</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Claudio Stamile]]></dc:creator>
            <pubDate>Mon, 28 Jan 2019 10:56:48 GMT</pubDate>
            <atom:updated>2019-01-28T13:33:06.202Z</atom:updated>
            <content:encoded><![CDATA[<p>We already know, machine learning is an amazing field and a large number of techniques exist in order to solve problems. Unfortunately, just a few subsets of algorithms are very popular and largely used in <a href="http://kaggle.com/">Kaggle competitions</a>. But, since the world is a large place, a lot of powerful, and not largely used algorithms/techniques, exist.</p><p>In this article, we will show a simple application of the Tensor Decomposition (TD) algorithm on graph data. More in details, we will show how to apply TF to analyze time changing graphs. The goal of this analysis is to extract latent information from graphs in order to find hidden information.</p><p>This article is divided into two main parts. In the first, we will give a quick overview of the tensor factorization algorithm. In the second, we will show how to use this technique on real data with Python.</p><h3>Tensor Factorization</h3><p>We start by the simple definition of a tensor. In this article, we will use different definitions extracted from the well-known paper on tensor decomposition: <a href="https://epubs.siam.org/doi/pdf/10.1137/07070111X">Kolda et. al, <strong>Tensor Decompositions and Applications</strong>, SIAM Review, 2009, Vol. 51(3): pp. 455–500.</a></p><blockquote>A tensor is a multidimensional array. More formally, an N-way or Nth-order tensor is an element of the tensor product of N vector spaces,<br>each of which has its own coordinate system. A third-order tensor has three indices, as shown in Figure 1. A first-order tensor is a vector, a second-order tensor is a matrix, and tensors of order three or higher are called higher-order tensors. (<a href="https://epubs.siam.org/doi/pdf/10.1137/07070111X">Kolda et. al, <strong>Tensor Decompositions and Applications</strong></a><strong>)</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*tt_-76Mqxfl96wysIaVJLg.png" /><figcaption><strong>Figure 1. </strong>A tensor</figcaption></figure><p>The goal of tensor decomposition is to obtain a compact representation of a given tensor. One of the first types of tensor factorization is the canonical polyadic decomposition (CPD). This decomposition factorizes a tensor into a sum of component rank-one tensors as described in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*7mXhSHKPE5QJJDV_UU1lbg.png" /><figcaption><strong>Figure 2</strong>. Non-Negative Canonical Polyadic Decomposition (NCPD) for Tensor Factorization</figcaption></figure><p>In Figure 2, we introduced a particular type of tensor decomposition, the Non-Negative Canonical Polyadic Decomposition (NCPD). The only difference with “classical” CPD is related to the non-negative constraints for the extracted factors.</p><p>One of the most important point to take into account from Figure 2 is the <em>Rank </em>(R) of the factorization.</p><blockquote>The rank of a tensor T, denoted rank(T), is defined as the smallest number of rank-one tensors that generate T as their sum. In other words, this is the smallest number of components in an exact CP decomposition, where “exact” means that there is equality in the factorization and the original tensor. An exact CP decomposition with R = rank(T) components is called the rank decomposition. (<a href="https://epubs.siam.org/doi/pdf/10.1137/07070111X">Kolda et. al, <strong>Tensor Decompositions and Applications</strong></a><strong>)</strong></blockquote><h3>Tensorize and Factorize Graphs with Python</h3><p>The data used in this article were selected among a large series of network datasets available at the <a href="https://icon.colorado.edu/#!/networks">Index of Complex Networks</a> website. We selected the <em>Golden Age of Hollywood </em>dataset, here its description extracted from the website:</p><blockquote>A network of all collaborations among 55 actors who were prominent during the Golden Age of Hollywood (1930–1959). Nodes are actors and each edge represents those actors appearing together in a movie. Edge direction points toward the actor with “better” billing status. Collaborations were extracted from the Internet Movie Database and span 1909–2009. Data available as both a temporal network (sequence of graphs, each spanning 10 years) and as single aggregated network.</blockquote><p>All graphs, except the aggregated network, were used during the analysis. As you already know a graph can be represented with an<a href="https://en.wikipedia.org/wiki/Adjacency_matrix"> adjacency matrix</a>. Using its matrix representation we can then built a tensor (3D matrix) where we stacked all the matrix along the 3rd mode as described in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*Twag50BqKJEnHYh4J3tXCg.png" /><figcaption><strong>Figure 3. </strong>The stack<strong> </strong>of adjacency matrix along the 3rd mode of the tensor</figcaption></figure><p>The first step is to load all the adjacency matrices in a 3D matrix using the following python code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z8stBDBPg_APiGXAZWOHHw.png" /><figcaption><strong>Figure 4. </strong>Load all matrices and stack in a 3D matrix</figcaption></figure><p>Before stacking the matrices we remove the self-loop in the graph simply putting zero on the diagonal. In the last step, in order to simplify the visualization, we loaded the file containing the names of all the Hollywood actors available in the dataset.</p><p>We are now able to start using tensor factorization. <a href="http://tensorly.org/stable/modules/api.html#module-tensorly.base">Tensorly</a> is the best choice to perform tensor factorization in Python since it offers a wide range of functions. Another incredible tool is <a href="https://www.tensorlab.net/">Tensorlab</a>, unfortunately, it is only available for MATLAB and will not be used in this article. Before running the factorization algorithm we need to transform the 3D numpy matrix in a tensor that Tensorly can process using the following code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2HblMiCqDG5oyEmvKqOz2w.png" /><figcaption><strong>Figure 5. </strong>Conversion from numpy matrix to tensorly tensor</figcaption></figure><p>Now we can perform the non-negative tensor factorization. In this article, we decided to extract 20 components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E98_PK8JryqbEh07n2JBKQ.png" /><figcaption><strong>Figure 6.</strong> Code to perform non-negative tensor factorization</figcaption></figure><p>In the factors array, we have all the factors extracted from the factorization. Let’s start an exploratory analysis in order to check, what we can see in the different components. We will just focus on a few components in order to describe a general way to perform the analysis.</p><p>In order to perform the plot, we used <a href="https://matplotlib.org/">matplotlib</a> and <a href="https://networkx.github.io/">NetworkX</a>. More in detail, we plot two different information: 1) The vector product between A and B. This information shows a subgraph of the original graph describing the sub connections more relevant for this specific source. 2) The factor C. This factor describes which temporal period a specific component contains.</p><p>The main idea behind these plots is to understand which subset of relevant information can be extracted from the original dataset. The python code to perform the visualization is available in the following image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MAN4OyuFFOad62dhHJyFmw.png" /><figcaption><strong>Figure 7. </strong>Code to plot the A times B component and C.</figcaption></figure><p>Here the plots</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6sPkfSZfOFw49TyojMcR4Q.png" /><figcaption><strong>Figure 8. </strong>Information obtained from component #0.<strong> Top. </strong>The plot of factor C.<strong> Bottom. </strong>The graph obtained from the product of A and B factors.</figcaption></figure><p>Now, let’s try to understand the meaning of those plots. Apparently, component #0 contains information about the filmography of Marx brothers. Indeed, if we focus our analysis on the plot of the product of factor A and B (Botton of Figure 8) we can see that the obtained graph contains the connection between the 4 Marx brothers. Moreover, if we analyze the component C we can see values different from zero in the range 1920–1959 with a spike in 1930–1939. This plot perfectly overlaps with their filmography as you can see in Figure 9. From 17 movies, 2 (11.8%) belong to 1920–1929, 9 (52.9%) belong to 1930–1939, 4 (23.5%) belong to 1940–1949 and 2 (11.8%) belong to 1950–1959.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9vBQI_di9ZV7ViJvbxD4Tg.png" /><figcaption><strong>Figure 9. </strong>From <a href="https://en.wikipedia.org/wiki/Marx_Brothers">Wikipedia</a> the filmography of Marx brothers.</figcaption></figure><p>Another interesting component is the #4. Plots are visible in the following image</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLJq9e04AdsHpIkr4-VXWQ.png" /><figcaption><strong>Figure 10.</strong> Information obtained from component #4.<strong> Top. </strong>The plot of factor C.<strong> Bottom. </strong>The graph obtained from the product of A and B factors.</figcaption></figure><p>This component is also significant since represents one of the most celebrated and successful collaborations of any actor/director, the one between Alfred Hitchcock and Cary Grant, visible in bottom part of Figure 10. If we also look at the Top part of the figure, we can see the same value in the range 1940–1949 and 1950–1959. This is a really amazing fact since this duo made 4 movies, 2 in 1940–1949 (“Suspicion” — 1941 and “Notorious” — 1949) and 2 in 1950–1959 (“To Catch a Thief” — 1941 and “North by Northwest” — 1949).</p><h3>Conclusions</h3><p>In this article, we presented a simple approach to analyzing longitudinal changing networks using non-negative tensor factorization. We started with a simple introduction to tensors and tensor factorization. More in details, we used a particular type of tensor factorization, namely Non-Negative Canonical Polyadic Decomposition (NCPD). The decomposition was applied to analyze time changing graphs obtained from the <a href="https://icon.colorado.edu/#!/networks">Index of Complex Networks</a> website. Analyzing different component we found interesting hidden information available in the dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=590df44c9f6c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yadb/tensor-factorization-for-graph-analysis-in-python-590df44c9f6c">Tensor Factorization for Graph Analysis in Python</a> was originally published in <a href="https://medium.com/yadb">YADB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPS Trajectories Clustering in Python]]></title>
            <link>https://medium.com/isiway-tech/gps-trajectories-clustering-in-python-2f5874204a53?source=rss-230e695d0813------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f5874204a53</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[gps-tracker]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Claudio Stamile]]></dc:creator>
            <pubDate>Fri, 07 Dec 2018 10:09:47 GMT</pubDate>
            <atom:updated>2018-12-07T10:09:47.968Z</atom:updated>
            <content:encoded><![CDATA[<p>GPS trajectories clustering is a common analysis to perform when we want to exploit GPS data generated by personal devices like smartphones or smartwatches.</p><p>In this article we will describe a fast and easy way to perform GPS trajectories clustering in Python. Main goal here is to create clusters containing “similar” trajectories. For instance, we want to put in the same cluster the path we follow each day from work to home.</p><p>This article is divided in two main parts. In the first, we will describe the clustering algorithm. In the second, we will show how to use and customize the algorithm in Python.</p><h3>Neuroimage Algorithms and GPS Trajectories Clustering</h3><p>Instead using classical clustering algorithms like K-Means or DBSCAN, in this article we use a clustering algorithm used in neuromaging.</p><p><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3518823/pdf/fnins-06-00175.pdf">QuickBundles (QB)</a> is a simple compact clustering algorithm used in magnetic resonance imaging to cluster white matter fibers obtained from the application of <a href="https://en.wikipedia.org/wiki/Tractography">tractography </a>algorithms.</p><p>Just by looking at the following image, we can see how the white matter fibers in our brain look like GPS trajectories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B3xA2ewbRldYmNS9uLekrQ.png" /><figcaption>Example of white matter fibers obtained after application of tractography algorithm. Image from <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3518823/pdf/fnins-06-00175.pdf">Garyfallidis et al. Frontiers In Neuroscience, vol 6, 2016</a>.</figcaption></figure><p>The main idea is to treat each GPS trajectory as a white matter fiber and then merge “similar” trajectories in the same cluster. In the rest of this article we will assume <em>GPS Trajectory = White Matter Fiber</em>.</p><p>With this assumption we can then use the description of the algorithm available in the original paper:</p><blockquote>The algorithm proceeds as follows. At any one step in the algorithm we have <strong>M </strong>clusters. Select the first streamline <strong>s1 </strong>and place it in the first cluster <strong>c1</strong> ←({1}, s1, 1); M = 1 at this point. For each remaining streamline in turn <strong>i</strong> = 2, . . ., N:</blockquote><blockquote>(i) calculate the distance between streamline <strong>si </strong>and the centroid streamlines <strong>ve </strong>of all the current clusters <strong>ce </strong>, <strong>e</strong> = 1, . . ., M, where <strong>v</strong> is defined on the fly as <strong>v = h/n</strong>;</blockquote><blockquote>(ii) if any of the distance values <strong>me </strong>are smaller than a clustering threshold <strong>θ</strong>, add streamline<strong> i</strong> to the cluster <strong>e</strong> with the minimum value for <strong>me </strong>; <strong>ce = (I, h, n)</strong>, and update <strong>ce </strong>←(append(I, i), h + s, n + 1); otherwise create a new cluster <strong>cM+1</strong> ←([i], si , 1), M ←M + 1. (Page 5 — <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3518823/pdf/fnins-06-00175.pdf">Garyfallidis et al. Frontiers In Neuroscience, vol 6, 2016</a>)</blockquote><p>In the following image we show an example how the algorithm merges the different streamlines in common centroids according to a given threshold value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*0vToDk1Y9C6vmZAuc891TA.png" /><figcaption>QuickBundle centroids with different thresholds. Image from <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3518823/pdf/fnins-06-00175.pdf">Garyfallidis et al. Frontiers In Neuroscience, vol 6, 2016</a>.</figcaption></figure><p>The threshold value is <strong>THE </strong>parameter to select in order to tune the behaviour of the clustering algorithm. If you want the “big” trajectories you can set high values for the threshold. Otherwise, if you want small clusters you need low values.</p><p>The authors of the paper provided a Python implementaton of the proposed method. The algorithm is available as part of <a href="http://nipy.org/dipy/">dipy </a>library while the documentation of the algorithm is available <a href="http://nipy.org/dipy/examples_built/segment_quickbundles.html#">here</a>.</p><h3>GPS Trajectory Clustering</h3><p>The dataset we used comes from the GeoLife GPS Trajectories Dataset released by Microsoft Research Asia and available <a href="https://www.microsoft.com/en-us/download/details.aspx?id=52367">here</a>. A nice overview of the dataset, with examples describing how to open and use it, is available <a href="https://yidatao.github.io/2016-12-23/geolife-dbscan/">here</a>.</p><p>Before starting to perform clustering, let’s plot all the trajectories on google maps using <a href="https://github.com/vgm64/gmplot">gmplot</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0zg5Ox0c-n2GOTB-F1RP5A.png" /><figcaption>GPS trajectories in dataset</figcaption></figure><p>We can now start by defining the distance function between two GPS trajectories (streamlines). Instead using classic Euclidean distance, available in QuickBundle, we will use the GPS distance defined in the <a href="https://geopy.readthedocs.io/en/stable/">GeoPy </a>library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2auMJN3e8dk24U2mtcJjpg.png" /><figcaption>Code to compute the average pointwise distance in GPS trajectories</figcaption></figure><p>We computed the average pointwise GPS distance between two trajectories. This way to compute the distance can be used if and only if the two trajectories have the same number of points, this is why we resampled all the trajectories using the ResampleFeature class.</p><p>Once the distance between two trajectories is defined, we can run the QuickBundle clustering algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pv0oCR0TbO33ZWXNN7EzIg.png" /><figcaption>Code to run the trajectory clustering</figcaption></figure><p>We can then plot the trajectories contained in the different clusters on google maps using, as before, gmplot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_uSVuHUrTjOviN0FBCd0A.png" /><figcaption>Clustering plot using gmplot</figcaption></figure><p>Here the result in <em>my_map.html </em>plot for different clusters</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*dRDc3B3gWKhqvK8g9Tl1bg.png" /><figcaption>Cluster #0</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5PpB-mIfxr8UR5KfBO1hbA.png" /><figcaption>Cluster #2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ON0BQgel8hx_MQ0zjpTfzQ.png" /><figcaption>Cluster #30</figcaption></figure><h3>Conclusions</h3><p>In this article we described a simple and fast way to perform trajectories clustering of GPS data. The goal was achieved using QuickBundles, a clustering algorithm applied in neuroimaging.</p><p>Main limitation of this algorithm is related to the tuning of the threshold parameters. But, as everything in data analysis, this parameter needs to be selected according to the type of clusters you want.</p><blockquote><strong><em>P.S</em></strong><em> We’re hiring, if you like us, we also like you. If you are interested, check our job opportunities </em><a href="https://humanativaspa.it/job-opportunity-specialists-in-erp-application-data-analytics/"><em>here</em></a><em>.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f5874204a53" width="1" height="1" alt=""><hr><p><a href="https://medium.com/isiway-tech/gps-trajectories-clustering-in-python-2f5874204a53">GPS Trajectories Clustering in Python</a> was originally published in <a href="https://medium.com/isiway-tech">isiway-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seq2Seq in Keras for Petrol Price Prediction using Italian Open Data]]></title>
            <link>https://medium.com/isiway-tech/seq2seq-in-keras-for-petrol-price-prediction-using-italian-open-data-767cb1016af3?source=rss-230e695d0813------2</link>
            <guid isPermaLink="false">https://medium.com/p/767cb1016af3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Claudio Stamile]]></dc:creator>
            <pubDate>Thu, 22 Nov 2018 08:30:36 GMT</pubDate>
            <atom:updated>2018-11-22T08:30:36.494Z</atom:updated>
            <content:encoded><![CDATA[<p>As you may know in Italy the price per liter of petrol is really high compared to the rest of Europe. In the following image you can see Italy is in the Top 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*H6ZNIXdbA_V1Fl5MaZEYWw.png" /><figcaption>Image from <a href="https://autotraveler.ru/en/spravka/fuel-price-in-europe.html#.W-xADuhKhPY">https://autotraveler.ru/en/spravka/fuel-price-in-europe.html#.W-xADuhKhPY</a> 14/11/2018</figcaption></figure><p>Unfortunatly, Italy is not in the Top 4 in the list of European countries by average wage sorted by purchasing power parity (PPP). So it is good to find a method to help people to save the money they burn in their cars every day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ETZ-gJtaXSz22drvqj1mbg.png" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/List_of_European_countries_by_average_wage">https://en.wikipedia.org/wiki/List_of_European_countries_by_average_wage</a> 14/11/2018</figcaption></figure><p>In this article we show how to use Machine Learning algorithms, in particular Seq2Seq Neural Networks, to predict petrol price at self service stations in Italy.</p><p>This article is divided in two main parts. In the first, we will perform some nice plots using Qlik Sense. In the second, we will show how to use Seq2Seq neural network to perform petrol price prediction.</p><h3>Italian Open Data</h3><p>In the last years, italian public administration put a lot of effort in the Open Data initiative. A nice overview of the project is available on the <a href="https://www.dati.gov.it/">official website</a> while, a collection of links with the most interesting datasets is available <a href="https://github.com/italia/awesome-italian-public-datasets">here</a>.</p><p>In order to write this post, we decided to use the dataset containing daily information of the petrol price for all the italian gas stations created by the ministry of economic development and available <a href="https://www.sviluppoeconomico.gov.it/index.php/it/open-data/elenco-dataset/2032336-carburanti-prezzi-praticati-e-anagrafica-degli-impianti">here</a>. In this dataset, all the quarters (trimesti) from 2015 to 2018 are available. For each quarter it is possibile to download its corresponding compressed directory. Each directory contains one .<em>csv </em>per day in the quarter. Each <em>.csv</em> file contains the price of the petrol and gasoline for all the gas stations available in that period. In the same link, the registry of each gas station, containing its description, is also available. From this file we can extract two important information: 1) municipality identifier and GPS coordinates and 2) the gasoline marketer brand for a specific gas station.</p><p>In order to merge all the price data, we created a big single file using the following bash command.</p><pre>awk &#39;NR &gt;1 {print $0 &quot;;&quot; substr(FILENAME,15,8)}&#39; prezzo*.csv &gt;&gt; all.csv</pre><p>Now that we have all the data we need, we can start to perfom our analysis.</p><h3>Visualization using Qlik Sense Desktop</h3><p>In order to obtain a nice geographical display of our data we used <a href="https://www.qlik.com/">Qlik Sense Desktop</a>. In the dataset, for each gas station geographical information about its municipality are available. Those information can be used to create a map that can help us to understand the gas price in specific municipalities in Italy.</p><p>Unfortunately, Qlik Sense does not provide, at least for Italy, geographical areas at the “municipality” level but only at “region” level. In order to solve this problem we need to import <a href="https://developers.google.com/kml/">KML</a> data containing the municipality borders. Luckily for us, <a href="https://localmapping.wordpress.com/">Rocco Giove</a> created a set of KML files with the <a href="https://localmapping.wordpress.com/2008/11/20/i-confini-amministrativi-degli-8101-comuni-ditalia-al-2001/">municipality border divided by region</a>.<br>In this part of the article we focus or analysis just for the Lazio region, so, in order to perform our plot, we imported the KML file just for that region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FnfPUMMLmG0I7jy0PgRlzw.png" /><figcaption>Municipalities in Lazio</figcaption></figure><p>Since we want a global overview of the last prices, we are going to plot just the last prices of the gas stations in all the municipalities in Lazio region. We imported the price and the description just for the last available day in the dataset: 2018–06–30.</p><blockquote>In this analysis we are not looking at the price for the single gas station, we just aggregate at the municipality level. In another post we will perform a specific analysis at the gas station level of granularity.</blockquote><p>All the three datasets (municipalities borders, price per gas station and gas station description) need to be joined. We used the following join keys:</p><p>KML File (<strong>Lazio.Name</strong>) — Anagrafica Impianti (<strong>Comune</strong>)<br>Anagrafica Impianti (<strong>idImpianto</strong>) — Prezzo Alle (<strong>idImpiato</strong>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FzP5c5PXo_6dClbpTskaDA.png" /><figcaption>Link between the different data sources</figcaption></figure><p>Here we show some plots (with the associated query) we did in order to show different information about the petrol price for each municipality. Let’s start by looking at the average petrol price only at the Self-Service using the following query</p><pre>=Avg(if(isSelf &gt; 0 and descCarburante=&#39;Benzina&#39;, prezzo))</pre><p>and as result we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R2dNf9c1qlJOVd4dKsllKg.png" /></figure><p>We can also count the number of distinct gasoline marketers for each municipality</p><pre>=Count(distinct(if(isSelf &gt; 0 and descCarburante=&#39;Benzina&#39;, Bandiera))</pre><p>as expected we see that in Rome we have 20 distinct gasoline marketers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CPCXT3T_YXdQ3RGEe0wacg.png" /><figcaption>Number of distinct gasoline marketers</figcaption></figure><p>Now we check if the are differences in price between branded (B) and not branded (NB) gasoline marketers.</p><pre>=Avg(if(isSelf &gt; 0 and not Bandiera = &#39;Pompe Bianche&#39; and descCarburante=&#39;Benzina&#39;, prezzo))<br>=Avg(if(isSelf &gt; 0 and Bandiera = &#39;Pompe Bianche&#39; and descCarburante=&#39;Benzina&#39;, prezzo))<br>==Avg(if(isSelf &gt; 0 and not Bandiera = &#39;Pompe Bianche&#39; and descCarburante=&#39;Benzina&#39;, prezzo)) - Avg(if(isSelf &gt; 0 and Bandiera = &#39;Pompe Bianche&#39; and descCarburante=&#39;Benzina&#39;, prezzo))</pre><p>We get the following map</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrt7fM1jLIajc-sidY1qxQ.png" /><figcaption>Average price of Branded (B) and Not Branded (NB) gasoline marketers and their differences</figcaption></figure><p>In this section we just showed a small number of examples to use Qlik with geographical data to exploit the dataset. Of course we can add more cool analysis but this post will be a <strong>TL;DR</strong> post.</p><h3>Seq2Seq Neural Network in Keras</h3><p>In order to predict the price of petrol for each gas station Seq2Seq neural nerworks were used.</p><p>But why Seq2Seq neural networks and not classical ARIMA models for time series forecasting ?</p><p>In the subset of data we used for this analysis, 27042 gas stations were available. In this case, application of ARIMA models is not trivial. Indeed, in order to predict the price for each station, we need to fit and perform parameter optimization of different ARIMA models, one for each gas station. This can be difficult to perform in real scenario.</p><p>Seq2Seq neural networks instead, can learn how to reproduce different type of sequences. Using this approach we can use just one single Seq2Seq neural network to fit all the time series together.</p><p>Before going deep with the neural network we need to load the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4nkUurtsVghU_xToeuj7ew.png" /><figcaption>Subset of dataset used for the analysis</figcaption></figure><p>We then create a table containing, for each gas station the petrol price for each day in the dataset. Since for certain gas station we have NaN values (for instance a gas station was not present in that period) we decided to fill those values with the mean price of all the gas stations for that day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tvu-RTMtTx8DbLojQDL9XA.png" /><figcaption>Shape of the dataset after pivoting</figcaption></figure><p>The dataset is now ready and can be used in a Seq2Seq neural network.</p><p>In this article we used the Seq2Seq neural network described by Joseph Eddy and available <a href="https://github.com/JEddy92/TimeSeries_Seq2Seq/blob/master/notebooks/TS_Seq2Seq_Conv_Intro.ipynb">here</a>. We are not going to give specific implementation details since all the information are well described in the previous link. Just to give you and intuition, the convolutional filters take different points of the temporal sequence to predict the next point in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHnUta3z2WhVXbAg_NbjmA.png" /><figcaption>1D Convolution Seq2Seq Neural Network. Credit: <a href="https://github.com/JEddy92/TimeSeries_Seq2Seq/blob/master/notebooks/TS_Seq2Seq_Conv_Intro.ipynb">https://github.com/JEddy92/TimeSeries_Seq2Seq/blob/master/notebooks/TS_Seq2Seq_Conv_Intro.ipynb</a></figcaption></figure><p>Starting with this assumption the network is trained to perdict the value t+1 using the previous values.</p><p>We used a 14 days prediction period. We split, for each gas station, the dataset in two sequence, the “Encoding Interval” used to “learn” the sequence and the “Prediction Interval” used to train the network to predict the next steps.</p><p>Since we need to know a previous time step to predict the next step, the Encoding and Precition interval have an entry in common. This entry is used as starting point to perform the prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0AbmPtDOF4WwL2eFadJ9Pg.png" /><figcaption>Encoding and Prediction Interval used to train the neural network</figcaption></figure><p>After the split, the dataset is used as input in the Seq2Seq neural network.</p><p>Once the training is completed, we performed some plots for different gas stations to show the real and the predicted price. Some examples are available here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qKU5TkjY2y-g_KDuuaynQA.png" /><figcaption>Real and predicted results with Seq2Seq neural network in 4 different gas station</figcaption></figure><p>As you can see from the results the neural network is quite good to predict the petrol price for single gas stations.</p><h3>Conclusions</h3><p>In this article we just show a simple example describing how to combine Qlik Sense and neural network to exploit one of the many Italian Open Datasets. We performed some high level steps just to show the potential of this approach.</p><p>In the next article we will provide a deep description of the analysis pipeline from the visualization to the prediction.</p><blockquote><strong>P.S</strong> We’re hiring, if you like us, we also like you. If you are interested, check our job opportunities <a href="https://humanativaspa.it/job-opportunity-specialists-in-erp-application-data-analytics/">here</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=767cb1016af3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/isiway-tech/seq2seq-in-keras-for-petrol-price-prediction-using-italian-open-data-767cb1016af3">Seq2Seq in Keras for Petrol Price Prediction using Italian Open Data</a> was originally published in <a href="https://medium.com/isiway-tech">isiway-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>