<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lorenzo Lancia on Medium]]></title>
        <description><![CDATA[Stories by Lorenzo Lancia on Medium]]></description>
        <link>https://medium.com/@llancia?source=rss-1c24b3f580a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ed-Q6vK_-rUW0ehG</url>
            <title>Stories by Lorenzo Lancia on Medium</title>
            <link>https://medium.com/@llancia?source=rss-1c24b3f580a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:44:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@llancia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A simple recommender system using PageRank]]></title>
            <link>https://medium.com/eni-digitalks/a-simple-recommender-system-using-pagerank-4a63071c8cbf?source=rss-1c24b3f580a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a63071c8cbf</guid>
            <category><![CDATA[networkx]]></category>
            <category><![CDATA[network-science]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Lorenzo Lancia]]></dc:creator>
            <pubDate>Wed, 26 Oct 2022 07:16:54 GMT</pubDate>
            <atom:updated>2022-10-26T07:16:54.389Z</atom:updated>
            <content:encoded><![CDATA[<p>Discover how a quick and easy recommender system can be built with PageRank, one of the most popular Network Science algorithms, combined with graph projection techniques</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*uQc59GEdGVQe1cu4" /><figcaption><a href="https://unsplash.com/photos/ZiQkhI7417A">Picture </a>under the <a href="https://unsplash.com/license">Unsplash License</a></figcaption></figure><p>Building recommender systems is one of the most basic tasks a data scientist has to face. Choosing the best approach and the best algorithm is highly dependent on the problem at hand. But let’s look at a very simple approach that exploit a very famous algorithm, the <strong>PageRank</strong>.</p><p>PageRank was developed by Larry Page (that’s why is called Page-Rank) and Sergey Brin to solve the following problem: on the Web, what is the most relevant page that contains a term?</p><p>If we were talking about a set of offline (not-linked) documents, a classical information retrieval approach like computing the</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/98/1*LnAR7jgmK2hXDl97ldy00Q.png" /></figure><p>(see <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">here</a>) would work.</p><p>But the documents on the www are not <em>independent</em> one from the other. They are <strong>connected</strong> by the underlying structure of <strong>hyperlinks</strong>, and this structure can be represented with a <strong>Graph</strong>.</p><h3>Graph definition</h3><p>A graph <em>G</em> = (<em>V</em>,<em>E</em>) is a discrete mathematical object that consists of a collection <em>V</em> of vertices and a collection of edges <em>E</em>. Each edge <em>e</em> ∈ <em>E</em> is said to join two vertices, which are called its end-points.</p><p>If <em>e</em> ∈ <em>E</em> joins <em>u</em>, <em>v</em> ∈ <em>V</em>, we write <em>e</em> = <em>u</em>, <em>v</em> and vertices <em>u</em>, <em>v</em> are said to be <strong>adjacent</strong>, while edge <em>e</em> is said to be <strong>incident</strong> with <em>u</em> and <em>v</em>, respectively.</p><p>A graph can be represented and stored by using a matrix called <strong>Adjacency Matrix.</strong></p><p>Considering <em>n</em> the number of nodes of a graph <em>G</em>, the Adjacency Matrix <em>A</em> is a square <em>n</em> × <em>n</em> matrix, such that its element <em>L</em>ij indicate whether node <em>i</em> and node <em>j</em> are adjacent or not in the graph. In practice the matrix <em>L</em> encodes the relation of vertex-vertex pairs.</p><p>From a memory-efficiency perspective the matrix-based data structures are highly inefficient to represent a graph. This is true especially in case of <strong>sparse</strong> graph (a graph which the number of links is much smaller rather than the numbers of vertex).</p><p>A more <strong>memory efficient way to store graph</strong> information is the <strong>adjacency list</strong>, that is a collection of unordered lists, where each list describes the set of neighbors of a particular vertex of the graph.</p><h3>Directed graphs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*NK_zSon_w7puzXxHKYkl8Q.png" /><figcaption><strong>Figure 1: A directed graph, source: </strong><a href="https://commons.wikimedia.org/wiki/File:Directed.svg"><strong>https://commons.wikimedia.org/wiki/File:Directed.svg</strong></a></figcaption></figure><ul><li>Edges have an orientation (<em>A</em> → <em>B</em> ≠ <em>B</em> → <em>A</em>)</li><li>Adjacency matrix <em>A</em> is <strong>not symmetric</strong></li><li>The sum over columns of <em>M</em> is 0 (if the graph is not weighted)</li><li>Example: following on Twitter</li></ul><h3>Bipartite graphs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iVTa_3To6j8oJ2-xyFlR2g.png" /><figcaption><strong>Figure 2: A complete bi-partite graph, source: </strong><a href="https://commons.wikimedia.org/wiki/File:Biclique_K_3_5_bicolor.svg"><strong>wikicommons</strong></a><strong> used under CC BY-SA 3.0 License</strong></figcaption></figure><ul><li>Vertices can be divided into two disjoint and independent sets <em>U</em> and <em>V</em> such that every edge connects a vertex in <em>U</em> to one in <em>V</em></li><li>They are used to model relations between different classes of objects</li><li>Example: consumer-product graph, affiliation networks</li></ul><h3>PageRank (PR)</h3><p>Let’s go back to Larry Page’s Problem: <strong>a query</strong> that gives me all the webpages that contains a <em>term</em> ranked by importance.</p><p>The idea of <strong>PageRank</strong> is simple: a webpage is important if it has a lot of incoming links, and its outgoing links must distribute that authority.</p><p>And like in <em>tf-idf</em>: webpages that link to a page, but link to a lot of other webpages, in general, should be given less weight with respect links that link to only that specific page.</p><h3>Broken Page Rank: a first approach to solve the problem.</h3><p>Let’s start from the adjacency matrix: <em>L</em>ᵢⱼ = 1 if webpage <em>j</em> links to webpage <em>i</em> (written <em>j</em> → <em>i</em>), and <em>L</em>ᵢⱼ = 0 otherwise.</p><p>Also let</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*fYzkTsLc6Zm6XLg3mrCJtA.png" /></figure><p>the total number of webpages that <em>j</em> links to. A possible <em>page rank score</em> for the page <em>i,</em> called <em>p</em>ᵢ, would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Yaneod9xZc0Iyyziqdtzkg.png" /><figcaption>Equation : Broken Page Rank</figcaption></figure><p>Moving to a matrix representation, the PageRank score is represented by the <em>eigenvector</em> of matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*saE7ROtzLmT_t6S3IPcn3Q.png" /></figure><p>with <em>eigenvalue</em> 1. (see <a href="https://www.stat.cmu.edu/~ryantibs/datamining/lectures/03-pr.pdf">here</a>)</p><p>Let’s think of the process as a guy starting from a random page and clicking on random links again and again: the probability to land to a specific page after a certain number of clicks is the PageRank score.</p><h3>Markov Chain</h3><p>That process is mathematically defined as a Markov Chain process characterized by a <em>transition matrix</em>: <em>A</em>ᵗ.</p><p>The stationary distribution of our Markov chain is a probability vector <em>p̄</em> (i.e., its entries are ≥ 0 and sum to 1), with <em>p̄</em> = <em>Ap̄</em>.</p><p>If the Markov chain is <em>ergodic</em>, meaning (among other things) that any state can be reached from any other state, then stationary distribution <em>p̄</em> exists and is unique.</p><h3>Teleporting and dampening</h3><p>This <strong>ergodicity</strong> is the requirement we have to compute the PageRank score. Issues arise with:</p><ul><li><em>Disconnected components:</em> part of the graph that you cannot reach from another</li><li><em>Dangling links</em> or <em>dead-ends</em>: nodes that have no outer links</li><li><em>Closed Loops</em>: part of the graph that force you to repeat paths.</li></ul><p>The presence of any of this condition makes the Markov chain non-ergodic: e.g., if you end up in a loop in your random surfing of the net you won’t’ be able to reach any possible node.</p><p>We can solve this (and restore the ergodicity) by adding a random teleport action in our random walk that increases in probability with each further step in the graph.</p><p>The new score become:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*YsHWDWTJ6w7hKY4f3H4aPw.png" /><figcaption>Equation : Page Rank</figcaption></figure><p>the general formula of PageRank where 1 − <em>d</em> is the probability to use the teleport action.</p><h3>Personalized PageRank (PPR)</h3><p>Now that we understood how PageRank works let’s take a look at a more generalized version of it (see more <a href="https://www.cs.cornell.edu/~bindel/blurbs/edgeppr.html">here</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*rpljZMfJYALTku-eURk6Rw.png" /><figcaption>Equation : generalized version of page rank with personalization</figcaption></figure><p>Considering:</p><ul><li>the graph with adjacency matrix L.<em>DL</em> is the diagonal matrix of out-degrees for nodes in W</li><li><em>gW</em> is the dangling node indicator vector</li><li>and <em>e</em> is the vector of all ones.</li></ul><p>The important part here is <em>k</em> which is the personalization vector: if it is a uniform distribution among the pages, that leads to the classic page rank. The more skewed it is, the most it forces the random walker to teleport itself to a specific set of nodes, thus introducing a <strong>personalization</strong>.</p><h3>Recommendations</h3><p>As PageRank can be used to rank webpages, we can use Personalized PR, for example, to rank suggested purchasable items based on specific user and its purchasing history.</p><p>Let’s see a quick example here. We’ll use the <a href="https://networkx.org/">networkx</a> library to handle graphs.</p><p>We define a bipartite graph connecting users to the items they have purchased.</p><pre>import networkx as nx</pre><pre>#Defining a directed graph<br>G = nx.DiGraph()</pre><pre>## Define a Bipartite Graph of people and items bought<br>G.add_nodes_from([&quot;Alice&quot;, &quot;Bob&quot;, &quot;Charlie&quot;, &quot;Diana&quot;,<br>                  &quot;Orange&quot;, &quot;Apple&quot;, &quot;Banana&quot;, &quot;Pineapple&quot;, &quot;Raspberry&quot;])<br>G.add_edges_from([<br>    (&quot;Alice&quot;, &quot;Orange&quot;),<br>    (&quot;Alice&quot;, &quot;Apple&quot;),<br>    (&quot;Bob&quot;, &quot;Orange&quot;),<br>    (&quot;Bob&quot;, &quot;Apple&quot;),<br>    (&quot;Bob&quot;, &quot;Banana&quot;),<br>    (&quot;Charlie&quot;, &quot;Orange&quot;),<br>    (&quot;Charlie&quot;, &quot;Apple&quot;),<br>    (&quot;Charlie&quot;, &quot;Banana&quot;),<br>    (&quot;Diana&quot;, &quot;Banana&quot;),<br>    (&quot;Diana&quot;, &quot;Pineapple&quot;),<br>    (&quot;Diana&quot;, &quot;Raspberry&quot;),<br>])</pre><p>the graph is something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sAMtqBejMvmHYbB0ZvVQQQ.png" /><figcaption>Figure 3: networkx drawing of a simple bipartite graph</figcaption></figure><p>Now we can make suggestions using the PPR algorithm, networkx already implement it, so we just need to create a personalization vector and run it.</p><pre>import operator</pre><pre>ppr = nx.pagerank(G, personalization={&quot;Alice&quot;: 1})<br>ppr = sorted(ppr.items(), key=operator.itemgetter(1), reverse=True)</pre><pre>for item, score in ppr:<br>    print(item, score)</pre><h3>Movielens Dataset Example</h3><p>We can try to follow this approach in a bigger dataset, using the <a href="https://grouplens.org/datasets/movielens/">movielens</a> dataset. The dataset describes 5-star rating and free-text tagging activity from <a href="http://movielens.org/">MovieLens</a>, a movie recommendation service.</p><p>Each user is represented by an id, and no other information is provided while we have full info on movies.</p><h3>Preprocessing</h3><p>We build a bi-partite graph structured like this:</p><p>User — vote → Movie</p><p>Each edge represents a movie seen by a user, with a weight corresponding to the vote given by the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zbjN9dRBHgARoJrhvS5QdA.png" /><figcaption><strong>Figure 4: Subset of the user-movie graph</strong></figcaption></figure><p>In this example I’ve scaled the votes using the rule:</p><pre>mapping_score = {1:-1,2:0, 3:0, 4:1, 5:1.2}</pre><p>the resulting graph is referred in the example below as user_movie_graph.</p><p>We then proceed to create another graph, this representing <em>closeness</em> between movies. We can achieve this by <em>projecting</em> the user_movie_graph. You can see more about the projection operation on <a href="https://en.wikipedia.org/wiki/Bipartite_network_projection">wikipedia</a>.</p><p>In this scenario we project the <em>weighted</em> bipartite graph by summing the scores (positive and negatives): the more couples of users like the same movie the higher the edge weight is.</p><p>The resulting graph has the form of movie--aggregated_score--&gt;movie and we refer to it in the code examples as movie_movie_graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*nVELCJQEz3qAFZplhtRihg.png" /><figcaption><strong>Figure 5: Subset of the projected graph.</strong></figcaption></figure><h3>Scoring of recommender system</h3><p>We can now compute the recommendations for each user by using the <em>Personalized Page Rank</em>. First, compute the preference vector by taking all the movies seen by a user and rated positively, an example function to do so is reported here.</p><pre>def create_preference_vector(user_id: int, user_movie_graph: nx.Graph):</pre><pre>    edges = {m: v for _, m, v in user_movie_graph.edges(user_id, data=”weight”)}</pre><pre>    tot = sum(edges.values())</pre><pre>    if tot &gt; 0:</pre><pre>        return {</pre><pre>        movie: edges.get(movie, 0) / tot</pre><pre>        for movie in filter_nodes(user_movie_graph, “movie”)</pre><pre>        }</pre><pre>    else:</pre><pre>        return {movie: 1 for movie in filter_nodes(user_movie_graph, “movie”)}</pre><p>Then we score the personalized page rank and display top scored movies that user have not seen:</p><pre>def predict_user(user_id, user_movie_graph: nx.Graph, movie_movie_graph: nx.Graph):</pre><pre>    p_vec = create_preference_vector(user_id,user_movie_graph)<br>    already_seen = [movie for movie,p in p_vec.items() if p&gt;0]</pre><pre>    if len(already_seen)&lt;1:<br>        return []</pre><pre>    item_rank = nx.pagerank_numpy(movie_movie_graph, personalization=p_vec, alpha=0.95, weight=&quot;weight&quot;)</pre><pre>    s_t = [x <br>        for x in<br>        sorted(movie_movie_graph.nodes(),key=lambda x: item_rank[x]      </pre><pre>    if x in item_rank else 0, reverse=True) <br>        if x not in already_seen<br>        ]<br>    <br>    return s_t</pre><p>If you run this code for a few user, we can see what the algorithm suggests to watch next.</p><p>For example, from a user who have seen:</p><pre>[‘Toy Story (1995)’, ‘Four Rooms (1995)’, ‘Twelve Monkeys (1995)’, ‘Braveheart (1995)’]</pre><p>We get those suggestions:</p><pre>[‘Raiders of the Lost Ark (1981)’, ‘Silence of the Lambs, The (1991)’, ‘Fargo (1996)’]</pre><p>Which is reasonable being very popular movies that are skewed towards action and thriller as the user past views suggested.</p><h3>Recap and further advancements</h3><p>While this very simple approach may not achieve any good performances and won’t be used in any real-world scenario, we showed that with just some graph operations we can compute recommendations and a lot of enhancement can be done to increase its capabilities.</p><p>Using different projections techniques may increase the capabilities to produce meaningful item-item graphs and this approach with personalized page rank can lead to interesting information even for smaller datasets.</p><p>But if you want to take a further look at the recommender system we suggest you take a deep dive in the world of <a href="https://en.wikipedia.org/wiki/Collaborative_filtering">collaborative filters</a> and check out this <a href="https://link.springer.com/book/10.1007/978-3-319-14142-8">book</a>.</p><h3>References</h3><p>The techniques and procedures here presented are based on resources from academic papers and books:</p><p>Graph Mining: <a href="https://link.springer.com/chapter/10.1007/978-3-319-14142-8_17">https://link.springer.com/chapter/10.1007/978-3-319-14142-8_17</a></p><p>Personalized Page Rank <a href="https://nlp.stanford.edu/IR-book/html/htmledition/topic-specific-pagerank-1.html">https://nlp.stanford.edu/IR-book/html/htmledition/topic-specific-pagerank-1.html</a></p><p>Graph projection: <a href="https://doc.rero.ch/record/8503/files/zhang_bnp.pdf">https://doc.rero.ch/record/8503/files/zhang_bnp.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a63071c8cbf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eni-digitalks/a-simple-recommender-system-using-pagerank-4a63071c8cbf">A simple recommender system using PageRank</a> was originally published in <a href="https://medium.com/eni-digitalks">Eni digiTALKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing data science code with Git and GitLab]]></title>
            <link>https://medium.com/eni-digitalks/managing-data-science-code-with-git-and-gitlab-2cd2e454952?source=rss-1c24b3f580a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cd2e454952</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Lorenzo Lancia]]></dc:creator>
            <pubDate>Mon, 04 Apr 2022 16:25:39 GMT</pubDate>
            <atom:updated>2022-04-04T16:25:39.445Z</atom:updated>
            <content:encoded><![CDATA[<p>To write a better code and to write code better in your data science projects will pay off in the long run, so let’s take a look at Git Source Code Management system and how to use it in your data science daily work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*TqDQ3oKrqtLd_9VJ6owZSQ.png" /></figure><p>Writing code is an essential part of most <em>data science</em> projects and, while a lot of those projects struggle to evolve towards a complete software solution, it is important to adopt some of the software engineering best practices while coding. The advantages may vary according to your company or project maturity and objectives. Organizing your code in functions and classes, instead of complex notebooks cells, will allow you to debug your code easily, re-use procedures in other projects and enable better collaboration among different <em>data professionals</em> working on the same task.</p><p>Collaborating on the same codebase is one of the most basic struggles that coders face every day and <strong>version control</strong> tools made it easy and almost hassle free.</p><p>In this brief article we describe how to install Git and get familiar with the most common commands and workflows before looking at some peculiarities of data science projects.</p><h3>1. Git: the most widely used version control system</h3><p>Git is an open source distributed version control system created by <a href="https://en.wikipedia.org/wiki/Linus_Torvalds">Linus Torvalds</a> in 2005 for the development of the <em>Linux kernel</em> (see <a href="https://git-scm.com/book/en/v2/Getting-Started-A-Short-History-of-Git">history of git</a>). It reached on January 2022 version 2.35, and it’s designed to handle any kind of code project with speed and efficiency.</p><p>Distributed means that multiple copies (called <em>remotes</em>) of your <em>repository</em> can simultaneously exist and Git will keep track and help you synchronize the changes. Services like GitHub and GitLab offer the chance to host your remote on their server and have it available via internet to store and synchronize your work among your team.</p><p>Knowing how to use <a href="https://git-scm.com/">Git</a> or any other version control tool or Source Code Management (SCM) are among the fundamental skills for any programmer or software engineer out there, and any person who write down code, alone or with other collaborators, may benefit from using it.</p><p>In this brief article we’ll focus on GitLab that, while less popular, offers the chance to install its suite of services on a company or personal server (see <a href="https://about.gitlab.com/install/ce-or-ee/">GitLab Community Edition and Enterprise Edition</a>).</p><h3>1.1. Installing Git</h3><p>To use the Git version control capability and interact with remote repositories you need to have Git installed on your system.</p><p>For <strong>Linux</strong>, Git is generally available from your distro’s favorite package manager (e.g., apt or rpm).</p><p>For <strong>OSX</strong> systems, it’s strongly recommended using the <a href="https://brew.sh/">homebrew</a> package manager that will enable you to quickly access tons of open-source tools like a Linux-like package manager; after installing homebrew you can proceed to retrieve and install Git with the command brew install git. Alternatively, Apple’s <em>XCode</em> comes with a Git pre-installed.</p><p>For <strong>Windows</strong> systems, a setup package can be downloaded <a href="https://gitforwindows.org/">here</a> or you can use the <a href="https://docs.microsoft.com/it-it/windows/package-manager/winget/">winget</a> tool with the command</p><pre>winget install — id Git.Git -e — source winget </pre><p>Several visual interfaces are available for the Git system, and most development environment like <a href="https://www.jetbrains.com/pycharm/">Pycharm</a> or <a href="https://code.visualstudio.com/">Visual Studio Code</a> integrated one. For reproducibility and simplicity, we will refer to Git command using the <em>command line interface</em> which is identical in every Operating System and less subject to changes.</p><h3>1.2. Basic usage of git</h3><p>Mastering Git may require some time and some practice, but the basic workflow is very simple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*rKKgr8aOxVSh2kUYK2OEYA.png" /><figcaption>A very common approach to git, xkcd (used under Creative Common License <a href="https://creativecommons.org/licenses/by-nc/2.5/">CC BY-NC 2.5</a>)</figcaption></figure><p>First, you must ensure you’re working inside a folder that represents a local version of a Git repository: use the command git status to check.</p><pre>$git status<br>fatal: not a git repository (or any of the parent directories): .git</pre><p>This happens if you’re not inside a Git repository, that can be easily solved by <em>initializing</em> the current folder as a new repo via the command git init.</p><p>Now a new git status command will return a correct photograph of the repo’s status. In the example below the folder contains a single file and it shows it’s not currently <em>tracked</em> by the Git system.</p><pre>On branch master<br> <br>No commits yet<br> <br>Untracked files:<br>(use “git add &lt;file&gt;…” to include in what will be committed)<br>article_git.md<br> <br>nothing added to commit but untracked files present (use “git add” to track)</pre><p>The git add command adds one or more file to the one tracked by git. If a file is already tracked by git, the same command adds all the changes to that file to the <em>staging</em> area.</p><pre>$ git add article_git.md<br> $ git status<br> On branch master<br> <br> No commits yet<br> <br> Changes to be committed:<br> (use “git rm — cached &lt;file&gt;…” to unstage)<br> new file: article_git.md</pre><p>Now it’s time to consolidate the changes made to tracked files inside a new <strong>commit</strong>. A commit is a snapshot of your repository along its timeline that Git can always recover and share with others. To create a new commit from <em>staged changes</em> the command is git commit.</p><p>The system will ask you for a commit message. You should use it to describe explicitly and briefly the changes introduced to the codebase by that commit. Do it for other people reading your code and for your future self. Another useful suggestion is to avoid keep the commit as simpler as you can to ensure that can be easily reverted if something goes wrong.</p><p>Now the commit is recorded, and you can continue your work adding more and more commit to the history of your project. You can always see the list of the commits in reverse chronological order (from the latter to the older) by using the git log command.</p><p>At any time, you may need to undo some changes you’ve added to your codebase. Git offers different ways to do it, you can either:</p><p>• git revert to introduce a commit that changes the status of the repository to the one of an older snapshot.</p><p>• or git reset to delete all the commits introduced after a past one.</p><p>An extensive description of all Git commands and their usage is beyond the scope of this article. If I’ve managed to convince you that Git is worth learning, you can continue your journey into learning all the main commands and their usage scenarios. A good interactive resource to learn it is <a href="https://learngitbranching.js.org/">learn Git branching</a>, while some classical Git struggles (and their solutions) can be found <a href="https://dangitgit.com/">here</a>.</p><h3>2. GitHub and GitLab: synchronizing your work</h3><p>Working alone is rarely the case for developers and data scientists. Using a centralized remote for your repository helps you to synchronize your work with your colleagues and team-members.</p><p>The Git commands git pull and git push enable you to get and send your local changes to a remote repository. Often this remote is hosted on a centralized server like the public GitHub or your own company server.</p><p>When setting up a project is important to configure the central repository according to your needs.</p><h3>2.1. Create a new repository</h3><p>A new repository can be created from a previous Git repository anywhere on the internet, empty or from a template available on GitLab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*K3JtG-V4FKgcoZc03xfJWQ.png" /><figcaption>Available options while creating a new repository on GitLab</figcaption></figure><p>After creating a repository, you can either:</p><p>• link your local work to the new remote</p><p>• clone the remote into a new local repository</p><p>In any way the next step will require you to choose the Visibility Level of your repository to ensure it can be only accessed by a selected group of users. Making a repository <em>public </em>will allow everyone that can reach the GitLab server to take a look at your source code without<strong> </strong>allowing them to push any changes.</p><h3>2.2. Roles and Collaboration</h3><p>At this point everything is up and running and it’s time to bring your teammates in the loop and enable <strong>collaboration features</strong>.</p><p>GitLab distinguishes between five level of users with different permissions on the project:</p><p>• Guest</p><p>• Reporter</p><p>• Developer</p><p>• Maintainer</p><p>• Owner</p><p>Developers and Maintainers are the only ones that can push new code on the project, so keep it in mind. (For other role-specific permission look <a href="https://gitlab-dgt.eni.com/help/user/permissions">here</a>).</p><p>After you’ve assigned the roles to all your team members, they can use the git push and git pull commands to sync with the repo. Their commit will appear on the timeline alongside yours.</p><h3>2.3. Organizing Work on Gitlab</h3><p>Having several people working on the same code can lead to some struggles in the form on <strong>concurrent updates</strong>. Git always tries to handle concurrent updates to the project history locally. The <em>default</em> strategy Git applies, when you pull new commits from your remote repository, is to create a <em>merge commit</em>. A <em>merge commit</em> is a new commit that integrate the differences between the code committed on your local repository and the code from remote repository.</p><p>If changes exist on different fragments of code files, Git is capable to handle things automatically. But when you and your teammate touched the same lines of code, Git needs help to decide what to include in the <em>merge commit</em>.</p><p>While this strategy is fast and can work out with smaller teams within a project of limited complexity, this is un-feasible for complex projects like open-source projects or company products. <em>Merge commits</em> may be hard to understand and an unclear project history makes it hard to roll back if things go sideways.</p><h3>2.4. Choose a Git workflow for your project</h3><p>Adopting a <strong>Git branching model</strong> may help you address complexity in your project workflow. A branch is an independent history line for your repository.</p><p>You can start a branch from your current commit and add new commit on this <em>new branch</em> without adding them to main history line (called <em>master</em> branch or <em>main</em> branch), see more <a href="https://git-scm.com/book/it/v2/Git-Branching-Basic-Branching-and-Merging">here</a>.</p><p>When you create a new branch, you can work independently on what other people are pushing on the <em>master branch</em> till you’re ready to <em>merge</em> your changes with them.</p><p>To create a new branch, you can use the command git checkout -b &lt;name of the new branch&gt;, this will automatically switch you to the new branch so any new commit you create are added to the new branch.</p><p>Anytime you want to move between branches you can use the command git checkout &lt;branch to move to&gt;.</p><p><em>Merging</em> a branch with another means to include all the changes introduced by the source branch’s commits on the target branch. Git may ask you to solve concurrency issues highlighting some <a href="https://www.atlassian.com/git/tutorials/using-branches/merge-conflicts">conflicts</a>. To merge branches, use git checkout to switch to the branch you want to merge into and then use the command git merge &lt;source branch name&gt;, se more <a href="https://git-scm.com/docs/git-merge">here</a>.</p><p>A possible branching model is one called <strong>Gitflow</strong>. To adopt it you must have (at least) two fixed branches:</p><p>· A <strong><em>master</em></strong> branch which only contains usable releases of your product.</p><p>· a <strong>develop</strong> branch which progress as soon as you <em>merge</em> new features on it from different <em>feature branches</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*zKk7PZweeJKLn_eKffG0Rw.png" /><figcaption>The Gitflow model with a master branch, a feature branch and multiple feature branches, a good description of the model is available on <a href="https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow">Atlassian.com (source of the picture)</a>, used under <a href="https://creativecommons.org/licenses/by/2.5/au/">Creative Commons Attribution 2.5 Australia License</a></figcaption></figure><p>This model is well suited for products that needs to be stable in complex teams with different level of experience even though can slow down the release of new features to the <em>main</em> branch.</p><h3>3. GitLab and Merge requests</h3><p>A correct way to enforce this solution is to avoid manually merging of the <em>feature</em> branches on the <em>develop</em> branch. GitLab (and other Git hosting services like GitHub) offers you an automated and controlled way of handling merges. The tool is called <strong>merge request</strong>.</p><p>When starting to work on a new feature branch you can open a new <em>Merge Request</em> (MR) on GitLab after pushing your new branch to remote. You need to specify the target branch (develop) and the source branch (our feature branch).</p><p>After it has been created any changes pushed on the feature branch will be tracked inside the merge request alongside any comment the people working on or maintains the code will add on the MR page.</p><p>When the work is completed and the feature branch is ready to be integrated into develop, you can mark the MR as ready and ask a <strong>maintainer</strong> to approve it and merge into <em>develop</em>. It may be enforced (via GitLab project settings) that only someone who did not worked on the feature branch can approve a MR to ensure that an external review is done on the work.</p><h3>3.1. Conflicts and rebasing</h3><p>It may happen that the develop branch evolved since the moment you detached your feature branch from develop, for example other feature branches were closed in the meantime. In this likely scenario it is impossible to merge the feature branch automatically and GitLab will tell you so. The correct way to handle this is to integrate the changes on develop on your feature branch before merging.</p><p>The best way to do it is git rebase your feature branch <em>onto</em> develop. <strong>Rebasing</strong> is a Git operation that may seem complex but it’s fundamental to keep the history of your project clean and readable.</p><p><em>Rebasing</em> branch A <em>onto</em> branch B means to roll back to the moment the two branches diverged; and re-introduce the changes included in A’s commits one by one onto the last status of B.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*WgbVmPZy9b42xFoLqoRaVQ.png" /><figcaption>Git rebasing a feature branch onto Main, picture from <a href="https://www.atlassian.com/git/tutorials/rewriting-history/git-rebase">Atlassian.org</a>, used under <a href="https://creativecommons.org/licenses/by/2.5/au/">Creative Commons Attribution 2.5 Australia License</a></figcaption></figure><p>Any conflict will be resolved during the application of the commit from B ensuring that the final merge of the feature branch can be done automatically. Since the rebase command does <em>rewrite history</em> (in other words overwrites previous commits) it cannot be pushed without the -f flag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*w00OhVmgcmi4jCq6KWvaqg.png" /><figcaption>Recap of the step needed to add a new feature to a project managed with Gitflow.</figcaption></figure><h3>4. Data Science Projects do’s and don’ts</h3><p>A Data Science project has some intrinsic characteristics that differ from a pure software project and that may impact the normal usage of a git-based version control system. Here there are some examples where standard data science practices clash with standard Git ones and some “negotiation” is needed.</p><h3>4.1. Managing data</h3><p>Data are the main protagonists and it’s important to add them to your Git repository to enable other people to access them and to reproduce experiments and results. However, data, as code, evolve with time: datasets become obsolete and need to be replaced.</p><p>In the same way as you want to track changes in your code, you want to track changes in your datasets. Unfortunately, Git is not suited to track changes to data files, either tabular or binary.</p><p>A straightforward solution that many people suggest is <strong>do not commit the datasets</strong>. Just avoid Git to tangle with data files and manage them without a VCS. This is very easily done with a proper use of the git ignore files.</p><p>Any file called .gitignore (with the dot), will be interpreted by Git as containing instructions regarding files to exclude from the tracking. See more <a href="https://git-scm.com/docs/gitignore">here.</a></p><p>A better solution would be to use a Git plugin that handles files that do not contain code: git lfs (large file storage). By using the command git lfs track “*.csv” you are telling Git to handle all the csv file in your repository. From now on, you can treat them as regular files without worrying about messing up with your Git history or repository space. If you have all the data inside a folder, you can track that specific folder with the command git lfs track “Data/”. Further information on git lfs can be found <a href="https://www.atlassian.com/git/tutorials/git-lfs">here</a>.</p><h3>4.2. Versioning Jupyter notebooks</h3><p>The <strong>Jupyter notebook</strong> is one of the most used development environments for data science projects. Its imperativeness makes it amazing for rapid prototyping and testing of machine learning or data visualization solutions.</p><p>From git’s point of view a Jupyter notebook file (usually identified by the extension .ipynb) is a json file which contains all the cells with both code and its produced output along with some metadata.</p><p>The main issue is that every time the cells are executed the output and the metadata change and Git will track those changes even if they are not relevant.</p><p>Before registering a new commit, you may <em>clear</em> all the output from the cells to ensure only code (and non-output) is saved to the file, so conflict won’t arise. Otherwise, you will end resolving them anytime you run the notebook without changing the cell code content.</p><p>There is a simple way to clear Jupyter cell’s outputs by using the command line:</p><pre>jupyter nbconvert — ClearOutputPreprocessor.enabled=True — inplace notebook.ipynb</pre><p>The nbconvert tool is available previous installation on your python environment via pip or conda.</p><p>We can even use the wildcard * to ensure all the notebooks are cleared.</p><pre>jupyter nbconvert — ClearOutputPreprocessor.enabled=True — inplace *.ipynb</pre><p>While this will speed up our Git workflow, we may even want to run the previous command automatically every time we create a new Git commit. This can be done using the Git <strong>pre-commit</strong> hooks.</p><p>In every Git repository folder exists a folder named .git/ that contains internal Git file. We can add a shell script that will be automatically executed before each commit, writing down a file called pre-commit in the hooks subfolder using any text editor, for example using <strong>vim</strong>: vim .git/hooks/pre-commit.</p><p>We write as the content of the file the commands we want to execute:</p><pre>#!/bin/sh<br> jupyter nbconvert — ClearOutputPreprocessor.enabled=True — inplace *.ipynb<br> git add *.ipynb</pre><p>Now every time a git commit command is executed the output of every cell inside the notebooks is erased.</p><h3>5. Recap and further topics</h3><p>Git is a powerful tool, ubiquitous, easy to install and to learn and it’s a <em>must-have</em> in the belt of any person who write down code, including <em>data scientists</em>. Setting up a correct workflow to collaborate on the same codebase will speed up your project development and this will be priceless when your code will ship to <strong>production,</strong> allowing you to control all the changes that will eventually occur.</p><p>Using a robust way of handling the evolution of your codebase, like <em>Gitflow</em>, will ensure your project to have a clear and readable history and that any change can be easily monitored and evaluated before hitting the final release.</p><p>Setting up your repository within a hosting service like GitLab will allow you to supercharge your development workflow with new and powerful features that you can exploit within your data-science development, like git-lfs. It will require some time to bring every member of the team on board to work with Git and GitLab, but the quality of your work will increase and you’ll find helpful in time of trouble when concurrent updates will clash on your code.</p><p>You can push this further and handle your data science <em>experiment tracking</em> in a Gitflow-like process as described in this article <a href="https://dagshub.com/blog/data-science-workflows-to-track-experiments/">here</a>. Any experiment will have its own feature branch and you’ll create merge request for the best model candidates and moving the first step towards a <a href="https://ml-ops.org/">Machine Learning Model Operationalization Management</a> (MLOPS) process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cd2e454952" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eni-digitalks/managing-data-science-code-with-git-and-gitlab-2cd2e454952">Managing data science code with Git and GitLab</a> was originally published in <a href="https://medium.com/eni-digitalks">Eni digiTALKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>