<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amy Li on Medium]]></title>
        <description><![CDATA[Stories by Amy Li on Medium]]></description>
        <link>https://medium.com/@liamy561t?source=rss-e0fbb3b20e73------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*B_nN9f18zrKogNkHu2puOw.jpeg</url>
            <title>Stories by Amy Li on Medium</title>
            <link>https://medium.com/@liamy561t?source=rss-e0fbb3b20e73------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:26:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@liamy561t/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When Addresses Alone Are Not Enough]]></title>
            <link>https://medium.com/analytics-vidhya/when-addresses-alone-are-not-enough-35da9b37bbea?source=rss-e0fbb3b20e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/35da9b37bbea</guid>
            <category><![CDATA[geopy]]></category>
            <category><![CDATA[mapping]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[folium]]></category>
            <category><![CDATA[geocoding]]></category>
            <dc:creator><![CDATA[Amy Li]]></dc:creator>
            <pubDate>Fri, 13 Sep 2019 17:51:59 GMT</pubDate>
            <atom:updated>2019-09-19T05:28:17.226Z</atom:updated>
            <content:encoded><![CDATA[<h4>Geocoding and Reverse Geocoding in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*MjKv0sQPfrYxNCP_6238bQ.png" /><figcaption>A cartoon representing what is geocoding from the ArcGIS website: <a href="https://developers.arcgis.com/assets/img/features/features-hero_geocoding.png">https://developers.arcgis.com/assets/img/features/features-hero_geocoding.png</a></figcaption></figure><p>In a prior project which attempted to predict if a restaurant would receive an inspection grade of A from the NYC Department of Health and Mental Hygiene, a thought I had was: wouldn’t it be great to visualize areas where there are restaurants that have been cited for critical violations to see if there is a pattern. The more critical the violations, the less of a chance the restaurants will receive an A. However, the dataset used for the project did not include longitude and latitude coordinates, which were necessary for plotting the locations on a map. Then I heard about geocoding, the solution to this problem.</p><h3><strong>What is geocoding?</strong></h3><p>According to the Google Maps Geocoding API’s <a href="https://developers.google.com/maps/documentation/geocoding/start">website</a>:</p><blockquote><strong>Geocoding</strong> is the process of converting addresses (like a street address) into geographic coordinates (like latitude and longitude), which you can use to place markers on a map, or position the map.</blockquote><p>Since there is a way to convert addresses into coordinates, there must be a way to work backwards and convert coordinates into addresses. This is called <strong>reverse geocoding.</strong> I will be focusing on geocoding here but the same steps can be used for reverse geocoding.</p><h3>How is geocoding done?</h3><p>Now that we understand what is geocoding, the next big question is: how is geocoding done? The answer is: geocoders!</p><p>A <strong>geocoder </strong>is a piece of software or web service made up of a reference dataset and an algorithm that implements a process to convert addresses to longitude and latitude coordinates. There are a variety of geocoders available such as ArcGIS, Google Maps Geocoding API, TIGER (US Census), and Nominatim (using data from OpenStreetMaps).</p><p>But which one is the best geocoder to use? Choosing a geocoder depends on the data and experience. There are a few ways to test out the geocoders to see which matches your purpose such as working with each geocoding API individually. As a newbie, I decided to use a library which can access a variety of geocoders.</p><h3>GeoPy — A Python Solution for Geocoding</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I9Miy4XzfFIewVJe5GclRg.png" /><figcaption>GeoPy logo from its website: <a href="https://geopy.readthedocs.io/en/stable/_images/logo-wide.png">https://geopy.readthedocs.io/en/stable/_images/logo-wide.png</a></figcaption></figure><p>As described in their documentation, <strong>GeoPy</strong> is a Python package that serves as a client to provide consistent access to some of the most popular geocoders available on the web. For those familiar with working with databases, this package is similar to using a Python driver to access a MySQL (ie. mysql connector) or PostgreSQL database (ie. psycopg2).</p><blockquote>“Geopy makes it easy for Python developers to locate the coordinates of addresses, cities, countries, and landmarks across the globe using third-party geocoders and other data sources.”</blockquote><p>The best part is that it works well with Pandas, which is really helpful for batch geocoding (imagine you have a large list of addresses, batch geocoding is more efficient than converting individual addresses).</p><h3>Working with GeoPy and Pandas</h3><p>The best way to learn about a package is to just jump in. Looping back to why I started looking into geocoding, I decided to use the NYC Restaurant Inspections data from my prior project to experiment with GeoPy. For those interested in the full code of this mini-project, it is available on Github <a href="https://github.com/lasoya/Restaurant_Inspections_Map">here</a>.</p><ol><li>To begin, first <strong>install GeoPy using pip</strong> — <strong><em>pip install geopy.</em></strong></li></ol><p>2. Next, <strong>clean the data</strong>. For batch geocoding with Pandas, this also involves isolating the addresses into its own column.</p><p><strong><em>Why is this step important?</em></strong></p><p>Well, if there is a difference in the spelling in the address or a difference in the order of the words in the address, this may cause an error. Because each address is compared to an entry on the reference table on the backend, if an address is not formatted the same as an entry on the table, it will not be geocoded.</p><p>3. <strong>Now that the data is clean and the addresses are isolated, it is time to run it through a geocoder.</strong> As it was my first time geocoding, for this mini-project, the default geocoder, Nominatim, was used. However, other geocoders can be used instead.</p><ul><li>Each geocoder uses its own reference dataset. If a geocoder does not produce any results, trying another geocoder may solve the problem. For example, when I used Nominatim to geocode the address: “1057 LEXINGTON AVENUE MANHATTAN NY 10021”, there was an error that the address was not found. However, when I looked it up on Google Maps, it was found.</li></ul><p>Here is the general code for batch geocoding with GeoPy and Pandas:</p><pre># import modules from geopy library<br>from geopy.geocoders import Nominatim<br>from geopy.extra.rate_limiter import RateLimiter<br><br># create an instance of the geocoder<br>geolocator = Nominatim(user_agent=&quot;inspections_locations&quot;)<br><br># create an instance or geocoder object with rate limit implemented<br>geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1, max_retries=0)<br><br># create a new column in dataframe for storing the location details from geocoding<br>df[&#39;LOCATION&#39;] = df[&#39;ADDRESS&#39;].apply(geocode)<br><br># create a new column called POINT and fill column with coordinates as tuple pulled from the Location column<br>df[&#39;POINT&#39;] = df[&#39;LOCATION&#39;].apply(lambda LOC: tuple(LOC.point) if LOC else None)</pre><ul><li>As you probably noticed in the above snippet, a RateLimiter object was used. This is because batch geocoding can overwhelm the geocoding service with its volume same with any API. To prevent this, it is necessary to add a delay either after each address or after a subset of addresses is processed. For this mini-project, the data were fed to the geocoder in subsets of 100 with a delay of 1 second after each address.</li><li>The number of retries can also be edited under the RateLimiter object. Running an address through the geocoder multiple times, which can be due to difficulty matching the address to a set of coordinates or due to connection issues, can delay the overall geocoding process. Therefore, it is helpful to have this parameter which can limit the number of retries, thus, reducing the overall time spent. In this case, I set the max number of retries to 0.</li></ul><p>In the end, out of 800 unique addresses, 486 addresses were able to be converted to geographic coordinates. Not perfect but these results were sufficient for my use.</p><p>Phew! Ok that’s the end to the geocoding step. Pretty simple right? Even for batch geocoding, there are just 3 major steps.</p><h3>Geocoding Results At Play</h3><p>Now that I have the coordinates, I can create the visualizations to see if there are certain areas that have restaurants with more critical violations. For this, I used Folium, which is a Python package built on a Javascript library called Leaflet.js for map visualizations.</p><p>The map below, which visualizes each restaurant location (in red), seems to show that most of the restaurants with critical violations are in Manhattan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*MC1Rxl3ViW7U894c1cHsKA.png" /><figcaption>A snapshot of the map visualization (individual locations) created using Folium</figcaption></figure><p>Another view of this is in the map below, which clusters the restaurant locations with critical violations in each area for a cleaner look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cPsglnJyLuCTSZcNlAANNg.png" /><figcaption>A snapshot of the map visualization (clustered locations) created using Folium</figcaption></figure><p>However, this is probably because most restaurants are located in Manhattan. Looking at the distribution of restaurant inspections by borough, the borough with the most inspections is Manhattan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*DruNNNDfK5zRl-LaJtnJ6w.png" /><figcaption>Distribution of restaurant inspections by borough in NYC</figcaption></figure><p>Therefore, seeing that most of the restaurants in Manhattan have critical violations does not really produce much insight on the areas to avoid when looking for a safe place to eat. In addition, while it is true that if a restaurant has more critical violations it is less likely to obtain an A, there are restaurants who still obtain an A despite having critical violations whether during the same inspection visit or at a subsequent inspection visit.</p><p>Despite not obtaining a lot of useful insights from this project, it was a great way to learn about geocoding and see how it can be used in addition with map visualization. Only by actually trying it out in a simple project like this that I was able to understand some of the challenges with geocoding.</p><h3>Resources</h3><ul><li><a href="https://geopy.readthedocs.io/en/stable/#">Welcome to GeoPy&#39;s documentation! - GeoPy 2.0.0 documentation</a></li><li><a href="https://sunnykrgupta.github.io/a-practical-guide-to-geopy.html">Daemon Blog - A Practical Guide to Geopy</a></li><li><a href="https://python-visualization.github.io/folium/quickstart.html#GeoJSON/TopoJSON-Overlays">Quickstart - Folium 0.9.0 documentation</a></li><li><a href="https://deparkes.co.uk/2016/06/10/folium-map-tiles/">Folium Map Tiles - deparkes</a></li><li><a href="https://deparkes.co.uk/2016/06/24/folium-marker-clusters/">Folium Marker Clusters - deparkes</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35da9b37bbea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/when-addresses-alone-are-not-enough-35da9b37bbea">When Addresses Alone Are Not Enough</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Peek Into Transactions and Concurrency Control]]></title>
            <link>https://medium.com/@liamy561t/a-peek-into-transactions-and-concurrency-control-97cf0c08e32e?source=rss-e0fbb3b20e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/97cf0c08e32e</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[transactions]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Amy Li]]></dc:creator>
            <pubDate>Tue, 21 May 2019 19:41:41 GMT</pubDate>
            <atom:updated>2019-05-21T20:12:14.071Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scratching the Surface on Transactions and Concurrency Control</h3><p>After being introduced to SQL databases at my data science bootcamp, I became interested in learning more about the underlying concepts that define how databases operate, which led me to reading about transactions and concurrency control. Transactions and concurrency control schemes are database concepts that regularly appear in real-life situations. There is still a lot I have to learn but I wanted to summarize what I understand so far to help me wrap my head around these concepts.</p><h3><strong>What is concurrency control?</strong></h3><p>Concurrency control is the management of simultaneously executing transactions.</p><h3><strong>Why is this important?</strong></h3><p>Concurrency control keeps each transaction isolated as it is executed which helps data remain consistent even after the transaction ends especially in multi-user systems. Imagine you are reading data from a database at the same time another user is writing into it. The copy of data you obtained will not include these updates and thus is an inaccurate or inconsistent set of data. Analysis completed on this data will yield inaccurate information for decision-making. To prevent this, it is important to have concurrency control. To understand how this works, we first need to talk about transactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*Vln73f03U2tuNoqgmCZ-rQ.gif" /><figcaption>Database transaction: <a href="http://maxdb.sap.com/doc/7_7/81/74b30edc2142658e510080ef6917f1/ppt_img.gif">http://maxdb.sap.com/doc/7_7/81/74b30edc2142658e510080ef6917f1/ppt_img.gif</a></figcaption></figure><p>A <strong>transaction</strong> is a unit of operations, usually read and write operations, which is executed from start to finish to access and update various data items. An example is transferring money from a savings account to a checking account. A good transaction has <strong>ACID</strong> properties:</p><ul><li><strong>Atomicity</strong>: “All or none” — All operations of the transaction are completely executed and reflected properly in the database or none at all.</li><li><strong>Consistency</strong>: A transaction is completed in a way that ensures that the database is in a consistent state after the execution.</li><li><strong>Isolation: </strong>Each transaction, even if there may be multiple transactions happening concurrently, must be executed in isolation so that it is unaware of other transactions happening in the system.</li><li><strong>Durability</strong>: The changes a fully executed transaction has on the database should persist regardless of any adverse events.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*76N2VelAHDFVggGyLOT5ng.png" /><figcaption>Transaction States: <a href="https://www.tutorialspoint.com/dbms/images/transaction_states.png">https://www.tutorialspoint.com/dbms/images/transaction_states.png</a></figcaption></figure><p>There are a couple of states a transaction goes through:</p><ul><li><strong>Active</strong>: the initial state when the operations in the transaction are being executed</li><li><strong>Partially Committed</strong>: the state after the final operation is executed but before the changes are saved to the database</li><li><strong>Failed</strong>: the state after the execution is discovered to be unable to proceed</li><li><strong>Aborted</strong>: the state after the transaction has been rolled back and the database is restored to the state prior to the transaction <em>(terminated)</em></li><li><strong>Committed</strong>: the state after completion of the transaction and changes are reflected in the database <em>(terminated)</em></li></ul><p>If we consider what is the best method to ensure consistency and isolation of a transaction, transactions should be conducted serially one after another. One transaction should complete its cycle before a new transaction begins on the same item. However, there are disadvantages to this approach such as reduced resource utilization and overall inefficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*itgTMOa2g48hZpRSuLK8cw.jpeg" /><figcaption><a href="https://image.slidesharecdn.com/lecture03transactionconcurrencycontrol-121201053857-phpapp01/95/transaction-concurrency-control-24-638.jpg?cb=1354340463">https://image.slidesharecdn.com/lecture03transactionconcurrencycontrol-121201053857-phpapp01/95/transaction-concurrency-control-24-638.jpg?cb=1354340463</a></figcaption></figure><p>Concurrency has two advantages:</p><ul><li><strong>Improved resource utilization and efficiency of the system</strong>: If transactions happen one after another, the number of transactions completed within a given amount of time would be greatly reduced, and work can pile up. As more transactions are waiting in queue, resources are allocated to keep them idling, which prevent them from being used elsewhere. With concurrency, these two problems would be prevented.</li><li><strong>Reduced waiting time</strong>: Transactions can be short or long. A short transaction may need to wait an indefinite amount of time to be executed as it waits for a long transaction to be completed if they are completed serially. However, if they can happen simultaneously, both transactions can be completed in a shorter period.</li></ul><p>So how can we combine the advantages of both serializability and concurrency? Schedules and concurrency control mechanisms help. Schedules arrange each step of the transaction in a serial pattern so the transaction can go to completion without worry of interference from other transactions. Concurrency control mechanisms manage the flow of transactions and improve the efficiency of the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*HOd7lQabpL3jOP4de3YH5Q.jpeg" /><figcaption><a href="https://image.slidesharecdn.com/transactions-and-concurrency-control-patterns-170223193756/95/transactions-and-concurrency-control-patterns-15-638.jpg?cb=1487878836">https://image.slidesharecdn.com/transactions-and-concurrency-control-patterns-170223193756/95/transactions-and-concurrency-control-patterns-15-638.jpg?cb=1487878836</a></figcaption></figure><h3><strong>Types of Concurrency Control Mechanisms</strong></h3><p>There are many types of concurrency control mechanisms. I will only introduce lock-based protocols here but other mechanisms are multiple granularity mechanisms, timestamp-based protocols such as Thomas’ Write Rule, validation-based protocols also associated with optimistic concurrency control mechanisms, multiversion schemes, and snapshot isolation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*IuuyEjtAGzozb4mWJAKeoQ.png" /><figcaption>Concurrency Control Algorithms: <a href="https://www.researchgate.net/profile/Mehdi_Asadi2/publication/202199234/figure/fig1/AS:305900326801408@1449943729411/Classification-of-Algorithms-in-the-Concurrency-Control.png">https://www.researchgate.net/profile/Mehdi_Asadi2/publication/202199234/figure/fig1/AS:305900326801408@1449943729411/Classification-of-Algorithms-in-the-Concurrency-Control.png</a></figcaption></figure><p><strong>Lock-Based Protocols</strong></p><p>Lock-based protocols, as its name suggests, utilizes locks to manage the access and changes to an item to ensure there is consistency and that each transaction is completed in isolation while allowing multiple transactions to happen at the same time in some situations. When a transaction holds a lock on a data item, it prevents other transactions from modifying it. Some important notes about this type of mechanism:</p><ul><li><strong>Shared-mode</strong> (read-access only): Multiple transactions with this lock can access the same item at the same time</li><li><strong>Exclusive-mode</strong> (read/write access): Only one transaction can have this lock at a time.</li><li>The workflow is: transaction request a lock in the appropriate mode depending on the operations it wants to perform on the item → request goes to concurrency control manager → concurrency control manager grants lock to the transaction.</li></ul><p><strong>Compatibility function:</strong></p><ul><li>If a transaction already has a shared mode lock and another transaction requests a lock to access the same item, one of two ways can happen:</li><li>If the lock being requested for is also a shared mode lock, the concurrency control manager will grant the lock.</li><li>If the lock being requested for is not a shared mode lock (thus an exclusive mode lock), the transaction will need to wait until those with the shared mode lock releases their lock.</li><li>Overall this concept shows that multiple transactions can concurrently read a data item but only one can write into it, only if no one else is accessing the item. This prevents those reading the items from reading inconsistent copies.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*d6PudVuzmwqwDfp0ha30Ww.jpeg" /><figcaption>Lock Compatibility Matrix: <a href="https://image.slidesharecdn.com/transactions-and-concurrency-control-patterns-170223193756/95/transactions-and-concurrency-control-patterns-19-638.jpg?cb=1487878836">https://image.slidesharecdn.com/transactions-and-concurrency-control-patterns-170223193756/95/transactions-and-concurrency-control-patterns-19-638.jpg?cb=1487878836</a></figcaption></figure><p><strong>Two-Phased Lock Protocol</strong></p><ul><li>A lock-based protocol used a lot in enterprise systems, which splits into two phases: growing phase (initial phase when the data is accessed), and shrinking phase (end phase when the lock is released).</li><li>Scheduling of concurrent transactions is based on the lock points, the moment when a lock is released.</li></ul><p>I am just barely scratching the surface in the world of transactions and concurrency control but already I can tell how important it is to learn about these concepts because more knowledge on these concepts will help with better understanding how best to manage the flow of data when working with databases to avoid conflict situations such as loss of data and inconsistency.</p><p><strong>References:</strong></p><ul><li>Abraham Silberschatz, Henry F. Korth, and S. Sudarshan: <strong><em>Database System Concepts, 6th Edition</em></strong> (McGraw-Hill 2010)</li><li>GeekforGeeks: <a href="https://www.geeksforgeeks.org/concurrency-control-introduction/">https://www.geeksforgeeks.org/concurrency-control-introduction/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97cf0c08e32e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning About Git Large File System (LFS)]]></title>
            <link>https://medium.com/swlh/learning-about-git-large-file-system-lfs-72e0c86cfbaf?source=rss-e0fbb3b20e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/72e0c86cfbaf</guid>
            <category><![CDATA[large-file-storage]]></category>
            <category><![CDATA[git-lfs]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Amy Li]]></dc:creator>
            <pubDate>Tue, 21 May 2019 19:11:37 GMT</pubDate>
            <atom:updated>2019-05-26T11:50:12.980Z</atom:updated>
            <content:encoded><![CDATA[<p>The first time I encountered Git LFS was in my third week of data science bootcamp. Some of you might already be thinking — whoa, just git is difficult enough, why learn about another feature of git? Or most importantly…what is Git LFS and why should I continue reading this article right now? Well, you are not alone. I was also asking myself these questions when I first started learning about it.</p><p>I guess I should provide some context around how I came to hear about Git LFS to explain why I decided to dive into this. My partner and I were working on a project involving movie data analysis of information from OMDB, Rotten Tomatoes, and IMDB. As we were pushing our work to Github, we received an error message while uploading the IMDB datasets, like the one below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZH5xf3YhBwxLs6CtcrP-FA.png" /><figcaption>Error message received in git when trying to push large files to remote repository on Github</figcaption></figure><p>It was at this time that I learned that git and Github have file size limits of 100MB. Files with a size of 50MB trigger a warning message but can still be pushed through.</p><p>At the time, we were under a deadline to submit our project and of course, as beginners to git, the first thing we thought was: “OK… let’s just try to undo it and push up the files one by one instead.”</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3orieSCjpTnfFa68Cs%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3orieSCjpTnfFa68Cs%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F3orieSCjpTnfFa68Cs%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="331" frameborder="0" scrolling="no"><a href="https://medium.com/media/d500972e275140254b5733caa72e733d/href">https://medium.com/media/d500972e275140254b5733caa72e733d/href</a></iframe><p>Spoiler: it did not work. So, we went with the next option, followed the error message and tried Git Large File Storage.</p><p>To understand Git LFS, it is helpful to know about git. So, before I dive into this topic, let’s talk briefly about git.</p><h3>What is Git?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*BoxIGVbunpIvAsp3SpymtQ.png" /><figcaption>Git logo from its website: <a href="https://git-scm.com/downloads/logos">https://git-scm.com/downloads/logos</a>.</figcaption></figure><p>According to Atlassian’s Bitbucket <a href="https://www.atlassian.com/git/tutorials/git-lfs">website</a>:</p><blockquote>Git is a <strong>distributed version control system</strong>, meaning the entire history of the repository is transferred to the client during the cloning process.</blockquote><p>Let’s back up a little. What is a version control system? A <strong>version control system </strong>is a tool which manages the <em>changes</em> to source code, files, and other forms of information. Changes are tracked as <strong>commits</strong>, which are <em>snapshots</em> of edits at a particular time. A <strong>distributed version control system</strong> is a <em>type of version control system</em>, which allows for the entire code base including all of its history (all of the changes) to be on each developer’s computer. This allows each developer working on the project to see the entire timeline of edits to the project.</p><p>Git was first created in 2005 by Linus Torvalds, the creator of Linux, after he and other developers stopped using Bitkeeper, a propriety source control management (SCM) system, when it was no longer available for free use. According to <a href="https://en.wikipedia.org/wiki/Git">Wikipedia</a>, after trying with no luck to find a free and open system to replace Bitkeeper, Linus decided to create a version control system that would be <strong>small</strong>, <strong>fast</strong> (“take no more than 3 seconds”), supports <strong>branching (</strong>helpful for team collaboration and software development), be the complete opposite of <a href="https://en.wikipedia.org/wiki/Concurrent_Versions_System">concurrent version systems</a>, and includes <strong>safeguards for data assurance</strong>.</p><p>Because git was created to be small and fast, it was primarily built to support only source code, not large files. Remember, it is a <em>distributed </em>version control system, which means the full history of changes of a project is transferred each time it is cloned or pulled. As a commit is added, it adds to the history, which increases the file size of the overall project, and over time, it can become unwieldy.</p><p>However, there are many fields that work with large files for their projects, such as those involving music files, picture files, and datasets. What can we do in these situations? That is where Git LFS comes in.</p><h3>What is Git Large File Storage (LFS)?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*g2RxGAoBAbuTsvB3UBtycw.png" /><figcaption>Git LFS logo from Atlassian: <a href="https://wac-cdn.atlassian.com/dam/jcr:83e62b1a-36ee-4fc1-82de-7a06640f7c87/hero.svg?cdnVersion=340">https://wac-cdn.atlassian.com/dam/jcr:83e62b1a-36ee-4fc1-82de-7a06640f7c87/hero.svg?cdnVersion=340</a></figcaption></figure><p>Git LFS is a git extension, programmed in Go, which was created by developers at Atlassian and Github as well as other open source collaborators to circumvent the file size restriction in git. It does this by storing large files in a separate location from your repository and placing a pointer file in your repository directing to its location.</p><p>The best way for me to understand how this works was to first forget about Github, Bitbucket, Gitlab, and remote repositories for a second. Let’s just focus on the local computer, which is depicted as a monitor below with three sections (“Working Copy”, “Local Repository”, “LFS Cache”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NowszVQWdcZpgVL7FiKacg.png" /><figcaption>Git LFS structure image from Tower: <a href="https://www.git-tower.com/learn/content/01-git/01-ebook/en/02-desktop-gui/05-advanced-topics/07-git-lfs/03-setup-git-lfs.png">https://www.git-tower.com/learn/content/01-git/01-ebook/en/02-desktop-gui/05-advanced-topics/07-git-lfs/03-setup-git-lfs.png</a>.</figcaption></figure><p>The <strong>local repository</strong> is a directory or folder you see on your computer, which has been initialized as a git repository using <em>git init</em><strong> </strong>or <em>cloned</em> from a remote repository. The <strong>working copy</strong> is the representation of the files or folders that are being edited in the local repository. The <strong>LFS cache</strong> is the separate storage for large files once they are pushed through git. Keep these terms in mind — they will come in handy as we go through the steps on how to work with Git LFS in the next section.</p><h3>Working with Git LFS</h3><p>The great thing about Git LFS is that we can continue to use the usual git commands and workflow we all know and love with it. The only changes are a few additional commands and another storage location to keep in mind.</p><p>Ok, now that we have some information about git and Git LFS, let’s walk through how to use it. I will go through two possible scenarios but first, download Git LFS via Homebrew (<strong>brew install git-lfs</strong>) or through their website (<a href="https://github.com/git-lfs/git-lfs/releases"><strong>https://github.com/git-lfs/git-lfs/releases</strong></a>).</p><p><strong><em>Scenario 1</em>: Using Git LFS after getting an error message with the usual git commands.</strong></p><p>Here I have a new repository in which I placed a large data file (1.9GB). I wanted to make sure any changes to the data file are tracked and eventually backed up remotely. First, I go through the usual git commands to stage the file (<strong>git add</strong>), save a copy of the changes on the local repository (<strong>git commit</strong>), and push the copy to the remote repository (<strong>git push</strong>). This is the output I get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*hVYxRJv8uvY93XOarDLpLw.png" /><figcaption>Error message after going through the usual git commands — note the error is because the file is too large</figcaption></figure><p>How should I resolve this error? One option is to undo the changes using git reset and either forget about saving the file, zip the file to compress it to a smaller size, or restart with Git LFS. Another option would be to stay where you are and integrate Git LFS so you can continue the process, which is what we will focus on here.</p><p><strong>Step 1: Once Git LFS is installed, enable the specific repository with Git LFS by running git lfs install.</strong></p><p>Although we have installed Git LFS on your computer, we will need to tell it which repositories need its service. A great analogy is a storage company. Storage companies are available throughout the city where we can choose to store your items but they do not automatically knock on the door and start storing the items. Instead, the first step is to start a relationship with the company by calling and setting up an agreement. It is the same here. To enable Git LFS “services” in a specific repository or to tell Git LFS the repository to initialize its “services”, run <strong>git lfs install</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*8E49xP_oMqTKvnv24rbfDg.png" /><figcaption>Run git lfs install to initialize your repository with Git LFS. Successful initialization of Git LFS in the repository.</figcaption></figure><p><strong>Step 2: Tell Git LFS which files to track with the command: git lfs track “*.file_extension”.</strong></p><p>Again, we need to tell Git LFS what files or what types of files we would like it to track so the files can be stored at a separate location instead of in git to avoiding getting the error message again. To do so, run <strong>git lfs track “*.file_extension”</strong>. For example, if all csv files need to be tracked, run git lfs track “*.csv” or if all jpeg image files need to be tracked, run git lfs track “*.jpg”. The asterisk (*) represents all files. The quotes (“ ”) are necessary when running this code. Without them, there will be an error later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*QHpRMsFUJImqi2B7uwEEHA.png" /><figcaption>Run git lfs track “*.csv” to tell Git LFS to start tracking all csv files. It is also possible to tell git lfs to track specific files like in the image above.</figcaption></figure><p>Same as how a receipt will be received when an order is placed with a storage company to start storing an item, when we track a file with Git LFS, a <strong>.gitattributes</strong> file will be created. If there is already a .gitattributes file, the file is added as a new line in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*1txZ8_wdY7zvEkQW3RTuXg.png" /><figcaption>Once a file is tracked by Git LFS, a .gitattributes file is created and the file being tracked will be listed in this file.</figcaption></figure><p><strong>Step 3: Git add, commit, &amp; push your .gitattributes file to your repo.</strong></p><p>Similar to the .gitignore file, as Git LFS tracks new files, updates are automatically made to the .gitattributes file. To make sure the changes are being tracked, each time the .gitattributes file is updated, it needs to be staged and commited, otherwise issues may occur later on.</p><p><strong>Step 4: Now the real secret in this scenario is to use git lfs migrate to move your commits from git to Git LFS.</strong></p><p>What allows us to stay in the current state, not have to undo our commits and restart, is a nifty line of code that lets us move or “migrate” our commits from git to Git LFS. To move our commits, we can run <strong>git lfs migrate import — include “*.file_extension”</strong>. In order to see what file types are in the commits and can be tracked by Git LFS, we can run <strong>git lfs migrate info</strong>. By moving our commits over, we can continue to the next step: pushing our changes Github. More details in the next section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*oi1ZG1F09-8yagdlHVzsnw.png" /><figcaption>Run git lfs migrate info to obtain information on the types of files that are in the commits being moved over to Git LFS. Run git lfs migrate import — include “*.csv” to move the commits of csv files to Git LFS.</figcaption></figure><p><em>Important Note: Moving commits involves rewriting the history. </em>A tag can be added to prevent overwriting the changes listed in the history but this will prevent this line of code from running.</p><p><strong>Step 5: Lastly, run git push to push the changes to Github and the large commits (ie. large files) to Git LFS.</strong></p><p>After migrating the commits to Git LFS, currently we have a local git repository which has been updated with a change (in this case, added a new data file, which is indicated by a pointer file directing to Git LFS) and a local Git LFS cache, which now stores the data file. In the next step, we push the changes to Github. The local git repo with files within the file size criteria (ie. source code, pointer file) will be stored to Github, which is the Git host indicated in the image below, and the Git LFS cache will be stored in the Git LFS store on the cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*0ApCZW7uT2eqNTzB9lqjJg.png" /><figcaption>With git push, a copy of the git repo is pushed to Github and the Git LFS cache is pushed to Git LFS store on the cloud. Image source: <a href="https://www.atlassian.com/git/tutorials/git-lfs">https://www.atlassian.com/git/tutorials/git-lfs</a></figcaption></figure><p><strong><em>Scenario 2:</em> Using Git LFS from the beginning.</strong></p><p>If it is known that there are large files in the repository, we can use Git LFS from the beginning by going through steps 1 to 3 listed above. After going through these steps, return to the usual git commands (git add, git commit) to stage and save the changes in the local repo. Then, complete step 5 listed above, to push the changes to Github or other Git host and the remote Git LFS store.</p><h3>Conclusion</h3><p>Seems pretty simple right? Just remember the five steps above and we should be good to go. Pulling down the changes from a remote repository is also straightforward. It is the same set of git commands we typically use: git pull or git fetch and git merge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*fCEtSBO1FuECRQs5VKFimw.png" /><figcaption>Using git pull to pull down the changes which include the data file saved in Git LFS. When pulling down changes from the remote repo, changes from the remote repo will be pulled (as usual) and any objects saved in Git LFS will replace the pointer file in the pull down to the local computer.</figcaption></figure><p>Well, it was actually pretty confusing to learn at first. Here are some notes on what I learned as I fumbled my way through this:</p><ul><li>For those uncomfortable with git, this can add another layer of complexity. This was my biggest challenge when learning about Git LFS. Learning more about the git commands, git workflow, and how Git LFS fits in with git was key to learning the steps. This <a href="https://www.atlassian.com/git/tutorials/git-lfs">video</a> on Atlassian’s website gave me the “epiphany” I needed to put it all together.</li><li>Even with Git LFS, there is still a file size limit of 2GB, which is a restriction placed by Github. Anything bigger, it is probably time to look into cloud storage.</li><li>Git LFS is an active open source project which is continuously being improved. Their github keeps a running list of current issues <a href="https://github.com/git-lfs/git-lfs/issues">here</a>.</li><li>There are still issues when trying to resolve merge conflicts. It is best to communicate within the team before pushing any changes and merging.</li><li>Larger files can still be a bit slow when being pushed to the remote repo.</li></ul><p>Overall, I really enjoyed learning about this resource and it was very helpful in making me more comfortable with using git.</p><h3>Resources</h3><p>I would not have been able to understand anything about this topic without the knowledge gleaned from the below resources. I recommend checking them out.</p><ul><li><strong>Git LFS Website</strong>: <a href="https://git-lfs.github.com/">https://git-lfs.github.com/</a></li><li><strong>Atlassian Git LFS Tutorial</strong>: <a href="https://www.atlassian.com/git/tutorials/git-lfs">https://www.atlassian.com/git/tutorials/git-lfs</a></li><li><strong>GitLab Git LFS Documentation:</strong> <a href="https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html">https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html</a></li><li><strong>Dzone — What is Git LFS</strong>: <a href="https://dzone.com/articles/learning-git-what-is-git-lfs">https://dzone.com/articles/learning-git-what-is-git-lfs</a></li><li><strong>Oh Shit Git</strong>: <a href="https://ohshitgit.com/">https://ohshitgit.com/</a></li><li><strong>Visualizing Git</strong>: <a href="https://git-school.github.io/visualizing-git/">https://git-school.github.io/visualizing-git/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72e0c86cfbaf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/learning-about-git-large-file-system-lfs-72e0c86cfbaf">Learning About Git Large File System (LFS)</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>