<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ag on Medium]]></title>
        <description><![CDATA[Stories by Ag on Medium]]></description>
        <link>https://medium.com/@ag4077?source=rss-35f98b92b4f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*n5myeuYh4wA8awVU.jpg</url>
            <title>Stories by Ag on Medium</title>
            <link>https://medium.com/@ag4077?source=rss-35f98b92b4f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:40:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ag4077/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Speed up Pandas through reduced memory usage]]></title>
            <link>https://medium.com/@ag4077/speed-up-analysis-while-using-pandas-200586c3b844?source=rss-35f98b92b4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/200586c3b844</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[writing]]></category>
            <dc:creator><![CDATA[Ag]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 21:46:58 GMT</pubDate>
            <atom:updated>2022-03-19T19:25:22.496Z</atom:updated>
            <content:encoded><![CDATA[<p>Pandas is data scientists’ best friend until you start working on large datasets. While most of the prototype code will get repurposed in a more efficient way, analyzing large data, and waiting for results to appear on your console is no fun.</p><p>Many of us start doing operations on Numpy rather, which makes things much faster, but Numpy, unlike Pandas, does not have all the cool easy-to-use features.</p><p>If you Google for “Optimize Pandas memory management” there will be plenty of articles on how you can downcast “float64” to “float32”, “object” to “category” etc. While all of these are valid steps, there are certain basic things that one could do before starting to play with the data types.</p><p>Following are some basic steps that one can take to help be more efficient while working with large datasets,</p><ol><li>Drop columns that you do not require</li><li>Remove duplicates</li><li>Deal with NA</li><li>Aggregate data if it suits your use case</li><li>Finally, we can play around with different data types</li></ol><h3>Data:</h3><p>We will be using the famous online retail dataset from UCI’s machine learning <a href="http://archive.ics.uci.edu/ml/machine-learning-databases/00352/">repository</a>. The data set is based on a UK Retailer and contains transnational transactions between 1/12/2010 to 9/12/2011. Included are 541,909 instances and 8 columns: InvoiceNo, StockCode, Description, Quantity, InvoiceDate, UnitPrice, CustomerID, and Country.</p><pre>df = pd.read_excel(‘Online Retail.xlsx’)<br>df[std_col.value] = df[‘UnitPrice’] * df[‘Quantity’]<br>df[std_col.id] = df[std_col.id].astype(str)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8R85N_wyzukIgr55IK3Z-Q.png" /></figure><p>We add another column by multiplying two of their existing columns, “Quantity” and “UnitPrice”. We also make the customer ID column as string, just for illustration purposes :) The size of the DataFrame is ~175MB. While this is not particularly large in a commercial context, this will do for our purpose.</p><p>I have saved some constants as a Class for ease of use,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*eshs6HPzcMfpR1nYBG8H3A.png" /><figcaption>Saving constants in class lets you access information more easily and reduces hard coding</figcaption></figure><h3>Objective:</h3><p>The objective is to do customer LTV analysis like the one done in this medium <a href="https://medium.com/geekculture/predicting-customer-life-time-value-cltv-via-beta-geometric-negative-binominal-distribution-59be07ac30bd">post</a>.</p><h3>Step 1: Drop columns</h3><p>This is the most obvious and the one that can have the most impact. In this case, except for “InvoiceDate”, “CustomerID”, “Country”, “InvoiceNo”, and “original_amount” I do not want any other column.</p><pre>df_select = df[[std_col.date, std_col.id,std_col.country,std_col.invoice, std_col.value]]</pre><p>This result’s in a 43% memory saving! The size of the new DataFrame is 100MB.</p><p>Small side note, I use the following function to compare the DataFrame size,</p><pre>def compare_pandas_size(_df_pre:pd.DataFrame, _df_post:pd.DataFrame)-&gt; None:</pre><pre>‘’’<br>This function is used to compare memory usage of two dataframes</pre><pre>Parameters:<br>_df_pre (pd.DataFrame): The reference df<br>_df_post (pd.DataFrame): The df whose memory usage needs to be evaluated</pre><pre>Returns:<br>None</pre><pre>‘’’</pre><pre>df_pre = _df_pre.copy()</pre><pre>df_post = _df_post.copy()</pre><pre><br># Get memory usage</pre><pre>new_space = sys.getsizeof(df_post)*std_val.b_to_mb</pre><pre>old_space = sys.getsizeof(df_pre)*std_val.b_to_mb</pre><pre>space_savings = 1 — (new_space/old_space)</pre><pre><br># Print output</pre><pre>print(f’Current size: {new_space}’)</pre><pre>print(f’{space_savings: .0%} space savings’)</pre><pre><br>return None</pre><h3>Step 2: Remove duplicates</h3><p>Once you understand the dataset you would know how to evaluate duplicates. In this case, before removing duplicates I also wanted to differentiate between normal orders and return orders. So that even by mistake I would not remove a return order by mistake.</p><p>In this dataset, the “InvoiceNo” is preceded by “c” in cases of canceled/returned orders. A little bit of regex to our rescue,</p><pre>df_select_rem_dup[std_col.return_order] = df_select_rem_dup[std_col.invoice].map(type) == ‘str’</pre><pre>df_select_rem_dup[std_col.invoice] = df_select_rem_dup[std_col.invoice].replace(regex=True,inplace=False,to_replace=r’\D’,value=r’’)</pre><p>Now, we remove duplicates!</p><pre>df_select_rem_dup = df_select_rem_dup[~(df_select_rem_dup.duplicated(subset=[std_col.date,std_col.id,std_col.invoice,std_col.value,std_col.return_order]))]</pre><p>We further saved 42% of memory usage and are using 58MB now. This is despite adding one additional column.</p><h3>Step 3: Deal with NA</h3><p>This step again requires some level of understanding of the dataset and does not have a “one-size fits all” sort of solution. We first evaluate where we might have NA,</p><pre>df_select_rem_dup.isnull().sum(axis=0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*-Qi6SgRwKbNRceGR39S9Gw.png" /></figure><p>Wow, no null values! It’s not always happy news when you have “object” columns though. NA can sneak up on you,</p><pre>df_select_rem_dup_na[std_col.id] = df_select_rem_dup_na[std_col.id].replace(‘nan’,np.nan)</pre><pre>df_select_rem_dup_na[std_col.country] = df_select_rem_dup_na[std_col.country].replace(‘nan’,np.nan)</pre><pre>df_select_rem_dup_na[std_col.invoice] = df_select_rem_dup_na[std_col.invoice].replace(‘nan’,np.nan)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*s_g_Pu9nItWwOf25eUC_Tw.png" /></figure><p>Now we are talking! We can now remove rows where there are no “CustomerID”, those are absolutely useless for this particular analysis.</p><pre>df_select_rem_dup_na = df_select_rem_dup.copy()</pre><pre># remove NA where the row might not be that useful for the purpose of analysis</pre><pre>df_select_rem_dup_na = df_select_rem_dup_na[(df_select_rem_dup_na[std_col.id].notnull())]</pre><p>We further saved 16% of memory usage and are using 48MB now.</p><h3>Step 4: <strong>Aggregate data if it suits you</strong></h3><p>This step also requires some level of understanding of the dataset and is unique for the kind of analysis that you intend to do with the dataset. In this particular situation, I am not quite interested in having individual product-level information, all I am interested is in order-level information. We there Groupby on “InvoiceDate” and “CustoemrID”,</p><pre>df_pre_final = df_select_rem_dup_na.copy()</pre><pre>df_pre_final = df_pre_final.groupby([std_col.invoice,std_col.id]).agg({std_col.value:’sum’,std_col.country:’unique’,std_col.date:’min’}).reset_index()</pre><pre>df_pre_final[std_col.country] = df_pre_final[std_col.country].apply(lambda x: str(x[0]))</pre><pre>compare_pandas_size(_df_pre = df_select_rem_dup_na,_df_post = df_pre_final)</pre><p>This step is one of the most rewarding steps! We saved 93% in terms of memory usage and are now using just 4MB.</p><h3>Step 5: Finally, playing with data types</h3><p>In most of the articles on memory management in Pandas people generally skip to this step while having multiple other vital things that one could do to make life easier.</p><p><a href="https://towardsdatascience.com/staying-sane-while-adopting-pandas-categorical-datatypes-78dbd19dcd8a">This </a>article does an amazing job in explaining the nitty-gritty details involved in dealing with “category” data type. I don’t think I can do a better job at it. What I can tell you is that “category” does not always help in saving space. This will be the case if there are too many unique values in your column, If there are a smaller number of unique values then the category data type cannot be beaten! Here is an example of it,</p><p>We know that there are a lot of “CustomerID”s, so if you were to try to convert it to “category” then see what happens,</p><pre>df_temp = df_pre_final[[std_col.id,std_col.date,std_col.value]]</pre><pre>get_memory_savings(_df = df_temp[[std_col.id,std_col.date,std_col.value]], dtypes_to_tackle = [‘object’])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*KBSX_xEjssYe0uxX5iP2KA.png" /></figure><p>In this case, we would have been better off having the column as an “object”. There is a misconception that we might be better off using “str” instead of “object”, but as seen above it is not always the case. There is a secret better option, which is what you see here as “int32”. What we essentially do there is, we actually do not care about the actual “CustomerID”, all we care about is that each of them should be uniquely distinguishable. Therefore, instead of using the actual “CustomerID”, we use a dummy integer which helps us save more space. The following function can be used for this purpose,</p><pre>def convert_str_to_uid_int(_df_input:pd.DataFrame, col_to_convert_to_int: str, int_type: str = ‘int32’) -&gt; tuple:</pre><pre>‘’’</pre><pre>This function is used to map unique values in a column to integer with objective of saving memory</pre><pre>Parameters</pre><pre> — — — — — — — — — — — — — — — — — — — — </pre><pre>_df_input (pd.DataFrame): Input dataframe</pre><pre>col_to_convert_to_int (str): column name in _df_input that needs to coded up using random numbers</pre><pre>int_type (str): Deafult: “int32” or altenatively use “int64”</pre><pre>Returns</pre><pre> — — — — — — — — — — — — — — — — — — — -</pre><pre>(pd.DataFrame, pd.DataFrame)</pre><pre>‘’’</pre><pre>df_input = _df_input.copy()</pre><pre># Input validation</pre><pre>if not isinstance(df_input, pd.DataFrame):</pre><pre>raise ValueError(f’df_input needs to be a dataframe but got {type(df_input)}’)</pre><pre>if col_to_convert_to_int not in _df_input.columns:</pre><pre>raise ValueError(f’The input dataframe does not have the column by the name {col_to_convert_to_int}’)</pre><pre>if int_type not in [‘int32’,’int64&#39;]:</pre><pre>raise ValueError(‘The function supports int32 and int64 alone’)</pre><pre># Get a dictionary that maps the column value to an unique int value</pre><pre>df_mapping = pd.DataFrame()</pre><pre>df_mapping[col_to_convert_to_int] = df_input[col_to_convert_to_int].unique()</pre><pre>df_mapping = df_mapping.reset_index()</pre><pre>df_mapping = df_mapping.rename(columns={‘index’:col_to_convert_to_int+’_int’})</pre><pre>df_mapping = df_mapping.set_index([col_to_convert_to_int])</pre><pre>dict_mapping = df_mapping.to_dict()[col_to_convert_to_int+’_int’]</pre><pre># Replace the col value with the int value</pre><pre>df_input[col_to_convert_to_int] = df_input[col_to_convert_to_int].map(dict_mapping)</pre><pre>df_input[col_to_convert_to_int] = df_input[col_to_convert_to_int].astype(int_type)</pre><pre>return df_input, df_mapping</pre><p>If we were to analyze the impact of different data types on “Country” column which would naturally have a much lesser number of unique values. In this case, as mentioned before, “category” data type cannot be beaten!</p><pre>get_memory_savings(_df = df_pre_final[[std_col.id,std_col.country,std_col.value]], dtypes_to_tackle = [‘object’])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*LLEYt8XisMRpiiBdvJZd3Q.png" /></figure><p>I typically am very skeptical about using the “category” data type since we need to do things differently as indicated by this <a href="https://towardsdatascience.com/staying-sane-while-adopting-pandas-categorical-datatypes-78dbd19dcd8a">article</a>. Since we anyway have a better option, we go with the strategy of mapping unique string values to unique int values. We saved 31% and are now left with 2.9MB.</p><p>Another place where we could potentially save up some memory is by downcasting “float64” to “float32” or “float16”.</p><p>The disadvantages of “float16”:</p><ol><li>Makes anything more than ~99999 as inf, so if your column could take on values &gt;99,999 then this wouldn’t be a good choice.</li><li>It can hold up to 3 digits perfectly well, after that it starts approximating</li></ol><p>The disadvantages of “float32”:</p><ol><li>This does not have limitations like “float16” in terms of values that it can take on, so you wouldn&#39;t end up with a bunch of inf in this case</li><li>It holds information till the 7th digit (with/without decimal) perfectly, after that it will start approximating. For example, 12,540,000,099 will be saved as 12,540,000,000.</li></ol><p>“float32” would work for most cases and it is ~50% smaller in comparison to float64 but the majority of pandas operations do not speed up with “float32”, maybe because float data type, in general, does not occupy much memory. On top of that, if we were to groupby on a “float32” column, Pandas would automatically convert it to “float64” so the effort would go to waste. Therefore, I am going to stick with “float62”.</p><p>Alright, we are done. Let’s evaluate where we stand. We started with a dataset that was 175MB and finally ended up with a DataFrame that is just 3MB in size. We were able to successfully shrink our memory usage by 98% without having to play around too much with data types. This can have significant performance improvement, for example, a simple groupby operation would be 28x faster because of the reduced memory usage.</p><p>That’s it from me, thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=200586c3b844" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A beginners guide to collaborating using GitHub (Part-II)]]></title>
            <link>https://medium.com/@ag4077/a-beginners-guide-to-collaborating-using-github-part-ii-9af40a97d34e?source=rss-35f98b92b4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9af40a97d34e</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Ag]]></dc:creator>
            <pubDate>Sat, 22 Jan 2022 13:04:55 GMT</pubDate>
            <atom:updated>2022-01-22T13:04:55.762Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/@ag4077/a-beginners-guide-to-python-set-up-part-i-1b196475a910">part-I</a> of this two-part series, we went through the steps required to set up a local Python environment using Conda and VSCode. We also saw how we can integrate with GitHub. In this article, we shall dig deeper in order to take advantage of GitHub’s version control and more importantly its collaboration features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kDF4BOeuDlpUS6iz" /><figcaption>Photo by <a href="https://unsplash.com/@synkevych?utm_source=medium&amp;utm_medium=referral">Roman Synkevych</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>How do you add collaborators to a project?</strong></p><p>This step is pretty straightforward, please follow the instructions in the video below to add a collaborator to your project,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fbnv8xwWiwOQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dbnv8xwWiwOQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fbnv8xwWiwOQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c08b3812dc3b19960c457fc5a58e7e6d/href">https://medium.com/media/c08b3812dc3b19960c457fc5a58e7e6d/href</a></iframe><p><strong>How does a collaborator replicate a project locally?</strong></p><p>Anyone with access to a project can replicate a repository on GitHub locally. You can do it in the following way,</p><ol><li>Copy the URL of the library that you are trying to clone</li><li>Open VSCode → source control (Ctrl + Shift + G) → clone repository → paste URL → click clone from GitHub → select repository location</li><li>We need to select a Python interpreter for the project. In order to isolate the installations for this project, create a virtual environment as indicated in <a href="https://medium.com/@ag4077/a-beginners-guide-to-python-set-up-part-i-1b196475a910">part-I</a> of this article.</li><li>Once a virtual environment is created, you can select it for this project in VSCode. You can do that by pressing [ctrl + shift + p] → Python: select interpreter → selects the virtual environment you created above.</li><li>We need to install all the dependencies of the project. You can do that by opening up Terminal → select Command Prompt by pressing the + sign → type <em>pip install -r requirements.txt</em></li></ol><p>That’s it! You have now successfully replicated a project locally.</p><p><strong>How do you collaborate?</strong></p><p>GitHub has its own nuances. Do a bit of research to understand it better. We will go through some basic must-know techniques to make this journey easier for you,</p><p>Step 1: Keep your master branch updated as much as possible → <em>git pull</em> in the terminal while on the master branch should do the trick</p><p>Step 2: To make some changes to a repository, the best practice is to create a separate <a href="https://www.youtube.com/watch?v=iJKIxrJ40ss&amp;ab_channel=TheCodex">branch</a>, make changes, and then commit (the tick mark)+push (will be accessible once you click the three dots) the changes from Source Control in VSCode.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZcaQCT0PiII%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZcaQCT0PiII&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZcaQCT0PiII%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/667b6e4dfae6b1197b98fd1ac96e0419/href">https://medium.com/media/667b6e4dfae6b1197b98fd1ac96e0419/href</a></iframe><p>Step 3: You can then visit the project’s GitHub page, and create a pull request. You can add reviews to the pull request as well.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhKmtVVRCrJs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhKmtVVRCrJs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhKmtVVRCrJs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/000529bde01805855689ec440fda57cc/href">https://medium.com/media/000529bde01805855689ec440fda57cc/href</a></iframe><p>Step 4: Once approved, the changes will get reflected in the master branch. Others collaborating in the project could do a <a href="https://www.youtube.com/watch?v=v4g6y_HsgpA&amp;ab_channel=GitKraken"><strong><em>git pull</em></strong></a><strong><em> </em></strong>in their terminal to replicate changes onto their respective local repository.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FEN0xV3n7_pw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEN0xV3n7_pw&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee236ce2139ae62c8d78c41b5742ee88/href">https://medium.com/media/ee236ce2139ae62c8d78c41b5742ee88/href</a></iframe><p>Thanks a lot for reading the article. Hope you found it useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9af40a97d34e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A beginners guide to Python set up — Part I]]></title>
            <link>https://medium.com/@ag4077/a-beginners-guide-to-python-set-up-part-i-1b196475a910?source=rss-35f98b92b4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b196475a910</guid>
            <category><![CDATA[virtual-environment]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ag]]></dc:creator>
            <pubDate>Sun, 16 Jan 2022 02:55:56 GMT</pubDate>
            <atom:updated>2022-01-22T13:05:50.883Z</atom:updated>
            <content:encoded><![CDATA[<h3>A beginners guide to Python set up — Part I</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RjRhagVdoYQly4bm" /><figcaption>Photo by <a href="https://unsplash.com/@florianolv?utm_source=medium&amp;utm_medium=referral">Florian Olivo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Are you relatively new to the world of coding? Is Python your go-to programing language? Do most of your code reside in JupyterHub and Jupyter Notebooks? If so, read on!</p><p>Please note that some parts of the instructions might be Windows-specific, but shouldn’t be very different for Mac.</p><p><strong>Why would this article be useful?</strong></p><p>While Python Notebook is a useful tool for data exploration and data analysis, Notebooks are cumbersome to manage once you have gone through a reasonable number of projects. Collaborating with others while using Notebooks is no fun as well.</p><p>There are industry-grade solutions for these problems, and we would be taking a leaf out of techies playbook. While there are some coding style tweaks that one needs to go through, including but not limited to, writing functional code, minimizing/eliminating hard coding, writing test cases, etc. The biggest mental block comes from the sheer number of software and platforms that one needs to navigate. That’s one of the main reasons why a lot of code lives and dies on Jupyter Notebooks. I thought I could share my rather minimal knowledge in this subject hoping that some of the readers find it useful.</p><p><strong>What do we intend to achieve?</strong></p><p>Before we get knee-deep into the details, let&#39;s set our agenda straight. We want to achieve the following,</p><ul><li><strong>Minimal disruption </strong>— Code in Python the way we normally do. Access to Python Notebook wherever required while having access to other handy tools.</li><li><strong>Maintain a project-specific virtual environment</strong> — This will help us maintain library dependencies of our code. For example, there are certain features of an older version of Numpy that become incompatible with a newer version. We cannot always be modifying our code to make sure it works with a different version. Watch this video (till 4:30) on virtual environments to understand this concept better.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FJWcrseHrh2w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DJWcrseHrh2w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FJWcrseHrh2w%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2a62185641923de235178e79742fd10/href">https://medium.com/media/f2a62185641923de235178e79742fd10/href</a></iframe><ul><li><strong>Version control</strong> — A way for us to properly version our code (instead of marking each version as v1, v20, etc) so that we can always go back to an earlier version if required. This will be useful even if we delete a file.</li><li><strong>Collaborate with others </strong>– Be it in an office or in a university setting, working with others on projects involving coding is very common. How do you split work? How do you integrate? How do you manage interdependencies? Just like groups in social media operate, there are tools to use for this as well – enter GitHub! Refer to the video below to get a better understanding of GitHub.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwpISo9TNjfU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwpISo9TNjfU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FwpISo9TNjfU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a59c5ef33be8870f72b247b8dc49bbde/href">https://medium.com/media/a59c5ef33be8870f72b247b8dc49bbde/href</a></iframe><p>Please note that we will address the last two points on version control and collaboration in part II of this article.</p><p><strong>What do we need?</strong></p><p>Now let&#39;s go through the installations that we need to do,</p><ol><li><strong>Download and install </strong><a href="https://code.visualstudio.com/docs/?dv=win"><strong>VSCode</strong></a> —VSCode is a lightweight IDE (Interactive Development Environment). In order to better understand what an IDE is and why one needs to transition from exclusively coding up in Jupyter and Python Notebooks please refer to the following articles and videos — <a href="https://towardsdatascience.com/top-3-vs-code-extensions-for-python-and-data-science-7462dd4ee530">Article 1</a>, <a href="https://towardsdatascience.com/top-3-vs-code-extensions-for-python-and-data-science-7462dd4ee530">Article 2</a>, <a href="https://www.youtube.com/watch?v=eXinDi55iOk&amp;ab_channel=TechWithTim">Video 1</a>.</li><li><strong>Install the Python extension for VS Code</strong> — Open VSCode, click on Extensions (Ctrl+Shift+X), type “Python” and install the extension. Make sure that the extension you are installing is <a href="https://marketplace.visualstudio.com/items?itemName=ms-python.python">this</a>.</li><li><strong>Create GitHub account – </strong>Go to their website and signup, it’s simple enough.</li><li><strong>Download and install </strong><a href="https://docs.anaconda.com/anaconda/install/windows/"><strong>Anaconda</strong></a><strong> </strong>— We will be using the Conda package management feature which comes along with the installation. This is how we would be creating and managing our virtual environment.</li><li><strong>Create virtual environment</strong> — Search and open “<em>Anaconda Prompt</em>”. Type <em>conda create -n type_your_environment_name python=3.7.5. </em>Please note that Version 3.7.5 is an example here. It is advisable to create a separate virtual environment in cases where that demarcation makes sense.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*-J4tbJ4jCiJ0s-FnjjeYUw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*0kuxkx2qXlEXA5oNkBfHwQ.jpeg" /><figcaption>This is one way to create virtual environment. Notice that you would have to click “y” once before the packages are downloaded and extracted.</figcaption></figure><p><strong>How to get started?</strong></p><p>Now that we have everything we need, let&#39;s get down to business!</p><ol><li>Create a folder on your computer where you want all your code to live.</li><li>Open VSCode. File → Click Open Folder → Navigate to the folder mentioned above and select it.</li><li>Now, let&#39;s select our virtual environment and do some package imports. Open up a new terminal (Ctrl + `), select “cmd” by pressing “+” sign on the right-hand side of the terminal. Type <strong><em>conda activate type_your_environment_name </em></strong>and press enter. Next, type <strong><em>pip install numpy==1.21.5 </em></strong>to install NumPy. Note — If the above command does not work then you might have to assign the path to the location where you installed Anaconda, refer to this <a href="https://stackoverflow.com/a/44597801/17944019">link</a> for further details.</li><li>Open up explorer (Ctrl + Shift +E). You will see your folder name there. Now let&#39;s go ahead and create two files, a Python Notebook file (with .ipynb extension) and a Python script file (with .py extension) under that folder.</li><li>Create a simple function inside the .py file, make sure to install the required packages. Now open up the .ipynb file and import the function that you just wrote by using <strong><em>from your_py_file_name import your_function_name</em></strong><em>. </em>Now try using that function in the .ipynb file it should work. You might have to install ipykernel to run the .ipynb file but VSCode will prompt you to do so. By having your function in a separate .py file you can call them in multiple places. Focus on organizing your functions well so that it is easier to find them later on.</li><li>Let&#39;s get ready to share this project with others! Before we can share, we need to have a way to communicate to others the packages that our project is dependent on. We don’t actually share our virtual environment itself, rather we will create a text file that will communicate the same. In order to do this, go to Terminal (cmd), type <a href="https://stackoverflow.com/questions/62885911/pip-freeze-creates-some-weird-path-instead-of-the-package-version">pip list --format=freeze &gt; requirements.txt</a> this will write all our project dependencies onto the requirements.txt file. You should be able to see the file in the file explorer.</li><li>Let&#39;s bring GitHub into the picture now. Click on Source Control (Ctrl + Shift + G) → Click on <em>Publish to Github </em>→ Login onto Github when prompted and <em>publish to public repository </em>→ select all the files. Now your project is published on GitHub!</li></ol><p>Thanks a lot for reading. In the next article, we will try to go through some basic steps which will allow us to collaborate with others and leverage Github better. Stay tuned!</p><p>Part of this series is available <a href="https://medium.com/@ag4077/a-beginners-guide-to-collaborating-using-github-part-ii-9af40a97d34e">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b196475a910" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>