<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Julian West on Medium]]></title>
        <description><![CDATA[Stories by Julian West on Medium]]></description>
        <link>https://medium.com/@julianwest155?source=rss-257ac04bf615------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rwKpd2LDgiwJjzNriQjWjg.jpeg</url>
            <title>Stories by Julian West on Medium</title>
            <link>https://medium.com/@julianwest155?source=rss-257ac04bf615------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:57:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@julianwest155/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Google Search Console API with Python]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/codex/google-search-console-api-with-python-3146a9f9ea90?source=rss-257ac04bf615------2"><img src="https://cdn-images-1.medium.com/max/2600/1*lE1T48eWK4qvRCloUK90RA.jpeg" width="3456"></a></p><p class="medium-feed-snippet">How to use the Google Search Console API with Python</p><p class="medium-feed-link"><a href="https://medium.com/codex/google-search-console-api-with-python-3146a9f9ea90?source=rss-257ac04bf615------2">Continue reading on CodeX »</a></p></div>]]></description>
            <link>https://medium.com/codex/google-search-console-api-with-python-3146a9f9ea90?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/3146a9f9ea90</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Wed, 24 May 2023 09:46:47 GMT</pubDate>
            <atom:updated>2023-05-24T09:46:47.328Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Unit testing PySpark code using Pytest]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/unit-testing-pyspark-code-using-pytest-b5ab2fd54415?source=rss-257ac04bf615------2"><img src="https://cdn-images-1.medium.com/max/2600/1*sv67yR1CN18Jbv5Ht5G4Qg.jpeg" width="3862"></a></p><p class="medium-feed-snippet">When it comes to writing unit-tests for PySpark pipelines, writing focussed, fast, isolated and concise tests can be a challenge.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/unit-testing-pyspark-code-using-pytest-b5ab2fd54415?source=rss-257ac04bf615------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/unit-testing-pyspark-code-using-pytest-b5ab2fd54415?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5ab2fd54415</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Mon, 16 Jan 2023 13:45:43 GMT</pubDate>
            <atom:updated>2023-01-16T13:45:43.065Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[13 Tips for using PyTest]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/13-tips-for-using-pytest-5341e3366d2d?source=rss-257ac04bf615------2"><img src="https://cdn-images-1.medium.com/max/2600/1*B6oWzUw7FJZifK2E_pWPeA.jpeg" width="3886"></a></p><p class="medium-feed-snippet">Unit-testing is a really important skill for software development.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/13-tips-for-using-pytest-5341e3366d2d?source=rss-257ac04bf615------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/13-tips-for-using-pytest-5341e3366d2d?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/5341e3366d2d</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Mon, 28 Nov 2022 15:08:12 GMT</pubDate>
            <atom:updated>2022-11-28T15:08:12.233Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Why and How to Set Up Logging for Python Projects]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/why-and-how-to-set-up-logging-for-python-projects-bcdd4a374c7a?source=rss-257ac04bf615------2"><img src="https://cdn-images-1.medium.com/max/2600/1*PuJVDRdzwjDRJvaPj4Mvjg.jpeg" width="4770"></a></p><p class="medium-feed-snippet">Python&#x2019;s logging library is very powerful but generally under-utilised in data science projects.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/why-and-how-to-set-up-logging-for-python-projects-bcdd4a374c7a?source=rss-257ac04bf615------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/why-and-how-to-set-up-logging-for-python-projects-bcdd4a374c7a?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcdd4a374c7a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Mon, 17 Oct 2022 16:03:06 GMT</pubDate>
            <atom:updated>2022-10-17T16:03:06.084Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Reproducible ML: Maybe you shouldn’t be using Sklearn’s train_test_split]]></title>
            <link>https://medium.com/data-science/reproducible-ml-maybe-you-shouldnt-be-using-sklearn-s-train-test-split-ea8550ddd18d?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea8550ddd18d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Mon, 20 Jun 2022 18:06:28 GMT</pubDate>
            <atom:updated>2022-06-20T18:06:28.541Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eSN9m7VmBjisuV_xHnXxcw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@jdent?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jason Dent</a> on <a href="https://unsplash.com/s/photos/careful?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Reproducibility is critical for robust data science — after all, it is a science.</p><p>But reproducibility in ML can be surprisingly difficult:</p><blockquote><strong>The behaviour of your model doesn’t only depend on your code, but also the underlying dataset that was used to train it</strong></blockquote><p>Therefore, keeping tight control on which data points were used to train and test your model to is critical to ensure reproducibility.</p><h4><strong>How you split your data can have a big effect on the perceived model performance</strong></h4><p>If you split your data ‘randomly’ there is a statistical chance that more outliers end up in the test set than the training set. As your model won’t ‘see’ many outliers during training, it will perform poorly on the test set when predicting ‘outlier’ values.</p><p>Now imagine you randomly split the data again and the ‘outliers’ now all reside in the training set and none in the test set. It is likely that your ‘model performance’ will increase. This performance increase has little to do with the chosen model, just the statistical properties of the training/test sets.</p><p>Therefore, it is important to control and understand the training and test splits in order to effectively compare different model candidates across multiple training runs.</p><h4><strong>Sklearn train_test_split</strong></h4><p>Probably, the most popular way to split your dataset is to use Sklearn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">train_test_split</a> function.</p><p>Out of the box, the train_test_split function will randomly split your data into a training set and a test set.</p><p>Each time you run the function you will get a different split for your data. Not ideal for reproducibility.</p><blockquote>“Ah!” — you say</blockquote><blockquote>“I set the random seed so it is reproducible!”.</blockquote><p>Fair point.</p><p>Setting random seeds is certainly an excellent idea and goes a long way to improve reproducibility. I would highly recommend setting random seeds for any functions which have non-deterministic outputs.</p><blockquote><strong>However, random seeds might not be enough to ensure reproducibility</strong></blockquote><p><strong>This article will demonstrate that the </strong><strong>train_test_split function does not always guarantee reproducible splitting even with the random seed set. I will also suggest alternative solution — hashing — for more robust and repeatable splitting.</strong></p><p>This isn’t to say you should never use train_test_split , just to highlight it is perhaps more sensitive than you might think. In certain scenarios, this can lead to inconsistent splitting which is difficult to debug.</p><pre>This article was originally published on my blog, <a href="https://engineeringfordatascience.com/posts/ml_repeatable_splitting_using_hashing/">engineeringfordatascience.com</a></pre><h3>What is the problem with train_test_split?</h3><p><strong>Setting the random reed only guarantees reproducible splits if the underlying data does not change in any way.</strong></p><p>The train_test_split is <strong>not <em>deterministic</em>.</strong></p><p>The splits generated from train_test_split are sensitive to the *ordering* of the underlying data and any *new data* added to the existing dataset.</p><p><strong>If your dataset is shuffled or amended in any way, the data will be split completely differently. </strong>It cannot be guaranteed that an individual data point will *always<em>*</em> be in the training set or *always<em>*</em> be in the test set. This means data points that were in the original training set might now end up in the test set and visa versa if the data was shuffled.</p><p>Therefore, for the same dataset, you can get *completely* different splits depending on how the rows in the dataset are ordered. That’s quite worrying.</p><p>Even if one data point is removed, the order of two rows are switched, or a single data point is added you will get a *completely<em>*</em> different training and test split.</p><p>This ‘ultra sensitivity’ to the data might be surprising — it certainly was to me at first — and lead to unexpected model training results.</p><p>Let’s demonstrate the issue with a simple demo.</p><blockquote><em>💻 All code for this post is provided in the </em><a href="https://github.com/julian-west/e4ds-snippets/blob/master/best-practices/repeatable-splitting/Reproducible%20ML%20-%20Maybe%20you%20shouldn%27t%20be%20using%20sklearn%27s%20train_test_split.ipynb"><em>accompanying notebook</em></a><em> stored on GitHub 🚀</em></blockquote><p>We will first download an example dataset from sklearn.datasets and create an ‘index’ column to uniquely identify each row.</p><pre><strong>from</strong> <strong>sklearn.datasets</strong> <strong>import</strong> load_breast_cancer</pre><pre><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong></pre><pre><em># download an example dataset</em><br>data = load_breast_cancer()<br>df = pd.DataFrame(data[&quot;data&quot;], columns=data[&quot;feature_names&quot;])</pre><pre><em># create an &#39;index&#39; column to use to uniquely identify each row</em><br>df = df.reset_index(drop=False)</pre><pre>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QgusUzX6vatYoNpn.png" /></figure><p>Now let’s split the data using Sklearn’s train_test_split, setting the random state (seed).</p><pre><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> train_test_split</pre><pre>TEST_RATIO = 0.1<br>SEED = 42</pre><pre><em># split into training and test using a random seed</em><br>x_train_skl, x_test_skl = train_test_split(df, test_size=TEST_RATIO, random_state=SEED)</pre><p>Next, we shuffle the original dataframe and split the data again. We will still use the same random seed as before for consistency.</p><p>Note that no new data has been added, we have just reordered the rows.</p><pre><em># shuffle the orginal dataframe</em><br>df_shuffled = df.sample(frac=1)</pre><pre><em># split the shuffled dataframe using the same random seed</em><br>x_train_skl_shuffled, x_test_skl_shuffled = train_test_split(<br>    df_shuffled, test_size=TEST_RATIO, random_state=SEED<br>)</pre><p>Ideally the rows contained in the x_test_skl and x_test_skl_shuffled test sets should be identical as we used the same random seed.</p><p>However, when we compare the row ids contained in each test set, we notice they are different! Even though the random state (seed) was the same both times. Nothing in the data has changed, it was just shuffled.</p><pre><em># compare the row ids included in the original test set vs shuffled test set. Should return True if identical rows are included in each test set</em></pre><pre>set(x_test_skl[&quot;index&quot;]) == set(x_test_skl_shuffled[&quot;index&quot;])</pre><pre>False</pre><p>This highlights just how sensitive the train_test_split function is, even to reordering of the data.</p><p>More importantly, if there was a change in the underlying data, it would be extremely difficult to reproduce the original data splits and debug model performance.</p><h3>What are the consequences of relying on a random seed?</h3><h4>It is risky</h4><p>The random seed only guarantees reproducibility when the dataset has not changed in any way.</p><p>Can you be 100% sure the dataset has not changed between training runs? If a colleague has removed an outlier data point or if new rows have been added. Your data splits will be completely different to your original splits with no way to easily replicate the old data splits.</p><p>You can use data versioning tools, such as <a href="https://dvc.org/">dvc</a> to help keep track of changes, however, that doesn’t prevent your data splits from changing. It would be better to protect against split changes in your code.</p><h4><strong>Difficult to effectively compare models</strong></h4><p>When comparing models, we want to be able to control as many variables as possible. That should include which data points were used for training and testing.</p><p>If your data splits are significantly different between runs you might observe considerable differences in performance. For example if you have a couple of ‘outlier’ data points that were in your training set for the original training run, but are now in your test set, your model performance might ‘decline’ as it could not predict outlier values in the test set as well as before.</p><h4><strong>Difficult to debug</strong></h4><p>If you can’t effectively compare models, it can make it hard to debug performance issues.</p><p>Imagine you add some new data points to your dataset and retrain your model, but the performance of the model drops.</p><p>If you have used train_test_split with the random seed set, you will have completely different data splits as the underlying data has changed. It would be difficult to understand whether the model performance decline was due to the quality of the new data, or, as highlighted in the previous point, it was just because the data was split differently.</p><h3>When might train_test_split not be appropriate?</h3><h4>…if you need to retrain your model in the future on the original data &amp; new data</h4><p>As demonstrated, any underlying change to your existing data, be it reordering or even adding one additional data point will cause completely different data splits. Your original data splits will not be reproducible.</p><p>If you are retraining the model with a <em>completely</em> new dataset it isn’t a problem as obviously all the training and test data points will be different.</p><p>But if you are training again with a dataset that includes your original data points, ideally you should be able to replicate their original data splits during the new training run. Even with the random seed set, train_test_split will not guarantee this.</p><h4>…if you are sampling or retrieving your source data from an evolving data source</h4><p>In an ideal situation, you should have full control of your source dataset, however, sometimes this is not the case.</p><p>For example, if you are using a table stored in BigQuery as the source that is used by many teams. You cannot guarantee the order of the rows returned by a query and new rows might be appended to the table in the meantime.</p><p>Another example, would be if you are working with image data stored in a filesystem. If new images are added to your source folder you cannot guarantee the ordering of filepaths especially if new images are added.</p><h4>…if you have large datasets which do not fit in memory</h4><p>If you need to distribute your data across many machines in order to parallelise data processing, using a non-deterministic method for splitting your training and test data can be problematic and difficult to ensure reproducibility.</p><h4>…if your experimentation or production code will be rewritten in another language</h4><p>As the train_test_split is non-deterministic, the data splits will not be easily reproducible across languages.</p><p>For example, you might want to compare the performance of your custom Python model to a model created using BigQuery’s BQML which is defined using SQL. The splits from train_test_split Sklearn will not easily translate directly to SQL.</p><p>It can also be common for teams to prototype models using Python, but then write production systems in another language such as Java. To help the process of translating the prototype model into another language, ideally we should be able to split the data in the same way in both languages to ensure reproducibility and help debug any differences from the original model to the new model.</p><h3>The Solution: Hashing</h3><h4>What is hashing?</h4><blockquote><em>“A hash function is any function that can be used to map data of arbitrary size to fixed-size values” </em><a href="https://en.wikipedia.org/wiki/Hash_function"><em>Wikipedia</em></a></blockquote><p>There are many different hashing algorithms, but essentially they allow you to reproducibly convert an input into an arbitrary value.</p><p>The output of the hashing function is deterministic — it will always be the same for the same input.</p><h4>How does it work for splitting data reproducibly?</h4><p>In the context of data splitting, we can use hashing to reliably assign splits to individual data points. As this is a deterministic process, we can ensure that the data point is always assigned to the same split which aids reproducibility.</p><p>The process works as follows:</p><ul><li>Use a unique identifier for the data point (e.g. an ID or by concatenating multiple columns) and use a hashing algorithm to convert it to an arbitrary integer. Each unique data point will have a unique output from the hashing function.</li><li>Use a <a href="https://en.wikipedia.org/wiki/Modulo_operation">modulo operation</a> to arbitrarily split the data into ‘buckets’</li><li>Select all datapoints in a subset of buckets to be the training set and the rest to be in the test set</li></ul><h4><strong>Pseudo Code (90:10 data split)</strong></h4><pre>row_id = &quot;0001&quot;</pre><pre># convert id into a hashed integer value<br>hash_value = hash(row_id)</pre><pre># assign a bucket between 0 and 9<br>bucket = hash_value % 10</pre><pre># add id to train set if less than 9 (i.e. approx 90% of the data)<br>if bucket &lt; 9:<br>   train_set.append(row_id)<br>else:<br>   test_set.append(row_id)</pre><h3>Reasons to use hashing</h3><h4><strong>Deterministic</strong></h4><p>Hashing is robust to underlying changes in the data, unlike train_test_split.</p><p>Using this method, an individual data point will <em>always</em> be assigned to the same bucket. If the data is reordered or new data is added, the assigned bucket will not change. This is preferable as a data point’s train/test split assignment is now independent of the rest of the dataset.</p><h4><strong>Improves development and reduces chances of human error</strong></h4><p>When working on models in parallel with colleagues, it is very easy to accidentally forget to use random seeds or even use different random seeds. This leaves you open to the risk of human error.</p><p>Using the same hashing algorithm removes the need to control reproducibility explicitly in your code with random seeds. As long as you agree with your team on which hashing algorithm to use you will always recreate the same splits. No risk of human error.</p><h4><strong>Consistent splitting across raw and preprocessed data</strong></h4><p>During experimentation you might investigate different preprocessing steps and save intermediate and preprocessed data in a new file. You then might load this intermediate data at another stage to continue your analysis.</p><p>As the preprocessed data is different to the raw data, using train_test_split and random seeds will give different splits for the raw and preprocessed data when loaded from a new file.</p><p>Hashing will provide identical splits for raw and preprocessed data as long as the column used for calculating the hash value has not changed.</p><h4><strong>Storage and memory efficient</strong></h4><p>There are other strategies to combat reproducibility (discussed later in this article) such as explicitly saving data in a ‘training’ file and a ‘test’ file or adding a new column to your data to indicate which train/test split the data point belongs to.</p><p>However, sometimes you are not able to save data to new files and add columns — for example, if you don’t have permissions to copy or edit the original datasource or the data is too large.</p><p>Hashing is deterministic and the data point split can be calculated ‘on the fly’ in memory when required without needing to explicitly change the underlying data or save into a new file.</p><h3>The Farmhash Algorithm</h3><p>There are <a href="https://en.wikipedia.org/wiki/List_of_hash_functions">many different hashing algorithms</a> which are used for multiple use cases such as checksums and cryptography.</p><p>For the purpose of creating reproducible train/test splitting we need to use a ‘fingerprint’ hash function. <a href="https://devopedia.org/fingerprinting-algorithms">Fingerprint hash functions</a> are lightweight, efficient and deterministic — they will always return the same value for the same input.</p><p>Cryptographic hash functions, such as MD5 and SHA1, are not suitable for this use case as they are not deterministic and they are also purposefully made to be computationally expensive.</p><p><a href="https://github.com/google/farmhash">Farmhash is developed by Google</a> and <a href="https://towardsdatascience.com/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39">recommended for this use case</a> . It has a simple <a href="https://pypi.org/project/pyfarmhash/">Python library</a> implementation and is available across many other languages including <a href="https://towardsdatascience.com/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39">BigQuery SQL</a> .</p><blockquote><em>Another alternative to Farmhash is to use zlib and the crc32 checksum algorithm. An example of the implementation is shown in this notebook from </em><a href="https://github.com/ageron/handson-ml2/blob/master/02_end_to_end_machine_learning_project.ipynb"><em>Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow</em></a></blockquote><p>Below is a demo of farmhash and how we can use it to assign buckets to our data points.</p><h4>Demo</h4><p><a href="https://pypi.org/project/pyfarmhash/">Python package on PyPI</a></p><pre><em># install Python library</em><br><em># ! pip install pyfarmhash</em></pre><p>We can hash a data point using its unique identifier (i.e. an ID or concatenation of column values).</p><p>Let’s start by converting an individual ID into a hashed integer value using farmhash’s fingerprint64 function.</p><pre><strong>import</strong> <strong>farmhash</strong></pre><pre>example_id = &quot;0001&quot;<br>hashed_value = farmhash.fingerprint64(example_id)<br><strong>print</strong>(hashed_value)</pre><pre>6241004678967340495</pre><p>We can now assign this datapoint to a ‘bucket’ using an arbitrary function.</p><p>A useful method for this can be to use the modulo function. The integer outputs from the hashing algorithm are randomly distributed, therefore using the modulo function with a divisor of 10, for example, will split the data into 10 random buckets (from 0 to 9).</p><pre><em># assign a bucket using the modulo operation</em><br>bucket = hashed_value % 10<br><strong>print</strong>(bucket)</pre><pre>5</pre><p>Therefore, our datapoint ID of “0001” would be assigned to bucket 5. When we use a divisor of 10 we will have 10 different buckets. Therefore, for example, we could assign all data points with a bucket of ‘1’ to the test set to use 10% of the data for testing.</p><h4>Splitting the dataset using Farmhash</h4><p>Now let’s apply this splitting strategy to our dataset.</p><p>The hash_train_test_split function below can be used to split a dataframe into training and test sets using a specified hash function. In this example, the function creates a new column to store the bucket assignments. This is just for demonstration purposes, there is no need to actually store the bucket values in your data as the bucket can be calculated reproducibly ‘on the fly’ from the row ID.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb14ba662d4c255d432cf5d57f47c80f/href">https://medium.com/media/fb14ba662d4c255d432cf5d57f47c80f/href</a></iframe><p>As before, we will create a train/test split using the original breast cancer dataset, but use the hashing method with Farmhash instead of sklearn’s train_test_split with random seeds.</p><p>Then we will shuffle the data, split the data again and compare the test set IDs to ensure the splits are the same.</p><pre><em># create a training and test set from original dataset using hashing method</em><br>x_train_hash, x_test_hash = hash_train_test_split(<br>    df,<br>    split_col=&quot;index&quot;,<br>    approx_test_ratio=TEST_RATIO,<br>)<br></pre><pre><em># create a training and test set from shuffled dataset using hashing method</em><br>x_train_hash_shuffled, x_test_hash_shuffled = hash_train_test_split(<br>    df_shuffled,<br>    split_col=&quot;index&quot;,<br>    approx_test_ratio=TEST_RATIO,<br>)</pre><pre><em># show which bucket each row has been assigned for demo purposes</em><br>x_train_hash[[&quot;index&quot;, &quot;bucket&quot;]].head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/0*w2bo8T8CBqO4u_T3.png" /></figure><pre><em># compare the row ids included in each test set</em><br>set(x_test_hash[&quot;index&quot;]) == set(x_test_hash_shuffled[&quot;index&quot;])</pre><pre>True</pre><p>Problem solved! Even though the underlying dataframe was shuffled, the same row ids appear in the test dataset regardless.</p><h3>Considerations</h3><p>The hashing method is a little (although not much) more complicated than the common Sklearn train_test_split. As such, there are some additional important things to think about and be aware of when implementing this approach.</p><h4><strong>Hashing will not split your data exactly according to your specified train/test ratio</strong></h4><p>The output integers from the hashing algorithm are consistent but still random. By statistical chance, you may have slightly more outputs assigned to a particular bucket which means you might not get <em>exactly</em> 10% of your data being assigned to the test set. It might be slightly more or less. This is why I named the argument in the hash_train_test_split function ‘approx_test_ratio’ as the result will only be approximately that ratio.</p><p>In our example above, we specified a ratio of 0.1 and would expect a test set size of 56. However, we actually ended up with only 46 records in the test set (8%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/0*FDBphrWCmmnZ9Ycn.png" /><figcaption>Image by Author</figcaption></figure><p>The larger your dataset, the more uniform the bucket assignments will be and the closer to your desired ratio the splits will become (law of large numbers).</p><p>Generally this should not be a problem though. Using a 90:10 train/test split is arbitrary. In reality, it doesn’t matter if your split is a little over or under this target ratio.</p><h4><strong>How many buckets should you choose?</strong></h4><p>I.e. which number should you use as the divisor for the modulus operation.</p><p>This really depends on granularity of the desired test split. If you want to have a split of 90:10 you can split your data into 10 buckets using ‘10’ as the divisor. Then choose one of the buckets to be your test set which will be approximately 10% of your data points.</p><p>If you want 15% of your data for testing, you could use 100 as the divisor to split the data into 100 buckets. You could randomly choose 15 buckets from your data to get 15% of your data for testing.</p><h4><strong>Platform cross-compatibility quirks</strong></h4><p>Having said that the hashing approach is consistent across platforms. That is true to an extent.</p><p>I won’t go into extra detail here. A more detailed discussion on the ‘gotchas’ using hashing to split data in BigQuery vs Python can be found in the <a href="https://engineeringfordatascience.com/posts/ml_repeatable_splitting_using_hashing/#considerations">original article on my blog</a>:</p><blockquote>tl;dr: Just be careful about how different languages deal with applying the modulus function for negative numbers…</blockquote><h3>Alternatives to hashing</h3><p>For completeness, here are two other common more explicit approaches for ensuring consistent train/test splitting.</p><h4>Create an additional column in your data</h4><p>You could use train_test_split (or another random splitting method) to initially define the splits. Then create an additional column in your dataset to explicitly record whether the data point should be included for training or test (or specify the fold for K-Fold validation).</p><p>Here is an <a href="https://github.com/abhishekkrthakur/mlframework/blob/master/src/create_folds.py">example implementation by Abhishek Thakur</a> who uses it to define the ‘folds’ for cross validation.</p><p>This will ensure your splits are ‘remembered’ between training runs as they are explicitly defined.</p><p>On the positive side, it is very transparent which data points belong to each split. However, a downside is that it increases the total size of your dataset which may not be sustainable for very large datasets. Additionally, if you do not have full control of the dataset (e.g. shared database table) you may not be able to add columns to the original schema.</p><h4>Save your training and test data to different files</h4><p>Another common approach is to store your training data and test data to individual files after splitting the data for the first time. For example, into files called train.csv and test.csv. If the data is too large for individual files you could also save multiple files into folders named train and test respectively.</p><p>This can be a valid approach. However, sometimes it may not be feasible to make a copy of all your data and save into individual files. For example, if the dataset size is very large or you don’t have the permissions to make a copy from the original source.</p><p>The hashing approach can compute the deterministic data splits reproducibly on the fly and in-memory, preventing the need to copy data into individual files.</p><h3>Conclusion</h3><p>Sklearn’s train_test_split works well for introductory tutorials and small static datasets, however, in the real-world things get more complicated.</p><p>Reproducibility is key for sustainably deploying ML models. The Sklearn train_test_split is surprisingly sensitive to changes in the underlying data and may not be appropriate, particularly if you add new data to your existing data.</p><p><strong>Hashing decouples an individual data point’s train/test assignment from the rest of the dataset. </strong>This results in a more robust method for splitting your data.</p><p>Hashing is a great solution, however, it can be overkill. If you are completing a one-off training on a small static dataset, Sklearn’s train_test_split with a random seed will be sufficient. However, hashing is a great addition to a data scientist’s toolbox to improve reproducibility and protect against unexpected changes in model performance.</p><p>Happy coding!</p><pre>This article was originally published on my blog, <a href="https://engineeringfordatascience.com/posts/ml_repeatable_splitting_using_hashing/">engineeringfordatascience.com</a></pre><pre>Code for the article can be found in <a href="https://github.com/julian-west/e4ds-snippets/tree/master/best-practices/repeatable-splitting">this GitHub repository</a></pre><h3>References &amp; Resources</h3><p><strong>Hashing</strong></p><ul><li><a href="https://towardsdatascience.com/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39">ML Design Pattern: Repeatable sampling</a> (inspiration for this post)</li><li><a href="https://www.bi-kring.nl/192-data-science/1340-reusing-data-for-ml-hash-your-data-before-you-create-the-train-test-split">Hash your data before you create the train-test split</a></li><li><a href="https://github.com/google/farmhash/blob/master/Understanding_Hash_Functions">Farmhash Algorithm Description</a></li><li><a href="https://pypi.org/project/pyfarmhash/">Python Farmhash library</a></li><li><a href="https://www.danli.org/2021/06/06/hands-on-machine-learning/">Different hashing implementation from Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow</a> also <a href="https://github.com/ageron/handson-ml2/blob/master/02_end_to_end_machine_learning_project.ipynb">see notebook</a></li><li><a href="https://developers.google.com/machine-learning/data-prep/construct/sampling-splitting/randomization">Google documentation: Considerations for Hashing</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea8550ddd18d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/reproducible-ml-maybe-you-shouldnt-be-using-sklearn-s-train-test-split-ea8550ddd18d">Reproducible ML: Maybe you shouldn’t be using Sklearn’s train_test_split</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do programmers need to be able to type fast?]]></title>
            <link>https://medium.com/codex/do-programmers-need-to-be-able-to-type-fast-2b765181b7a6?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b765181b7a6</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Mon, 04 Apr 2022 08:38:11 GMT</pubDate>
            <atom:updated>2022-04-06T15:25:22.335Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Yes. But probably not for the reasons you are thinking</blockquote><p>There is a common stereotype that all programmers are extremely fast at typing.</p><p>In films and TV shows, any scene involving coding will inevitably be accompanied by a ‘hacker’ furiously typing away at their keyboard.</p><p>This leads to the logical deduction by many that typing fast is a prerequisite to being ‘good’ at programming.</p><p>The reality is nothing like the media’s depiction. Most of the time ‘coding’ is spent surfing StackOverflow or staring blankly at the screen and scratching your head — or at least that is my experience 👀</p><p><strong>You don’t need to be able to type fast in order to ‘code’. But you do if you want to be a highly productive programmer.</strong></p><p>The role of a developer is not restricted to only writing code. It includes ancillary tasks such as writing documentation and design documents, interacting with search engines, communicating issues to other developers over chat etc.</p><blockquote>The question shouldn’t be: How fast do you need to type to write code?</blockquote><blockquote>You should ask: How fast do you need to type for someone who spends most of their day at a computer and keyboard?</blockquote><p><strong>It makes just as much sense to improve your skills on a keyboard as it does to improve your knowledge/skills in your chosen programming language. Both make you more efficient as a programmer.</strong></p><p>In this post I argue why you should invest the time to improve your typing speed (spoiler: it isn’t to write code faster) and the knock on benefits to your job as a developer.</p><p>Additionally, I share my top tips which I used to double my typing speed in a few months from 55wpm to over 110wpm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dANPKaxEulC9SJee.gif" /><figcaption>Example typing test using MonkeyType.com — Image by Author</figcaption></figure><h3>You don’t need to type quickly to write code…</h3><h4>Modern IDEs eliminate the need to type quickly while coding</h4><p>Let’s first cover the obvious — modern IDEs (Integrated Development Environment).</p><p>Tab completion has killed the need to be able to type quickly when writing code. You are never going to be able to type quicker than your tab completer. Even if you can, tab completion comes with other benefits such as code assist and hinting.</p><p>In general, tab completion allows you to code ‘at the speed of thought’. This dramatically reduces cognitive load so you can focus on the logic of the problem at hand rather than the act of typing out each individual letter.</p><p>With tab completion in your IDE, typing speed is unlikely to be a blocker to your thought processes when writing code.</p><h4>Most time is spent reading code, not writing it</h4><blockquote>“Indeed, the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code.”</blockquote><blockquote>Robert C. Martin <a href="https://www.amazon.co.uk/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882">Clean Code: A Handbook of Agile Software Craftsmanship</a></blockquote><p>Typing speed is not the limiting factor when developing software. Most of your time is spent reading your own or other people’s code. Ultimately, it doesn’t matter if you can smash out ten of lines of code, faster than your colleague, if it then takes 20 minutes to debug it.</p><h3>…But why should you bother?</h3><p>So what am I on about? Why is typing speed important for programmers if it isn’t to write code?</p><h4>Write at the speed of thought. Increase creativity.</h4><p>The closer your typing speed gets to your thinking speed the more creative you become. When there’s no gap between you wanting to type something and actually doing it, you don’t interrupt the stream of thoughts.</p><p>Programmers can have a lot of plates spinning in their head when solving issues — I like to think of this as your personal cache. It takes a lot of effort to stay in the flow in order to piece together various dependencies to solve the problem. Any disruption to the thought process can set you back minutes or hours in the problem solving process — you have to go back to the ‘hard drive’ to pick up all relevant the info again.</p><p>Thinking about the problem should be separate from typing the actual characters in a code editor. Using up cognitive bandwidth to look down at your keyboard and think about typing can be a killer for productivity and your ability to solve the complex issues. Particularly if you type significantly slower than you think.</p><p>You should try and reduce any blockers to your stream of thought, this includes typing.</p><h4>Reduce friction. Build better habits.</h4><p>Additionally, improving your typing speed reduces friction which makes other tasks easier.</p><p>Writing documentation, commenting in your code (e.g. doc strings), compiling a document of scope requirements etc. are all common tasks which necessitate efficient typing. They also happen to be the less glamorous of developer tasks and often don’t receive the attention they should.</p><p>There is a great book by James Clear called <a href="https://www.amazon.co.uk/dp/B01N5AX61W/ref=dp-kindle-redirect?_encoding=UTF8&amp;btkr=1">Atomic Habits</a> which explains the concept of ‘friction’ in relation to building good habits. “The greater the friction, the less likely the habit”. Slow typing speed, or even discomfort when typing, builds a barrier to the tasks mentioned above.</p><blockquote>“The greater the friction, the less likely the habit.”</blockquote><blockquote>James Clear <a href="https://www.amazon.co.uk/dp/B01N5AX61W/ref=dp-kindle-redirect?_encoding=UTF8&amp;btkr=1">Atomic Habits</a></blockquote><p>If you can type quicker, it is easier for you to write comments and detailed documentation for your code as the friction is reduced — increasing the chances of you actually writing any documentation in the first place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/0*q78jofv0x-a_P4HC.png" /><figcaption>Image by Author: Positive feedback loop by improving your typing speed</figcaption></figure><h4>Get more done</h4><p>It is a matter of simple arithmetic. If you can type twice as fast (while remaining accurate) you will double your output when it comes to writing documentation, emails, chatting on Slack etc. You will also spend less time checking for and correcting errors, saving you even more time.</p><p>Don’t underestimate <a href="https://jamesclear.com/marginal-gains">the power of marginal gains</a> . Small improvements in the speed of some tasks compound greatly over the course of a day or year.</p><h3>How to improve your typing speed</h3><p>Over the last year, I have made a concerted effort to improve my typing speed. I was inspired by <a href="https://aliabdaal.com/how-to-type-faster">Ali Abdaal</a> and his — quite ridiculous — typing speed of 155 words per minute (wpm).</p><p>Little by little. Day by day. I was able to improve my typing speed from <a href="https://www.typing.com/blog/typing-speed">a very average</a> 50wpm up to around 110wpm.</p><p>I am not suggesting you need to reach 100 wpm (although setting a target certainly helps keep motivated to improve). I started noticing tangible improvements to the speed and comfort of my workflow when I reached 80–90 wpm.</p><p><strong>Here are my top 5 tips for boosting your typing speed 🚀.</strong></p><h4>1. Learn to Touch Type 👌</h4><p>The biggest drain on productivity and cognitive overload is when you need to look down at the keyboard while typing.</p><p>Learning to touch type will likely lead to the biggest improvement in the your typing speed and comfort.</p><h4>2. Focus on Accuracy 🎯</h4><p>There is little benefit to typing quickly if your accuracy is terrible.</p><p>Accuracy is just as important as speed and should be at least &gt;95% to not be a blocker to your thought processes. Going back to correct words is a productivity and thought process killer.</p><h4>3. Productive Procrastination 😇</h4><p>The best way to improve is by practicing regularly. There are many great websites which help you practice your typing and benchmark your speed.</p><p>Searching ‘typing tests’ on Google will provide you with a number of great testing websites. My personal favourite is MonkeyType — an aesthetic UI, highly customisable and doesn’t spam you with adverts.</p><p>Use these typing tests regularly and often to consistently practice and benchmark you typing speed. I used these websites as a form of ‘productive procrastination’, for 5 minutes here and there every day when I was bored. I quickly noticed the benefits of regular and deliberate practice.</p><h4>4. Track your progress 📈</h4><p>Track your progress and work towards a target. Progress tracking will help keep you motivated and highlight your improvements over time.</p><p>Most typing websites have some functionality to track progress. Alternatively you can maintain a progress log yourself.</p><p>A note of warning: these websites will give you a rather inflated typing speed figure compared to your ‘real world’ performance.</p><h4>5. Build Muscle Memory 💪</h4><p>Building muscle memory, particularly for common words, can dramatically increase your speed and reduce cognitive load while typing.</p><p>In an ideal situation, you should be able to see a word and your fingers just subconsciously type the word without any cognitive effort — in much the same way as humans read by recognising word patterns instead of reading individual letters.</p><p>You can practice building muscle memory using the following exercise:</p><ul><li>Prepare your fingers to type the word. Don’t type anything yet, just place your fingers over the right keys.</li><li>Try typing the word as quickly as possible</li><li>Pause before the next word (e.g for 1 second)</li><li>Repeat from step 1</li></ul><p>Don’t worry about your overall speed. For the moment, only worry about the speed of typing the individual word.</p><p>This exercise will train your fingers to learn the motions for keystroke patterns in common words. Once you get good at it, you will read a word and your fingers will subconsciously arrive in the correct positions and you will be able to press the keys in the right order without even thinking. Trust me on this. It really does work.</p><h3>Conclusion</h3><p>Investing the time learning to type faster was one of the best decisions I have made in the last year.</p><p>You don’t need to type quickly to ‘code’. Most of your time is spent reading and debugging rather than writing code, and besides, tab completion in modern IDEs takes care of most of your keystrokes.</p><p><strong>Increasing typing speed reduces friction and cognitive load allowing you to type ‘at the speed of thought’.</strong></p><p>This brings benefits such as being able to stay ‘in the flow’ without breaking your thought processes. It also improves the chances of you writing more detailed documentation and code comments. These marginal gains all contribute to increasing your effectiveness as a programmer.</p><p>So, it comes as no surprise what my concluding piece of advice is:</p><blockquote>If you are a developer, learn to improve your typing speed. It is an investment I guarantee you won’t regret. <strong>Typing fast is a superpower.</strong></blockquote><pre>This post was originally published on my blog: <a href="https://engineeringfordatascience.com/posts/improve_typing_speed_for_programmers/">https://engineeringfordatascience.com/posts/improve_typing_speed_for_programmers/</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b765181b7a6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/do-programmers-need-to-be-able-to-type-fast-2b765181b7a6">Do programmers need to be able to type fast?</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Resources to Learn Data Science on Google Cloud]]></title>
            <link>https://medium.com/data-science/10-resources-to-learn-data-science-on-google-cloud-c19fb3033df5?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/c19fb3033df5</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Mon, 14 Mar 2022 19:35:55 GMT</pubDate>
            <atom:updated>2022-03-14T19:35:55.894Z</atom:updated>
            <content:encoded><![CDATA[<h3>10 Handy Resources for Learning Data Science on Google Cloud</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsAv-DCDGVyL0H-lO7lEMw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@laurenedvalson?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Lauren Edvalson</a> on <a href="https://unsplash.com/s/photos/google?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><blockquote>My top piece of advice for aspiring data scientists is to learn about cloud computing.</blockquote><p><strong>All businesses are moving data workloads to the cloud.</strong></p><p>If you are ever going to build an ML model and put it into production at a company, I almost guarantee it will involve either accessing or processing data on the cloud.</p><p>I have noticed that most job adverts for data science positions now include cloud skills as a core part of the job description. I highly recommend you invest the time to up-skill.</p><p><strong>It doesn’t matter which cloud provider</strong> <strong>— AWS, GCP (Google Cloud Platform), or Azure.</strong></p><p><strong>Pick one and learn the basics of data storage, processing, analysis and ML workflows using the platform.</strong></p><p>In my case, I landed with GCP.</p><p>One of the main reasons I love using GCP is the vast amount of easily digestible documentation and quality learning resources created by the Google Developer Advocate Teams.</p><p>In this post, I will share my top 10 ‘go to’ sources for information and inspiration when developing on Google Cloud.</p><p>I have categorised these resources by: blogs, GitHub repositories, Documentation pages, YouTube channels, and content creators.</p><blockquote>Almost every day I seem to find a new valuable source of information to help me understand a topic or implement a solution. I’m sure I must have missed some great resources — please comment if you think I have missed any!</blockquote><h3>Blogs</h3><h4>1. <a href="https://cloud.google.com/blog/topics/developers-practitioners">Developers and Practitioners Google Cloud Blog</a></h4><p>Google Cloud runs a very active <a href="https://cloud.google.com/blog/">blog</a> on their main website covering many Google Cloud related topics including both high-level trends as well as more detailed technical content.</p><p>The Developers &amp; Practitioners blog is a subtopic within the main Google Cloud blog and focuses on detailed tutorials and demonstrations of GCP tooling.</p><p>The high-quality blog posts are written by members of Google’s exceptional Developer Advocate team.</p><p><strong>I find this blog particularly useful for learning about best practices.</strong></p><p>There are also many excellent <a href="https://cloud.google.com/blog/products/gcp/5-google-cloud-product-cheat-sheets-2021">cheat sheets</a> that are great to refresh your memory on different GCP tools . The ‘<a href="https://cloud.google.com/blog/topics/developers-practitioners/back-popular-demand-google-cloud-products-4-words-or-less-2022-edition">Google Cloud Products in 4 Words or Less</a>’ blog post is great for getting up to speed with the GCP platform offerings.</p><h4>2. <a href="https://medium.com/google-cloud">Google Cloud Medium Blog</a></h4><p>In addition to the blog on Google Cloud’s website, Google regularly update their ‘community’ blog on Medium. This consists of a number of curated blog posts from Google Cloud employees as well as other practitioners in the field.</p><p>Similar to the Developers &amp; Practitioners blog, this resource contains a number of in-depth tutorials on various GCP use cases and best practices.</p><p>Checking this blog for new updates has become a staple of my morning routine.</p><h3>GitHub Repositories</h3><h4>3. <a href="https://github.com/GoogleCloudPlatform">GoogleCloudPlatform GitHub repo</a> (⭐️ Top Resource)</h4><p>Google Cloud maintains a very active GitHub account with hundreds of repositories containing tutorials, workshop materials and GCP tools.</p><p>This is a great resource for finding code snippets for already implemented solutions on GCP and observing coding best practices from Google engineers.</p><p>Many of these repositories are referenced in the GCP documentation, however, I recommend browsing through to see if there are any repositories related to the tools and languages you tend to work on.</p><p>My particular highlights include:</p><ul><li><a href="https://github.com/GoogleCloudPlatform/professional-services">Professional Services</a> — Tutorials and code for common solutions faced by Google Cloud’s Professional Services team</li><li><a href="https://github.com/GoogleCloudPlatform/ml-on-gcp">ML on GCP</a> — Guides for various machine learning frameworks on Google Cloud</li><li><a href="https://github.com/GoogleCloudPlatform/mlops-on-gcp">MLOps on GCP</a> — Demonstration of a number of design and code patterns for a variety of ML Engineering topics</li><li><a href="https://github.com/GoogleCloudPlatform/mlops-with-vertex-ai">MLOps with Vertex AI </a>— End-to-end MLOps process using Vertex AI platform</li><li><a href="https://github.com/GoogleCloudPlatform/ml-design-patterns">ML Design Patterns</a> — Source code and examples for O’Reilly’s <a href="https://www.oreilly.com/library/view/machine-learning-design/9781098115777/">ML Design Patterns</a> book</li><li><a href="https://github.com/GoogleCloudPlatform/data-science-on-gcp">Data Science on GCP</a> — Source code and examples for O’Reilly’s <a href="https://www.oreilly.com/library/view/data-science-on/9781491974551/">Data Science on Google Cloud Platform</a></li><li><a href="https://github.com/GoogleCloudPlatform/training-data-analyst">Training data analyst</a> — Labs and demos for Google Cloud’s <a href="http://cloud.google.com/training">training courses</a></li></ul><h4>4. <a href="https://github.com/googleapis">Google APIs Source Code</a></h4><p>Google provides a number of client libraries to interact with its services using your language of choice.</p><p><strong>Throughout the Google Cloud documentation, you will find various code snippets for each tool demonstrating how to use the API for your chosen client library.</strong></p><p>These code snippets are actually available in the Google API source code hosted on Github — normally in a folder called samples. I find this really handy as you can access all these code samples in one place rather than having to navigate through various pages in the documentation trying to find that useful snippet you vaguely remember.</p><blockquote>For example, <a href="https://github.com/googleapis/python-bigquery/tree/master/samples">here are all the code snippets</a> for using BigQuery’s Python API.</blockquote><p>Additionally, you can access the various <a href="https://github.com/googleapis/python-bigquery/tree/master/samples/tests">tests for these samples</a> Google uses to test these code samples. <strong>These can be a useful reference when writing units tests for your own functions based off these snippets.</strong></p><h3>Documentation</h3><h4>5. Product Documentation Pages — Key Concepts</h4><p>As you would expect, each product on Google Cloud has extensive documentation with explanations and step-by-step example tutorials.</p><p>In particular, each product documentation page has a section called ‘Concepts’.</p><p><strong>The concepts pages explain the most important aspects of each product</strong>. They normally call out the key features and best practices. This is extremely helpful for getting up to speed quickly, particularly if you have used a similar tool before on another cloud provider.</p><p>For example, here is a link to the key concepts page for <a href="https://cloud.google.com/storage/docs/concepts">Google Cloud Storage</a> and <a href="https://cloud.google.com/sql/docs/concepts">Cloud SQL</a>.</p><h4>6. <a href="https://cloud.google.com/architecture">Google Cloud Architecture Center</a></h4><p>For the most part, Google’s <a href="https://cloud.google.com/docs">documentation</a> is an excellent resource for understanding the key concepts for each GCP tool with useful examples.</p><p>For a long time, as the documentation pages were so comprehensive, I never even thought to look whether there were any more resources to help me design solutions on Google Cloud.</p><p>That was until I stumbled across the Google Cloud Architecture Center.</p><blockquote>In most cases, the data problem you are tackling has previously been solved (at least in part) by someone else. There is no need to reinvent the wheel.</blockquote><p><strong>The Google Cloud Architecture Center contains reference architectures, diagrams, design patterns, guidance, and best practices for building or migrating your workloads on Google Cloud.</strong></p><p>Whenever designing a new solution on Google Cloud, I always check the Architecture Center for similar solutions already documented. If there isn’t already a reference architecture, it still provides inspiration for tackling my specific problem.</p><p>For example, there is a ready-made <a href="https://cloud.google.com/architecture/processing-streaming-time-series-data">solution for processing streaming time series data</a> including anomaly detection with pub/sub and dataflow.</p><p>Many of these reference architectures also link to the <a href="https://cloud.google.com/bigquery/docs/tutorials">tutorial sections</a> in each part of the documentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OMAWbM_fhPmJL1CF.png" /><figcaption>Screenshot of Cloud Architecture Center captured by author</figcaption></figure><h3>YouTube Channels</h3><h4>7. <a href="https://www.youtube.com/channel/UCJS9pqu9BzkAMNTmzNMNhvg">Google Cloud Tech</a></h4><p>Google Cloud Tech is Google’s main YouTube channel for developers and has over 800k subscribers. The channel features professionally produced videos from the Google Developer Advocate team introducing GCP tools and best practices as well as technical tutorial walkthroughs.</p><p><strong>This channel is different to the other Google Cloud YouTube channels (</strong><a href="https://www.youtube.com/user/GoogleDevelopers"><strong>Google Developers</strong></a><strong> , </strong><a href="https://www.youtube.com/channel/UCTMRxtyHoE3LPcrl-kT4AQQ"><strong>Google Cloud</strong></a><strong> ) as it focuses on in-depth technical material for practitioners.</strong> The other channels provide more of a ‘C-suite level’ view on trends and concepts which are less useful for the day to day implementation of solutions on GCP.</p><p>If I ever need a walkthrough of a tool, this is normally my first port of call to get familiar with the main concepts and best practices.</p><p>I recommend going to the <a href="https://www.youtube.com/user/googlecloudplatform/playlists">playlists</a> page to browse for the playlists most relevant to the tool you are interested in. I particularly enjoyed the <a href="https://www.youtube.com/watch?v=HcqpanDadyQ&amp;list=PLIivdWyY5sqJxnwJhe3etaK7utrBiPBQ2">AI Adventures series by Yufeng Guo</a></p><h4>8. <a href="https://www.youtube.com/channel/UCAkiVNZfRKgxerudxbVkHfg">Adventures in the cloud</a></h4><p>Adventures in the cloud is a much smaller channel (currently only 2k subscribers) run by Yufeng Guo, who is also a Google Cloud Developer Advocate. Despite its size, it is a gold mine of content.</p><p>The channel mostly features long-form and real-time tutorials on machine learning and MLOps. Yufeng Guo runs through the tutorials in real-time, which doesn’t always go to plan, but this is a really valuable way to learn as you can see how to debug common issues as he comes across them.</p><p>I really enjoyed his video on <a href="https://www.youtube.com/watch?v=NKN6HawFuts">Productionizing Machine Learning with AI Platform Pipelines</a> .</p><p>Unfortunately, it looks like he has not uploaded any new content for a while, however, there is a lot of older content to keep you busy.</p><h3>Content Creators</h3><p>Finally, I want to highlight a few of my favourite personalities to follow who create great content in the Google Cloud space. I’m sure I have missed a lot of other excellent content creators, but here are two that keep coming up again and again with really useful material.</p><p>I have linked to their Medium blogs, however, I recommend you follow them on LinkedIn as well for regular updates and insights.</p><h4>9. <a href="https://lakshmanok.medium.com/">Lak Lakshmanan</a></h4><p>Lak is a Director of Data Analytics and AI Solutions at Google Cloud. He provides insightful thinking and clear tutorials mostly around BigQuery and machine learning.</p><p>He is the author of many books including the O’Reilly <a href="https://www.oreilly.com/library/view/google-bigquery-the/9781492044451/">Definitive Guide to BigQuery</a> (<a href="https://github.com/GoogleCloudPlatform/bigquery-oreilly-book">GitHub link</a>).</p><h4>10. <a href="https://pvergadia.medium.com/">Priyanka Vergadia</a></h4><p>Priyanka is a Developer Relation Engineer at Google and is very active on Google’s Cloud Tech YouTube channel as well as LinkedIn.</p><p>Priyanka creates content on topics across the whole of Google Cloud with many entertaining and informative videos.</p><p>She recently produced a great series explaining <a href="https://medium.com/google-cloud/top-13-google-cloud-reference-architectures-c8a697546505">13 reference architectures</a> for Google Cloud.</p><h3>Conclusion</h3><p>In this post, I have shared my top 10 technical resources for developing with Google Cloud as a data scientist.</p><p>I have been really impressed with the quality of Google’s resources for developers which makes Google Cloud by far my favourite cloud provider to work with.</p><p>Many of these resources I only came across in the last few months, which makes me think there must be many more that I have overlooked. I would love to hear what your favourite resources are — I’m sure I have missed a few!</p><pre>This post was originally published on my blog: <a href="https://engineeringfordatascience.com/posts/top_10_google_cloud_resources/">https://engineeringfordatascience.com/posts/top_10_google_cloud_resources/</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c19fb3033df5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/10-resources-to-learn-data-science-on-google-cloud-c19fb3033df5">10 Resources to Learn Data Science on Google Cloud</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to set up an amazing terminal for data science with oh-my-zsh-plugins]]></title>
            <link>https://medium.com/data-science/how-to-set-up-an-amazing-terminal-for-data-science-with-oh-my-zsh-plugins-2518dfeeab5d?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/2518dfeeab5d</guid>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[command-line-tools]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[macos]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Thu, 03 Mar 2022 08:42:31 GMT</pubDate>
            <atom:updated>2022-03-03T08:42:31.860Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Set Up an Amazing Terminal for Data Science with oh-my-zsh-Plugins</h3><h4>Increase your productivity and user experience in the terminal by installing some key plugins.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iJcVzoBFABx4xlcg.png" /><figcaption>A preview of the terminal described in this post. Image by Author</figcaption></figure><h3>Why should you use the terminal as a data scientist?</h3><p>There is not a more productive or versatile tool for developers to use — the terminal is the Swiss Army Knife of the developers’ tool kit.</p><p>The terminal can be a one-stop-shop for your daily tasks. Code editing, launching programs, multi-tasking, scripting, system admin, and much more.</p><blockquote>Command line expertise frees you from restrictive GUI tools and endless mouse clicks, opening a world of efficiency and automation.</blockquote><p><strong>But the command line can be intimidating and unforgiving for the uninitiated.</strong></p><p>The monochrome UI of the default terminal takes no prisoners.</p><p>No syntax highlighting and no autocompletion to hold your hand. Who knows whether the command you are typing is correct?</p><p><strong>Yet, small tweaks to the configuration go a long way to improving the user experience. </strong>Turning the command line into a powerful tool to supercharge your workflow.</p><p><strong>The key is to invest a little bit of time customising your terminal with plugins.</strong> A couple of hours tailoring your terminal will pay huge dividends down the line.</p><h3>Key features of a terminal that enable productivity</h3><p>Customising the terminal is a very personal thing. Everyone has aesthetic preferences and different functional requirements depending on their tech stack.</p><p><strong>But I would consider the following terminal attributes important for boosting productivity</strong>:</p><ul><li>a command line (prompt) displaying key information about your environment — e.g. git information, virtual environment, Python version etc.</li><li>syntax highlighting</li><li>auto-completion</li><li>window and session management</li><li>efficient file system navigation and directory jumping</li><li>a nice aesthetic — if you don’t enjoy using it, you probably won’t bother</li></ul><p><strong>In this post I will go through the key oh-my-zsh plugins I use to achieve the above functionality and transform the command line into a productive environment.</strong></p><p>Let’s get into it!</p><h3>Prerequisites</h3><p>I am using MacOS, <a href="https://iterm2.com/">iterm2</a>, and <a href="https://engineeringfordatascience.com/posts/setting_up_a_macbook_for_data_science/">Homebrew</a> for this setup.</p><p>However, it should be possible to install and configure this setup across any operating system.</p><p>I will provide links to the documentation of each program and plugin. Read the documentation pages for more information on how to setup each component for your specific operating system.</p><h3>Configure the Command Line Prompt</h3><h3>1. oh-my-zsh</h3><p>The MacOS terminal prompt ships with <a href="https://zsh.sourceforge.io/">z shell</a> (zsh) as the default shell.</p><p><a href="https://ohmyz.sh/"><strong>Oh My Zsh</strong></a><strong> is an open source framework for managing your zsh configuration.</strong> It provides easy access to many great plugins which can improve your terminal experience and supercharge your productivity.</p><blockquote>“Oh My Zsh will not make you a 10x developer…but you may feel like one!”</blockquote><blockquote><strong><em>— oh-my-zsh documentation</em></strong></blockquote><p>We will use oh-my-zsh to manage and install plugins for the rest of the terminal setup described in this post.</p><p>You can install oh-my-zsh using the following command (full instructions and documentation on their <a href="https://ohmyz.sh/">website</a> ):</p><pre><em># install oh-my-zsh</em><br>sh -c &quot;<strong>$(</strong>curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh<strong>)</strong>&quot;</pre><p>Installing oh-my-zsh will create/update your ~/.zshrc file – located in your home directory – which defines your zsh configuration. We will edit the ~/.zshrc file throughout this tutorial to customise our terminal.</p><h3>2. Download a Nerd-Font</h3><p>Next, download a <a href="https://github.com/ryanoasis/nerd-fonts">Nerd-font</a> of your choice.</p><p><strong>Nerd fonts provide many great fonts for your terminal which are patched to support icons.</strong> We will need a patched font later to render dev icons in the terminal and get that great aesthetic look.</p><p>You can follow instructions on the <a href="https://github.com/ryanoasis/nerd-fonts">Nerd-font GitHub page</a> to install a patched font of your choice on your operating system.</p><p>I use the <a href="https://github.com/ryanoasis/nerd-fonts/tree/master/patched-fonts/Hack">Hack font</a> which I install using <a href="https://brew.sh/">Homebrew</a> :</p><pre><em># install &#39;hack&#39; font using Homebrew</em><br>brew tap homebrew/cask-fonts<br>brew install --cask font-hack-nerd-font</pre><p>You can change the font in your terminal to the one you just installed by updating your item2 preferences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5S8_bI69cDFjAb-u.png" /><figcaption>iterm2 preferences UI. Image by Author</figcaption></figure><h3>3. Customising the prompt (Starship)</h3><p>The default theme for oh-my-zsh is called robbyrussell – named after the creator of oh-my-zsh.</p><p>I really like this theme’s minimalistic aesthetic and I used it for a long time before migrating to Starship.</p><p>While it is only the default theme, it is already a great upgrade on the featureless monochrome default zsh prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y2L-5EDOOWD5ZA5W.png" /><figcaption>Default robbyrussel oh-my-zsh theme. Image by Author.</figcaption></figure><p>There are a number of other themes that come with oh-my-zsh. A full list can be found in the <a href="https://github.com/ohmyzsh/ohmyzsh/wiki/Themes">GitHub Wiki</a> . In order to change the theme to one of the inbuilt themes, you just need to update the ZSH_THEME value in your ~/.zshrc and restart your terminal for the changes to take effect.</p><p>Alternatively, there are external themes that you can install and use instead of an inbuilt theme.</p><blockquote><a href="https://github.com/romkatv/powerlevel10k#installation">Powerlevel10K</a> is a popular choice, however, I prefer to use <a href="https://starship.rs/">Starship</a> .</blockquote><p><a href="https://starship.rs/">Starship</a> is a highly customisable prompt that is compatible across shells — you can use the same prompt config for no matter if you are using bash, zsh, fish or Powershell as your shell.</p><p>Starship comes with sensible defaults, can be configured with a user friendly toml file and comes with a CLI tool that makes updating your prompt on the fly a breeze (more on that later…).</p><p>To install Starship you can follow the instructions on the <a href="https://starship.rs/">documentation homepage</a> .</p><pre><em># install starship</em><br>brew install starship</pre><p>Then add the following line to the end of your ~/.zshrc to use the Starship prompt in zsh.</p><pre><em># ~/.zshrc</em></pre><pre>eval &quot;<strong>$(</strong>starship init zsh<strong>)</strong>&quot;</pre><p>Restart your terminal for the changes to take effect.</p><h4><strong>Configuring the Starship Prompt</strong></h4><p>Full documentation for configuring Starship can be found on <a href="https://starship.rs/config/">their website</a> .</p><p>The Starship prompt configuration is controlled using a file in your .config directory called starship.toml</p><p>For example, below is a simple configuration I use to show the Python version and virtual environment in the prompt when I navigate to a Python project directory:</p><pre><em># example ~/.config/starship.toml</em></pre><pre>add_newline = <strong>false</strong></pre><pre>[python]<br>format = &#39;via [${symbol}${pyenv_prefix}(${version} )(\($virtualenv\) )]($style)&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5f2tC_0edkR8dRp5.png" /><figcaption>Starship prompt with the example starship.toml configuration file. Image by Author.</figcaption></figure><h3>4. Choose a terminal colour scheme</h3><p>Ok, so this part doesn’t necessarily affect your ‘productivity’, but it is important to get the best looking terminal that you love coming back to 🎨.</p><p>This is completely down to personal preference — you might completely disagree with my choice — but I like using the Solarized Dark colour scheme. I also add some transparency/blur to the background.</p><p>There are many other preset colour schemes available and I recommend you play around to find your favourite.</p><p>These settings can be changed in the preferences GUI of iterm2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wAheTLQS9bBTHDXv.png" /><figcaption>iterm2 preferences UI, set the colour scheme. Image by Author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kQu5uljA-vy8KeqK.png" /><figcaption>iterm2 preferences UI, set the background transparency and blur. Image by Author</figcaption></figure><p>My final prompt using Starship and Solarized Dark colour scheme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uQ2Zlh3wW-tG0eJx.png" /><figcaption>Final Starship prompt with Solarized Dark colour scheme and some transparency. Image by Author.</figcaption></figure><h3>Essential Tools: Best oh-my-zsh plugins</h3><h3>5. Syntax highlighting</h3><p>Syntax highlighting is critical for improving the usability of the terminal.</p><p>It helps you identify if you have a program installed correctly and can alert you to a misspelling or invalid command.</p><p>To enable syntax highlighting in zsh you need to install a plugin called <a href="https://github.com/zsh-users/zsh-syntax-highlighting">zsh-syntax-highlighting</a> .</p><p>Installation instructions are provided in the <a href="https://github.com/zsh-users/zsh-syntax-highlighting/blob/master/INSTALL.md">GitHub repo</a> . You can install zsh-syntax highlighting using the following command:</p><pre><em># download syntax highlighting extension</em><br>git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM<strong>:-</strong>~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting</pre><p>Then activate the plugin by updating the .zshrc file.</p><pre><em># add syntax highlighting to the list of plugins in your ~/.zshrc file</em><br>plugins=(zsh-syntax-highlighting)</pre><p>Run source ~/.zshrc or restart your terminal for the changes to take effect.</p><p>Now when you type a keyword command it will turn green if it is correct or red if it is incorrect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*5I_3YaInIon4ivCm.gif" /><figcaption>Syntax highlighting. Spot errors before submitting your command. Image by Author.</figcaption></figure><h3>6. Auto-completion</h3><p>Auto-completion saves a lot of time when typing long commands and is very helpful for remembering commands you have previously run.</p><p>The <a href="https://github.com/zsh-users/zsh-autosuggestions">zsh-autosuggestions</a> plugin is required to enable autocompletions in zsh.</p><p>Installation instructions are provided on the <a href="https://github.com/zsh-users/zsh-autosuggestions/blob/master/INSTALL.md">installation wiki page</a> of the GitHub repo. The installation is very similar to the syntax highlighting.</p><pre><em># install autocompletions extension</em><br>git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM<strong>:-</strong>~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions</pre><pre><em># update plugins in your ~/.zshrc file</em><br>plugins=(<br>    zsh-syntax-highlighting <br>    zsh-autosuggestions<br>)</pre><p>Restart you terminal or run source ~/.zshrc to see the changes.</p><h3>7. Fuzzy file finding (<a href="https://github.com/junegunn/fzf">FZF</a> )</h3><p><a href="https://github.com/junegunn/fzf">FZF</a> is a fuzzy file finder that enables you to search for files interactively from the command line. It is really useful for quickly finding files when you know the name but can’t remember exactly where you saved it.</p><pre><em># install FZF (say yes to each option during installation)</em><br>git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf<br>~/.fzf/install</pre><p>The installation script will automatically update your .zshrc with the following line:</p><pre>[ -f ~/.fzf.zsh ] &amp;&amp; source ~/.fzf.zsh</pre><p>Additionally, to get fzf to play nice with oh-my-zsh you will need to add fzf to your list of plugins and set an environment variable with the fzf installation location:</p><pre># update your ~/.zshrc file<br>FZF_BASE=&quot;$HOME/.fzf&quot;</pre><pre>plugins=(<br>    zsh-syntax-highlighting <br>    zsh-autosuggestions<br>    fzf<br>)</pre><p>You can now activate the fuzzy finder by either typing fzf as a command to your terminal or pressing CTRL+T. You can also trigger the fuzzy finder using the ** prefix and pressing &lt;TAB&gt; (see <a href="https://github.com/junegunn/fzf#files-and-directories">documentation</a> ). For example:</p><pre><em># search which file to open in vim using the fuzzy finder</em><br>vim **&lt;TAB&gt;</pre><p>FZF is highly customisable. I won’t go into too much detail here on how to configure it — it probably requires an entire blog post in itself. But there are plenty of great tutorials online explaining how to configure FZF which I recommend you investigate to get the most out of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*ZUxHOwRwomNaA3aU.gif" /><figcaption>Fuzzy file finding using FZF. Image by Author.</figcaption></figure><h3>8. Window &amp; session management (tmux)</h3><p>It is common to have multiple terminal processes running at the same time.</p><p>A use case that immediately comes to mind as a data scientist is needing to run a Jupyter notebook server from the terminal, but also needing access to the terminal for other tasks like making git commits and running other python scripts.</p><p>Without the ability to open multiple windows or terminal sessions at once, your terminal will be blocked by the long running process.</p><p>The remedy for this problem is using a window management tool to manage multiple windows and terminal sessions at the same time.</p><p>iterm2 has an inbuilt ability to have multiple tabs open for different sessions. However, I love to use <a href="https://github.com/tmux/tmux">tmux</a> instead as it is very customisable, powerful and works across multiple different terminal emulators so I can use the same keybindings no matter which terminal I am using.</p><p>Tmux can be installed via Homebrew:</p><pre># install tmux</pre><pre>brew install tmux</pre><p>Again, you could write a whole blog post about configuring tmux. For this post, I’ll just provide a screenshot of an example tmux session and strongly encourage you look into a session/window management solution for managing multiple terminal sessions, whether that be using iterm2 tabs or tmux.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_gWuD1-ygbrppFds.png" /><figcaption>Tmux window management. Three sessions open in the same window: vim, command line and Jupyter Notebok server. Image by Author.</figcaption></figure><blockquote><em>👨‍🎓 There is a great crash course for learning tmux by Chris Toomey on </em><a href="https://thoughtbot.com/upcase/tmux"><em>thoughtbot</em></a><em> to get you up and running quickly.</em></blockquote><h3>Extras: Useful plugins and command line tools</h3><p>The following programs are some additional zsh plugins and command line tools that I also find useful. Your own mileage may vary depending on your preferred workflow and use case.</p><h3><a href="https://github.com/agkozak/zsh-z">z</a> — a directory jumping tool</h3><p>zsh-z is a command line tool that allows you to quickly jump between directories you have most recently visited.</p><p>One of the pain points of using the terminal is navigating through directories which can be trickier than navigating using a familiar file explorer GUI. Tools like z (and FZF described earlier) can make traversing your file system a breeze.</p><p>To install zsh-z we can clone the GitHub repository and add zsh-z to our list of plugins in the .zshrc file as before:</p><pre><em># download the plugin then update your ~/.zshrc file</em><br>git clone https://github.com/agkozak/zsh-z $ZSH_CUSTOM/plugins/zsh-z</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*8jDp3c0FnOjegKoJ.gif" /><figcaption>Directory jumping using z plugin. Image by Author.</figcaption></figure><h3><a href="https://github.com/sharkdp/bat">bat</a> — a better alternative to cat</h3><p>When you want to quickly see the contents of a file in a terminal without actually opening it (e.g. showing a configuration file) you can use the inbuilt cat command.</p><p>However, the output on the command line is just plain text which can be difficult to scan read, especially if you are used to nice syntax highlighting in IDEs and code editors.</p><p>bat is an alternative to cat which supports syntax-highlighting among other functions and also integrates with other tools like fzf.</p><p>You can install bat using Homebrew (full instructions for other operating systems available on their <a href="https://github.com/sharkdp/bat#installation">GitHub documentation page</a> ):</p><pre><em># install bat from Homebrew</em><br>brew install bat</pre><p>You can use bat in the same way as you would use cat:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mLAlW9Iachn4tguG.png" /><figcaption>Comparison of cat and bat output. (Not a real AWS token and secret key 😅). Image by Author.</figcaption></figure><p>You can also integrate it with fzf to show a preview of each file complete with syntax highlighting while searching files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KB2eTUbuuPi5qblX.png" /><figcaption>Fuzzy file finding and file preview using FZF and bat. Image by Author.</figcaption></figure><h3><a href="https://the.exa.website/">exa</a> — a modern replacement to ls</h3><p>In a similar vein, exa is a command line tool to replace ls.</p><p>The output of ls can be difficult to scan as there is no visual differentiation between files and file information. Everything is listed in the same colour.</p><p>exa is a more aesthetic file ‘lister’ with colour coding and support for icons which can make it much easier to understand the output.</p><p>Installation instructions are on the <a href="https://the.exa.website/">exa website</a> .</p><pre><em># install exa</em><br>brew install exa</pre><p>To use exa to list out files in a directory you can use the exa keyword followed by some arguments (see <a href="https://the.exa.website/docs">documentation</a> ). I really like the support for file icons which makes it easy to identify each file type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*efbp1svvQQWR6Sw0.png" /></figure><blockquote><em>💡</em><strong><em>Top tip</em></strong><em>: Create an alias in your </em><em>~/.zshrc file to map </em><em>exa --icons to </em><em>ls so you don’t need to relearn any syntax</em></blockquote><h3>Personal Touches: Aliases</h3><p>Another big time saver is creating aliases for long commands that you use frequently.</p><p>You can specify aliases in your .zshrc file.</p><p>Aliases will be personal to you and your workflows. I don’t use that many but here are the ones I like to use:</p><pre><em># ~/.zshrc</em></pre><pre><em># python</em><br>alias python=python3</pre><pre><em># git</em><br>alias gs=&quot;git status&quot;</pre><pre><em># map exa commands to normal ls commands</em><br>alias ll=&quot;exa -l -g --icons&quot;<br>alias ls=&quot;exa --icons&quot;<br>alias lt=&quot;exa --tree --icons -a -I &#39;.git|__pycache__|.mypy_cache|.ipynb_checkpoints&#39;&quot;</pre><pre><em># show file previews for fzf using bat</em><br>alias fp=&quot;fzf --preview &#39;bat --style=numbers --color=always --line-range :500 {}&#39;&quot;</pre><pre><em># starship prompt</em><br>alias gt=&quot;starship toggle gcloud disabled&quot; <em># toggle gcloud info on the prompt</em></pre><blockquote><strong>Side Note for those who use GCP</strong></blockquote><blockquote>I work a lot with Google Cloud to deploy machine learning pipelines and models.</blockquote><blockquote>When using the glcoud command line tool I get paranoid about accidentally deploying infrastructure in the wrong region (#GDPR) or even to the wrong client project.</blockquote><blockquote>Starship allows you to ‘toggle’ the prompt settings using their CLI tool.</blockquote><blockquote>I have created an alias, gt, which toggles the display of important information about my current gcloud profile in my prompt.</blockquote><blockquote>This is great as I don’t need it all the time, but being able to change my prompt on the fly to check is really useful!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ulo6iD6Bi62Va2az.png" /><figcaption>Using an alias to toggle GCP profile information on the prompt. Image by Author.</figcaption></figure><h3>Conclusion</h3><blockquote>For the uninitiated, the terminal can be unforgiving, stressful and annoying to use.</blockquote><p>Default settings have an appalling UX. Enough for some developers to retreat to the cosy embrace of a familiar GUI tool and never return.</p><blockquote>However, a couple of hours invested in configuring your terminal will pay huge dividends for your productivity, workflow and overall programming effectiveness.</blockquote><p>Oh-my-zsh is a great framework for managing your zsh shell configuration and plugins. By installing three or four key plugins and adding a few lines to the .zshrc file you can transform your experience when using the command line and boost productivity.</p><p>Happy coding!</p><pre>This article was originally published on my blog, <a href="https://engineeringfordatascience.com/posts/configure_terminal_for_data_science_with_oh_my_zsh/">engineeringfordatascience.com</a></pre><h3>Resources</h3><ul><li>A full list of oh-my-zsh plugins is available on the <a href="https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins">zsh GitHub wiki page</a></li></ul><p><strong>Full .zshrc file created from this tutorial</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/359568a75c19344a5f60ad4ca463cfd7/href">https://medium.com/media/359568a75c19344a5f60ad4ca463cfd7/href</a></iframe><p><strong>Basic starship.toml configuration file from this tutorial</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62546c6d8624f473635212356bc7a2d6/href">https://medium.com/media/62546c6d8624f473635212356bc7a2d6/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2518dfeeab5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-set-up-an-amazing-terminal-for-data-science-with-oh-my-zsh-plugins-2518dfeeab5d">How to set up an amazing terminal for data science with oh-my-zsh-plugins</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Five Tips to Elevate the Readability of your Python Code]]></title>
            <link>https://medium.com/data-science/five-tips-to-elevate-the-readability-of-your-python-code-7b049bbf72e6?source=rss-257ac04bf615------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b049bbf72e6</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Julian West]]></dc:creator>
            <pubDate>Tue, 15 Feb 2022 05:57:05 GMT</pubDate>
            <atom:updated>2022-02-15T05:57:05.978Z</atom:updated>
            <content:encoded><![CDATA[<h3>Five Tips to Elevate the Readability of Your Python Code</h3><h4>Apply these to your Python code today to improve readability—your coworkers and future self will thank you!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cu1yyN_pTZy9M8Bv6eg-cQ.jpeg" /><figcaption>Image: <a href="https://unsplash.com/photos/82TpEld0_e4">Unsplash</a></figcaption></figure><blockquote><strong>tl;dr</strong></blockquote><blockquote>1. Utilise auto-formatters such as black and isort</blockquote><blockquote>2. Use code checkers such as flake8 and pylint</blockquote><blockquote>3. Add type hints to remove ambiguity in your function arguments</blockquote><blockquote>4. Automate code quality checking with pre-commit</blockquote><blockquote>5. Write good documentation using Python doc strings</blockquote><p>I never used to worry too much about the readability of my code.</p><p>Obviously, I cared that I could read it during development. But I never gave too much consideration to whether I would still be able to read it in 6 months time.</p><blockquote><strong>“Of course, I know what type of data that variable holds and what that function is responsible for — it’s obvious, right!?”</strong></blockquote><p><strong>Note to past self: You will forget 🤦‍♂️.</strong></p><p>In contrast to Software engineers, as Data Scientists we tend to work quite independently on code.</p><p>Yes, we work in teams, but it is unusual for you to be working on the same code file as someone else. Normally you will have individual responsibility and ownership over a particular part of the project — for example, your individual notebooks — rather than the collective responsibility from multiple team members.</p><p>This tends to result in project code bases with inconsistent code formatting and less focus on readability and maintainability by others.</p><p>Since the beginning of my career I have mostly been working in consulting roles for ‘greenfield’ projects. These projects generally involved writing new code that only I was responsible for. It was only after a couple of years that I worked on my first project that involved an inherited legacy code base.</p><p>This was when I was finally struck by the importance of writing readable code. Not just for yourself, but for developers on-boarded onto the project in the future. It also made me reflect on my own bad coding habits on previous projects and what I wished previous developers had added to their code to save me hours of head scratching.</p><p><strong>Luckily, in Python there are several simple cosmetic changes you can apply to make your code significantly more readable and manageable going forwards</strong>.</p><p>In this post I give my top five actionable tips you can apply while coding to improve the readability of your Python code, without needing to refactor.</p><p>Let’s get into it!</p><h3>1. Use auto-formatters to standardise code 🧹</h3><p>The lowest effort method to improve code readability is using auto code formatting tools to standardise your code.</p><p>There are a number tools in Python to help you with code formatting. Some of the most popular include: <a href="https://github.com/psf/black">Black</a> , <a href="https://github.com/PyCQA/isort">isort</a> , <a href="https://github.com/hhatto/autopep8">autopep8</a> and <a href="https://github.com/google/yapf">YAPF</a> .</p><p>I personally use black for code formatting and isort for bringing order to my library import statements.</p><p>Defining and adhering to a consistent coding style/format throughout a project greatly <strong>increases the readability of your code</strong> for other developers.</p><p>Reading code with poor formatting is like reading an article with bad grammar. You can still understand the text but the meaning can be ambiguous or confusing.</p><p><strong>Consistent formatting that adheres to PEP8 standards helps orientate developers when reading code</strong>. They know what to expect from the code layout, and can instead focus on the important bit — understanding the code logic.</p><p>Standardised and automated <strong>code formatting also helps version control</strong>.</p><p>Imagine the scenario where you create a new file which does not follow PEP8 conventions. Your coworker then checks out the file on version control and makes a change to <em>one line of code</em>. Upon saving the file, their IDE autoformats the original file to adhere to PEP8 (e.g. max line length 79 characters, remove unnecessary blank lines between functions, extraneous whitespace between function arguments etc…). They then check the file back into the repository. There is now a problem. There will be a very large difference (‘diff’) between the old and updated code, even though only a very small change was made to the underlying code logic.</p><p>Standardised, automated code formatting reduces the risk of large diffs in version control that are irrelevant to the functionality of the code being changed.</p><h4>How to use code formatters</h4><p>You can run auto code formatters against your code from the command line. Alternatively, most IDEs (such as VSCode) have settings and plugins that will auto format your code upon saving your code.</p><p>For example, on the command line, you can apply the <a href="https://github.com/psf/black">Black</a> code style to your code and organise your imports with <a href="https://github.com/PyCQA/isort">isort</a> using the following:</p><pre><em># install the black and isort libraries</em><br>pip  install black isort</pre><pre><em># use black command line tool to format code</em><br>black SOURCE_FILE_OR_DIRECTORY</pre><pre><em># organise import statements with isort</em><br>isort SOURCE_FILE_OR_DIRECTORY</pre><p>Below is an extract from my VSCode settings.json file which automatically applies black formatting to my Python files upon saving files in VSCode. It also organises my imports in alphabetical order (similar to the functionality of <a href="https://github.com/PyCQA/isort">isort</a> ) to make imports statements more readable when there are many imported libraries or functions.</p><pre># vscode settings.json<br>{<br>    <strong>&quot;editor.formatOnSaveMode&quot;</strong>:&quot;file&quot;,<br>    <strong>&quot;editor.formatOnSave&quot;</strong>:<strong>true</strong>,<br>    <strong>&quot;python.formatting.provider&quot;</strong>: &quot;black&quot;,<br>    <strong>&quot;editor.codeActionsOnSave&quot;</strong>: { <strong>&quot;source.organizeImports&quot;</strong>: <strong>true</strong>},<br>}</pre><blockquote><strong><em>Note</em></strong></blockquote><blockquote><em>While it is recommended to follow </em><a href="https://www.python.org/dev/peps/pep-0008/"><em>PEP8</em></a><em> coding conventions, there are no universally accepted Python code styling rules. You will notice variations between different projects as a result of the developing team’s own preferences.</em></blockquote><blockquote><em>It is vitally important that all members of your team working on the project are adhering to the same code styling rules. Therefore, it is best to be explicit about which style guide you are following.</em></blockquote><blockquote><em>You can describe the coding style your project uses in a </em><em>README.md, but better still, you can explicitly define the style using </em><a href="https://pycqa.github.io/isort/docs/configuration/config_files.html"><em>configuration files</em></a><em> in your project directory. Each of the tools mentioned above can automatically read from a configuration file to ensure all developers on the project are following the same rules.</em></blockquote><h3>2. Use code checking tools (linters) to catch bugs and follow best practices 🔎</h3><p>Code checking tools, commonly referred to as <a href="https://code.visualstudio.com/docs/python/linting">linters</a> , help highlight syntactical and stylistic problems with your Python code.</p><p>Linters help you to keep your code PEP8 compliant, catch some errors before running your code and warn against bad practices. For example:</p><ul><li>spotting unused variables or variables used before assignment</li><li>identifying unused imported libraries</li><li>ensuring all functions/classes have docstrings</li><li>warning against using <a href="https://www.flake8rules.com/rules/E722.html">bare exceptions</a></li></ul><p>In some cases, the errors highlighted by the linter are not really a problem. For example, pylint suggests each Python module should have a docstring to describe it’s purpose. You might decide for your project that this is not necessary. You can control the behaviour of the linters using <a href="https://pylint.pycqa.org/en/latest/">configuration files or inline comments</a> which can be set to ignore certain errors.</p><p>Once again, there are many linting tools for Python. The two most common are: <a href="https://github.com/PyCQA/flake8">flake8</a> and <a href="https://pylint.pycqa.org/en/latest/">pylint</a> .</p><p>I tend to use flake8 for my pre-commit hooks (see tip below). I also use Pylint during development to help catch issues and identify possible improvements, however, compared to flake8, it is much stricter on seemingly benign issues which can make it impractical to use as the gatekeeper for committing new code to your repo.</p><p>Pylint also gives your code a rating out of 10 which is quite cool. Although it can hurt your ego when your fully working code initially gets rated a 3 out of 10 😞.</p><h4>How to use linters to check your code</h4><p>As above, linters can be run from the command line or built into your IDE.</p><pre><em># install flake8</em><br>pip install flake8</pre><pre><em># run flake8 checks</em><br>flake8 SOURCE_FILE_OR_DIRECTORY</pre><p>You can enable linting in VSCode using the settings.json file. For example, using pylint:</p><pre># vscode settings.json<br>{<br>    <strong>&quot;python.linting.pylintEnabled&quot;</strong>:<strong>true</strong>,<br>    <strong>&quot;python.linting.enabled&quot;</strong>: <strong>true</strong>,<br>}</pre><blockquote><strong><em>Top Tip 💡</em></strong></blockquote><blockquote><strong><em>Extend code checking tools to your Jupyter notebooks using the </em></strong><a href="https://github.com/nbQA-dev/nbQA"><strong><em>nbQA</em></strong></a><strong><em> library.</em></strong></blockquote><blockquote><em>If you use Jupyter notebooks, using code checking tools becomes even more important than simple </em><em>.py files.</em></blockquote><blockquote><em>By nature, work in Jupyter notebooks is experimental and can involve running cells in different orders when you are testing or debugging code. As a result, it can be very easy to accidentally leave unassigned variables, or use variables before they are defined. It is also hard to catch these errors manually before committing the notebook to your git repo.</em></blockquote><blockquote><em>nbQA runs the Python linting tools against your notebooks to help protect against broken notebooks being committed to your repository.</em></blockquote><h3>3. Static type hinting 💡</h3><p>Using type hinting in my code was a game changer for me.</p><p>I cannot emphasise enough how much the readability of your code improves when you start using <a href="https://realpython.com/lessons/type-hinting/">type hinting</a> . If you take anything away from this post, it should be to start using type hinting.</p><h4>What is type hinting?</h4><p>Python is a <a href="https://realpython.com/lessons/dynamic-vs-static/">dynamically typed language</a> . This means that you don’t need to explicitly declare the variable data type in your code. Data types are automatically assigned to variables based on the specific data structure passed to them when the code is run.</p><p>The lower syntactical overhead of dynamically typed languages can be more forgiving and appealing for beginners, however, it can lead to bad habits and less readable code particularly in larger projects.</p><p>Type hints in Python (version 3.6+) are optional comments which signal to the reader of your code what data structure/type is expected for a particular variable or function. E.g str, list, dict, int etc.</p><p>The Python interpreter ignores type hints. Your code will still run if you do not have type hints — or even if you have incorrectly specified type hints. <strong>The purpose of type hinting is purely for the benefit of the developer</strong></p><p><strong>Example</strong></p><p>Take the following function as an example where type hints can be used to improve the developer experience.</p><pre><strong>def</strong> get_largest(items):<br>    <strong>return</strong> max(items)</pre><p>The function above is intended to return the maximum value in a list of numbers. Simple, right?</p><p>Now imagine you are a new developer to the project and see this code. How should you use this function?</p><p>When the function was originally written, I’m sure the author knew exactly what that function did and what it should be used for — it is very simple Python code. However, as a new developer there is ambiguity which could lead to the function being used incorrectly or getting unexpected results.</p><p>The biggest question in this example is: what data structure should I pass into the function argument called items?</p><p>The max Python keyword would work with many different data structures including: dictionaries, lists, strings, lists of strings, sets, tuples. The function above would not throw an error if any of these data structures were passed to it. However, only a list of integers (or perhaps floats) would give the result intended by the original developer.</p><p>Type hinting (and improved variable naming) can dramatically improve the readability and reduce the chance of errors further down the line:</p><pre><em># improved with type hints (python 3.9+ syntax)</em><br><strong>def</strong> get_largest(input_list: list[int]) -&gt; int:<br>    &quot;&quot;&quot;Return largest number in a list&quot;&quot;&quot;<br>    <strong>return</strong> max(input_list)</pre><p>The snippet above uses the : syntax to specify that the data structure expected by the function is a list of integers, and the -&gt; syntax indicates that the function will return an integer.</p><blockquote><em>You can read more about the Python type hint syntax in the </em><a href="https://docs.python.org/3/library/typing.html"><em>documentation</em></a></blockquote><p>While there is still nothing preventing future developers from accidentally passing a string or list of strings to this function, the type hints greatly aid the developer and signal how the function was intended to be used.</p><h4>Type hinting tools</h4><p>You can use tools like <a href="https://github.com/python/mypy">mypy</a> and <a href="https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance">pylance</a> (in VSCode) to check for suspected typing errors.</p><p>If you specify type hints, these tools will check for instances of that variable/function in your code and ensure the correct data structure is being used.</p><p>For example, in my VSCode editor below I have specified that the function expects a list of integers and returns an integer. VSCode has identified two issues with this. Firstly, I am returning a string from the function and not an integer. Secondly, when I try and use the function elsewhere I get a red line to warn me that I am passing a list of strings to the function instead of a list of integers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/0*sYe4Hzsk2uUJjg5V.png" /><figcaption>Image by author</figcaption></figure><p>These warnings can be very useful during development to help catch potential bugs quickly.</p><p>The relevant settings in my vscode settings.json file are:</p><pre>{<br>    <strong>&quot;python.linting.pylintEnabled&quot;</strong>:<strong>true</strong>,<br>    <strong>&quot;python.linting.enabled&quot;</strong>: <strong>true</strong>,<br>    <strong>&quot;python.analysis.typeCheckingMode&quot;</strong>: &quot;strict&quot;,<br>}</pre><p>Alternatively you can use mypy as a command line utility to run static type analysis on your code:</p><pre><em>#install mypy</em><br>pip install mypy</pre><pre><em># run mypy</em><br>mypy SOURCE_FILE_OR_DIRECTORY</pre><h3>4. Automate everything with pre-commit hooks 🪝</h3><p>As programmers, we love automating things. Code formatting and checking should be no exception.</p><p>You can automate code styling and checking using <a href="https://pre-commit.com/">pre-commit</a> hooks. The pre-commit framework allows you to specify which checks you want to run against your code before committing changes to your git repository. These checks are run automatically when committing the code. If any checks fail, the code is rejected until they are fixed.</p><p>This helps protect against bad code being added to your project history and ensures all code in the repo complies with your code styling guidelines.</p><h4>Getting started with pre-commit hooks</h4><ol><li>Install pre-commit</li></ol><pre>pip install pre-commit</pre><p>2. Add .pre-commit-config.yaml configuration file to your project directory</p><pre><em># .pre-commit-config.yaml</em><br><strong>repos</strong>:<br>-   <strong>repo</strong>: https://github.com/pre-commit/pre-commit-hooks<br>    <strong>rev</strong>: v2.3.0<br>    <strong>hooks</strong>:<br>    -   <strong>id</strong>: trailing-whitespace<br>-   <strong>repo</strong>: https://github.com/psf/black<br>    <strong>rev</strong>: 19.3b0<br>    <strong>hooks</strong>:<br>    -   <strong>id</strong>: black</pre><p>3. Install git hooks</p><pre>pre-commit install</pre><p>After following these instructions, next time you try to commit files to the repo, the pre-commit hooks will run and ensure your code is compliant with the black formatter. In the case above, pre-commit will also check for and remove any trailing whitespace in the files</p><p>Check out my <a href="https://engineeringfordatascience.com/posts/pre_commit_yaml/">article on pre-commit hooks</a> for a more detailed tutorial and my full template .pre-commit-config.yaml file that I use for my Data Science projects.</p><h3>5. Write useful documentation ✍️</h3><p>We all want our code to live on beyond our input on the project. But it doesn’t matter how good your software is, if the documentation is not good, people will not use it.</p><p>It is very easy to skip on the documentation during development and is commonly the last thing that gets our attention at the end of the project.</p><p>While there is no point writing detailed documentation for a feature still under development — bad or incorrect documentation is worse than no documentation — there are some simple strategies that will help you keep on top of things without too much overhead. <strong>With better documentation you increase the chances of other people using your code and will make your life easier in the future.</strong></p><p>In Python, documentation generally comes in three forms: inline comments, docstrings and external documentation (e.g. a README.md).</p><h4>Inline Comments</h4><p>In general, you should try to follow the inline comments guidelines set by <a href="https://www.python.org/dev/peps/pep-0008/#comments">PEP8</a> .</p><p>Writing inline comments in code is a surprisingly <a href="https://stackoverflow.blog/2021/12/23/best-practices-for-writing-code-comments/">controversial topic</a> . But I like to take the advice of Martin Fowler in his book, <a href="https://engineeringfordatascience.com/book-notes/refactoring/">Refactoring </a>:</p><blockquote><em>“Comments should explain ‘why’ the code is doing something rather than ‘what’ it is doing” — </em><strong><em>Martin Fowler</em></strong><em>, </em><a href="https://engineeringfordatascience.com/book-notes/refactoring/"><em>Refactoring</em></a></blockquote><p><em>What</em> the code is doing should be self explanatory. However, <em>why</em> the developer chose to use that ‘magic’ number, unusual method or workaround in the code may not be obvious. This is where inline comments make a lot of sense.</p><p>I like to add inline comments where the intention of my code is not immediately obvious. A good example of this is when you are using <a href="https://en.wikipedia.org/wiki/Web_colors">HEX codes</a> for colours — you are never going to remember the colour just by looking at the hex code.</p><p>Below is a snippet from my post on <a href="https://engineeringfordatascience.com/posts/visualising_asset_price_correlations/">Visualising Asset Price Correlations</a> where I used inline comments for this purpose.</p><pre><strong>def</strong> assign_colour(correlation: float) -&gt; str:<br>    &quot;&quot;&quot;Assign hex code colour string based on correlation value&quot;&quot;&quot;<br>    <strong>if</strong> correlation &lt;= 0.0:<br>        <strong>return</strong> &quot;#ffa09b&quot;  <em># red</em><br>    <strong>else</strong>:<br>        <strong>return</strong> &quot;#9eccb7&quot;  <em># green</em></pre><h4>Doc strings</h4><p>Doc strings are used to describe the operation of the function or class. They are defined using the three quotes syntax &quot;&quot;&quot; below the function declaration.</p><pre><strong>def</strong> add(a,b):<br>    &quot;&quot;&quot;Add two numbers together&quot;&quot;&quot;<br>    <strong>return</strong> a + b</pre><p>Adding simple one line doc strings to your code functions, even if you think the function is trivial, goes a long way to improving readability and reducing ambiguity for future developers.</p><p>It is important to note that <strong>doc strings and inline/block comments are not interchangeable</strong>.</p><p>Unlike inline/block comments, doc strings can be accessible at code runtime using either the __doc__ dunder attribute or the built in help() function. This means the doc string can be interpreted by your IDEs and displayed as hints while typing new code.</p><p>I notice programmers coming from other languages do not make proper use of Python’s doc string features. For example, using block comments <em>above</em> the Python function instead of using the doc string syntax:</p><pre><em># function to add two numbers together</em><br><strong>def</strong> add(a,b):<br>    <strong>return</strong> a + b</pre><p>Not only is this not ‘Pythonic’ (see tip 1 — code formatting), but by writing comments like this you surrender your access to doc strings at runtime. As a result, you will not be able to view the function description when writing in your IDE or be able to use auto documentation tools like <a href="https://www.sphinx-doc.org/en/master/">sphinx</a> (see below).</p><p>Depending on the complexity of the function, a simple one-line doc string can be appropriate. Don’t add unnecessary detail if the function name and arguments provide enough descriptive detail in themselves.</p><blockquote><strong><em>Top Tip</em></strong><em> 💡</em></blockquote><blockquote><em>You can use tools like the </em><a href="https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring"><em>Python Docstring Generator</em></a><em> in VSCode to automatically generate doc string templates</em></blockquote><h4>External Documentation</h4><p>The most detailed documentation should be reserved for the project README.md.</p><p>The README should give general information to maintainers and users of the project about how to set up the environment, the purpose of the project, the project directory structure etc.</p><p>You can make use of <a href="https://www.makeareadme.com/">README templates</a> available on <a href="https://github.com/othneildrew/Best-README-Template">GitHub</a> .</p><p>Additionally, if you have been documenting your code with doc strings you can automatically generate documentation in a beautiful user interface using tools like <a href="https://www.sphinx-doc.org/en/master/">Sphinx</a> .</p><h3>Bonus: Read other people’s code 👨‍🎓</h3><p>So this piece of advice does not relate directly improving your own Python code. <strong>However, one of the best ways to learn how to write readable and maintainable code is to read other people’s code.</strong></p><p>You can get hold of other people’s code by looking at popular open source repositories on GitHub, or if you work in a team, by reviewing other people’s code.</p><p><strong>By reading other people’s code, you will be able to spot things that frustrate you when trying to understand their code base</strong>. This could be poorly named variables, unclear data structures required for a function, insufficient documentation (e.g. no docstrings for functions, README) inconsistent or non PEP8 compliant code making it harder to read.</p><p>With this outside perspective looking at someone else’s code, you can apply these learnings to your own code from the perspective of another programmer with no prior knowledge of the code base.</p><p>For example, it was only when I was on a project that inherited a legacy code base that I realised how useful type hints can be for helping other developers (and yourself!) to understand the data structures being passed around your code.</p><h3>Conclusion</h3><p>You can dramatically improve the readability and maintainability of your projects without needing to refactor any code logic, simply by utilising tools developed by Python’s open source community.</p><p>In my experience, you should assume that you will forget the purpose of every function or variable at some point in the future. Therefore, it is always better to be as explicit as possible when writing code the first time round (e.g. Writing descriptive doc-strings, adding explicit type hints etc.). Applying these strategies to your own Python projects will greatly improve the developer experience for collaborators, if not, it will at least make life easier for your future self.</p><p>My top five tips covered in this post include:</p><ul><li>use auto-formatters such as black and isort</li><li>use code checkers such as flake8 and pylint</li><li>use type hints to remove ambiguity in your function arguments</li><li>automate checking with pre-commit</li><li>write good documentation using Python doc strings</li></ul><p>Applying these principles to your development will dramatically improve the maintainability of your code and make your life so much easier in the future!</p><h3>Resources</h3><ul><li><a href="https://gist.github.com/julian-west/e50b25a9aa10551e3452f5a6cfaa6aa3">My template .pre-commit-config.yaml file I use as a base for most projects</a></li><li><a href="https://realpython.com/lessons/type-hinting/">RealPython — Type hinting tutorial</a></li><li><a href="https://www.youtube.com/c/ArjanCodes/videos">Arjan Codes YouTube Channel — software design thinking and clean code</a></li><li><a href="https://docs.python-guide.org/writing/documentation/">Writing Good Python Documentation</a></li></ul><blockquote>This article was originally published on my blog, <a href="https://engineeringfordatascience.com/">engineeringfordatascience.com</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b049bbf72e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/five-tips-to-elevate-the-readability-of-your-python-code-7b049bbf72e6">Five Tips to Elevate the Readability of your Python Code</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>