<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Madhushan Buwaneswaran on Medium]]></title>
        <description><![CDATA[Stories by Madhushan Buwaneswaran on Medium]]></description>
        <link>https://medium.com/@bmshan?source=rss-491f7ade5f15------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*nsTzKio9QCptJ2Sf.</url>
            <title>Stories by Madhushan Buwaneswaran on Medium</title>
            <link>https://medium.com/@bmshan?source=rss-491f7ade5f15------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:46:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bmshan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[“8 More” Python Best Practices for Writing Industry Standard Code]]></title>
            <link>https://medium.com/data-science/8-more-python-best-practices-for-writing-industry-standard-code-64d97f42da5e?source=rss-491f7ade5f15------2</link>
            <guid isPermaLink="false">https://medium.com/p/64d97f42da5e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[Madhushan Buwaneswaran]]></dc:creator>
            <pubDate>Tue, 14 Jun 2022 04:08:25 GMT</pubDate>
            <atom:updated>2022-06-22T16:52:12.008Z</atom:updated>
            <content:encoded><![CDATA[<h3>8 More Python Best Practices for Writing Industry-Standard Code</h3><h4>A few Python best practices I learnt after entering the software industry</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X7s2Wo3VgQBNPKl_bKhzbA.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@divinetechygirl/">Christina Morillo</a> from <a href="https://www.pexels.com/photo/woman-programming-on-a-notebook-1181359/">Pexels</a></figcaption></figure><p>I recently read a good <a href="https://towardsdatascience.com/python-best-practices-for-writing-industry-standard-code-66b848783e0b">article</a> by Anmol Tomar on Python Best Practices. It touches on simple yet effective best practices that can improve your code quality. Then I felt like “Well, I can add a few more to this list by Anmol.” Hence, here I am, writing this blog post. I did not follow these best practices before going to the industry. But after being part of a team, I learnt these best practices from my colleagues and wished I knew and followed these small steps early on.</p><p>“Industry standard code” may sound fancy. But it is nothing other than readable, reusable, modifiable code. In industry, we work in a team. If I write some piece of code while working for a firm, the code I write is owned by the firm. Sometime later, a completely different person, working for the same firm, should be able to read my code, understand it, make improvements on it, or fix a bug in it, or integrate it with another piece of software. If that is not possible, software industry won’t scale beyond individuals. So as long as you are writing simple, understandable code that someone else can improve upon, you are writing “Industry standard code”.</p><p>The 8 best practices I have listed here are in addition to what Anmol has mentioned in his <a href="https://towardsdatascience.com/python-best-practices-for-writing-industry-standard-code-66b848783e0b">article</a>. Do read it as well. There is never an exhaustive list of “best practices.” These are a few simple ones I learnt in the industry, that helped me improve my coding and work in a collaborative environment. So never treat it as a finite set of best practices.</p><h4>1. Don’t use random values in your code. Define them as constants.</h4><p>Let’s say, the piece of code you are writing has one line that converts mass (kg) to weight (N). It is a simple one-liner, where you have to multiply the mass variable by 9.8 ms⁻². So we tend to write a line of code like below.</p><blockquote>Bad practice</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af5918f68cfe278c3e0a1a1345079462/href">https://medium.com/media/af5918f68cfe278c3e0a1a1345079462/href</a></iframe><p>It is technically correct. But when someone reads your code without context, they will wonder why it is 9.8 and where it came from. So it is always good to define it as a named constant and use it subsequently.</p><blockquote>Good practice</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39d784bb0bc50c83ad48c74a53caafcc/href">https://medium.com/media/39d784bb0bc50c83ad48c74a53caafcc/href</a></iframe><p>Python doesn’t have constant definitions like most other languages. Hence we define it as just another variable. But we use ALL_CAPS as a naming convention to indicate that it is a constant. When defining constants, it is a good practice to define those at the top of the source file (after imports). In case the project is large and if many source files share project level constants, you can define all of the constants in a constants.py file and import them as necessary in the required source files.</p><h4>2. Use verbs as function and method names</h4><p>This might sound not so important, but hear me out. Let’s say you are writing a function to calculate the prime factors of a given number. Most probably we tend to define the function like:</p><blockquote>Bad practice</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee51edd19a9becd2decf21c91f07c30c/href">https://medium.com/media/ee51edd19a9becd2decf21c91f07c30c/href</a></iframe><p>Then when calling the function and assigning its output to a variable, we will be wondering what to name the variable because prime_factors is already taken. Same goes if we want to define a local variable inside the function. But if we ponder a little, almost always we write a function or method to “do something”. Functions and methods perform an action on the data. Hence it is a good idea to name your functions and methods as &lt;do&gt;_&lt;something&gt;. In this particular example I would prefer the following as function name.</p><blockquote>Good practice</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/133d2b68a0d423e0718a377144f41ec0/href">https://medium.com/media/133d2b68a0d423e0718a377144f41ec0/href</a></iframe><p>After that I would use prime_factors as the variable name to capture the result of the function when calling it.</p><h4>3. Define members as private or protected as per the access scope requirements</h4><p>Defining attributes and methods of a class as private or protected is not a language feature of Python. But it can be done through a universal convention. It is always a best practice to expose attributes and methods that are only necessary. Everything else should be protected or private members of the class. Protected members are the attributes and methods which are accessible from within the class and its sub-classes, where as private members are accessible only from within the class itself.</p><p>In Python, the following conventions are used to define protected and private members. Members prefixed with single underscore _ are protected and the ones prefixed with double-underscores __ are private members.</p><blockquote>Good practice</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fd86f6429ced3ad7dcf7c1b8f269ca2/href">https://medium.com/media/9fd86f6429ced3ad7dcf7c1b8f269ca2/href</a></iframe><p>As I mentioned above, in Python this is just a convention. Still we have means to access these members. But a good programmer won’t. This indicates to subsequent programmers, who may extend your program, that these attributes and methods are not meant to be accessed from outside.</p><p>Even if you are not using Python in an object-oriented fashion (because most people don’t), you still can adopt the same convention for functions which are not meant to be imported elsewhere. Simply prefix such function names with underscores. This will indicate that this function is local to the file and not supposed to be imported elsewhere.</p><h4>4. Don’t do import *</h4><p>I have done it many times in the past. When I want to import few components that I defined elsewhere, I would simply do:</p><blockquote>Bad practice</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47d27da6f4de5298fb574c8f2e339466/href">https://medium.com/media/47d27da6f4de5298fb574c8f2e339466/href</a></iframe><p>When we do an import * in our code, all the entities from the module will be imported to the current namespace. This severely affects the readability of your code. When you have two such lines, the reader will loose track of where the entities you have used are imported from. Even worse, such import may shadow the names from an earlier import or local definitions.</p><p>The best thing to do is to import the whole package or module if you are accessing a lot of entities. then you can later access the classes or functions as, &lt;module&gt;.&lt;class&gt;. If you are importing only one or two components, use the from &lt;package&gt; import &lt;class&gt; format. But use it scarcely.</p><blockquote>Good practice</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51b05e3545ffb6f022b00d7eccd1f483/href">https://medium.com/media/51b05e3545ffb6f022b00d7eccd1f483/href</a></iframe><h4>5. Use code formatters (or linters)</h4><p>What if I say a lot of the styling (formatting) best practices can be enforced or automatically applied to your code? Yes start using a code formatter or linter. In industrial setting, most organizations have a linter as part of the CI (Continuous Integration) pipeline. When you try to push some code to the organization’s repo, it will be automatically linted and if any prescribed best practices aren’t followed it will raise an error preventing merging of the code.</p><p>So it is better to start using it. <a href="https://github.com/psf/black">Black</a> is a good code formatter. It can help you format your code as prescribed by PEP8 standard. Some IDEs have in built code formatters which you can use with a simple shortcut. Linters are a bit more than code formatters. They are static code analyzers. They can be used to check if proper variable naming conventions are used, check for bugs etc. <a href="https://pypi.org/project/pylint/">Pylint</a> can be a staring point to get introduced to linters.</p><h4>6. Write unit tests</h4><p>This can be boooooooooring and time consuming. But it is essential, when working in a team or on a long term project. Writing proper unit tests makes your code “standardized”. Unit testing is the mechanism through which the functionality of individual logical units of source code is tested. You can simply isolate logical errors and prevent them from propagating up. When working in a team, this is essential, because if a functionally faulty code is merged, it can cause errors when interacting with other components. Trust me, writing proper unit tests is in fact a time saver in the long run. Python’s inbuilt <a href="https://docs.python.org/3/library/unittest.html">unittest</a> library is very good for unit testing your python code.</p><h4>7. Log the errors</h4><p>Your code isn’t bug free. Even after linting, unit testing, manual testing, it is highly likely to fail in production occasionally. Trust me, it is okay. It happens quite often in the industry. What matters is whether you have preemptively employed mechanisms to capture what went wrong and where. You can fix an error only if you have a pointer to what went wrong and where. So enable mandatory error logging in your code. It is simple - use python <a href="https://docs.python.org/3/library/logging.html">logging</a>.</p><blockquote>Good practice</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3d7c7cf379ee14b4b1c7f0a4f13d50f/href">https://medium.com/media/b3d7c7cf379ee14b4b1c7f0a4f13d50f/href</a></iframe><p>Also use the warning, info, and debug level logging as required by your project. Remember the log file is your “blackbox” which you will have to use for a “postmortem”, if your code failed in production.</p><h4>8. Generate requirements.txt file with versions</h4><p>This is more relatable if you are working in the domain of machine learning, or data science or any other domain with packages that are continuously evolving and getting major updates. You will create a virtual environment, install some packages and work on a project, and deliver. Then you won’t touch the project for sometime. After that, you are supposed to make some improvements. Now you try to recreate the environment, there is a significant possibility for your code to break. Well, the reason is packages go through updates which may, at times, contain breaking changes.</p><p>You can simply overcome this by following the best practices mentioned below.</p><ol><li>Have a separate virtual environment for each project</li><li>Track the installed packages using a requirements.txt file per project</li><li>When tracking the packages track them with the exact package versions.</li></ol><p>The third point is very important. If you have the working package version always in the repo, every time you try to recreate the environment, the exact version of the package will be installed. Hence, there won’t be any breaking changes.</p><p>You can manually track the packages you install or if you are maintaining a proper minimal environment you can use the following shell command to generate the file automatically.</p><pre>pip freeze &gt; requirements.txt</pre><blockquote><strong>Edit </strong>: As mentioned by <a href="https://medium.com/u/37afeaaf9b9a">Cesar Flores</a> in the comments, using pip freeze may cause issues as it tracks second level of dependencies as well. So, manually tracking the packages would be the best option.</blockquote><p>If you need more information on requirements.txt, here is a <a href="https://learnpython.com/blog/python-requirements-file/">good read</a>.</p><p>So there you have it, 8 basic Python best practices that I learn after entering the software industry. I hope that you found this article useful. Also, if there are any comments/more best practices that you think should be included, please do so in the comments section.</p><h4>References</h4><ul><li><a href="https://towardsdatascience.com/python-best-practices-for-writing-industry-standard-code-66b848783e0b">Python Best Practices for Writing Industry Standard Code</a></li><li><a href="https://www.tutorialsteacher.com/python/public-private-protected-modifiers">public, protected, private members in Python</a></li><li><a href="https://stackoverflow.com/questions/9916878/importing-modules-in-python-best-practice">Importing modules in Python - best practice</a></li><li><a href="https://github.com/psf/black">GitHub - psf/black: The uncompromising Python code formatter</a></li><li><a href="https://pypi.org/project/pylint/">pylint</a></li><li><a href="https://docs.python.org/3/library/unittest.html">unittest - Unit testing framework</a></li><li><a href="https://docs.python.org/3/library/logging.html">logging - Logging facility for Python</a></li><li><a href="https://learnpython.com/blog/python-requirements-file/">The Python Requirements File and How to Create it</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64d97f42da5e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/8-more-python-best-practices-for-writing-industry-standard-code-64d97f42da5e">“8 More” Python Best Practices for Writing Industry Standard Code</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monty Hall Problem — An empirical proof]]></title>
            <link>https://medium.com/data-science/monty-hall-problem-an-empirical-proof-60e7e0761503?source=rss-491f7ade5f15------2</link>
            <guid isPermaLink="false">https://medium.com/p/60e7e0761503</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[simulation]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Madhushan Buwaneswaran]]></dc:creator>
            <pubDate>Sat, 26 Jun 2021 18:11:47 GMT</pubDate>
            <atom:updated>2021-06-26T18:11:47.142Z</atom:updated>
            <content:encoded><![CDATA[<h3>Monty Hall Problem — An empirical proof</h3><h4>A Python simulation of Monty Hall problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfWlh16wFKZ792wNkd3hBA.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@oleg-magni">Oleg Magni</a> from <a href="https://www.pexels.com/photo/three-yellow-wooden-doors-2726517/">pexels.com</a></figcaption></figure><p>Monty Hall problem is a mathematical brain teaser dealing with probabilistic decision making. It originated from a TV show hosted by Monty Hall in 1963. It is a very good example of how probabilistic scenarios may seem simple but yet at times can be difficult to wrap our minds around them.</p><h3>Scenario…</h3><p>The problem is simple, you go to Monty’s game show “Let’s make a deal”. There are 3 doors in front of you. Behind 1 of those doors, there is a car and behind the remaining two doors are goats! If you select the door behind which the car is there, you drive it home. The host asks you to select a door, you pick one. Then comes the fun part. After you picked a door, the host will open one of the two remaining doors to <strong>always</strong> reveal a goat. Now he would ask you, are you going to switch your selection or are you staying with your original choice? This is where the dilemma kicks in.</p><p>On the surface, it seems there is no clear advantage in staying with your original decision or switching it. Since there are now two unopened doors, we see it as a 50–50 choice. But in fact, if you ponder enough you will see that there is a clear advantage of changing your decision when the host asks you whether you wanna switch!!!</p><p>The key reason this is not a 50–50 scenario is that the host, who opens one of the remaining doors <strong>after you made</strong> your initial choice, has the information that you don’t — which door has the car behind it. And he <strong>never</strong> opens the door which has the car behind it. If we incorporate this piece of information into calculating the probabilities, we end up with the number that you have a 2/3 probability of winning if you switch.</p><h3>Explanation</h3><p>There are a lot of good explanations using multiple methodologies on the internet. (I find <a href="https://www.youtube.com/watch?v=4Lb-6rxZxx0">this</a>, <a href="https://www.youtube.com/watch?v=hdo4CZC76fA">that</a> and <a href="https://www.statisticshowto.com/probability-and-statistics/monty-hall-problem/">the other</a> one to be interesting.) But personally, I feel, in probability, anything can be simply and intuitively explained using a tree diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6dM33P7b4qm25XsxqDZ6qQ.png" /><figcaption>Figure 1 — Tree diagram showing the probabilities associated with the Monty Hall Problem (Diagram by the Author)</figcaption></figure><p>When you are asked to make your first choice, there is an equal probability that the car is behind any one of the three doors. So you have a 1/3 chance of guessing it correctly. This implies that 2/3 of the times your guesses are wrong. So now comes the host and he opens a door to reveal a goat. At this point his choice of opening a door is not random. It is governed by your previous choice, since he has a rule not to reveal the car when opening the door.</p><p>Let’s first consider the case where your first choice is correct. Then the remaining two doors have goats behind them. In this scenario, the host can open any of those two doors. So now if you change your initial choice you will definitely loose regardless of the host’s choice of door. But remember this happens only 1/3 of the times because your initial choice is only correct that many times.</p><p>Now consider the case when your initial choice is wrong. You choose a door behind which a goat is there. Now out of the two remaining doors, one has the car behind it, and one has a goat behind it. Now the host cannot open a door randomly. He cannot open the door with the car behind it. So he is <strong>forced</strong> to open the door with the other goat behind it. At this point, if you change your initial choice, you’ll definitely win. If we look back to initial probabilities, we can realize that this is the most likely scenario, since 2/3 of the times our initial guess is wrong. So you should always make the switch! Statistically, if you play this game infinitely many times you will win in 2/3 of the games you play, if you always make the switch.</p><p>I was discussing this with one of my friends and he was like, “yeah that seems understandable but it is still a bit dizzy to get a feel for this explanation”. So I thought, “Hey let’s simulate it and get the same number!”. That way, seeing empirical proof would be more convincing!</p><h3>The Pythonic way!</h3><p>So let’s get to the fun part! I created the whole program with number of doors as configurable, so that if someone wants to simulate scenarios with more doors (as explained <a href="https://www.statisticshowto.com/probability-and-statistics/monty-hall-problem/">here</a>), it can be done.</p><p>First, let’s look at the Contestant class. It has one constructor parameter switch. This parameter defines the response of the contestant when asked “Do you wanna switch?”. By changing this parameter when instantiating the contestant object, we can simulate a contestant who always switches and a contestant who never switches.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/328d28e41011474b9f1f86bea3384160/href">https://medium.com/media/328d28e41011474b9f1f86bea3384160/href</a></iframe><p>The Contestant class has two methods. The first one is select_door(). This is used to make the initial choice. When given the number of doors (door_count) as input, it selects and returns one door (selection) with a random uniform probability. Next method is wanna_switch(). This method is to be invoked when the host opens all remaining doors, presents one alternative door, and asks the contestant if she/he wants to change her/his decision? It takes one input: the new_door. If the contestant’s attribute switch is False, this method returns the old selection (sticking with the original decision). If the contestant’s attribute switch is true, it returns the new_door (changing the original choice).</p><p>Next we have the Host class. It takes one constructor parameter door_count. During instantiation the Host class randomly ‘places’ the imaginary car behind one of the doors by setting the private attribute _car_index. <em>I have made the </em><em>_car_index, </em><em>door_counts attributes of the </em><em>Host class, since the host has knowledge of these.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc5b9e22a48045b0fb05559cf54049b6/href">https://medium.com/media/bc5b9e22a48045b0fb05559cf54049b6/href</a></iframe><p>The Host class has three methods. The first one is present_doors(), which is used to show the doors to the contestant by returning the door_count. The next method is eliminate_others_and_present_one_door(). This method takes the contestant’s first selection (contestant_selection) as an input and returns an alternative_door as output, to ask the contestant whether she/he wants to switch? If the contestant&#39;s first selection is the door with the car behind it, it randomly returns another door as alternative_door. If the contestant’s first choice is not the door with the car behind it, this method returns the _car_index (door with the car behind it) as alternative_door. (<em>This returning of </em><em>alternative_door is analogous to opening all other doors and revealing goats behind them.</em>) The final method is reveal_if_winner(). It takes the contestant’s final_selection as the input and returns if the contestant has won or not.</p><p>Putting everything together we can simulate one iteration of the competition using the snippet shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a88a6b83e141735751dd84c44d9e6fd1/href">https://medium.com/media/a88a6b83e141735751dd84c44d9e6fd1/href</a></iframe><p>However to get the actual probability of winning we need to simulate this infinitely many times (which is impossible). Hence I have simulated this for 1 million times and used the resulting number as a proxy to actual probability. As we increase the number of iterations, we can see in fig. 2, that the empirical probability is converging towards the (expected) actual probability.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c67b3d563c44954a8a3489b911de9ec/href">https://medium.com/media/1c67b3d563c44954a8a3489b911de9ec/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iOCI9OOxFNDf3V7k1JLaug.png" /><figcaption>Figure 2 — Plot showing the convergence of empirical win probabilities towards the theoretical probabilities with more iterations (Diagram by the author)</figcaption></figure><p>I have simulated two contestants: one who always changes her/his decision and one who always sticks with the original decision. From the above results, we can clearly see, that if we switch our decision when asked, we have a better chance of driving the imaginary car home.</p><p>Safe driving… 😁</p><p>In case if you want to access the full code (I don’t know why someone would want that), it can be found <a href="https://github.com/bmsknight/monty-hall-problem">here</a>.</p><h3>References</h3><ul><li><a href="https://en.wikipedia.org/wiki/Monty_Hall_problem">Monty Hall problem - Wikipedia</a></li><li><a href="https://www.statisticshowto.com/probability-and-statistics/monty-hall-problem/">Monty Hall Problem: Solution Explained Simply</a></li></ul><p>“Numberphile” explanation video — <a href="https://www.youtube.com/watch?v=4Lb-6rxZxx0">https://www.youtube.com/watch?v=4Lb-6rxZxx0</a></p><p>“The Infographics Show” explanation video — <a href="https://www.youtube.com/watch?v=hdo4CZC76fA&amp;t=3s">https://www.youtube.com/watch?v=hdo4CZC76fA</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60e7e0761503" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/monty-hall-problem-an-empirical-proof-60e7e0761503">Monty Hall Problem — An empirical proof</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementation differences in LSTM layers— Tensorflow vs Pytorch]]></title>
            <link>https://medium.com/data-science/implementation-differences-in-lstm-layers-tensorflow-vs-pytorch-77a31d742f74?source=rss-491f7ade5f15------2</link>
            <guid isPermaLink="false">https://medium.com/p/77a31d742f74</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Madhushan Buwaneswaran]]></dc:creator>
            <pubDate>Sun, 25 Apr 2021 18:18:55 GMT</pubDate>
            <atom:updated>2021-04-28T04:14:50.314Z</atom:updated>
            <content:encoded><![CDATA[<h3>Implementation Differences in LSTM Layers: TensorFlow vs PyTorch</h3><h4>Drawing parallels between TensorFlow LSTM layer and PyTorch LSTM layer.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kp4XYg7obqvn4VVjOXi7-g.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@anna-nekrashevich">Anna Nekrashevich</a> from <a href="https://www.pexels.com/photo/pen-business-eyewear-research-6801648/">Pexels</a></figcaption></figure><p>Tensorflow and Pytorch are the two most widely used libraries in deep learning. Both these libraries have different approaches when it comes to implementing neural networks. Both these libraries started with major differences. Usually, TF was more preferred for developing production-ready models due to its better optimizations, and Pytorch was preferred for research works due to its more “pythonic” syntax and eager execution. But with Torchscript and TF 2.0, the gap between the two libraries has shrunk.</p><p>Both of these libraries have good community support and active contributions. So we can easily implement different types of neural networks without any major issues in both these libraries. But these two libraries have architectural differences. Hence, there are certain differences we need to focus on when implementing a neural network.</p><p>One such difference is in the layers API. A neural network (NN) can be designed using either the sub-classing approach or sequential model API approach in both these libraries. The sub-classing approach is more preferred out of the two since it is object-oriented and extensible. When implementing a NN in either of these libraries, we can use already designed layers — linear (fully connected) layers, convolution layers, recurrent layers, etc. and extend them into our model. In TF, we use tensorflow.keras.layers, and in Pytorch, we use torch.nn to access these layers. As I mentioned earlier, there are subtle differences in the implementation of these layers. Most of the time, they are minor and intuitive. But in LSTM (Long Short-Term Memory) layers, these differences are somewhat major and significant. The way the layer is parameterized, the default values for the parameters, and the default output of the layer mostly differ between Pytorch and TF in LSTM layers.</p><p>In this post, I will try to explain the differences between LSTM layers in these two libraries. I hope it will be of help if someone refers to a NN (with LSTM layers) implemented in one library and tries to reproduce it in the other.</p><h3>An Intro to LSTM</h3><blockquote><em>I have added this section only for the sake of completion and as a refresher (in case one needs it). But if you are trying to understand the implementation differences when using LSTM layers, then I hope you already have a background in deep learning and know the fundamentals of LSTMs. So you can skip this part.</em></blockquote><p>A recurrent neural network (RNN) is a special type of NN which is designed to learn from sequential data. A conventional NN would take an input and give a prediction based on that input only. It doesn’t look into previous inputs/outputs and make a decision. But when predicting the next value in sequential data (eg. — sentences, the daily average temperature in a city), we cannot do it by looking at the current data point only. We have to look at the behavior of the series (up to the current point) as a whole and get the ‘context of the series’ to make a meaningful prediction. This is what RNNs are specialized for. RNNs are used for tasks like language modeling, machine translation, sequence prediction, etc.</p><p>Fig. 1 shows how an RNN takes the current value and previous lateral output as input. The lateral output of the previous timestep is passed by the recurrent mechanism (as aᵗ). It is expected (and proven) that the RNN layer will learn to capture the ‘context’ of the sequence in the lateral output. Please note that the unrolled version of the RNN shown in fig. 1 (b) depicts the <strong>same</strong> <strong>layer</strong> as copies for better understanding and doesn’t represent multiple layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CQrovkr1mK2njvnKgsZCPQ.png" /><figcaption>Figure 1 — (a) A general RNN with loop (z⁻¹ symbolizes a unit time delay). (b) Same RNN with time series loop unrolled (diagram by the author, inspired by <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">Chris</a>’s work)</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b741fec0b4505890cee53471b9ec9ba6/href">https://medium.com/media/b741fec0b4505890cee53471b9ec9ba6/href</a></iframe><p>The design principle of RNN expects that the ‘context’ derived from earlier values in the series persists along the sequence for a long range. But it was observed in practical scenarios, the RNN’s ability to learn the ‘context’ of a sequence deteriorates with distance. That is, RNNs are not good enough to capture long-term dependencies in sequences. Long Short-Term Memory (LSTM) networks are a specialized version of RNN that was introduced with the intention of preserving long-term ‘context’ information in sequences. The inherent design of LSTMs enables them to capture the long-term context through what is known as cell state (denoted by cᵗ). The diagram below (Fig. 2) shows a typical LSTM layer. The diagram I used is based on the diagrams used in a blog post on <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">understanding LSTMs</a> by Chris Olah. (It is a wonderful blog post. Do check it out.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*ACJoB7r1T4qNUQulNqQqLQ.png" /><figcaption>Figure 2 — A basic LSTM layer (diagram by the author, inspired by <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">Chris</a>’s work)</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/771aefa0e2e126de7852a4c2e5800f43/href">https://medium.com/media/771aefa0e2e126de7852a4c2e5800f43/href</a></iframe><p>As Chris explains in his blog, the cell state can be thought of as a conveyor belt, which carries the long-term ‘context’ from start to end of the sequence. At each timestep based on the current input (<em>x</em>ᵗ) and the previous output (hᵗ⁻¹), the LSTM layer decides what information to forget from the ‘context’ and what information to add to the ‘context’. The forgetting is handled by the forget gate (Γ𝒻). Γ𝒻 can be thought of as a mask that is applied to cᵗ⁻¹. If a value in a certain position in Γ𝒻 is 0 (or closer), when multiplied with cᵗ⁻¹, it removes the information in that position from the context. If the value of the mask is 1 (or closer) in some other position, after multiplication with cᵗ⁻¹, it allows the information in that position to persist in the context. The operation Γ𝒻.*cᵗ⁻¹ (<em>.* — element-wise multiplication</em>) makes sure that the unwanted part of the context is forgotten. The addition of new information to the cell state is handled in two steps. First, a candidate vector (c̃ᵗ) is created through the sub-layer with tanh activation, based on <em>x</em>ᵗ and hᵗ⁻¹. Then the update gate (Γᵤ) generates a mask (just like in forget gate) which decides what part of the candidate vector is added to the cell state. ThenΓᵤ.*c̃ᵗ is added to cᵗ⁻¹ to generate cᵗ. Finally, to generate the output, we pass cᵗ through a tanh non-linearity and use it. But to decide what part of it to give out, we make use of the update gate (Γₒ). Γₒ.*tanh(cᵗ) is given out as the output of the LSTM layer at timestep t.</p><p>Please note that both cᵗ⁻¹ and hᵗ⁻¹ are given as lateral inputs in an LSTM layer, compared to the vanilla RNN layer where only aᵗ⁻¹ is given.</p><p>If you need additional information on the principles of RNN and LSTM, I would recommend going through the first week of the <a href="https://www.coursera.org/learn/nlp-sequence-models">Sequence Models</a> course by Andrew Ng (it is free to audit) and reading this wonderful <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">blog post</a> by Chris Olah.</p><h3>LSTM layer in Tensorflow</h3><blockquote>At the time of writing Tensorflow version was 2.4.1</blockquote><p>In TF, we can use tf.keras.layers.LSTM and create an LSTM layer. When initializing an LSTM layer, the only required parameter is units. The parameter units corresponds to the number of output features of that layer. That is units = nₕ in our terminology. nₓ will be inferred from the output of the previous layer. Hence the library can initialize all the weight and bias terms in the LSTM layer.</p><p>TF LSTM layer expects a 3 dimensional tensor as input during forward propagation. This input should be of the shape (batch, timesteps, input_features). This is shown in the code snippet below. Suppose we are using this LSTM layer to train a language model. Our input will be sentences. The first dimension corresponds to how many sentences we use as one batch to train the model. The second dimension corresponds to how many words are present in one such sentence. In practical setting, the number of words in each sentence varies from sentence to sentence. So, in order to batch these sentences, we can select the length of the longest sentence in the training corpus as this dimension and pad the other sentences with trailing zeros. The last dimension corresponds to the number of features used to represent each word. For simplicity, if we say, we are using one-hot encoding and there are 10000 words in our vocabulary, then this dimension will be 10000.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea5b2e091b9421a3e3c6115e96525aac/href">https://medium.com/media/ea5b2e091b9421a3e3c6115e96525aac/href</a></iframe><p>But when initializing the layer, if we set time_major = True, then the input will be expected in the shape - (timesteps, batch, feature).</p><p>As seen from the above code snippet, the output of the LSTM (with default parameters) is of shape (32,4), which corresponds to (batch, output_features). So if we go back to the example of the language model, the output has one vector per sentence, with nₕ number of features per sentence (nₕ = units = no. of output features). This one vector (per sentence) is the output of the LSTM layer corresponding to the last timestep T (last word of the sentence). This output is hᵀ in our notation. This is depicted in fig. 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*l9k82VuwtSrdg_a6EF2gMA.png" /><figcaption>Figure 3 — Default output of Tensorflow LSTM layer (diagram by the author)</figcaption></figure><p>However, if we want to stack multiple LSTM layers, the next layer also expects a time series as input. In such situations, we can set return_sequences=True when initiating the layer. Then the output will be of shape (32,10,4), which corresponds to (batch, timesteps, output_features). If return_sequence is set to True, then hᵗ : ∀t = 1,2…T will be returned as output. This is shown in the code snippet below and the first LSTM layer in fig. 4.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/076ed5f749abc7106d596850572a4ec6/href">https://medium.com/media/076ed5f749abc7106d596850572a4ec6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*xRe8z9Znm2CqGeeYRwfmBw.png" /><figcaption>Figure 4 — A simple model with 2 LSTM layers and 2 fully connected layers. Note that LSTM 1 layer outputs a sequence and the LSTM 2 outputs a single vector. (diagram by the author)</figcaption></figure><p>If we want to get the cell state (cᵗ) as an output, we need to set return_state=True when initiating the layer. Then We get a list of 3 tensors as output. According to <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM">documentation</a>, if we set both return_sequences=True and return_state=True, these three tensors will be — whole_seq_output, final_memory_state,and final_carry_state. This is shown in the code snippet below.</p><p>In our notation,</p><ul><li>whole_seq_output — output corresponding to all timesteps. <br>hᵗ : ∀t = 1,2…T ; Shape — (batch, timesteps, output_features)</li><li>final_memory_state — Output corresponding to the last timestep.<br>hᵀ; Shape — (batch, output_features)</li><li>final_carry_state — Last cell state.<br>cᵀ; Shape — (batch, output_features)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/922e4820197d3ed5a85a33ab1fdcbee9/href">https://medium.com/media/922e4820197d3ed5a85a33ab1fdcbee9/href</a></iframe><p>If we set return_sequences=False and return_state=True, then these three tensors will be — final_memory_state, final_memory_state, and final_carry_state.</p><p>A single LSTM layer has five places where activation functions are used. But if we look at the parameters, we see only two parameters to set activation functions — activation and recurrent_activation. If we set a value to the activation parameter, it changes the activation applied to the candidate vector and the activation applied to the cell state just before element-wise multiplication with the output gate. Setting a value to recurrent_activation will change the activation functions of forget gate, update gate, and output gate.</p><p>Other parameters are quite self-explanatory or seldom used. One other thing to note is, we can set unroll=True, and the network will be unrolled. It will speed up the training process but will be memory intensive (since the same layer is copied multiple times).</p><p>The following code snippet implements the model shown in fig. 4 using TF. Note the output shape of each layer and the number of trainable parameters in each layer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b752738926584c1ca441ad0a9e73c9a5/href">https://medium.com/media/b752738926584c1ca441ad0a9e73c9a5/href</a></iframe><h3>LSTM layer in Pytorch</h3><blockquote>At the time of writing, Pytorch version was 1.8.1</blockquote><p>In Pytorch, an LSTM layer can be created using torch.nn.LSTM. It requires two parameters at initiation input_size and hidden_size.input_size and hidden_size correspond to the number of input features to the layer and the number of output features of that layer, respectively. In our terminology, hidden_size = nₕ and input_size = nₓ.</p><p>During forward propagation, the Pytorch LSTM layer expects a 3 dimensional tensor as input (similar to TF). But the default ordering of the dimensions changes. The input tensor should be of shape (timesteps, batch, input_features). If we want to get the same order of dimensions as TF, we should set batch_first=True at layer initiation.</p><p>Another major difference that can be seen in Pytorch LSTM API is that, at initiation, we can set num_layers=k and initiate a block of k LSTM layers stacked as a single object. However, I personally do not prefer this approach since it makes the overall implementation less readable and less maintainable.</p><p>The next big difference is the output of the Pytorch LSTM layer. The output of the Pytorch LSTM layer is a tuple with two elements. The first element of the tuple is LSTM’s output corresponding to all timesteps (hᵗ : ∀t = 1,2…T) with shape (timesteps, batch, output_features). The second element of the tuple is another tuple with two elements. The first element of this second tuple is the output corresponding to the last timestep (hᵀ). It has the shape (1, batch, output_features). The second element of this second tuple is the cell state corresponding to the last timestep (cᵀ). It also has the shape (1, batch, output_features). If we had initiated the LSTM as a block of stacked layers by setting num_layers=k, then hᵀ and cᵀ would have the shape (k, batch, output_features). Here, both hᵀ and cᵀ would have the last states of all the k layers in the stack. Further at initiation, had we set batch_first=True, then the timesteps and batch dimensions would swap in the output (similar to the input).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03251f97fc7d15e71b7c9a4765130c57/href">https://medium.com/media/03251f97fc7d15e71b7c9a4765130c57/href</a></iframe><p>As far as I know, changing the activation functions inside the LSTM layer is not possible in Pytorch. Also, it is not possible to limit the LSTM layer to give only one output (as in TF). However, we can assign the outputs to variables, use the required output, and ignore the others, as shown in the code segment below. Apart from that, other parameters should be self-explanatory.</p><p>Note that if you set the LSTM layer to be bidirectional (which I didn’t talk about in this article), then the output shape will be different from what I have mentioned above. Please refer to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html">documentation</a> for such a scenario.</p><p>The following code snippet implements the model shown in fig. 4 using Pytorch. Note the output shape of each layer and the number of trainable parameters in each layer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f04a26805380ddaaa71abbd24359b77/href">https://medium.com/media/1f04a26805380ddaaa71abbd24359b77/href</a></iframe><p>If we look at the number of parameters in the two implementations of the model in fig. 4., it can be observed that there is a difference in the number of parameters in LSTM layers. This is again a design choice. Pytorch does a minor change when implementing the LSTM equations (1), (2), (3), and (4). TF adds a single bias vector (as in our equations) in each of these equations. But Pytorch (as shown <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html">here</a>) adds <strong>two bias vectors </strong>per equation. Since the shape of each bias vector is (nₕ,1) and four such additional vectors are there per layer, there will be 4*nₕ more parameters in the Pytorch LSTM layers. In LSTM1 layer nₕ=8, so there are 32 additional parameters. In LSTM2 layer nₕ=4, so there are 16 more parameters.</p><h3>References</h3><ul><li><a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">Understanding LSTM Networks</a></li><li><a href="https://www.coursera.org/learn/nlp-sequence-models">Sequence Models</a></li><li><a href="https://medium.com/analytics-vidhya/demystifying-lstm-weights-and-biases-dimensions-c47dbd39b30a">Demystifying LSTM Weights and Biases Dimensions.</a></li><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM">tf.keras.layers.LSTM | TensorFlow v2.16.1</a></li><li><a href="https://stackoverflow.com/questions/44947842/can-someone-explain-to-me-the-difference-between-activation-and-recurrent-activa">Can someone explain to me the difference between activation and recurrent activation arguments passed in initialising keras lstm layer?</a></li><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html">LSTM - PyTorch 2.6 documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77a31d742f74" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/implementation-differences-in-lstm-layers-tensorflow-vs-pytorch-77a31d742f74">Implementation differences in LSTM layers— Tensorflow vs Pytorch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>