<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oleg Sinavski on Medium]]></title>
        <description><![CDATA[Stories by Oleg Sinavski on Medium]]></description>
        <link>https://medium.com/@oleg.sinavski?source=rss-b1c6a0c2d9b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*deaYEQ4QixWrmqU1IeVZBA.png</url>
            <title>Stories by Oleg Sinavski on Medium</title>
            <link>https://medium.com/@oleg.sinavski?source=rss-b1c6a0c2d9b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:33:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@oleg.sinavski/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My talks on Visual Language Model in Autonomous Driving]]></title>
            <link>https://medium.com/@oleg.sinavski/my-talks-on-visual-language-model-in-autonomous-driving-9a1295bef161?source=rss-b1c6a0c2d9b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a1295bef161</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[vision-language-model]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Oleg Sinavski]]></dc:creator>
            <pubDate>Sun, 29 Sep 2024 20:12:27 GMT</pubDate>
            <atom:updated>2024-09-29T20:12:27.590Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t9JSLK-Mrcnc5__b5YEmag.jpeg" /></figure><p>Hello! Here is a small collection of talks I gave in various conferences and workshops on VLMs in Autonomous Driving.</p><h3>CVPR 24: Language models in end-to-end autonomous driving</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FoeGis1cz_zM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DoeGis1cz_zM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FoeGis1cz_zM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1449ce1d8f0ef5a63a1f1c502b2dd263/href">https://medium.com/media/1449ce1d8f0ef5a63a1f1c502b2dd263/href</a></iframe><ul><li>Why use Language models in end-2-end driving?</li><li>Measuring LLM capabilities in driving</li><li>Paper survey: Non-visual LLMs in self-driving</li><li>Paper survey: VLMs for perception in self-driving</li><li>Why is it important to go end-2-end with VLMs?</li><li>Paper survey: End-2-end VLMs in self-driving</li><li>Unsolved problems, conclusions, Q&amp;A</li></ul><p>Don’t forget to check out other great talks from the <a href="https://wayve.ai/cvpr-e2ead-tutorial/">E2E Autonomy Tutorials session</a>!</p><h3>BMVA 2024: Trustworthy Multimodal Learning with Foundation Models</h3><p>Talk at a <a href="https://www.bmva.org/meetings/24-04-24-Multimodal%20Learning.html">Trustworthy Multimodal Learning with Foundation Models</a> meeting.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKK5gkkWjySI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKK5gkkWjySI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKK5gkkWjySI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed899ee14d5af761279fc4e6248c949d/href">https://medium.com/media/ed899ee14d5af761279fc4e6248c949d/href</a></iframe><ul><li>Why use Language models in end-2-end driving?</li><li><a href="https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/">Lingo-1</a></li><li>LingoQA and a trainable LLM judge</li><li>Trustworthiness and explainability of end-2-end models</li><li><a href="https://wayve.ai/thinking/lingo-2-driving-with-language/">Lingo-2</a></li></ul><h3>WandB Fully Connected: Vision language action models for autonomous driving at Wayve</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FNJNYkSEwkLM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DNJNYkSEwkLM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FNJNYkSEwkLM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bcd818ab77dece251e916b58bfc67158/href">https://medium.com/media/bcd818ab77dece251e916b58bfc67158/href</a></iframe><p>This is a short version of the BMVA 2024 talk above at the <a href="https://wandb.ai/site/resources/events/fully-connected/london">WandB event</a>.</p><h3>WACV 2024: Large language and Vision models for Autonomous driving</h3><p>Keynote talk at <a href="https://llvm-ad.github.io/previous_workshop/">LLVM-AD workshop</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FWNaJPyVNPdw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWNaJPyVNPdw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FWNaJPyVNPdw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2deb5c76c46691e411223f95dc682e0/href">https://medium.com/media/e2deb5c76c46691e411223f95dc682e0/href</a></iframe><ul><li>Software 2.0, Wayve’s end-2-end approach</li><li><a href="https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/">Lingo-1,</a> LingoQA</li><li>GAIA-1</li></ul><h3>Language and Video generative AI in Autonomous Driving</h3><p>Chalmers University of Technology, guest lecture for <em>SSY340 — Deep Machine Learning</em> course.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FdQ-HsAgPFGw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdQ-HsAgPFGw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FdQ-HsAgPFGw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6e369f53a30fc45e6512c61ecdc1328/href">https://medium.com/media/a6e369f53a30fc45e6512c61ecdc1328/href</a></iframe><ul><li>Autonomous driving introduction and previous systems</li><li>Software 2.0, Wayve’s end-2-end approach</li><li>Limitations of end-2-end systems</li><li><a href="https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/">Lingo-1</a></li><li><a href="https://wayve.ai/thinking/introducing-gaia1/">GAIA-1</a></li></ul><p><em>Originally published at </em><a href="https://www.sinavski.com/page/talks/index.html"><em>https://sinavski.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a1295bef161" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Software for exploration]]></title>
            <link>https://levelup.gitconnected.com/software-for-exploration-c526d44851ae?source=rss-b1c6a0c2d9b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c526d44851ae</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[management-and-leadership]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Oleg Sinavski]]></dc:creator>
            <pubDate>Tue, 19 Mar 2024 16:31:44 GMT</pubDate>
            <atom:updated>2024-03-20T17:55:59.692Z</atom:updated>
            <content:encoded><![CDATA[<h4>When to take shortcuts in research software</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bt9kDypcKFDhpkI6" /><figcaption>A researcher discovered production software techniques. Image by author</figcaption></figure><p>I worked in research all my life, so I know a stereotype that researchers write ugly code (e.g. see <a href="https://www.the-scientist.com/how-to-fix-science-s-code-problem-70447">here</a>, <a href="https://techblog.bozho.net/the-astonishingly-low-quality-of-scientific-code/">here</a>, or <a href="https://www.quora.com/Why-do-many-people-from-academia-write-bad-code#:~:text=It&#39;s%20human%20nature%3A%20short%2Dterm,feel%20enough%20pressure%20to%20learn">here</a>). But I thought: we can fix it, right? So multiple times I tried to design nice research frameworks. I tried to bring in interfaces and create nice abstractions using software engineering books and blogs I liked reading.</p><p>But over and over again all those efforts went in vain. The majority of research software I worked on never went to production (although some did). It would have been great for my mental health if someone told me a simple truth: <em>dying research code is actually what is supposed to happen.</em> Researchers should not spend much time engineering it in the first place.</p><p>Professional software engineers always look down on researchers who are not using the best software practices. There are several posts trying to raise the bar of research code (e.g. <a href="https://invenia.github.io/blog/2020/07/07/software-engineering/">this great post</a> and a <a href="https://goodresearch.dev/">research code handbook</a>). But this post goes other way around: it argues how to not overdo the best software practices and instead invest only in fast exploration. It is targeted for research-oriented companies where your goal is to try out many ideas fast.</p><h3>1. Take on some strategic tech debt</h3><p>A successful research project at a company has two phases: exploration and exploitation. In “exploration” you want to try out as many diverse paths as you can. During “exploitation” you want to robustify the best solution and turn it into a useful product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7dxzMOKbt6brttEt3z2_w.png" /><figcaption>During exploration many projects die out. You should build a robust solution only during exploitation. Image by author</figcaption></figure><p>Optimal software practices are quite different between the two. That’s why companies often have separate research and product divisions. All the books you might typically read on software design are mainly about the second “exploitation” phase. In this phase you are building foundations for a scalable product. This is where all the design patterns come in: nice APIs, logging, error handling and so on.</p><p>But in the first “exploration” phase you are <strong>not</strong> building foundations that will live forever. In fact, if the majority of your efforts survive, then you (by definition) did not explore enough.</p><p>Many practices in this post are examples of what would normally become “tech debt”. It’s what you get by not writing clean reusable well-abstracted code. Is debt always bad? We prefer never getting a loan or a mortgage, but borrowing money is often a good strategy in life. It’s ok to get into debt to move fast and profit later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*2EOfr7FJNd1UzUks4DVoxQ.png" /><figcaption>It’s ok to get into software debt in research — you don’t have to repay all of it, only for the minority on the successful research paths. Image by author</figcaption></figure><p>Similarly, by not taking technical debt you might be slowing down your research. The good news is that the majority of the time you don’t have to pay it back. Most your research code is likely to die anyway. So on average, you will not be suffering from whole tech debt you’ve taken.</p><h4>The case against code reuse</h4><p>Many software architectures and refactoring techniques are specifically oriented to improve code reusability. There are generic <a href="https://dev.to/pluralsight/reusable-code-the-good-the-bad-and-the-ugly-3do6">downsides</a> to code reuse. But in production they are outweighed by the well-known benefits (for example, see <a href="https://blog.filestack.com/the-benefits-of-reusing-code-for-software-development/">this typical post</a>). In research projects, the majority of the code is destined to sink to oblivion. Striving for code reuse could actually slow you down.</p><p>Limiting code reuse is the type of technical debt that is ok to take in research. There are several patterns of code reuse I want to discuss: adding an unneeded dependency, copypasting code, maintaining a lot of shared research code, premature design investments.</p><h4>Think twice before importing something new</h4><p>If you know a well-maintained versioned library that is going to speed you up — go for it! But before taking in a new dependency, try to make a judgment call whether it’s worth it. Every additional one brings you closer to dependency hell. It makes you invest time into learning and troubleshooting it. See more pitfalls of dependencies in this <a href="https://builtin.com/software-engineering-perspectives/software-dependency-costs">concise post</a>.</p><p>It’s probably fine to depend on something if:</p><ul><li>you used it already, there is not much to learn, it has a large community, good docs and tests</li><li>it is versioned, easy to install</li><li>and finally, there is no way you can implement it yourself.</li></ul><p>But be wary about a dependency if:</p><ul><li>you can’t figure out how to use it quickly, it is very new (or very old) or no one seems to know about it; there are no docs or tests</li><li>it is from your monorepo and is constantly being changed by other teams</li><li>it pulls in many other dependencies and tools; or it’s just hard to install</li><li>and finally, you feel that you (or some LLM) can write this code in a few hours.</li></ul><p>Instead of an explicit dependency, you can follow a nice Go proverb: “<a href="https://www.efekarakus.com/2021/09/23/a-little-copying-is-better-than-a-little-dependency.html">a little copying is better than a little dependency</a>”, which is our next topic.</p><h4>Copypaste gives you freedom of experimentation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NR7rh_2okRPNrSwBrdsQ9g.png" /><figcaption>Copypasting is fast and it sometimes the best tool during research. Image by author</figcaption></figure><p>Some say that “<a href="https://thecodist.com/copy_and_pasting_code_should_be_illegal/">copy-paste should be illegal</a>”. But to my surprise I found myself arguing in favor of it quite often. Copypaste could be the optimal choice during the exploration phase.</p><p>If you depend on a heavily-used function from another part of the codebase you can forget about easily changing it. You’re likely to break something for someone and have to spend precious time in code reviews and fixes. But if you copypaste the necessary code into your folder, you are free to do anything you want with it. This is a big deal in research projects where experimentation is a norm rather than an exception. Especially if you are not sure if changes are going to be useful for everyone.</p><p>I find that deep learning codebases are suitable for copypasting the most. Typically, the amount of code needed to describe a model and its training is not so huge. But at the same it could be very nuanced and hard to generalize. Shareable training scripts tend to grow to an unmanageable size: e.g. Hugging Face transformers <a href="https://github.com/huggingface/transformers/blob/v4.38.2/src/transformers/trainer.py">Trainer</a> has +4k lines. Interestingly enough, transformers opted for copypaste on the model level. Please check out their <a href="https://huggingface.co/blog/transformers-design-philosophy">post</a> with the reasoning behind their &quot;single file model&quot; policy.</p><p>An alternative to copypaste is staying on a branch. But I feel like it brings too much overhead in teamwork. Also, I found several more posts about the beauty of copypaste — check it out in “Resources” at the end.</p><h4>Maintaining shared research code is hard</h4><p>Maintenance of heavily used shared code requires a lot of work. Take a look at the torch.nn.Module number of file lines plotted against the Pytorch version. You can see that even the most advanced research teams struggle to keep the complexity in check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*btrUwI4vHNAgCeNhK2OgPg.png" /><figcaption>PyTorch.nn.Module file length depending on the version. It is not getting simpler. Image by author</figcaption></figure><p>Don’t underestimate time and resources needed to maintain a large shared research code. The more a research library is used the more complicated it becomes. It happens faster than for a typical library because every research direction has a slightly different usecase. Establish very strict rules of what could be contributed back. Otherwise, the shared code becomes fragile and overgrown with a slew of options, buggy optimizations and edgecases. Since the majority of research code dies out, all this extra complexity will never be used again. Dropping some of your shared code will free up some time to do actual research.</p><h4>Design for exploration, not for code reuse</h4><p>It is somewhat true that you don’t want <a href="https://blog.developerpurpose.com/dont-future-proof-your-code-aad04ef75584#:~:text=If%20you%20find%20yourself%20future,do%20is%20build%20something%20simple">to future-proof your code</a> too much even in production. Try to implement the simplest possible solution that meets the requirements. But in production code there are always aspects of maintainability to consider. For example, error handling, speed, logging, modularization is what you typically need to think about.</p><p>In research code, none of that matters. You just want to quickly prove that an idea is good or bad in the fastest possible manner and move on. So the dirty simplicity that achieves it without any modules or APIs is totally ok!</p><p>Don’t waste valuable time on premature software investments such as:</p><ul><li>creating component interfaces too early in the project. You’ll spend too much time fitting into self-made artificial constraints</li><li>optimizing deep learning training infrastructure before committing to deep learning solution</li><li>using production config/factories/serialization systems or base classes. Quite often you don’t need their functionality during prototyping</li><li>overly-strict linting and type checking systems. No reason to slow down fast-changing throw-away research code.</li></ul><h3>2. Invest into fast exploration</h3><p>The goal of a research project is to find a novel solution. No one knows (by definition) what it looks like. It is akin to an optimization process in a complicated research landscape with limited information. To find a good minimum, you need to try many paths, recognize good and bad paths and not get stuck in local minima. To do all of it fast, you sometimes need to make software investments instead of taking tech debt.</p><h4>Speed up common paths</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*y2-xRdrjMI590mlrGuqA9A.png" /><figcaption>Invest into speed in up common parts of your research projects. Image by author</figcaption></figure><p>There are several different research paths you want to try. Is there a design, a library or an optimization that would shave off time from the majority of paths? You should be careful to not over-engineer anything because you don’t always know all the ideas you’re about to try. This is very custom to every project, but here are some examples:</p><ul><li>if you train deep networks, invest into training infrastructure. Figure out hyperparameters allowing you to converge during training quickly and reliably</li><li>if every experiment requires you to use a different model, figure out how you can quickly swap them (e.g. by using a simple factory system or just copypaste)</li><li>if every experiment has too many parameters and is hard to manage, invest into a nice configuration library.</li></ul><h4>Branch out quickly</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*gDg406jXfdY89j2Q7CK6vw.png" /><figcaption>Invest into velocity of initiating new research paths. You need many diverse directions to find the solution. Image by author</figcaption></figure><p>Researchers should be able to initiate new diverse ideas quickly. It seems easy at the start of the project. But then it gradually becomes harder and harder as people get entrenched in their favourite research paths. To address this, cultural and organizational changes are essential. There should be a process stopping non-promising research before sinking too much money and emotion into it. Regular demo days and technical peer reviews can serve as effective strategies for this purpose. It’s also important to find a balance between people jumping on a new shiny idea and properly closing current projects.</p><p>But this is a software post, so here are some practices to make branching out new projects easy:</p><ul><li>keep evaluation code disentangled from algorithms. Evaluations are typically more stable than research directions</li><li>embrace starting a new project from a blank slate, but then watch out which components are reused. Modularizing and cleaning them up is a good investment</li><li>in a new research project, implement the most innovative and risky component first. Doing so identifies the majority of bottlenecks guiding future software design.</li></ul><h4>Increase signal to noise ratio</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*z2RS3lRFfezzSMMSOZAeow.png" /><figcaption>Bugs and non-determinism can derail research projects and make results inconclusive. Image by author</figcaption></figure><p>Noisy and buggy code makes results so ambiguous and inconclusive that the whole project is going to be a waste of time. While you shouldn’t over engineer things, you can easily follow these simple rules of thumb to avoid messy code:</p><ul><li>avoid code with <a href="https://medium.com/gitconnected/all-dangers-of-side-effects-for-python-coders-fdf0743457a3">side-effects</a></li><li>default to functions rather than classes; and with classes, prefer encapsulation vs. inheritance</li><li>minimize length of functions/classes/modules; minimize number of if-statements</li><li>know python well, but use simple techniques. Resist the temptation going into intellectual weeds of metaclasses, decorators and functional programming.</li></ul><p>Software that produces different results during different runs is tricky to work with. If you made an important but wrong decision based on an unlucky seed, you are going to waste a lot of time recovering. Here are some tips when dealing with non-deterministic software:</p><ul><li>understand whether noisy is coming from the algorithm or it’s evaluation. Noise sources compound and you should strive for completely deterministic evaluation.</li><li>don’t stop finding sources of randomness until you really get a reproducible script. Remember that after finding all random seeds, noise might come from data or from generic functions with side-effects.</li><li>vary seeds and determine baseline variance of your results. Don’t take decisions on non-statistically significant results.</li></ul><h3>Conclusion</h3><p>The punchline comes from <a href="https://qr.ae/psI7J4">this post</a> about research code:</p><blockquote>You don’t bother with good software design because the code is not the point. The code is a tool that gets you the answer that you need</blockquote><p>It is extremely important to have great coding foundations. But at the end of the day, the exploration and actually useful product is what matters. If you use too much production software in research, you waste time needed to discover something new. Instead, find what slows down your exploration process. Speed up research paths by investing in fast branching, time to results and clean noiseless code.</p><p>It would be crazy to argue completely against code reuse. I just want to point out that code reuse should be a well-balanced activity. In research, the ratio of throw-away code is larger than in production. The balance is tilted further against reuse. See more great posts with pitfalls of code reuse below.</p><p>Thank you for reading! You can find me on <a href="https://www.linkedin.com/in/oleg-sinavski/">LinkedIn</a> or <a href="https://twitter.com/SinavskiOleg">Twitter</a>.</p><h4>Resources</h4><p>More posts about balancing code reuse:</p><ul><li><a href="https://dev.to/pluralsight/reusable-code-the-good-the-bad-and-the-ugly-3do6">Reusable Code: The Good, The Bad, and The Ugly</a></li><li><a href="https://thinkingsideways.net/code/code-duplication.html">When It Makes Sense to Repeat Yourself</a></li><li><a href="https://softwareengineering.stackexchange.com/questions/448168/why-should-we-reuse-code-as-binary-modules-instead-of-copy-pasting/448178#448178?newreg=3302e0728d254fe38f5bf964d9c1c787">Balanced StackExchange with pros and cons of reuse</a></li></ul><p>More posts arguing for copypasting practices:</p><ul><li><a href="https://dzone.com/articles/copy-and-paste-programming">Is copypaste really a problem?</a></li><li><a href="https://betterprogramming.pub/when-developers-are-allowed-to-copy-and-paste-the-code-d8db988c78ae">When Should You Copy-Paste Code?</a></li><li><a href="https://javascript.plainenglish.io/the-good-and-the-bad-of-copy-paste-programming-6483d7ecb340">The Good and the Bad of Copy-Paste Programming</a></li><li><a href="https://www.reddit.com/r/datascience/comments/10lz6vf/how_often_do_you_copy_and_paste_code/">“As a senior with 5 YOE, I feel like part of my pay is knowing what code to copy lol”</a></li></ul><p><em>Originally published at </em><a href="https://sinavski.com/post/5_research_software/"><em>https://sinavski.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c526d44851ae" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/software-for-exploration-c526d44851ae">Software for exploration</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dealing with side effects step by step]]></title>
            <link>https://levelup.gitconnected.com/dealing-with-side-effects-step-by-step-76c100426b89?source=rss-b1c6a0c2d9b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/76c100426b89</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[refactoring]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Oleg Sinavski]]></dc:creator>
            <pubDate>Thu, 03 Aug 2023 14:40:14 GMT</pubDate>
            <atom:updated>2023-08-03T14:40:14.215Z</atom:updated>
            <content:encoded><![CDATA[<h4>A visual guide on how to clean up your Python code</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NTRG97gJ80xYSukcf3xjKQ.png" /><figcaption>Navigating in a codebase with hidden factors — Image by author</figcaption></figure><p>After reading Part 1 — “<a href="https://medium.com/gitconnected/all-dangers-of-side-effects-for-python-coders-fdf0743457a3">All dangers of side effects</a>,” you might wonder what is the best way to deal with side effects in your code. This is a set of recipes I found helpful throughout my long Python career. Are there enough posts about unnecessary side effects? Here is a sales pitch for this post:</p><ul><li>Many visuals for dealing with side-effects</li><li>Python-pragmatic and not overloaded by functional programming jargon</li><li>Many links to other great posts and advanced discussions</li></ul><h3>What are we striving for?</h3><p>A good (<a href="https://betterprogramming.pub/what-is-a-pure-function-3b4af9352f6f">“pure”</a>) function should</p><ul><li>not cause any changes outside its scope (avoid side effects)</li><li>produce the same output for the same set of inputs (avoid relying on hidden factors)</li></ul><p>People <a href="https://stackoverflow.com/questions/4391524/is-returning-a-random-value-from-function-a-side-effect">debate</a> whether the second property should be called a “side effect”. I will call incoming hidden factors <strong>“side arguments”</strong> and outgoing external side effects <strong>“side results.”</strong></p><p>It seems like 65% of people are <a href="https://en.wikipedia.org/wiki/Visual_thinking">visual thinkers</a>, so here are some pictures for you (the other 35% have to digest the code)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g0OMO1g3RF17BEK2-HlhOw.png" /><figcaption>A diagram of a dirty function with side effects</figcaption></figure><p>Calling system time.time() is an &quot;input&quot; side effect from the external system on the function. The print statement is an &quot;output&quot; side effect from the function to the external world. If one removes all side effects from this red &quot;dirty&quot; function, we will get a side-effect-free green one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JUibJU7UFcYqtz9gs_kMfw.png" /><figcaption>A diagram of a clean, pure function</figcaption></figure><p>If you don’t bother cleaning up side effects, your software will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5cUoomCLaiVdgj2psVHdkA.png" /><figcaption>A typical application with side-effects throughout</figcaption></figure><p>There are many dirty red functions called by top functions (larger circles). Both have side effects (side-arguments and side-results). Occasionally, a nice clean green function is buried underneath.</p><p>It’s important to understand what is the goal state here. <strong>We do not want to remove all side effects.</strong> Software that doesn’t interact with the world is truly useless. What we want to strive for is a picture like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXz9KxHV4tIj_NqWqRmXug.png" /><figcaption>Functional core, imperative shell application</figcaption></figure><p>All necessary side effects are pushed to the application’s boundary, but all the internal core logic is green and clean! This is what’s called <a href="https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell">“Functional core, imperative shell”</a>. There are at least a dozen posts I found about that idea. After digging around, it seems like <a href="https://uvsmtid.info/shells-cores/">this one</a> is quite clean and has nice pictures. But also see this <a href="https://kennethlange.com/functional-core-imperative-shell/">summary post</a> and a <a href="https://news.ycombinator.com/item?id=18043058">discussion</a>.</p><h3>Start from reusable utility functions</h3><p>Figuring out the correct order of refactoring can speed up the cleanup dramatically.</p><p>Notice that a dirty function down the stack infects all functions that call it. If there are ten-layer-deep function calls and a small utility at the bottom is dirty, the whole ten-layer stack is spoiled.</p><p>The first thing to do is to clean up frequently reused elementary functions and classes.</p><p>Hence, here is a good refactoring order:</p><ul><li>Identify highly reused dirty functions that are lowest in the call stack</li><li>Clean them up (see below)</li><li>Keep going until you run out of simple functions</li><li>Move a level up the call stack</li><li>Find reused dirty functions that are new lowest in the call stack</li><li>…</li><li>Strive for the “Functional core, Imperative shell” ideal!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HJCMj7XsiTGVvjWdV7pUew.gif" /><figcaption>Good refactoring order</figcaption></figure><h3>Remove some side effects altogether</h3><p>Can you rewrite your function without affecting any external world whatsoever? That would be the best option! Sometimes neglected, but the first step is to remove unnecessary stuff. Here are some common examples.</p><p>Often, people think their logging statements will be helpful for others forever:</p><pre>def inverse(x):<br>    print(&quot;A message that I feel everyone&quot; <br>        &quot;would benefit from!&quot;)<br>    return -x</pre><p>In reality, those logs are probably helpful just for you and only this week. A good rule of thumb is to remove most prints and logs from your branch before merging into the mainline. Most likely, you wouldn’t even notice the lack of a log message.</p><p>Another typical case is initializing an external resource for no reason. Quite often in machine learning, I see algorithms that create folders for saving results</p><pre>def algorithm_1(x):<br>   os.mkdir(&quot;folder_with_results&quot;)<br>   return -x</pre><p>It is not an algorithm’s job to create folders. Instead, it should be moved into the application initialization logic.</p><p>Sometimes people forget to remove a previously useful side effect after refactoring or debugging.</p><pre>def algorithm(argument):<br>    set_theano_flags(current_time=time.time())  # Init something used 5 years ago<br>    result = (... complicated logic based on argument)<br>    return result</pre><p>To summarize, a decent percentage of side effects could be just removed completely.</p><h3>Simple refactoring flow</h3><p>What is the simplest refactoring recipe to make functions pure? Here is where the split on side arguments and results comes in handy:</p><ul><li>move side-arguments up the function body. Then transform it into a regular argument.</li><li>move side-results down the function body. Then return the side effect together with other results.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*QPO2FYflkaX4PTcUXamfqQ.gif" /><figcaption>A refactoring flow for a single function</figcaption></figure><p>Let’s look at the real-life example I encountered. The following function performs some random image augmentations. The researcher sometimes wants to debug intermediate random masks and dumps them into temp files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-1exZMlo7qGTUQv9kyq2w.jpeg" /><figcaption>Image augmentation with side-arguments and results</figcaption></figure><p>This function has 2 side arguments: it reads fom the global random generator for the random mask and a cutout. It has 4 side-results:</p><ul><li>change global random generator state 2 times</li><li>writes a temp debug file</li><li>prints out debug file name</li></ul><p>This function is hard to test and will always bring you trouble by polluting /tmp and the console.</p><p>First, let’s move side-arguments up and side-results down. This step could be skipped, but I found it very useful in more convoluted and long functions. We create a random generator once from a global state and move debug outputs down:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KA8HTCLD6Q086lGCxZCqdQ.jpeg" /><figcaption>Side-channels are moved to the borders of the function</figcaption></figure><p>Finally, let’s merge “side” and “regular” arguments and results. We arrive at a clean image augmentation logic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*URSKzdhbF1r61vtHM1GMTQ.jpeg" /><figcaption>Clean random augmentation function</figcaption></figure><p>We are taking a random generator in and returning the debug mask. The high-level function is free to choose whether to pass seeded rng for reproducibility or use the global one for convenience. It will also decide how to save the debug masks if still needed.</p><h3>Short-circuit side effects</h3><p>Side-effects volume grows non-linearly. When one developer adds a side argument or a result, it encourages others to use a similar side channel.</p><p>When a function reads globals (envvars, files), it compels other functions to write to globals (envvars, files) so as to influence its behavior.</p><p>On the flip side, the more you remove, the easier it is to deal with the rest. After you move all side effects from the bottom up to upper functions, you typically find that you can completely “short-circuit” some. Here is a visual:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*obF9aNGoYAQ7JjlJNALEbw.gif" /><figcaption>Short-curcuit side channels</figcaption></figure><p>Let’s look at a real but a bit longer example. Here is an often-encountered pattern in research code to use the filesystem for passing parameters around:</p><ol><li>create a default config file -&gt;</li><li>modify the config file -&gt;</li><li>read out from a config file</li></ol><p>Below is an extract from the actual code I encountered:</p><pre>def dump_default_config(path):<br>   default_config = {&#39;hidden_size&#39;: 128, &#39;learning_coeff&#39;: 0.01}<br>   with open(path, &#39;wb&#39;) as f:<br>       pickle.dump(default_config, f)  # side result!<br><br>def run_network(network_config_path, image):<br>   with open(network_config_path, &#39;rb&#39;) as f:<br>       config = pickle.load(f)  # side argument!<br>   network = create_network(**config)<br>   prediction = network(image)<br>   return prediction<br><br>def network_main(image):<br>   config_path = &#39;my_config.pkl&#39;<br>   dump_default_config(config_path)  # side result!<br>   # update the learning coefficient in the config file<br>   with open(config_path, &#39;rb&#39;) as f:<br>       config = pickle.load(f)  # side argument!<br>   config[&#39;learning_coeff&#39;] = 1e-4<br>   with open(config_path, &#39;wb&#39;) as f:<br>       pickle.dump(config)  # side result!<br>   return run_network(config_path, image)</pre><p>As we discussed, let’s move side arguments and results up the stack. First, move up the saving of the default config, and second, move up the config loading.</p><pre>def create_default_config():<br>   return {&#39;hidden_size&#39;: 128, &#39;learning_coeff&#39;: 0.01}<br><br>def run_network(config, image):<br>   network = create_network(**config)<br>   prediction = network(image)<br>   return prediction<br><br>def network_main(image):<br>   config_path = &#39;my_config.pkl&#39;<br>   default_config = create_default_config()<br>   with open(config_path, &#39;wb&#39;) as f:<br>       pickle.dump(default_config, f)  # side result!<br>   # update learning coefficient in the config file<br>   with open(config_path, &#39;rb&#39;) as f:<br>       config = pickle.load(f)  # side argument!<br>   config[&#39;learning_coeff&#39;] = 1e-4<br>   with open(config_path, &#39;wb&#39;) as f:<br>       pickle.dump(config)  # side result!<br>   with open(config_path, &#39;rb&#39;) as f:<br>       config = pickle.load(f)  # side argument!<br>   return run_network(config, image)</pre><p>Now we can short-circuit all the file system calls and arrive at neat, side-effect-free code. On top of being considerably smaller and simpler, it is also much faster:</p><pre>def create_default_config():<br>   return {&#39;hidden_size&#39;: 128, &#39;learning_coeff&#39;: 0.01}<br><br>def run_network(config, image):<br>   network = create_network(**config)<br>   prediction = network(image)<br>   return prediction<br><br>def network_main(image):<br>   config = create_default_config()<br>   config[&#39;learning_coeff&#39;] = 1e-4<br>   return run_network(config, image)<br></pre><p>To short-circuit a side effect, you should first identify input and output side effects of the same type (files, globals, envvars). Then you can lift all of them up the stack and remove them together.</p><h3>Expose side effects to developers</h3><p>Side effects might bring you problems, but <strong>hidden</strong> side effects are the worst. Imagine you decided to use an external library to make a friendly math-related application:</p><pre>from external_library import compute_optimal_solution<br><br>def main():<br>   x = input(&quot;Enter the number&quot;)<br>   value = compute_optimal_solution(x)<br>   print(&quot;Optimal value is :&quot;, exp(value))</pre><p>You happily deploy it only to receive user complaints about database-related crashes. You’re really surprised since you just wanted to provide some math utility and never intended to deal with databases. Looking into the source of compute_optimal_solution, you might find something like:</p><pre>def compute_optimal_solution(x):<br>   result = 0<br>   for i in range(1000):<br>       result += i*x - log(i*x) + sin(i*x)<br><br>   # to understand how people use our function, <br>   # we log the results in the debug database<br>   database_cache = sqlite3.connect(DEFAULT_DB)<br>   cursor = database_cache.cursor()<br>   cursor.execute(f&quot;INSERT INTO MyDB (argument, solution) VALUES ({x}, {result})&quot;)<br>   database_cache.commit()<br>   cursor.close()<br><br>   return result</pre><p>You respect the developer’s desire to collect debugging data, but you would never have guessed it upfront. It would have saved you so much time if this function was named appropriately:</p><pre>compute_optimal_solution_and_cache_solution_in_database(x)</pre><p>You’d quickly realize that this function isn’t suitable for your basic math script.</p><p>It is a generic rule that the name of the function should describe what this function does. The same goes for side effects — if your function has a side effect, you better put it in its name. Using this naming rule, you’ll find that the ugliest and the most dangerous functions have the longest names!</p><p>You also can expose side effects by splitting “clean” and “dirty” code on the module level. For example, a library-like folder should have only clean side-effect-free code. All side effects should go into an application-like folder (e.g., scripts, app, or runners). Here is another <a href="https://medium.com/nerd-for-tech/what-are-side-effects-in-programming-51f7ef340f98">nice short post</a> reinforcing this point.</p><h3>Misc and advanced</h3><p>The points above should work for the majority of side-effect cleanups. But there are many exceptions and advanced concepts that wouldn’t fit this post. Here are some follow-ups.</p><p><em>Dependency injection?</em> Injecting an object that <em>might</em> produce a side effect instead of producing it yourself is a common way to kick a can down the road:</p><ul><li>pass random generator instead of querying it</li><li>pass Timer instead of time.time()</li><li>pass logging.Logger instead of a print</li></ul><p>After reading many resources, <a href="https://stackoverflow.com/questions/130794/what-is-dependency-injection">this thread</a> is probably stays the best. Here is a great <a href="https://jrsinclair.com/articles/2018/how-to-deal-with-dirty-side-effects-in-your-pure-functional-javascript/">JavaScript post</a> about it and side effects. In a liberal interpretation, replacing a side-argument by a regular one <a href="https://www.jamesshore.com/v2/blog/2006/dependency-injection-demystified">could be called</a> a dependency injection. One should be careful not to overdo it (<a href="https://stackoverflow.com/questions/2407540/what-are-the-downsides-to-using-dependency-injection">1</a>, <a href="https://softwareengineering.stackexchange.com/questions/371722/criticism-and-disadvantages-of-dependency-injection">2</a>).</p><p><em>Return a functor?</em></p><p>Instead of causing a side effect right away, you can return a “lazy” function that would do it later. See <a href="https://dev.to/ruizb/side-effects-21fc">the post</a> about this technique. Also, see “Lazy functions” section in <a href="https://jrsinclair.com/articles/2018/how-to-deal-with-dirty-side-effects-in-your-pure-functional-javascript/">this post</a>. It’s a fun long read, but it probably goes beyond the needs of a regular Python mortal.</p><p><em>Copy an input container?</em></p><p>Modifying an incoming list or dict is also a side effect. Quite often, it&#39;s worth copying, modifying and returning it instead. See the discussion <a href="https://stackoverflow.com/questions/20569142/is-there-any-way-to-prevent-side-effects-in-python">here</a> and <a href="https://goodresearch.dev/decoupled.html#kernighan">this post</a>. Is it slower? Probably, yes. But the actual question should be this: will your company spend more money on (A) executing slower code or (B) debugging bugs caused by side-effects? Without picking (A) or (B), there is no basis for fear of slow but robust code.</p><p><em>prints and loggers?</em></p><p>While logging is a side effect, it&#39;s not the worst one. At least, the majority of developers don&#39;t treat it as such. It&#39;s hard to advise anything specific without going on a long tangent. You can adopt configurable logging, pass a logger as a dependency to every function, return string messages or stick withprint (e.g. <a href="https://pythonspeed.com/articles/python-multiprocessing/">if you use multiprocessing</a>).</p><h3>Conclusion:</h3><p>At the end of the day, remember to be pragmatic: you don’t want to remove all side effects, you just want to remove unnecessary ones. In practice though, many side effects are unnecessary. So you better allocate some time for refactoring and deal with them using some recipes from this post:</p><ul><li>make a pass on your code and remove some side effects altogether</li><li>identify side-arguments and side-results and merge them with arguments or results</li><li>identify reused utilities and clean them up first</li><li>notice that some side-effects are caused by each other and short-circuit them</li><li>expose side effects to developers.</li></ul><p>Finally, thank you for reading!</p><p><em>Originally published at </em><a href="https://sinavski.com/post/4_dealing_with_side_effects/"><em>https://sinavski.com.</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76c100426b89" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/dealing-with-side-effects-step-by-step-76c100426b89">Dealing with side effects step by step</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All Dangers of Side Effects for Python Coders]]></title>
            <link>https://levelup.gitconnected.com/all-dangers-of-side-effects-for-python-coders-fdf0743457a3?source=rss-b1c6a0c2d9b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdf0743457a3</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[side-effects]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Oleg Sinavski]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 13:04:44 GMT</pubDate>
            <atom:updated>2023-03-21T08:27:10.294Z</atom:updated>
            <content:encoded><![CDATA[<h4>List of reasons to avoid side effects collected in a single place</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8N_-hBNfIyMRxD9M81_M8w.jpeg" /><figcaption>Side effects polluting the codebase — Photo by <a href="https://unsplash.com/@sandym10?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Sandy Millar</a> on <a href="https://unsplash.com/photos/yvpexJFLTSU?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Are there enough posts about unnecessary side effects? Unlikely — you still see them everywhere. Here is a sales pitch for this post:</p><ul><li>no functional programming jargon</li><li>a go-to list of reasons to avoid side effects</li><li>Python-oriented</li></ul><h3>Spotting side effects</h3><p>Some posts describe side effects in the context of functional programming (mainly in JavaScript): <a href="https://jessewarden.com/books/real-world-functional-programming/part1/02_what_is_a_side_effect.html">one</a>, <a href="https://www.yld.io/blog/the-not-so-scary-guide-to-functional-programming/">two</a>, <a href="https://thejs.dev/jmitchell/what-are-side-effects-and-what-you-can-do-about-them-jws">three</a> and a <a href="https://en.wikipedia.org/wiki/Side_effect_(computer_science">wiki</a>. You don’t have to study functional programming to grasp this concept. Unfortunately, unnecessary side effects are prevalent in all coding paradigms, especially OOP.</p><p>Let’s take a simple function that computes the absolute value of a number:</p><pre>def abs(x):<br>   if x &lt; 0:<br>       return -x<br>   return x</pre><p>What comes out from the return statement is called a “primary effect”. Now let’s take a look at another function:</p><pre>def abs_effectful(x):<br>    if x &lt; 0:<br>        return -x<br><br>    twitter_api = TwitterAPI(some_twitter_id())<br>    chat_bot = LanguageModel()<br>    tweet_content = twitter_api.read_first_tweet()<br>    response = chat_bot.make_response(tweet_content,<br>       f&quot;{x} is important because &quot;)<br>    twitter_api.respond(response)<br>    twitter_api.close()<br><br>    return x</pre><p>Here we also have the same primary effect returned by the abs_effectful function. But a section in the middle performs pretty surprising actions and influences the external world. It has side effects.</p><p>You may ask: “Why do you have a weird unrelated piece of code in your function??” It’s easy to spot side effects in this contrived example. But it is not so far from what happens in large codebases. It’s just a set of refactorings, misunderstandings, and a stream of people joining and leaving teams. Time passes, and simple code accumulates <strong>unnecessary</strong> baggage.</p><p>Here are some real examples of side effects that I’ve encountered in my Python ML career:</p><ul><li>publishing a number to the Weights-and-Biases server out of the loss function</li><li>dumping debug images into a home folder from a computer vision algorithm</li><li>changing an environmental variable inside a function</li><li>dropping to pdb on if condition</li><li>changing a global variable</li><li>calling sys.exit inside an algorithm</li><li>sending a Slack message if nan is encountered during training</li><li>… and so on.</li></ul><p>After seeing many of those, you get a <a href="https://marvelanimated.fandom.com/wiki/Spider-Sense">Spider-Sense</a> for spotting side effects. If you get surprised by what function does, quite often the function has a side effect. In fact, the code with unnecessary side effects <strong>is not as simple as it could be</strong>. See <a href="https://medium.com/javascript-scene/the-single-biggest-mistake-programmers-make-every-day-62366b432308">a great post from Eric Elliot</a> about simplicity, surprises and side effects.</p><p>Side effects are unavoidable because software must have some influence on the world. But are unnecessary side effects harmful? If there are no visible differences, is there a reason to prefer one script over another?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Blimr_SE3vJZrbIYPi4n2A.jpeg" /><figcaption>More than 7 plagues of side effects (Credit: <a href="https://commons.wikimedia.org/wiki/File:Martin,_John_-_The_Seventh_Plague_-_1823.jpg">John Martin, public domain</a>)</figcaption></figure><h3>Unexpected crashes</h3><p>How can abs crash? It pretty much can’t. What about abs_effectful? It might crash in almost every line:</p><ul><li>a Twitter API might fail to initialize</li><li>a language model can’t allocate GPU memory</li><li>it might fail to parse or generate a phrase</li><li>a Twitter API might fail to publish a response or crash upon closing</li></ul><p>Every side-effect code line might crash since it deals with things you don’t control. Now, from the real examples above:</p><ul><li>Weights-and-Biases or Slack might be unavailable</li><li>you might run out of space in the home folder</li><li>an environmental variable is deleted before setting, and so on …</li></ul><p>The code with side effects is more fragile.</p><h3>Unexpectedly slow code</h3><p>Every side effect might bring computational costs. But most importantly, it brings timing variability into your software. Because side-effect code typically deals with uncontrolled external resources, you get uncontrolled slowdowns. Every side-effect code line in abs_effectful can bring you a massive slowdown:</p><ul><li>a Twitter API could be slow to connect</li><li>a language model might wait for free GPU memory</li><li>computational load on the system might make phrase parsing too slow</li></ul><p>In real-world code, similar things can happen:</p><ul><li>database access could be very slow because the network is busy</li><li>writing to a hard drive could be slow because of fragmentation</li><li>logging could be too slow because other loggers compete for file access, and so on …</li></ul><p>The bottom line is that the code with side effects is slower.</p><h3>Hard to use it concurrently</h3><p>You must be careful about any state you modify in concurrent (or even worse <a href="https://stackoverflow.com/questions/1897993/what-is-the-difference-between-concurrent-programming-and-parallel-programming">parallel</a>) software to avoid <a href="https://verdagon.dev/blog/python-data-races">data races</a> and <a href="https://superfastpython.com/thread-deadlock-in-python/">deadlocks</a>. Code with side effects modifies the external state. It will often require a significant redesign to be used in a concurrent context. You need to ensure that only one function will use a global resource simultaneously and that the order of operations doesn’t break the logic.</p><p>The timing variability mentioned above makes it even worse. It’s a big topic, so here is a <a href="https://thejs.dev/jmitchell/what-are-side-effects-and-what-you-can-do-about-them-jws">good post</a> discussing concurrency and shared state access in depth.</p><h3>Hard to optimize</h3><p>Optimizing a function that doesn’t modify anything externally is more straightforward. You can reorder operations inside it and cache the computation results. Since you’re not constrained by an external call for a side effect, you can choose any tool for the job. Many side-effect-free functions can be easily rewritten in numpy/torch/jax and Pybind.</p><p>So on top of being slow, code with side effects is hard to speed up.</p><h3>Hard to understand</h3><p>What if I ask you to make a docstring for abs_effectful? You will definitely struggle to write a concise story about the purpose of that function.</p><p>A function with a single return statement like absachieves a single result. At a minimum, you can concisely write a docstring as</p><p>This function computes an absolute value.</p><p>If there are side effects, you have to add them all one by one:</p><p>This function computes an absolute value AND opens Twitter API and creates a model AND generates and sends a tweet AND closes connections.</p><p>In a large company, many people are joining and changing teams. New people have to understand obscure side effects. A company with less-readable code will waste days of cumulative onboarding time. A large docstring will not help: the longer the doc, the greater the chance it will be ignored completely.</p><h3>Hard to release and debug</h3><p>The code with side effects interacts with stuff which is not under your direct control. After thinking about all imaginable failure modes of abs_effectful, you might still find that it crashes after the release.</p><p>There is a crazy variability in the external world. It is tough to reproduce, debug and fix all edge cases of side effects. Often it happens stochastically, and you just can’t reproduce a single crash in-house (e.g., maybe when abs_effectfulruns on an old Windows version, GPU kernels behave differently?).</p><p>The code with side effects takes much longer to release and debug.</p><h3>Hard to test</h3><p>Writing exhaustive unit tests for code with side effects is very hard. On top of the main result of abs_stateful, you should better test that all side effects behave as expected: did the person get a tweet? Was the tweet parsed correctly? Not to say of some weird unit test interactions: side effects from one test could make other tests fail.</p><p>Interacting with the external world on a test server is often impossible. Developers have to resort to mocking all external entities. See more in this <a href="https://blog.thecodewhisperer.com/permalink/you-dont-hate-mocks-you-hate-side-effects">beautiful post</a> from CodeWhisperer.</p><h3>Hard to reuse</h3><p>What if your colleague wants to compute the absolute value of a number? There is no problem with abs. You import the function and use it! What about abs_effectful? They wouldn&#39;t know a priori about any side effects (ignoring the hint in the name :). The code can happily work until some unfortunate day when some crazy issues from above start coming up.</p><p>After finding the root cause, they’ll anticipate all the problems we mentioned. Your colleague will just leave such a function alone and write their own simpler version.</p><p>A codebase that relies on side effects would typically have much less code reuse.</p><h3>Conclusion</h3><p>Hopefully, you’re convinced that <strong>unnecessary </strong>side effects are also the root of all evil (just like <a href="https://web.archive.org/web/20130731202547/http://pplab.snu.ac.kr/courses/adv_pl05/papers/p261-knuth.pdf">premature optimization</a>). Now here is a compilation of all the reasons above:</p><pre>Code with side effects<br> - is fragile - leads to unexpected crashes<br> - is unexpectedly slow and is hard to optimize<br> - is hard to use concurrently<br> - is hard to read and understand<br> - it is hard to reuse<br> - is hard to debug, release and write tests</pre><p>For more discussion on the harms of side effects, please see the following great thread on <a href="https://softwareengineering.stackexchange.com/questions/15269/why-are-side-effects-considered-evil-in-functional-programming">stackexchange</a>.</p><p>Meanwhile, thank you for reading!</p><p><em>Originally published at </em><a href="https://sinavski.com/post/3_side_effects/"><em>https://sinavski.com.</em></a></p><h3>Level Up Coding</h3><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://levelup.gitconnected.com/?utm_source=pub&amp;utm_medium=post">Level Up Coding publication</a></li><li>💰 Free coding interview course ⇒ <a href="https://skilled.dev/?utm_source=luc&amp;utm_medium=article">View Course</a></li><li>🔔 Follow us: <a href="https://twitter.com/gitconnected">Twitter</a> | <a href="https://www.linkedin.com/company/gitconnected">LinkedIn</a> | <a href="https://newsletter.levelup.dev/">Newsletter</a></li></ul><p>🚀👉 <a href="https://jobs.levelup.dev/talent/welcome?referral=true"><strong>Join the Level Up talent collective and find an amazing job</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdf0743457a3" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/all-dangers-of-side-effects-for-python-coders-fdf0743457a3">All Dangers of Side Effects for Python Coders</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neat Python Interface]]></title>
            <link>https://medium.com/better-programming/neat-python-interface-e1371de63406?source=rss-b1c6a0c2d9b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1371de63406</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[interfaces]]></category>
            <category><![CDATA[interface-design]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Oleg Sinavski]]></dc:creator>
            <pubDate>Fri, 17 Feb 2023 20:13:25 GMT</pubDate>
            <atom:updated>2023-02-24T10:38:45.258Z</atom:updated>
            <content:encoded><![CDATA[<h4>Tips to avoid a bloated OOP interface</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yxVvss-0v6V86XTud71WeA.png" /><figcaption>Image by author</figcaption></figure><p>I’ve seen many Python interfaces in my career (<a href="https://www.quora.com/How-do-API-and-GUI-differ">as in API, not GUI</a>). You can quickly spot an ugly one by its size. Below are some recipes on how to make a neat one!</p><h3>Interfaces in Python</h3><p>An “interface” is a nebulous concept that might be initially hard to grasp. It is a blueprint of interactions with an object (reading a few answers <a href="https://stackoverflow.com/questions/2866987/what-is-the-definition-of-interface-in-object-oriented-programming">here</a> is not going to hurt). Do you need interfaces in Python at all? There are several good posts about it:</p><ul><li><a href="https://chelseatroy.com/2020/11/14/why-use-or-not-use-interfaces-in-python/">Chelsea Troy</a></li><li><a href="https://stackoverflow.com/questions/2124190/how-do-i-implement-interfaces-in-python">Stack Overflow</a> <em>(please read several answers!)</em></li><li><a href="https://towardsdatascience.com/python-interfaces-why-should-a-data-scientist-care-2ed7ff80f225">Diego Barba</a></li></ul><p>I’m going to assume that you read all that and decided to go with interfaces. I think Protocols are <a href="https://levelup.gitconnected.com/python-interfaces-choose-protocols-over-abc-3982e112342e">the best way to create interfaces</a> in Python. This is what we are going to use here.</p><p>Here, we are going to discuss what makes a good interface.</p><h3>An Interface Has To Be Small</h3><p>You’re designing a Zoo simulator and made an interface for “a generic animal”:</p><pre>class Animal(Protocol):<br>   def current_weight(self) -&gt; float: ...<br>   def price(self) -&gt; float: ...<br>   def sleep(self, hours): ...<br>   def eat(self, food): ...<br>   def draw(self, context): ...<br>   def look_for_food(self): ...<br>   def hunger_level(self) -&gt; float: ...<br>   def is_asleep(self) -&gt; bool: ...<br>   def is_awake(self) -&gt; bool: ...<br>   def current_position() -&gt; Tuple[float, float, float]: ...<br>   def current_orientation() -&gt; RotationMatrix: ...<br>   def current_transform3d() -&gt; HomogeneousMatrix: ...</pre><p>You should be alarmed when you see an interface with more than 4–5 methods. Chances are that it was not well-designed, and the project it is used in will be overly convoluted:</p><ul><li>This interface will be used in completely unrelated parts of the software. It will couple them together.</li><li>There will be extensive use of class hierarchies.</li><li>Interface implementations are going to be overly complicated.</li><li>It will be hard to document and confusing to use.</li></ul><p>You might think: “Clearly, animals do tons of different things, so it’s justified to have many methods.” But you can come up with ten more methods to add to a “generic animal,” and there will never be an end to it. The art is to model a complicated thing with a set of small decoupled interfaces.</p><p>What are the recipes for reducing an interface’s size?</p><h3>Semantic Duplication</h3><p>Sometimes there could be some obvious duplication methods (e.g., get_weights vs. current_weight). But more often, you’ll find methods that somewhat overlap in semantics. After a bit of thought, removing some of them is possible. In the example below:</p><pre>def current_position() -&gt; Tuple[float, float, float]:<br>    pass<br>def current_orientation() -&gt; RotationMatrix:<br>    pass<br>def current_transform3d() -&gt; HomogeneousMatrix:<br>    pass</pre><p>The user extract position and orientation from the HomogeneousMatrix returned by current_transform3d. The other two getters could be replaced by <a href="https://alexsm.com/homogeneous-transforms/">utility functions.</a></p><p>A similar situation is probably happening with the following:</p><pre>def is_asleep(self) -&gt; bool:<br>       pass<br>def is_awake(self) -&gt; bool:<br>       pass</pre><p>Is it true that animal.is_asleep() == not animal.is_awake()? If yes, then you better remove one of them.</p><h3>Decoupling Interfaces</h3><p>After removing duplicates, you should try to split the interface into independent parts. Most likely, you don’t use all Animal features in every part of your software:</p><ol><li>There could be a function that handles the drawing of animals that uses only current_transform3d and draw:</li></ol><pre>def draw_entity(animal: Animal):<br>   t = animal.current_transform3d()<br>   c = some_drawing_context(t)<br>   animal.draw(c) </pre><p>2. There could be an animal life cycle algorithm:</p><pre>def life_management(animal: Animal):<br>    eating_logic(<br>      animal.hunger_level(), animal.look_for_food(), <br>      animal.eat())<br>    sleeping_logic(animal.is_awake(), animal.sleep())</pre><p>3. And a Zoo management system:</p><pre>def purchase_decision(animal: Animal, budget: float) -&gt; bool:<br>   w = animal.current_weight()<br>   p = animal.price()<br>   decide_to_buy(p, w, budget)</pre><p>The “generic animal” is a collection of quite different interfaces that could be disentangled:</p><pre>class VisualEntity(Protocol):<br>   def current_transform3d() -&gt; HomogeneousMatrix: ...<br>   def draw(self, context): ...<br><br>class BehavingAgent:<br>   def sleep(self, hours): ...<br>   def eat(self, food): ...<br>   def look_for_food(self): ...<br>   def hunger_level(self) -&gt; float: ...<br>   def is_awake(self) -&gt; bool: ...<br><br>class ZooAsset:<br>   def current_weight(self) -&gt; float: ...<br>   def price(self) -&gt; float: ...</pre><p>Now you can see the advantages of many small interfaces over a single big one:</p><ul><li>They decouple parts of your codebase (while the large ones have a high chance of coupling unrelated parts). This in turn decouples company workflows (e.g. teams don’t break each other’s stuff too much)</li><li>You don’t have to write many implementations when instantiating a short interface. This ripples through your design because now you more easily avoid base classes and class hierarchies. Which in turn promotes composition over inheritance (there are many resources on why it’s a good idea, e.g. see <a href="https://stackoverflow.com/questions/49002/prefer-composition-over-inheritance">here</a>)</li><li>You would be able to reuse small interfaces in more places than large ones. This in turn makes the total surface area of APIs in your codebase smaller, easier to learn with simpler onboarding.</li></ul><h3>Recursive Semantics</h3><p>Next, you should be alarmed if one function calls another in the same interface. Continuing with the Zoo example:</p><pre>class BehavingAnimal(Protocol):<br>   def lifecycle(self):<br>       &quot;&quot;&quot; What happens during 24 hours period &quot;&quot;&quot;<br>   def eat(self, food):<br>       &quot;&quot;&quot; What happens when the animal consumes food &quot;&quot;&quot;<br>   def sleep(self, hours):<br>       &quot;&quot;&quot; What happens when the animal sleeps &quot;&quot;&quot;</pre><p>It is unlikely that an animal will spend a day without eating and sleeping. So there is a high chance that eat and sleep will be called from the lifecycle implementation.</p><p>Such an interface couples together two levels of abstraction. These are two interfaces in one, similar to the previous section. A “lifecycle” part is used in some global application contexts, but eat and sleep are probably used only locally inside it. Let’s split them apart:</p><pre>class ElementaryAnimal(Protocol):<br>   def eat(self, food): ...<br>   def sleep(self, hours): ...<br><br>class LifecycleManagement(Protocol):<br>   def lifecycle(self, animal: ElementaryAnimal): ...</pre><p>Notice that now lifecycle takes ElementaryAnimal as an argument. This clearly states that a lifecycle depends on something that can eat and sleep.</p><h3>Too Generic</h3><p>Yet another thing to avoid is an overly-generic interface:</p><pre>class Creature(Protocol):<br>   def act(self, *args, **kwargs) -&gt; object:<br>       &quot;&quot;&quot; Do anything you want &quot;&quot;&quot;<br><br> def perform_action(creature: Creature):<br>     custom_args = ...<br>     creature.act(*custom_args)</pre><p>An interface is like a contract between two parties. But a contract that says “do whatever” is as good as no contract. It is too easy to make such interfaces in an optional/dynamic-typed language like Python with its wildcard argument features.</p><p>Unfortunately, such interfaces do come up in some situations. We can summarize it as “passing a black box around.” Passing “blackboxes” is typically worse than explicit components. But if you have to do it, it’s better to acknowledge its “blackness” by passing a functor and moving to a more functional design (here is an <a href="https://medium.com/@olxc/switching-from-oop-to-functional-programming-4187698d4d3">in-depth tutorial</a>):</p><pre>def perform_action(actor: Callable[..., Any]):<br>    custom_args = ...<br>    actor(*custom_args)</pre><h3>Constructors in an Interface</h3><p>Related bonus topic: have you noticed that __init__ is typically not a part of any interface? Why not? It seems like it is just another method you can have.</p><p>It goes back to C++, where constructors are not virtual. Bjarne Stroustrup <a href="https://www.stroustrup.com/bs_faq2.html#virtual-ctor">gives an answer</a> on why it is so:</p><p><em>“A virtual call is a mechanism to get work done given partial information. To create an object, you need complete information. Consequently, a “call to a constructor” cannot be virtual.”</em></p><p>I like high-level reasoning more: a constructor is a function for making objects. It belongs to the “object-makers” realm. It can’t belong to an already-made object: you shouldn’t make an object out of itself (I’m not a fan of <a href="https://en.wikipedia.org/wiki/Prototype_pattern">cloning</a>).</p><p>A constructor could be viewed as a method of <a href="https://en.wikipedia.org/wiki/Factory_(object-oriented_programming">a factory</a> interface (similar ideas can be found <a href="https://stackoverflow.com/questions/5365348/can-zope-interface-define-how-a-class-init-method-should-look/5367884#5367884">here</a>). A constructor and an object interface are yet another example of two coupled levels of abstraction.</p><h3>Conclusion</h3><p>To summarize, a good interface should:</p><ul><li>be small and concise</li><li>avoid semantic duplication</li><li>avoid coupling several parts in one</li><li>avoid mixing multiple abstraction levels</li><li>be specific and not too generic</li><li>not include __init__</li></ul><p>At the same time, real life has many exceptions and complications. But those must be justified exceptions, not excuses to keep messy interfaces around.</p><p>If you found this article helpful, stay tuned for more. Thank you for reading!</p><h3>Resources</h3><ul><li>Why use interfaces in Python: <a href="https://chelseatroy.com/2020/11/14/why-use-or-not-use-interfaces-in-python/">Chelsea Troy</a>, <a href="https://stackoverflow.com/questions/2124190/how-do-i-implement-interfaces-in-python">Stack Overflow</a>, <a href="https://towardsdatascience.com/python-interfaces-why-should-a-data-scientist-care-2ed7ff80f225">Diego Barba</a></li><li><a href="https://levelup.gitconnected.com/python-interfaces-choose-protocols-over-abc-3982e112342e">Why use Protocols instead of ABC</a></li><li><a href="https://medium.com/@olxc/switching-from-oop-to-functional-programming-4187698d4d3">How to switch to a functional paradigm from OOP</a></li><li><a href="https://stackoverflow.com/questions/5365348/can-zope-interface-define-how-a-class-init-method-should-look/5367884#5367884">Discussion on constructors and factories in Python</a></li></ul><pre><strong>Want to Connect?</strong><br><br>You can find me on <a href="https://www.linkedin.com/in/oleg-sinavski/">LinkedIn</a> or <a href="https://twitter.com/SinavskiOleg">Twitter</a>.</pre><p><em>Originally published at </em><a href="https://sinavski.com/post/2_good_interface/"><em>https://sinavski.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1371de63406" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/neat-python-interface-e1371de63406">Neat Python Interface</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Interfaces: Choose Protocols Over ABC]]></title>
            <link>https://levelup.gitconnected.com/python-interfaces-choose-protocols-over-abc-3982e112342e?source=rss-b1c6a0c2d9b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3982e112342e</guid>
            <category><![CDATA[interface-design]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[interfaces]]></category>
            <dc:creator><![CDATA[Oleg Sinavski]]></dc:creator>
            <pubDate>Thu, 19 Jan 2023 14:38:26 GMT</pubDate>
            <atom:updated>2023-01-19T14:38:26.602Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python interfaces: abandon ABC and switch to Protocols</h3><h4>Several practical reasons to prefer new shiny Protocols</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OigNoAIlGNKT7ZfyQ5L7UQ.jpeg" /><figcaption>Beautiful interfaces will pay off in the future — Credit <a href="https://unsplash.com/@barkiple?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">John Barkiple</a> on <a href="https://unsplash.com/photos/l090uFWoPaI?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>I used a standard Python library abc to define interfaces for the last 10 years of my career. But recently, I found that relatively new Python <a href="https://peps.python.org/pep-0544/">Protocols</a> are way nicer. People find uses for both technologies. But I want to convince you to completely jump ships and start using them instead of more traditional techniques.</p><h3>Python interfaces: what can you use?</h3><p>Python is somewhat different from other popular languages since there are no interfaces on a language level. But there are several library implementations. The abc package is probably the most popular:</p><pre>from abc import ABC, abstractmethod<br><br>class Animal(ABC):<br>   @abstractmethod<br>   def eat(self, food) -&gt; float:<br>       pass<br>   @abstractmethod<br>   def sleep(self, hours) -&gt; float:<br>       pass</pre><p>Next, the most frequently mentioned package seems to be <a href="https://zopeinterface.readthedocs.io/en/latest/README.html#declaring-interfaces">Zope</a>:</p><pre>from zope.interface import Interface<br><br>class Animal(Interface):<br>   def eat(self, food) -&gt; float:<br>       pass<br>   def sleep(self, hours) -&gt; float:<br>       pass</pre><p>Zope is a web-related library, and its interfaces have a lot of advanced features. Also, there are more custom packages and tutorials on the web on how to make an interface system yourself (<a href="https://realpython.com/python-interface/">see example</a>).</p><p>Finally, there are protocols:</p><pre>from typing import Protocol<br><br>class Animal(Protocol):<br>   def eat(self, food) -&gt; float:<br>       ...<br>   def sleep(self, hours) -&gt; float:<br>       ...</pre><p>A protocol is a formalization of Python’s <a href="https://en.wikipedia.org/wiki/Duck_typing">“duck-typing”</a> ideology. There are many great articles on structural typing in Python (for example, see <a href="https://trm.io/2017/01/29/structural-subtyping-python">this tutorial</a>). Protocols and interfaces are different beasts in theory, but a protocol does the job. I had great success replacing abc with Protocols without any downsides.</p><h3>What should you use?</h3><p>You should know that an interface system will not be localized to a small part of your codebase. After you choose to go with one, you’ll see it everywhere, and it’s going to be hard to change in the future. So I would immediately dismiss any custom implementations or Zope. It’s an extra dependency you have to deal with forever: installation, versions, support, and so on. For example, you have to install a <a href="https://github.com/Shoobx/mypy-zope#plugin-for-mypy-to-support-zopeinterface">Mypy plugin</a> to support a zope.interface well. Additionally, a new developer in the team might not know this custom package, and you&#39;ll have to explain what it is and why you chose it.</p><p>The main battle will happen between abc and Protocols. But if you really want a zope vs Protocols battle, please read <a href="https://glyph.twistedmatrix.com/2021/03/interfaces-and-protocols.html#runtime-support">this detailed analysis</a>.</p><h3>Static checking first</h3><p>The big assumption I’m going to make is that you’re already convinced that static checking is a <em>must</em>: you are not going to run the code that fails pylint/mypy. Both checkers support abc and Protocols equally well. Also, just know that both abc and Protocols allow runtime checking, in case you need it.</p><h3>Both support explicit syntax</h3><p>First, note that you still can explicitly inherit from an abc <em>and</em> a Protocol. Many arguments in a <a href="https://www.youtube.com/watch?v=xvb5hGLoK0A">very good video</a> (and comments) from <a href="https://www.arjancodes.com/">Arjan</a> revolve around the misconception that you can’t do that with protocols. You totally can:</p><pre>class Giraffe(Animal):<br>   ...</pre><p>So in that regard, abc and protocols could be used the same way. But Protocols give you an extra degree of design freedom by default. You can avoid explicit inheritance but still enjoy full interface checking:</p><pre>class Giraffe:  # no base class needed!<br>   def eat(self, food) -&gt; float:<br>       return 0.<br> <br>   def sleep(self, hours) -&gt; float:<br>       return 1.<br><br>def feed_animal(animal: Animal):<br>    ... <br><br>giraffe = Giraffe()<br>feed_animal(giraffe)</pre><p>This allows you to make an interface for the code you don’t control and loosen the dependencies between modules in your codebase. Whether to choose an implicit or explicit option is a subtle choice decided on a case-by-case basis. A good example in favour of explicit “opt-in” for an interface is described <a href="https://glyph.twistedmatrix.com/2021/03/interfaces-and-protocols.html#where-interfaces-work-best-hidden-and-complex-meanings">here</a>. Protocols do not <em>force</em> you to opt-in. But you can still establish a company-wide agreement to always explicitly inherit from any protocol.</p><p>abc also support implicit interfaces through the concept of “<a href="https://docs.python.org/3/library/abc.html#abc.ABCMeta.register">virtual subclasses</a>” . But you have to call register for every implementation:</p><pre>class Giraffe:  # no base class needed!<br>   def eat(self, food) -&gt; float:<br>       return 0.<br><br>class Animal(ABC):<br>    ...<br><br>Animal.register(Giraffe)  # achieves the same as implicit Protocol</pre><p>Procotol supports implicit and explicit variants without extra syntax and works with mypy. Also, <a href="https://github.com/python/mypy/issues/2922">mypy does not support </a><a href="https://github.com/python/mypy/issues/2922">register</a> as of the end of 2022. I&#39;m not sure if we can fully count that in favour of abc.</p><p>Protocols allow you to define an interface for a <em>function</em> (not only a class) — see <a href="https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols">callback protocols</a>. It is a very cool feature that is worthy of a separate post.</p><h3>Both support default methods (unfortunately)</h3><p>There is a big downside to both abc and Protocols. In the real world, many people work in a single codebase. Abstract base classes sometimes tend to acquire default method implementations. This is what it might look like:</p><pre>class Animal(Protocol):  # the same holds for Animal(ABC):<br>   def eat(self, food) -&gt; float:<br>       ...  # this is still abstract<br> <br>   def sleep(self, hours) -&gt; float:<br>       return 3.</pre><p>In that case, they stop being “abstract” and become just base classes. Python and static checkers do not catch that. A software design with inheritance is not really the same as a design with interfaces. I would love Python to separate them on a language level, but it is unlikely to happen. Implicit protocols have an advantage here since they allow you to avoid messy inheritance altogether.</p><h3>Protocols are shorter</h3><p>Last but not least, you can count the number of lines of code you need to define an interface. With abc, you must have an abstractmethod decorator for every method. But with Protocols without runtime checking, you don&#39;t have to use any decorators at all. So here, Protocols win hands down.</p><h3>Conclusion</h3><p>Let’s add up the scores:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Em8M0KwmDHXsliHOaBwJ7Q.png" /></figure><p>Hopefully, I’m not missing anything huge in this analysis. Looking at the results, team <strong>“Protocols”</strong> wins, and you probably should just start using it!</p><p>If you found this article helpful, give me some claps. Thank you for reading!</p><p>You can find me on <a href="https://www.linkedin.com/in/oleg-sinavski/">LinkedIn</a> or <a href="https://twitter.com/SinavskiOleg">Twitter</a>.</p><h4><strong>Resources</strong></h4><ul><li><a href="https://en.wikipedia.org/wiki/Duck_typing">Duck-typing</a> wiki</li><li><a href="https://trm.io/2017/01/29/structural-subtyping-python">Good duck-typing tutorial</a> from Taylor Manderson</li><li><a href="https://glyph.twistedmatrix.com/2021/03/interfaces-and-protocols.html#runtime-support">A similar comparison between Zope and Protocols</a> from Glyph</li><li><a href="https://www.youtube.com/watch?v=xvb5hGLoK0A">ABC vs Protocols comparison</a> from Arjan</li><li><a href="https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols">Functional interfaces with callback protocols</a></li></ul><p><em>Originally published at </em><a href="https://sinavski.com/post/1_abc_vs_protocols/"><em>https://sinavski.com</em></a><em>.</em></p><h3>Level Up Coding</h3><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://levelup.gitconnected.com/?utm_source=pub&amp;utm_medium=post">Level Up Coding publication</a></li><li>🔔 Follow us: <a href="https://twitter.com/gitconnected">Twitter</a> | <a href="https://www.linkedin.com/company/gitconnected">LinkedIn</a> | <a href="https://newsletter.levelup.dev/">Newsletter</a></li></ul><p>🚀👉 <a href="https://jobs.levelup.dev/talent/welcome?referral=true"><strong>Join the Level Up talent collective and find an amazing job</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3982e112342e" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/python-interfaces-choose-protocols-over-abc-3982e112342e">Python Interfaces: Choose Protocols Over ABC</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>