<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Krupa Hebbar on Medium]]></title>
        <description><![CDATA[Stories by Krupa Hebbar on Medium]]></description>
        <link>https://medium.com/@krupahebbar?source=rss-e20777954378------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NTLFnjv4DHjDlLjkz9anBQ.jpeg</url>
            <title>Stories by Krupa Hebbar on Medium</title>
            <link>https://medium.com/@krupahebbar?source=rss-e20777954378------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:10:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@krupahebbar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MongoDB $lookup — the favours earned and lessons learned]]></title>
            <link>https://medium.com/@krupahebbar/mongodb-lookup-the-favours-earned-and-lessons-learned-b96c071d8ad9?source=rss-e20777954378------2</link>
            <guid isPermaLink="false">https://medium.com/p/b96c071d8ad9</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[database-design]]></category>
            <category><![CDATA[query-optimization]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[Krupa Hebbar]]></dc:creator>
            <pubDate>Tue, 03 Sep 2024 12:58:35 GMT</pubDate>
            <atom:updated>2024-09-03T16:16:01.857Z</atom:updated>
            <content:encoded><![CDATA[<h3>MongoDB $lookup — favours earned and lessons learned</h3><p>Has your Google search history ever looked like this?</p><blockquote><em>Joins with MongoDB<br>Why are my MongoDB joins slow?<br>Improve performance of $lookup in MongoDB</em></blockquote><p>I’ve been there, too, and judging by the search results — so have many others. What do you do when you have millions of documents already on production with 3N normalization on a relational database and your joins are getting slower by the day? Do you throw the whole DB away and start from scratch with some new database? Do you create views only to realize they aren’t updating fast enough? Do you write a custom ETL pipeline from scratch to do your joins in application code because MongoDB documentation says $lookup can be slow only to realize that your application now needs 20X the memory and compute that it did before? (Look, we didn’t <em>want</em> to do it that way, I swear.)</p><p>Let me tell you a story of my adventures and misadventures with the MongoDB $lookup operator, and the tentative peace treaty we have since negotiated with each other.</p><figure><img alt="image depicting a simplistic approach getting into a tangled web of complexity and emerging as elegant simplicity" src="https://cdn-images-1.medium.com/max/1024/1*GAD9WTkuMwkCbSk-jcm4Uw.jpeg" /><figcaption>A pictorial representation of this whole endeavour</figcaption></figure><h4>Table of Contents</h4><p>· <a href="#0839">Backstory</a><br>· <a href="#1a87">The Good</a><br>· <a href="#def1">The Lessons</a><br> ∘ <a href="#e5b1">Lesson 1: Where you do your grouping matters</a><br> ∘ <a href="#4d3c">Lesson 2: You do not know which indexes $lookup will use, don’t pretend otherwise</a><br> ∘ <a href="#0958">Lesson 3: Don’t skip the MongoDB basics</a><br>· <a href="#5ffc">In conclusion</a></p><h3>Backstory</h3><p>I’ll keep this part short. We recently migrated from MySQL to MongoDB to address a steady increase in the number of columns in our 3NF tables. We have a collection in MongoDB that houses a large volume of client data. This data is semi-structured and could largely be grouped into documents with 4–5 schemas.</p><p>We needed to show this data on a dashboard with the user retaining all the flexibility of what columns to read and which data to join. The dashboard had to reflect changes in real-time and be easily navigable.</p><p>After a year of trying to optimize an application-level ETL pipeline from when this data was separated into multiple collections, I gave up on increasingly unhinged code optimization activities and decided to migrate all the data to a single denormalized collection and use MongoDB’s $lookup operator and evaluate the performance myself.</p><h3>The Good</h3><p>To my surprise, it worked. Our query times dropped from 30–40 seconds to under a second. The resources required by the application dropped by nearly 95% and our existing MongoDB infrastructure could handle the $lookup pipelines with no additional scaling.</p><p>Big Win.</p><p>Of course, we prepared for this by adding the appropriate indexes since the query patterns were known.</p><p>It worked so well that we tested and retested for weeks and finally concluded that all our edge cases were accounted for and the performance boost was <em>real</em>.</p><p>What we did right:</p><ul><li>We denormalized our data, adding a single “document_type” key to indicate the semi-structured schema that each document would follow. This schema was validated and enforced at the application layer.</li><li>We moved all the related data into a single collection to better leverage the kinds of queries MongoDB is good at. Bonus — this allows us to easily shard this collection when we grow to a scale that needs it.</li><li>We adapted our indexes based on <a href="https://www.mongodb.com/blog/post/performance-best-practices-indexing">MongoDB’s Best Practices documentation</a> which is an excellent resource.</li><li>We experimented with multiple versions of aggregation pipelines for different query patterns and compared actual performance numbers.</li><li>We built an application-layer query builder that built the most optimized pipeline for our use-case, allowing us to ensure all queries were covered and optimized, and ensured all internal DB access went through this application layer.</li></ul><h3>The Lessons</h3><h4>Lesson 1: Where you do your grouping matters</h4><p>We made some minor adjustments and pushed to staging, and suddenly, performance dropped. We saw queries that were expected to take 300ms taking 100X longer. It was somehow <em>even slower</em> than our custom ETL pipelines. Think 14 seconds -&gt; 68 seconds.</p><p>Our original, tested, beautiful favourite-child query looked something like this, with one match query, multiple $lookup stages, followed by a projection:</p><pre>[<br>  {<br>    &quot;$match&quot;: {<br>      &quot;document_type&quot;: &quot;schema 1&quot;,<br>      &quot;client_id&quot;: &quot;client 1&quot;<br>    }<br>  },<br>  {<br>    &quot;$lookup&quot;: {<br>      &quot;$from&quot;: &quot;MyCollection&quot;,<br>      &quot;localField&quot;: &quot;product_id&quot;,<br>      &quot;foreignField&quot;: &quot;product_id&quot;,<br>      &quot;as&quot;: &quot;schema 2&quot;,<br>      &quot;$pipeline&quot;: [<br>        {<br>          &quot;$match&quot;: {<br>            &quot;document_type&quot;: &quot;schema 2&quot;,<br>            &quot;client_id&quot;: &quot;client 1&quot;<br>          }<br>        }<br>      ]<br>    }<br>  },<br>  {<br>    &quot;$lookup&quot;: {<br>      &quot;$from&quot;: &quot;MyCollection&quot;,<br>      &quot;localField&quot;: &quot;product_id&quot;,<br>      &quot;foreignField&quot;: &quot;product_id&quot;,<br>      &quot;as&quot;: &quot;schema 3&quot;,<br>      &quot;$pipeline&quot;: [<br>        {<br>          &quot;$match&quot;: {<br>            &quot;document_type&quot;: &quot;schema 3&quot;,<br>            &quot;client_id&quot;: &quot;client 1&quot;<br>          }<br>        }<br>      ]<br>    }<br>  },<br>  {<br>    &quot;$project&quot;: {<br>      &quot;fields&quot;: 1,<br>      &quot;schema 2.user&quot;: 1,<br>      &quot;schema 3.selects&quot;: 1<br>    }<br>  }<br>]</pre><p>Our new query just added a little grouping stage right at the end to “clean up” the documents and group related ones together:</p><pre>{&quot;$group&quot;: {&quot;_id&quot;: &quot;parent_id&quot;, &quot;docs&quot;: {&quot;$push&quot;: &quot;$$ROOT&quot;}}}</pre><p>Seemed completely harmless, at the time.</p><blockquote>Dear Reader, <br>Beware the feature request which needs an innocuous one-line code change. <br>That one line is always more dangerous than you realize.</blockquote><p>We added this $group stage <em>after</em> evaluating the pipeline performance. Suddenly, to load page zero of the dashboard, MongoDB went from having to scan 50 documents to scanning over 100,000 documents. The modified query took 68 seconds to run on 100K documents!</p><p>I eventually found out that to do the grouping at the end, MongoDB had to do all the lookups across the entire matching result set, meaning 100K*3 lookups instead of just looking up to fill the first page of documents. We were essentially getting worst-case performance in our best-case scenario (page 1 load time).</p><p>Our solution was to move the grouping stage <em>before</em> all the lookups. The outcome? For a client having 100,000 records, to load page 1:</p><ul><li>the old custom ETL pipeline: 14 seconds</li><li>with grouping after $lookups: 68 seconds</li><li><strong>with grouping <em>before</em> $lookups: 1.2 seconds</strong></li></ul><p>This is with the same indexes, the same database size, and the same infrastructure. The only difference was where the $group logic was placed. Hence the title of this section — where you do your grouping <em>matters.</em></p><h4>Lesson 2: You do not know which indexes $lookup will use, don’t pretend otherwise</h4><p>That’s it. That’s the lesson. We achieved significant optimizations by just running the aggregations with an $explain command and checking whether every stage was using the indexes designed for it:</p><pre>debug = db.command(&#39;explain&#39;, {<br>        &#39;aggregate&#39;: collection_name,<br>        &#39;pipeline&#39;: q,<br>        &#39;cursor&#39;: {}<br>    }, verbosity=&#39;executionStats&#39;)</pre><p>We saw many cases where the $lookup stage was completely ignoring the indexes we had designed for these specific query patterns and was instead using a suboptimal index or worse — not using any index at all and doing COLLSCANs. The optimizations we were able to make by combing through the execution stats:</p><ol><li><strong>Using localField and foreignField:</strong> Index use is much more reliable when using localField and foreignField (which are both indexed) compared to using expr stages in the lookup $pipeline.</li><li><strong>Reduce the number of $lookup documents:</strong> Using a $pipeline with a $match inside the $lookup drastically reduced the candidate set for our joins, thereby improving performance exponentially with our multiple $lookup stages (after ensuring the common $match patterns inside the $lookup were also covered by indexes)</li><li><strong>Skipping $unwind where we could:</strong> In places where we needed an outer join anyway, we saw better performance by skipping the $unwind stage in the query and handling the nested data in our application code. (Note: your mileage may vary on this one, this could be specific to our data model)</li><li><strong>Index ordering matters:</strong> In several cases, changing the order of indexes and removing unnecessary ones (eg indexes which were fully covered by other existing indexes) also reduced the number of “bad” indexes used by Mongo’s query planner.</li></ol><h4>Lesson 3: Don’t skip the MongoDB basics</h4><p>If you are using MongoDB on production for the first time, there are some basics that you need to have covered before you even think of query optimization. The best practices are pretty well covered across MongoDB’s excellent documentation, but <a href="https://www.mongodb.com/resources/products/capabilities/performance-best-practices">here is one to get you started.</a> In short — invest time in data modelling and planning your infrastructure and available memory for your specific data volume, you’ll regret it if you don’t. There’s very little query optimization can do when your data model itself is counter to what MongoDB is designed for. Avoid the uphill battle and design for the database you are using.</p><h3>In conclusion</h3><p>Dedicating a few weeks to this exercise helped me remember something critical — don’t take the abstractions for granted. Sure, the ease-of-use is great, but digging deeper into exactly <em>how</em> MongoDB’s query planner and index planner work led to huge payoffs with minimal cost, and spending a few weeks designing our data model around MongoDB’s most performant query patterns led to big wins. We were able to serve queries several orders of magnitude faster with no additional database infrastructure, no learning curve of new database technologies and query languages, and fewer server resources for our application servers, landing us in the idyllic scenario of saving both time and money — for ourselves and our users.</p><p>Granted, $lookups are still not optimal and our worst-case query performance has not improved nearly as much as our average case, but that’s a problem for another time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b96c071d8ad9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conquering the spectre of “intermittent issues” in Python applications — debugging strategies for…]]></title>
            <link>https://blog.stackademic.com/conquering-the-spectre-of-intermittent-issues-in-python-applications-31bbfe25cc32?source=rss-e20777954378------2</link>
            <guid isPermaLink="false">https://medium.com/p/31bbfe25cc32</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[profiling]]></category>
            <category><![CDATA[debugging]]></category>
            <dc:creator><![CDATA[Krupa Hebbar]]></dc:creator>
            <pubDate>Sun, 21 Jul 2024 08:57:38 GMT</pubDate>
            <atom:updated>2024-08-05T03:46:43.284Z</atom:updated>
            <content:encoded><![CDATA[<h3>Conquering the spectre of “intermittent issues” in Python applications — debugging strategies for when you don’t know where to start</h3><p>If you use Python on production systems at non-trivial scale, you’ve likely spent hours or days debugging problems that happen only on production even though “the code is fine”. This is what I call the spectre of “intermittent production issues” and “it works on my machine!”. At this point, you’re also (most likely) in too deep to rewrite your entire codebase in a more production-friendly language, so what can you do?</p><p>From my experience, once a codebase is well-tested and on production, most hard-to-debug intermittent issues boil down to one of three things:</p><ol><li><strong>Resource-related issues</strong> (eg. process hangs/is killed by OOM killer, high CPU load, timeouts due to long iowait times, downtime in parts of the application, downtime in external resources)</li><li><strong>Network or cloud provider issues </strong>(eg. DNS/IP changes not updated in firewalls, session timeouts, cloud provider issues unrelated to you, issues related to autoscaling latency during high traffic)</li><li><strong>Edge-cases in code</strong> (eg. exception handling code that is rarely encountered, rare paths in highly branched code, unhandled exceptions, false state assumptions)</li><li><strong>Library versions</strong> — no special tools here. Just please save yourself hours and check if the Python and library versions on your machine and production are the same. That’s all. Nothing fancy. I will never admit how many hours of debugging have led to version differences being the root cause of failed deployments early in my career.</li></ol><p>It’s often difficult to know where to even start debugging issues like this so in this post, I will be covering some methodologies to address each of these points. The tools I cover here are not going to solve all your issues, of course, but they will hopefully make the whole problem more approachable and feel more achievable and give you a framework to start thinking about what causes intermittent issues and how you can solve them. Think of them as silver bullets — finding the target and having good aim is still on you.</p><p><strong><em>Please note: I will not be providing detailed step-by-step instructions on using these libraries in this article as this has already been done well by others — I have linked some of these resources in the Recommended Reading section below and many others are just one search away. In this article, I will be covering some tools that I have personally found most useful to profile, test and debug resource-related and intermittent issues in my own Python APIs.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3aVPdbAknruFhAKyzgk9CA.jpeg" /><figcaption>Edited lyrics from Disney’s Mulan showing profiling and monitoring tools as the tools of a warrior</figcaption></figure><h3>Table of Contents</h3><p>· <a href="#1dcd">Profiling resource issues</a><br> ∘ <a href="#e842">Scalene</a><br> ∘ <a href="#6275">Locust</a><br> ∘ <a href="#0801">What next?</a><br>· <a href="#0594">Identifying network issues and edge cases: OpenTelemetry with Signoz</a><br> ∘ <a href="#133e">How does it help?</a><br> ∘ <a href="#1cbe">What OpenTelementry cannot tell you</a><br>· <a href="#9a46">Recommended reading</a></p><h3>Profiling resource issues</h3><p>If you haven’t encountered the libraries <a href="https://pypi.org/project/locust/">locust</a> and <a href="https://pypi.org/project/scalene/">scalene</a>, I have great news for you. Profiling your Python applications for CPU bottlenecks, memory leaks, high GPU usage, and all of these with individual requests vs at high loads has never been easier.</p><h4><a href="https://pypi.org/project/scalene/">Scalene</a></h4><p><a href="https://pypi.org/project/scalene/">Scalene</a> is a fast and accurate profiler for CPU, GPU and memory profiling of your Python application. Running it is as simple as replacing python with scalene in your run command:</p><pre>scalene app/main.py</pre><p>All you have to do is run your script or API server using the scalene library and run through the scenarios you want to profile. Hit the APIs you want to hit with the payloads you want to profile for and run through the common and edge cases. Once you stop the application or script, scalene provides a super convenient and detailed GUI interface that shows all the parts of your code that took the most time or the most memory, and even flags potential memory leaks or unreleased resources.</p><p>Scalene offers a sample GUI <a href="https://plasma-umass.org/scalene-gui/">here</a> that you can explore if you’re unsure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tbYnff-Ner6T8nKIp64gLg.png" /><figcaption>Screenshot from <a href="https://plasma-umass.org/scalene-gui/">https://plasma-umass.org/scalene-gui/</a> of a sample profiling page</figcaption></figure><p>The ease of use along with being able to profile CPU, memory, data copies and memory leaks all at once with low overhead and high accuracy makes scalene the best possible tool to get fast results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/0*joMeBSVlGDT0hH5S" /><figcaption>Comparison of scalene to other popular profilers, from <a href="https://pypi.org/project/scalene/">https://pypi.org/project/scalene/</a></figcaption></figure><p>Now, at this stage, you’ll likely catch the biggest culprits — unused library imports, inefficient data transformations, unnecessary data copies leading to higher memory use, potential memory leaks from unreleased resources, IO bottlenecks that could be removed with higher concurrency/<a href="https://docs.python.org/3/library/asyncio.html">asyncio</a>. These should be easy, low-risk high-reward fixes (I highly recommend having good logical coverage on your unit tests to validate that logic changes didn’t break your code — see <a href="https://medium.com/@krupahebbar/pytest-with-python-from-beginner-to-expert-aeb1152c2981">my post on unit testing</a> if this is something you’re new to).</p><p><strong>Limitations of Scalene</strong></p><ul><li>Running scalene on multi-worker apps (like gunicorn with FastAPI) is not very straightforward — some possible approaches are on <a href="https://github.com/plasma-umass/scalene/issues/35">this thread</a> if this is relevant to you. I have personally only debugged single-worker performance bottlenecks as that covers a vast majority of cases for a simple application with no shared memory/shared resources between workers.</li><li>GPU profiling currently works only on NVIDIA-based systems. If you know of alternatives, please leave them in the comments!</li></ul><p>Once you’ve made progress here, it’s time to move on to beast-mode testing, cue…</p><h4><a href="https://github.com/locustio/locust">Locust</a></h4><p><a href="https://github.com/locustio/locust">Locust</a> is a load-testing tool that’s so easy to use that it makes you question why you don’t load-test everything yourself in the first place. It lets you write load tests as regular Python code and then execute with as many concurrent users for as many seconds/minutes/hours as you want. You can be your own QA engineer and ensure the code you ship is in great shape before it ever leaves your hands.</p><p>Locust provides an extremely user-friendly browser GUI to monitor the testing in real-time, and CLI support to run load tests remotely and with a significantly lower footprint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2oJxu0vs6AoJcPsI.png" /><figcaption>load test stats on the Locust browser interface for 50 concurrent users (from <a href="https://docs.locust.io/en/stable/quickstart.html">https://docs.locust.io/en/stable/quickstart.html</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4lvoplpthmCsppkP8H0B5g.png" /><figcaption>Charts displayed on the Locust browser GUI during test runs (from <a href="https://docs.locust.io/en/stable/quickstart.html">https://docs.locust.io/en/stable/quickstart.html</a>)</figcaption></figure><p>I have linked a step-by-step guide to using locust in the Recommended Reading section below. TLDR version:</p><ol><li>Write a locustfile.py that describes your load-testing tasks/users.</li><li>Run the locust command from the directory containing the locustfile.</li><li>On the GUI (typically <a href="http://localhost:8080/">http://localhost:8089/</a> on your browser) put in the host, number of concurrent users, ramp-up rate, and optionally the duration of the run.</li></ol><p>For the scope of this article, my recommendation is:</p><ol><li>Write your locustfile to include the same tasks that you did on your manual run — the individual API calls, a sequence of operations, whatever it may be.</li><li>Run your API or application with scalene app/main.py (or if you are testing a standalone script/function, <a href="https://pypi.org/project/scalene/#:~:text=Using%20Scalene%20programmatically%20in%20your%20code%3A">explicitly use the scalene profiler in your code</a>).</li><li>Run locust and start your load test.</li><li>Once your load test runs for 30s/1m/2m/30m/1 hour for 1/50/100/1000/10000 concurrent users (based on your use case), stop locust and scalene. (<em>Note: these are reference points as a place to start — exact time and concurrency are heavily dependent on your application’s specs and that is best left to your own judgement)</em></li></ol><p>When you do this, you may see some new insights on the scalene GUI related to shared memory, unreleased memory leading to memory spikes, higher CPU usage in certain statements at high concurrency, and other issues that may happen only at higher load/higher concurrency that may not happen with a single user.</p><h4>What next?</h4><p>Once you feel comfortable using scalene and locust locally with the GUI, you can also explore their CLIs and run these remotely to stress-test under much more extreme situations or on remote servers with higher capacity.</p><h3>Identifying network issues and edge cases: OpenTelemetry with <a href="https://signoz.io/docs/instrumentation/python/">SigNoz</a></h3><p>We finally arrive at the promised silver bullet for those pesky intermittent issues that happen only on production. I have good news and bad news, and they are both that with telemetry, you can never use “it works on my machine” as an excuse ever again.</p><p>(From their <a href="https://opentelemetry.io/docs/">docs</a>) <strong>OpenTelemetry</strong>, also known as OTel, is a vendor-neutral open source <a href="https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability">Observability</a> framework for instrumenting, generating, collecting, and exporting telemetry data such as <a href="https://opentelemetry.io/docs/concepts/signals/traces/">traces</a>, <a href="https://opentelemetry.io/docs/concepts/signals/metrics/">metrics</a>, and <a href="https://opentelemetry.io/docs/concepts/signals/logs/">logs</a>.</p><p><a href="https://signoz.io/docs/instrumentation/python/"><strong>SigNoz</strong></a> is an open-source application monitoring tool that uses OpenTelemetry to give you traces of every request that your application serves in real-time. The dashboard offers real-time latency metrics as well as HTTP status codes and relationships between different monitored microservices used over the course of a request. If you’re new to telemetry and monitoring, I have included some recommended reading below.</p><p><strong><em>Note 1: the points below also apply to any other APM tool (Application Performance Monitoring tool) like DataDog, NewRelic, etc.</em></strong></p><p><strong><em>Note 2: Setting up SigNoz is a non-trivial task but is well worth the investment — if self-hosted, you will require separate infrastructure with sufficient storage and bandwidth on which SigNoz will be installed and deployed, and uptime will have to be ensured for reliability. Set up of SigNoz is outside the scope of this article, but their </em></strong><a href="https://signoz.io/docs/install/"><strong><em>docs on this are well-written</em></strong></a><strong><em> and should be enough to get you up and running.</em></strong></p><p>Assuming we are still talking about Python APIs running on production, a <a href="https://signoz.io/docs/instrumentation/python/#send-traces-to-signoz-cloud">2-line configuration</a> is all you need to get your application sending live traces to SigNoz.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YuQ_sQYgcwwr7VrxZO3hrA.png" /></figure><p>The auto-instrumentation works very well and you can drill down all the way to DB-level metrics very easily. They have a free community-edition self-hosted tier and cloud tiers starting at $199/month.</p><h4>How does it help?</h4><p>When you have reports of intermittent production issues coming in, all you need is the timestamp, and you can go into signoz and look at the requests being served around that timestamp. You can filter by APIs that returned non-200 responses and see exactly what error or exception occurred at that point in time — whether the application encountered network issues and was unable to connect to some external resource, or it encountered an edge-case exception in the code, or it timed out, your RCA will be rock-solid with telemetry evidence.</p><p>If you do the extra steps to also track application logs in SigNoz, you can also associate each trace with the corresponding application logs for an even richer debugging experience.</p><h4>What OpenTelementry cannot tell you</h4><p>It is not impossible to encounter intermittent issues on production that don’t even show up on your telemetry. In fact, if you are scaling fast, chances are this happens quite often. The fundamental reason is that telemetry is directly attached to your application code, so if there are issues in either</p><ol><li>The connection between your application and signoz, leading to lost traces/missing data, or</li><li>Some requests never reached your application due to issues in networks, load balancers, or simply due to your application being overloaded or the application being down,</li></ol><p>You won’t see these on a tool like Signoz. You will, however, know where to direct your debugging energy when you encounter these scenarios, which could save you hours of hunting.</p><h3>Recommended reading</h3><ol><li><a href="https://medium.com/nerd-for-tech/load-testing-using-locust-io-f3e6e247c74e">https://medium.com/nerd-for-tech/load-testing-using-locust-io-f3e6e247c74e</a> is a great resource for a step-by-step tutorial on how to use locust.io</li><li>The scalene GitHub page (<a href="https://github.com/plasma-umass/scalene">https://github.com/plasma-umass/scalene</a>) has great step-by-step instructions as well</li><li>If you are new to telemetry and monitoring, I recommend reading: <a href="https://opentelemetry.io/docs/what-is-opentelemetry/">https://opentelemetry.io/docs/what-is-opentelemetry/</a></li><li>Signoz also has great documentation: <a href="https://opentelemetry.io/docs/what-is-opentelemetry/">https://signoz.io/docs/introduction/</a></li></ol><h3>Stackademic 🎓</h3><p>Thank you for reading until the end. Before you go:</p><ul><li>Please consider <strong>clapping</strong> and <strong>following</strong> the writer! 👏</li><li>Follow us <a href="https://twitter.com/stackademichq"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/stackademic"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/c/stackademic"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a></li><li>Visit our other platforms: <a href="https://plainenglish.io/"><strong>In Plain English</strong></a> | <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://stackademic.com/"><strong>Stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31bbfe25cc32" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/conquering-the-spectre-of-intermittent-issues-in-python-applications-31bbfe25cc32">Conquering the spectre of “intermittent issues” in Python applications — debugging strategies for…</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pytest with Python — from beginner to expert]]></title>
            <link>https://blog.stackademic.com/pytest-with-python-from-beginner-to-expert-aeb1152c2981?source=rss-e20777954378------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeb1152c2981</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[unit-testing]]></category>
            <dc:creator><![CDATA[Krupa Hebbar]]></dc:creator>
            <pubDate>Wed, 20 Mar 2024 11:34:12 GMT</pubDate>
            <atom:updated>2024-08-07T00:08:40.543Z</atom:updated>
            <content:encoded><![CDATA[<h3>Pytest with Python — from beginner to expert in no time flat</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mIjuBCjpv6X4Ae1b" /><figcaption>Photo by <a href="https://unsplash.com/@cdr6934?utm_source=medium&amp;utm_medium=referral">Chris Ried</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Note: This article was edited on 26 March 2024 to include more reading materials.</em></p><p>If you are looking for an end-to-end guide for how to use pytest in real-world Python codebases, look no further.</p><p>Before we get started — pytest’s documentation is great but can be slightly hard to navigate at first. I highly recommend going through the <a href="https://docs.pytest.org/en/8.0.x/getting-started.html">Getting Started</a> page at least. Go ahead, this post will still be here when you get back.</p><p><em>Note: You can find runnable code with versions of all the following examples on </em><a href="https://github.com/KrupaH/pytest_examples"><em>GitHub</em></a><em>.</em></p><h3>Contents:</h3><p>· <a href="#9fc1">ABCs of testing with Python</a><br>· <a href="#bdfc">Basic unit tests</a><br>· <a href="#9b10">Testing exceptions</a><br>· <a href="#7abb">Built-in Fixtures: Capturing logs and output in tests</a><br>· <a href="#9665">Built-in Fixtures: Higher, Further, Faster</a><br>· <a href="#ef99">Built-in Fixtures: monkeypatch</a><br> ∘ <a href="#f52d">Patching configurations</a><br> ∘ <a href="#1177">Mocking functions to test error scenarios</a><br> ∘ <a href="#2eee">Testing exception handling with mocking</a><br>· <a href="#2714">Custom fixtures</a><br> ∘ <a href="#2f99">Data Fixtures</a><br> ∘ <a href="#b719">Fixtures for API responses to be used by tests</a><br> ∘ <a href="#aaa4">FastAPI test clients</a><br> ∘ <a href="#4167">Monkeypatched fixtures</a><br>· <a href="#4cde">Coverage of your test suite</a><br>· <a href="#51f9">Testcontainers for testing real dependencies</a><br>· <a href="#b619">Organizing your test suite</a><br>· <a href="#dc6c">Further reading</a><br>· <a href="#892f">That’s all, folks!</a></p><h3>ABCs of testing with Python</h3><p>If you’re reading this post, I hope I don’t need to get into why you need to write tests —you already know. What I will say is that I remember the confusion of getting started with Pytest and not knowing what any of the words meant or how to make the leap from manual testing to automated testing.</p><p>So in this guide, I will be going over all the features of pytest that I have personally used over the years and found to be incredibly useful in maintaining a stable and reliable codebase, whether you are writing thousands of lines of code each month or maintaining a legacy codebase.</p><p>Some of the terminology may be new if you are completely new to testing, but in the interest of keeping this to a reasonable length, I’ve linked to external resources throughout this article that should help you get a better grasp on things. I’ve focused more on code examples here because that’s how I learn best!</p><p>My goal is to take you from wherever you currently are to being comfortable around testing in no time flat!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F4626063&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fzerotohero-zero-hero-disney-hercules-gif-4626063&amp;image=https%3A%2F%2Fmedia.tenor.com%2FJWy6-9j6Lr8AAAAF%2Fzerotohero-zero.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/58d7cf44522137b99e8b8d4a2e203636/href">https://medium.com/media/58d7cf44522137b99e8b8d4a2e203636/href</a></iframe><h3>Basic unit tests</h3><p>These are largely covered in pytest’s own <a href="https://docs.pytest.org/en/8.0.x/getting-started.html">Getting Started</a> documentation, but I’ll cover it here again. Assuming you’ve abstracted out logical code into separate methods, you’re one step ahead — every logical method you have can have its own comprehensive set of unit tests.</p><p>Say you have a super simple atomic method to yield a subset of a list:</p><pre>def chunks(lst, n):<br>    &quot;&quot;&quot;Yield successive n-sized chunks from lst.&quot;&quot;&quot;<br>    for i in range(0, len(lst), n):<br>        yield lst[i:i + n]</pre><p>It’s pretty easy to look at this code and say it’ll work. But what if someone replaces the 0 with 1 tomorrow without anyone knowing? That’s where tests come in. Here’s an example of basic tests with pytest:</p><pre>def test_chunks_success():<br>    &quot;&quot;&quot;Method to test chunking logic&quot;&quot;&quot;<br>    # general case<br>    arr = range(101)<br>    res = chunks(list(arr), 10)<br>    assert isinstance(res, Generator)  # ensure method returns a Generator<br>    res = list(res) # get all chunks<br>    assert len(res) == 11  # for 101 elements in groups of 10, there should be 11 chunks<br>    assert res[-1] == [100]  # last chunk should have only one value with the last element<br><br>def test_chunks_empty_input_no_error():<br>    &quot;&quot;&quot;Method to test edge case where input is empty&quot;&quot;&quot;<br>    # edge case: empty list input<br>    res = chunks([], 10)<br>    assert isinstance(res, Generator)  # ensure method still returns a generator<br>    res = list(res)  # get all chunks<br>    assert not res  # should be empty</pre><p>Now you have covered your bases — the type of returned value, the expected behaviour, and an edge case.</p><p>Basic tests use vanilla Python and simple assert statements to ensure your methods work as expected in every scenario. The more of these you have (particularly for edge cases and rare scenarios), the more stable and reliable your codebase will be.</p><h3>Testing exceptions</h3><p>Now, if someone were to call this method with a non-iterable object, it would raise an exception. You may update the test suite to add the following test:</p><pre># edge case: invalid datatype<br>res = chunks(12, 10)<br>with pytest.raises(TypeError) as e:<br>    res = list(res)<br>e.match(&quot;object of type &#39;int&#39; has no len()&quot;)</pre><p>(and if you need to, you could always update the method to handle these exceptions and instead, test that this <em>doesn’t</em> raise an exception)</p><h3>Built-in Fixtures: Capturing logs and output in tests</h3><p>Pytest has handy <a href="https://docs.pytest.org/en/8.0.x/how-to/logging.html">built-in fixtures</a> which allow you to trap the logs and output statements generated by your code. This is particularly useful to ensure that your application logs are getting generated as expected in certain scenarios, so you know that when you go into debug a production issue at 2 am on a Saturday, the logs you want are going to be there.</p><p>Let us update our function with some logs and print statements:</p><pre>def chunks(lst, n):<br>    &quot;&quot;&quot;Yield successive n-sized chunks from lst.&quot;&quot;&quot;<br>    try:<br>        for i in range(0, len(lst), n):<br>            yield lst[i:i + n]<br>    except TypeError as e:<br>        logging.error(f&quot;Type {type(lst)} not supported&quot;)<br>        print(&quot;Error: invalid datatype input to chunks(lst, n)&quot;, file=sys.stderr)<br>        yield None</pre><p>In our test case, we can verify this log gets generated in the expected scenario. The fixture caplog allows us to check logger statements and capsys lets us inspect stdout/stderr.</p><p>Note — just like with any other Pytest fixtures, you don’t need to import anything explicitly for caplog and capsys to be accessible to your test functions. As long as you run the tests with the pytest command, your tests can access the fixtures.</p><pre>def test_chunks_invalid_datatype_verify_error_logs(caplog, capsys):<br>    &quot;&quot;&quot;<br>    test method to verify that expected logs are printed in case of<br>    invalid datatype<br>    &quot;&quot;&quot;<br>    # edge case: invalid datatype<br>    capsys.readouterr()  # clears any stdout/stderr statements so far<br>    caplog.clear()  # clears all log statements so far<br>    expected_log = f&quot;Type {type(12)} not supported&quot;<br><br>    # caplog.at_level overrides default log levels that your app may have<br>    with caplog.at_level(logging.INFO):  <br>        res = chunks(12, 10)<br>        for i in res:<br>            assert i is None<br><br>        # ensure errors are logged as expected<br>        error_logs = [record.message for record in caplog.records <br>                      if record.levelno == logging.ERROR]<br>        assert expected_log in error_logs<br><br>    # ensure stderr contains the expected log line<br>    captured = capsys.readouterr()<br>    assert &quot;Error: invalid datatype input to chunks(lst, n)&quot; in captured.err</pre><h3>Built-in Fixtures: Higher, Further, Faster</h3><p>You can do so much more with built-in fixtures. In the interest of time, here’s a short list of possibilities, which you can read more about in the <a href="https://docs.pytest.org/en/8.0.x/reference/fixtures.html">pytest documentation</a>.</p><ol><li>Use temporary directories, paths, and files for your tests: <a href="https://docs.pytest.org/en/8.0.x/reference/reference.html#std-fixture-testdir"><strong>testdir</strong></a><strong>, </strong><a href="https://docs.pytest.org/en/8.0.x/reference/reference.html#std-fixture-tmp_path"><strong>tmp_path</strong></a><strong>, </strong><a href="https://docs.pytest.org/en/8.0.x/reference/reference.html#std-fixture-tmp_path_factory"><strong>tmp_path_factory</strong></a><strong>, </strong><a href="https://docs.pytest.org/en/8.0.x/reference/reference.html#std-fixture-tmpdir"><strong>tmpdir</strong></a><strong>, </strong><a href="https://docs.pytest.org/en/8.0.x/reference/reference.html#std-fixture-tmpdir_factory"><strong>tmpdir_factory</strong></a></li><li>Configure the behaviour of your test functions based on command line inputs: <a href="https://docs.pytest.org/en/8.0.x/reference/reference.html#std-fixture-request"><strong>request</strong></a></li><li>monkeypatch — which we will get into more next.</li></ol><h3>Built-in Fixtures: monkeypatch</h3><p>This is where testing starts to get fun. What do you do when you have external dependencies in the methods you want to test? External APIs that may cost money, external databases that you don’t want to pollute with test data, or simply slow dependencies that you don’t want to slow down your test execution?</p><p>At its core, monkeypatching is a way to override functions, attributes, environment variables etc so that you can run your tests exactly the way you want, without any dependencies on external systems. How you can use monkeypatch is limited only by your imagination, but I will share the three most useful examples from my experience.</p><p>Consider the below application code snippet:</p><pre>import requests<br>import logging<br><br>class Settings:<br>    api_url = &quot;https://randomuser.me/api/&quot;<br><br>def make_request():<br>    &quot;&quot;&quot;Method to make API call to get information about a random user&quot;&quot;&quot;<br>    try:<br>        response = requests.get(Settings.api_url, timeout=5)<br>        return response.json()<br>    except requests.exceptions.JSONDecodeError as e:<br>        logging.error(&quot;Invalid response format; unable to parse json&quot;)<br>        return None<br>    except requests.exceptions.Timeout as e:<br>        logging.error(f&quot;{Settings.api_url} timed out: {e}&quot;)<br>        return None</pre><h4>Patching configurations</h4><p>Supposing you have an application whose behaviour can be configured with some settings. You want to test what happens if there’s a problem with the settings without changing your application code. The following test demonstrates the usage of monkeypatching for the same —</p><pre>def test_invalid_json_logs_error(monkeypatch, caplog):<br>    &quot;&quot;&quot;Test case for if api gives non-json response&quot;&quot;&quot;<br>    # patch to api which does not give json response<br>    monkeypatch.setattr(Settings, &quot;api_url&quot;, &quot;https://randomuser.me/&quot;)<br><br>    # make request uses above patched api<br>    resp = make_request()<br>    assert resp is None<br><br>    errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]<br>    assert &quot;Invalid response format; unable to parse json&quot; in errs<br>    caplog.clear()</pre><p>This test patches the api_url to one that gives a non-JSON response. We then verify that the application shows the expected logs.</p><h4>Mocking functions to test error scenarios</h4><p>Alternatively, say you have written some excellent exception handling, but you want to test the exception handling to ensure your app works like it should in case of timeouts, errors, etc.</p><p>The following test uses a “mock” method to override the default requests.get method with one that enforces a timeout error. We can then verify that this error is handled appropriately.</p><pre>def test_api_timeout_logs_error(monkeypatch, caplog):<br>    &quot;&quot;&quot;Test case for if api times out&quot;&quot;&quot;<br><br>    # mock function to simulate timeout from requests.get<br>    def mock_get_timeout(*args, **kwargs):<br>        raise requests.exceptions.Timeout(&quot;simulated timeout&quot;)<br><br>    monkeypatch.setattr(requests, &quot;get&quot;, mock_get_timeout)<br><br>    # this will use the above mock_get_timeout instead of requests.get<br>    resp = make_request()<br>    assert resp is None<br><br>    # verify that appropriate logs are present<br>    errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]<br>    assert f&quot;{Settings.api_url} timed out: simulated timeout&quot; in errs<br><br>    caplog.clear()</pre><h4>Testing exception handling with mocking</h4><p>Consider you have a method that raises certain known exceptions. If you want to verify that these are being raised as expected, we can use a combination of mocking, monkeypatching, and exception testing —</p><pre>def make_request():<br>    &quot;&quot;&quot;Method to make API call to get information about a random user&quot;&quot;&quot;<br>    try:<br>        response = requests.get(Settings.api_url, timeout=5)<br>        return response.json()<br>    except requests.exceptions.Timeout as e:<br>        logging.error(f&quot;{Settings.api_url} timed out: {e}&quot;)<br>        raise RuntimeError(f&quot;Timeout: {e}&quot;) from e</pre><p>We can verify the appropriate RuntimeError is raised with the following test:</p><pre>def test_api_timeout_raises_runtimeerror(monkeypatch, caplog):<br>    &quot;&quot;&quot;Test case for if api times out&quot;&quot;&quot;<br>    # mock function to simulate timeout from requests.get<br>    def mock_get_timeout(*args, **kwargs):<br>        raise requests.exceptions.Timeout(&quot;simulated timeout&quot;)<br>    monkeypatch.setattr(requests, &quot;get&quot;, mock_get_timeout)<br><br>    # this will use the above mock_get_timeout instead of requests.get<br>    with pytest.raises(RuntimeError) as e:<br>        resp = make_request()<br>        assert resp is None<br>    e.match(&quot;Timeout: simulated timeout&quot;)<br><br>    # verify that appropriate logs are present<br>    errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]<br>    assert f&quot;{Settings.api_url} timed out: simulated timeout&quot; in errs<br><br>    caplog.clear()</pre><p><em>For more examples and info about mocking and monkeypatching, you can refer to </em><a href="https://docs.pytest.org/en/latest/how-to/monkeypatch.html"><em>this section of the pytest docs</em></a><em>. While the examples in this post are trivial, a combination of mocking and monkeypatching can help you write truly great tests, particularly when used alongside testcontainers as described later in this post.</em></p><p><em>I would, however, suggest using mocking and monkeypatching with caution as tests which diverge from actual production behaviour will end up being redundant. These are good backup tools for when testing with production services is not feasible.</em></p><h3>Custom fixtures</h3><p>Pytest has extensive support for test configurations which is outside the scope of this post. If this interests you, you can see the possibilities in <a href="https://docs.pytest.org/en/8.0.x/reference/customize.html">the documentation</a>. I will go over the most useful test configurations I’ve used — <a href="https://docs.pytest.org/en/8.0.x/how-to/fixtures.html">custom fixtures</a>.</p><p>Custom fixtures can be used for test data, in combination with monkeypatch to yield versions of your application with different settings, and are a super clean way to test FastAPI applications.</p><p>Fixtures are defined in a file commonly named conftest.py , typically in the root directory of your tests or the root directory of your application. The pytest.fixture decorator is what makes them accessible in all your tests without any extra imports.</p><h4>Data Fixtures</h4><p>You can create a fixture to return fixed data to be used by any test method, such as fixed test IDs or sample data.</p><pre>@pytest.fixture(scope=&quot;session&quot;)<br>def sample_user_id():<br>    &quot;&quot;&quot;Pytest fixture to get user id to be used for testing&quot;&quot;&quot;<br>    return &quot;test_u_2b5e3801a8624f2d854127ae019e0714&quot;</pre><h4>Fixtures for API responses to be used by tests</h4><p>In case your test suite relies on some configurations that are read from an API, you can create an appropriately scoped fixture to abstract out the API call and make the API response accessible to any test:</p><pre>@pytest.fixture(scope=&quot;module&quot;)<br>def versions():<br>    &quot;&quot;&quot;Pytest fixture to yield latest version numbers as per API&quot;&quot;&quot;<br>    versions = requests.get(&quot;https://get/versions/url&quot;, timeout=10).json()<br>    yield versions</pre><p>This has the bonus of ensuring the API call happens only once in case of session- or module-scoped fixtures.</p><h4>FastAPI test clients</h4><p>If you have APIs developed with FastAPI, having your test client be part of your fixtures is an easy way to avoid reinitialization of the TestClient with each test.</p><pre>from app.server import fastapi_app<br><br>@pytest.fixture(scope=&quot;session&quot;)<br>def test_client():<br>    &quot;&quot;&quot;Pytest fixture to get test client of FastAPI app&quot;&quot;&quot;<br>    client = TestClient(fastapi_app)<br>    yield client</pre><h4>Monkeypatched fixtures</h4><p>If you find yourself repeating the same monkeypatching steps in many tests, it may be worth creating a fixture out of it. Following is an example where the versions used by the FastAPI client are patched to fixed values by mocking the function which returns the versions to be used —</p><pre>from app.server import fastapi_app<br>from app import api_helpers<br><br>@pytest.fixture(scope=&quot;function&quot;)<br>def test_client_patch_versions(monkeypatch):<br>    &quot;&quot;&quot;Pytest fixture to get API client that uses fixed version numbers&quot;&quot;&quot;<br>    # monkeypatch fn to simulate version change<br>    def mock_version_change(channels):<br>        return {k: 10 for k in channels}<br><br>    # replace utilities fn<br>    monkeypatch.setattr(api_helpers, &#39;get_latest_versions&#39;, mock_version_change)<br><br>    client = TestClient(app)<br>    yield client</pre><h3>Coverage of your test suite</h3><p>You can use the <a href="https://coverage.readthedocs.io/">coverage library</a> to see how much of your code is covered by tests (the results may surprise you).</p><p>Many organizations have conditions like the codebase must always have &gt;85%, &gt;90%, or &gt;95% of the application code covered by tests. This may or may not be overkill depending on how much critical code is in your application, but this metric is still good to know.</p><p>Even better — it helps you identify parts of your code that you thought may have been tested but aren’t. It could be the difference between one small if condition with an undefined variable breaking production because none of the tests ever reached that code (not that that has ever happened to me, of course).</p><p>To check your code coverage, all you need to do is run your tests with coverage run -m pytest . instead of pytest . , and then generate the report with coverage report (or coverage html if you want a clickable HTML report that you can navigate and search for uncovered code — I recommend this one for local development). Any flags you usually pass to pytest can be passed here as well.</p><h3>Testcontainers for testing real dependencies</h3><p><a href="http://testcontainers.com/">This is a super useful open-source project</a> which can help you work with external dependencies like databases, message queues, etc without actually touching your production systems, needing database access, or needing firewall access. The full list of supported packages is listed in the <a href="https://testcontainers-python.readthedocs.io/en/latest/README.html">documentation</a>.</p><p><em>Note: As of the writing of this article, the Python documentation for testcontainers is limited and contains only interface definitions. Reader discretion is advised.</em></p><p>For example, if your application code uses a Redis client to save some data during runtime and you want to write unit tests that won’t unnecessarily burden your production systems —</p><pre>import redis<br><br>class Settings:<br>    REDIS_HOST = &quot;prod.redis.host&quot;<br>    REDIS_PORT = 6379<br><br>def save_to_redis(key: str, value: str) -&gt; None:<br>    &quot;&quot;&quot;Method to save specified key-value pair to Redis&quot;&quot;&quot;<br>    client = redis.StrictRedis(host=Settings.REDIS_HOST,<br>                               port=Settings.REDIS_PORT)<br>    client.set(key, value)</pre><p>The following test shows you how easy it is to spin up a Redis instance for your test to use, which gets conveniently destroyed at the end of the test —</p><pre>from testcontainers.redis import RedisContainer<br><br>def test_save_to_redis_success(monkeypatch):<br>    &quot;&quot;&quot;Method to test saving to redis with a testcontainer&quot;&quot;&quot;<br>    with RedisContainer() as redis_container:<br>        host = redis_container.get_container_host_ip()<br>        port = redis_container.get_exposed_port(redis_container.port)<br><br>        # patch host and port configs to the testcontainer host and port<br>        monkeypatch.setattr(Settings, &quot;REDIS_HOST&quot;, host)<br>        monkeypatch.setattr(Settings, &quot;REDIS_PORT&quot;, port)<br><br>        # test save_to_redis method<br>        save_to_redis(&quot;test_key&quot;, &quot;test_value&quot;)<br><br>        # verify that data is saved in this redis instance<br>        client = redis_container.get_client()<br>        assert client.get(&quot;test_key&quot;) == b&quot;test_value&quot;</pre><p>Similarly, you can test your interfaces to MongoDB, MySQL, Kafka, Pub/Sub, or any other <a href="https://testcontainers-python.readthedocs.io/en/latest/README.html">supported technologies</a>.</p><h3>Organizing your test suite</h3><p>There is some great documentation about test layouts <a href="https://docs.pytest.org/en/latest/explanation/goodpractices.html#choosing-a-test-layout">here</a> that I recommend going through. What works best for you depends on the nature and structure of your application and its deployment.</p><p>In my personal opinion, having a separate tests/ directory is a good way to keep all the tests together. I also find that for unit testing, simple Python test functions as shown above work best. For integration testing where each method may have multiple associated tests, the class structure is most useful. It keeps each scenario separate while grouping tests related to the same method.</p><p>Considering the same example above, this is how I would organize the tests in a class structure:</p><p>app/main.py :</p><pre>import requests<br>import logging<br><br><br>class Settings:<br>    api_url = &quot;https://randomuser.me/api/&quot;<br><br><br>def make_request():<br>    &quot;&quot;&quot;Method to make get JSON details of a single user from an API&quot;&quot;&quot;<br>    try:<br>        response = requests.get(Settings.api_url, timeout=5)<br>        return response.json()<br>    except requests.exceptions.JSONDecodeError as e:<br>        logging.error(&quot;Invalid response format; unable to parse json&quot;)<br>        return None<br>    except requests.exceptions.Timeout as e:<br>        logging.error(f&quot;{Settings.api_url} timed out: {e}&quot;)<br>        raise RuntimeError(f&quot;Timeout: {e}&quot;) from e</pre><p>tests/test_main.py</p><pre>&quot;&quot;&quot;Tests related to app/main.py&quot;&quot;&quot;<br>import pytest<br><br><br>class TestMakeRequest:<br>    &quot;&quot;&quot;Tests related to make_request() method&quot;&quot;&quot;<br>    @staticmethod<br>    def test_make_request():<br>        # test basic success case<br>        resp = make_request()<br>        assert resp[&quot;results&quot;]<br>        <br>    @staticmethod<br>    def test_invalid_json_logs_error(monkeypatch, caplog):<br>        &quot;&quot;&quot;Test case for if api gives non-json response&quot;&quot;&quot;<br>        # patch to api which does not give json response<br>        monkeypatch.setattr(Settings, &quot;api_url&quot;, &quot;https://randomuser.me/&quot;)<br>        resp = make_request()<br>        assert resp is None<br><br>        errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]<br>        assert &quot;Invalid response format; unable to parse json&quot; in errs<br>        caplog.clear()<br><br>    @staticmethod<br>    def test_api_timeout_raises_runtimerror(monkeypatch, caplog):<br>        &quot;&quot;&quot;Test case for if api times out&quot;&quot;&quot;<br>        # mock function to simulate timeout from requests.get<br>        def mock_get_timeout(*args, **kwargs):<br>            raise requests.exceptions.Timeout(&quot;simulated timeout&quot;)<br>        monkeypatch.setattr(requests, &quot;get&quot;, mock_get_timeout)<br><br>        # this will use the above mock_get_timeout instead of requests.get<br>        with pytest.raises(RuntimeError) as e:<br>            resp = make_request()<br>            assert resp is None<br>        e.match(&quot;Timeout: simulated timeout&quot;)<br><br>        # verify that appropriate logs are present<br>        errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]<br>        assert f&quot;{Settings.api_url} timed out: simulated timeout&quot; in errs<br><br>        caplog.clear()</pre><p>In the case of a large project with multiple folders, following a similar folder structure in the tests/ directory also keeps things quite pleasant to work with.</p><p>I have created a GitHub project that you can use as a reference point <a href="https://github.com/KrupaH/pytest_examples">here</a>.</p><h3>Further reading</h3><p>If you want to learn more about pytest and testing in general, here are some resources I’ve found most useful over the years:</p><ol><li><strong>Test Driven Development by Kent Beck</strong>: this book was given to me once by a professor at college and my coding style changed forever.</li><li><a href="https://www.datacamp.com/courses/introduction-to-testing-in-python"><strong>Introduction to Testing on Datacamp</strong></a><strong>:</strong> If you already have a subscription or are planning to get one, this course is great for hands-on practice. (I wouldn’t pay for the subscription <em>only</em> for this though)</li><li><a href="https://www.datacamp.com/tutorial/pytest-tutorial-a-hands-on-guide-to-unit-testing"><strong>Datacamp’s Hands-On Guide to Unit Testing</strong></a><strong>:</strong> another article like this one, if you want more! :)</li><li><a href="https://docs.pytest.org/en/8.0.x/contents.html"><strong>Pytest documentation</strong></a><strong>:</strong> not the easiest to navigate, but everything is there if you know what to look for (which after this article, I hope you do!)</li><li><a href="https://pytest-with-eric.com/"><strong>Pytest with Eric</strong></a> — a series of great, in-depth posts about all things pytest. I particularly recommend <a href="https://pytest-with-eric.com/pytest-advanced/pytest-improve-runtime/">13 Proven Ways to Improve Test Runtime with Pytest</a> and <a href="https://pytest-with-eric.com/pytest-best-practices/pytest-plugins/">8 Useful Pytest Plugins To Make Your Python Unit Tests Easier, Faster and Prettier</a></li></ol><h3>That’s all, folks!</h3><p>Every test case I’ve had to write in the last 5+ years has largely used one of the above techniques, with testcontainers being the newest entrant. I will always advocate for high test coverage in production systems because as developers, it makes our own lives easier. My midnight production debugging sessions have drastically decreased in frequency ever since I introduced tests into all my codebases. Adding new features is stress-free and bugs are caught long before they reach production. The extra upfront time required to add tests will save literal <em>days</em> in debugging time over the years — particularly in legacy codebases.</p><p>That’s all from me, now go write some tests!</p><h3>Stackademic 🎓</h3><p>Thank you for reading until the end. Before you go:</p><ul><li>Please consider <strong>clapping</strong> and <strong>following</strong> the writer! 👏</li><li>Follow us <a href="https://twitter.com/stackademichq"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/stackademic"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/c/stackademic"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a></li><li>Visit our other platforms: <a href="https://plainenglish.io/"><strong>In Plain English</strong></a> | <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://stackademic.com/"><strong>Stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeb1152c2981" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/pytest-with-python-from-beginner-to-expert-aeb1152c2981">Pytest with Python — from beginner to expert</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>