<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Juliana Oliveira on Medium]]></title>
        <description><![CDATA[Stories by Juliana Oliveira on Medium]]></description>
        <link>https://medium.com/@jwnx?source=rss-fe772e577177------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xW8NNiG9g7o4FF_-C3HTpg.jpeg</url>
            <title>Stories by Juliana Oliveira on Medium</title>
            <link>https://medium.com/@jwnx?source=rss-fe772e577177------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:25:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jwnx/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Multiprocessing Serialization in Python with Pickle]]></title>
            <link>https://medium.com/@jwnx/multiprocessing-serialization-in-python-with-pickle-9844f6fa1812?source=rss-fe772e577177------2</link>
            <guid isPermaLink="false">https://medium.com/p/9844f6fa1812</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[outreachy]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Juliana Oliveira]]></dc:creator>
            <pubDate>Sun, 21 Jan 2018 04:44:28 GMT</pubDate>
            <atom:updated>2018-01-21T04:44:28.207Z</atom:updated>
            <content:encoded><![CDATA[<h4>What can go wrong?</h4><p>On the last few weeks working on <a href="https://diffoscope.org/"><em>diffoscope</em></a> I’ve came to realise how far we, as humankind, still are from being able to easily parallel large or complex chunks of code. In this article I’ll be exploring a little about what <em>multiprocessing</em> does with your data behind the curtains and why this knowledge makes our lives easier when trying to parallelize code.</p><p>Since we’re diving into specifics, these are some sources to learn more about therminology and basic concepts of parallelism on python:</p><ul><li><a href="https://medium.com/@bfortuner/python-multithreading-vs-multiprocessing-73072ce5600b">https://medium.com/@bfortuner/python-multithreading-vs-multiprocessing-73072ce5600b</a></li><li><a href="https://www.ploggingdev.com/2017/01/multiprocessing-and-multithreading-in-python-3/">https://www.ploggingdev.com/2017/01/multiprocessing-and-multithreading-in-python-3/</a></li></ul><h3>A Briefing on Processes and Threads</h3><p><strong>Processes</strong> have at least one <strong>thread</strong>. These threads, inside a process, share memory space. Threads are not interruptible or killable, whereas processes are heavier and take longer to spawn. Parallelism is achieved when two or more tasks are executed simultaneously; this can be done through threads or processes.</p><p>In Python, threading is the module to work with <strong>threads</strong>, while multiprocessing works with <strong>processes</strong> (and threads underneath it). The latter has many abstractions to make our lives easier, such as <strong>Pools</strong> and <strong>Managers, </strong>and is able to supervise thread execution without us having to deal with <a href="https://jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/">Pythons Hardest Problem</a>.</p><p>Since processes do not share memory space, they need a different and more complex ways of sending information than threads. As mentioned by <em>Matthew Rockling</em> in his article <a href="http://matthewrocklin.com/blog/work/2013/12/05/Parallelism-and-Serialization">Parallelism and Serialization</a>, processes send information much like the teleporters in Star Trek: a package is converted to something understandable by the teleporter, sent, and transformed back into its package form. This process is called <strong>serialization</strong>, and Python uses the <strong>Pickle</strong> library to handle translating the object back and forth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZXVbbKY-Zj_JnXHapRetvQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/OnAwTs0tu3k?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Natalie Rhea Riggs</a> on <a href="https://unsplash.com/search/photos/pickle?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>How far does Pickling go?</h3><p>Pickle is able to serialize and deserialize Python objects into bytestream. It does work well on most cases — with <a href="http://www.benfrederickson.com/dont-pickle-your-data/">reservations</a>. When multiprocessing spawns a process, <strong>Pickle</strong> is called by default to handle data transfer. A simple example of how this is done follows:</p><pre>import pickle</pre><pre>class Foo(object):<br>    @property<br>    def bar(self):<br>        return ‘bar’</pre><pre>&gt;&gt;&gt; f = Foo()<br>&gt;&gt;&gt; p = pickle.dumps(f)<br>&gt;&gt;&gt; p<br>b&#39;\x80\x03c__main__\nFoo\nq\x00)\x81q\x01.&#39;<br>&gt;&gt;&gt; u = pickle.loads(p)<br>&gt;&gt;&gt; u.bar<br>&#39;bar&#39;</pre><p>As you may notice, a Pickled Foo()is its function name. According to <a href="https://docs.python.org/3.5/library/pickle.html">Pickle’s documentation</a>:</p><blockquote>Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. <a href="https://docs.python.org/3.5/library/pickle.html#id7">[2]</a> This means that<strong> only the function name is pickled, along with the name of the module the function is defined in</strong>. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module <strong>must be importable in the unpickling environment</strong>, and the module must contain the named object, otherwise an exception will be raised.</blockquote><p>That said, we can imagine scenarios in which some objects won’t be picklable. Lambdas are one of them, since they are not importable.</p><pre>&gt;&gt;&gt; f = lambda x: x**2<br>&gt;&gt;&gt; p = pickle.dumps(f)<br>Traceback (most recent call last):<br>  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;<br>_pickle.PicklingError: Can&#39;t pickle &lt;function &lt;lambda&gt; at 0x7fbd12f29510&gt;: attribute lookup &lt;lambda&gt; on __main__ failed</pre><p>And more comonly, Inner Classes:</p><pre>&gt;&gt;&gt; class Foo():<br>...     def __init__(self):<br>...             class Bar():<br>...                     <a href="http://twitter.com/property">@property</a><br>...                     def bar(self):<br>...                             return &#39;bar&#39;<br>...             self.bar = Bar()<br>... <br>&gt;&gt;&gt; f = Foo()<br>&gt;&gt;&gt; p = pickle.dumps(f)<br>Traceback (most recent call last):<br>  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;<br>AttributeError: Can&#39;t pickle local object &#39;Foo.__init__.&lt;locals&gt;.Bar&#39;</pre><h3>Back to Multiprocessing</h3><p>Ok, so we know Pickle’s limitations. What about multiprocessing? Well, since multiprocessing uses Pickle, it inherits the same flaws. And, as much as we’d like to believe that there are just a couple of cases in which Pickle fails, achieving parallelization is a complex task that may expose many non expected errors.</p><p>When your code is not picklabe by default, you have two options: rewrite your code in order to make classes importable by name or try some alternatives. I’ll be talking about the latter on a next post.</p><p>This post is part of a series about <a href="https://diffoscope.org/"><em>diffoscope</em></a><em> </em>development for the <a href="https://www.outreachy.org/">Outreachy Project</a>. <a href="https://diffoscope.org/"><em>Diffoscope</em></a> is a comparison tool, part of the <a href="https://reproducible-builds.org/">Reproducible Builds effort</a>. In the following weeks, I’ll be writing about development details, techniques and general discussion around <a href="https://diffoscope.org/"><em>diffoscope</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9844f6fa1812" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction on Regression Testing]]></title>
            <link>https://medium.com/@jwnx/an-introduction-on-regression-testing-a894fad15ea6?source=rss-fe772e577177------2</link>
            <guid isPermaLink="false">https://medium.com/p/a894fad15ea6</guid>
            <category><![CDATA[outreachy]]></category>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Juliana Oliveira]]></dc:creator>
            <pubDate>Thu, 21 Dec 2017 20:34:54 GMT</pubDate>
            <atom:updated>2017-12-21T20:34:54.992Z</atom:updated>
            <content:encoded><![CDATA[<h4>When everything else fails</h4><p>One of the most known types of testing is <em>unit testing</em>. It hopes to guarantee that every module works individually. When an interaction is needed outside of its scope, this interaction is stubbed or mocked. But what happens when a new dependency is included, a new system is run or a new feature is written?</p><p>Bugs.</p><p><em>Regression</em>, popularly known as bug, is the name given to all non-expected software behavior. When this behavior is found — sometimes outside the testing scope — and fixed, new tests need to be written to ensure this behavior will not be seen again. Those tests are called <em>regression tests</em>.</p><p>In large and public projects, <em>regression tests</em> are needed to ensure the intended behavior is kept between libraries, operating systems and machines. Moreover, this tends to be a deal breaker for global software usage. In a Libre Software project, your software being usable across many OS distributions is likely to be your main goal.</p><p>It is important to notice that <em>regression tests</em> can be broadly referred to as <em>unit tests</em>. <em>Unit testing</em> against desired and non-desired software behavior <a href="https://en.wikipedia.org/wiki/Regression_testing#cite_ref-6">tends to reduce the need of future <em>regression testing</em></a><em>.</em></p><h3>A Simple Start</h3><p>In the ideal scenario, <em>regression testing</em> will work as described. Let us illustrate with a code snippet.</p><pre>import Couch</pre><pre>class Cat:<br>    def scratch(self, Couch):<br>        self.happiness += Couch.damage(self.claws)</pre><p>In the above example, there is a Catand a Couch class. Each of them are managed by different developers. Cat uses Couch, but Couch doesn’t need (or want) Cat. Moreover, Couch documentation states that damage() should return a value from 0 to 10 which corresponds to the hole-creating hability of the given sharp object. In our case, Cat.claws.</p><p>In this scenario, it’s easy to see that a change of behavior in Couch may cause Cat to crash. Maybe <em>unit testing</em> will help us?</p><pre>class TestCat:<br>    def test_scratch(self, cat, couch):<br>        happiness_before = cat.happiness<br>        couch.damage(cat.claws)<br>        happiness_after = cat.happiness<br> <br>        assert happiness_before &lt;= happiness_after</pre><p>Good enough, this test will run and pass. Unfortunately, as predictable as it can be, Couch developers wanted a way to optimize its damage taking hability and avoid possible holes. In their newest release, they updated the damage() function to also include different and unsual kinds of couch materials — such as glass and bees. As a consequence, damage() may now return values from -10 to 10.</p><p>With failing tests, we’re compeled to fix our code. By adding a few lines, it is possible to avoid this kind of error.</p><pre>import Couch</pre><pre>class Cat:<br>    def scratch(self, Couch):<br>        damage = Couch.damage(self.claws)<br> <br>        if damage &gt; 0:<br>            self.hapiness += damage</pre><p>However, there’s another major problem: we haven’t predicted this behavior in our <em>unit test</em> case. Now, we need to update our tests to foresee and avoid this output.</p><pre>class TestCat:<br>    def test_scratch_when_damage_is<br>        bigger_than_zero_happiness_increases(self, cat,<br>                                             regular_couch):<br>        happiness_before = cat.happiness<br>        regular_couch.damage(cat.claws)<br>        happiness_after = cat.happiness<br> <br>        assert happiness_before &lt;= happiness_after<br> <br>    def test_scratch_when_damage_is<br>        lower_than_zero_happiness_doesnt_change(self, cat,<br>                                                bee_couch):<br>        happiness_before = cat.happiness<br>        bee_couch.damage(cat.claws)<br>        happiness_after = cat.happiness<br> <br>        assert happiness_before == happiness_after</pre><h3>DEV Community 👩‍💻👨‍💻 on Twitter: &quot;2 unit tests. 0 integration tests pic.twitter.com/V2Z9F4G1sJ / Twitter&quot;</h3><p>2 unit tests. 0 integration tests pic.twitter.com/V2Z9F4G1sJ</p><p>In larger projects, this kind of errors may not be as elementary to predict. Quite on the contrary, as complexity grows, they may get impossible to predict in a <em>unit test</em> suite. Moreover, when dealing with many dependencies, writing <em>regression tests</em> gets a lot more tricky.</p><h3>Debug, Test, Rinse and Repeat</h3><p>During the past week, I’ve come across some interesting bugs in <a href="https://diffoscope.org/"><em>diffoscope</em></a> which illustrate this concept. One of them being an optional dependency version issue: <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877728">[#877728 — binutils 2.29.0 on x86–64]</a>.</p><p>In short, this bug is seen when running <a href="https://diffoscope.org/"><em>diffoscope</em></a> with <em>binutils 2.29.0</em> version. It’s rather common to see a change in software behavior from one major version to the other — from 2 → 3 — <a href="https://semver.org/">not that common in minor releases</a>. In this case, let’s analyse <em>binutils</em> previous behavior versus the current one to see which one is wrong so we can fix it.</p><p>Acording to the bug report, this regression was found while running tests/comparators/test_elf.py, which means the culprit is <em>readelf</em>. <em>Readelf</em> is distributed along with <em>binutils</em> and matches its version. In this case, we know that we will probably find more clues running <em>readelf </em>&lt; 2.29.0 and == 2.29.0.</p><p>After a few hours of investigation, it came to light that <em>readelf</em> used to return exit code 0 when no information was found and started returning code 1, which means error in computer language. Going a little further, it was found that this behavior was immediately fixed in version 2.29.1, fixing <a href="https://diffoscope.org/"><em>diffoscope</em></a><em>’</em>s behavior all together.</p><p>In this case, when running version binutils == 2.29.0 we found a regression. Since <a href="https://diffoscope.org/"><em>diffoscope</em></a><em> </em>depends on <em>readelf</em> behavior for comparisons, the best we can do is accomodate our software to require binutils != 2.29.0. After making this change and rerunning our test suit, we ensure this regression will not be seen.</p><h3>Conclusion</h3><p>As seen here, <em>testing</em> is not perfect and we must accept that. The actual main goal in keeping tests running is to guarantee working compatibility between software versions and platforms, and not for the sake of having tests. We must work towards this goal, while not — intentionaly — breaking userspace.</p><h3>jessie frazelle on Twitter</h3><p>If you liked it then you should have put a regression test on it.</p><p>This post is part of a series about <a href="https://diffoscope.org/"><em>diffoscope</em></a><em> </em>development for the <a href="https://www.outreachy.org/">Outreachy Project</a>. <a href="https://diffoscope.org/"><em>Diffoscope</em></a> is a comparison tool, part of the <a href="https://reproducible-builds.org/">Reproducible Builds effort</a>. In the following weeks, I’ll be writing about development details, techniques and general discussion around <a href="https://diffoscope.org/"><em>diffoscope</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a894fad15ea6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>