<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tómas Árni Jónasson on Medium]]></title>
        <description><![CDATA[Stories by Tómas Árni Jónasson on Medium]]></description>
        <link>https://medium.com/@tomagee?source=rss-3ecc69a308f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*3A6gxMvSGv8k0wki.jpg</url>
            <title>Stories by Tómas Árni Jónasson on Medium</title>
            <link>https://medium.com/@tomagee?source=rss-3ecc69a308f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 00:09:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tomagee/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[One weird trick for when filtering by datetime]]></title>
            <link>https://medium.com/@tomagee/one-weird-trick-for-when-filtering-by-datetime-5bb01ef215d5?source=rss-3ecc69a308f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bb01ef215d5</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tómas Árni Jónasson]]></dc:creator>
            <pubDate>Fri, 20 Apr 2018 21:28:48 GMT</pubDate>
            <atom:updated>2018-04-20T23:22:50.504Z</atom:updated>
            <content:encoded><![CDATA[<p>Some things in computer science are surprisingly tricky. Among these are the seemingly innocuous pair of <strong>date</strong> and <strong>time</strong>.</p><p>There’s a <em>lot</em> to discuss when it comes to dates and times in programming (time-zones being the <a href="http://www.creativedeletion.com/2015/01/28/falsehoods-programmers-date-time-zones.html">main offender</a>), but in this post I wan’t to focus in a fairly simple task, that’s easy to get wrong:</p><p>Filtering data based on an interval in time (date+time, that is).</p><h4>Definitions</h4><p>People often use certain terms interchangeably in human languages. Further, depending on programming languages, terms may differ (Date() in JavaScript returns date and time, as an example). Thus, I want to start with the following definitions:</p><p><strong>Date</strong>: Particular day in a particular month, on a particular year. I.e. “29th of October 2017” or “2017–10–29”.<br><strong>Time</strong>: Particular moment in time during a day. I.e. “just after noon”, or “12:05:12.4”.<br><strong>Datetime</strong>: Particular moment in time during a date. I.e. “just after noon on the 29th of October 2017”, or “2017–10–29 12:05:12.4”.</p><h4>Further setup</h4><p>In order to avoid the ambiguity that sometimes plagues pseudocode, let’s frame this in a fairly concrete setting: Let’s assume we’re working in Python, and let’s further assume we’re working in a typical “model+ORM” environment, say, <a href="https://docs.djangoproject.com/en/1.11/topics/db/queries/#retrieving-specific-objects-with-filters">Django</a>. Let’s call our model Event and the datetime attribute of interest happened_at.</p><p>The ideas discussed below should work with any technology stacks. Further, the examples ought be very clear, even for someone with no experience of Django nor Python.</p><p>Lastly, let’s assume a specific execution of this task: The end-user wants to get all the Event objects for a given week in the US/Pacific timezone: 43rd week of 2017, starting on Monday the 23rd up-to <em>and including</em> 29th of October that year.</p><h4>Basic Implementation</h4><p>Let’s assume the datetime variables denoting the interval are called a and b in the following first attempt:</p><pre># (Assuming tz_us_pacific has been set somewhere above)<br>from datetime import datetime<br># (The &quot;0, 0, 0, 0&quot; is not needed, but put in for clarity.<br>#  It stand&#39;s for &quot;hour, minute, second, microsecond&quot;<br>#  respectively.)<br>a = datetime(2017, 10, 23, 0, 0, 0, 0, tzinfo=tz_us_pacific)<br>b = datetime(2017, 10, 29, 23, 59, 59, 999999,<br>             tzinfo=tz_us_pacific)<br># (Note: There are smoother ways to get the last possibly moment in<br>#  Python, but for sake of clarity, I entered the last microsecond<br>#  manually above)<br>events = Event.objects.filter(<br>    happened_at__gte=a,<br>    happened_at__lte=b,<br>)</pre><p>On the surface, this seems fine. We start at the beginning of the 23rd, and filter up to, and including, the last moment of the 29th.</p><p>But there’s a precarious problem lurking there: While b from above is the last possible moment of a particular date, this unfortunately is only true for some implementations of some programming languages.</p><p>This will pose a problem if the values of the interval are generated from, for example, certain implementations of JavaScript, where the resolution of datetimes are 1 millisecond, or 1000 microseconds.</p><p>In other words: A “<em>last moment</em>” of a day is dependent on the programming language and it’s implementation.</p><p>This means that if the up-to value is constructed using JavaScript, we will never cover the last 999 microseconds of any interval we choose.</p><h4>The solution</h4><p>The reader has probably figured out the solution to this, which is pretty simple: Make the right-hand side of the interval be open/non-inclusive.</p><p>Let’s rewrite the example above to reflect this:</p><pre>a = datetime(2017, 10, 23, 0, 0, 0, 0, tzinfo=tz_us_pacific)<br>b = datetime(2017, 10, 30, 0, 0, 0, 0, tzinfo=tz_us_pacific)<br>events = Event.objects.filter(<br>    happened_at__gte=a,<br>    happened_at__lt=b,<br>)</pre><p>Two things happened here: We’re filtering up-to the midnight of the next day, and we changed the filtering to &lt; instead of ≤.</p><p>To some, this may look ugly for one reason: The asymmetry between left- and right-hand-side of the interval. One is inclusive, the other not.</p><p>Symmetry is usually a fair indicator that something is “correct”. Symmetry is very useful in many fields, such as engineering and <a href="http://www.feynmanlectures.caltech.edu/I_52.html">physics</a>, but at the heart of it are usually some basic mathematical ideas. Since programming is really just application of math, symmetry can be used to deem if certain fragments of code are well written/sensible/correct/etc.</p><p>So why then do I suggest breaking symmetry in this case?</p><p>The answer is that the definition of days <em>is</em> asymmetrical!</p><p>Days are defined to start at midnight, and end when the next day starts. In other words: <em>There is no last moments to a day</em>.</p><p>Thus, this solution is a perfect match to our use-case! Indeed, it works great for different levels too: I.e. filtering on specific hour, minute and so forth, as they are all asymmetrically defined on the time-axis.</p><p>Of course, this applies to any numerical range in reality. There’s no last real number coming just before we get to 2. There’s infinite real (and imaginary) numbers between 1.99999 and 2.</p><h4>Let’s get mathy!</h4><p>We can also compare the initial implementation to the improved one in more mathematical terms, and see if we can find a more rigorous justification of this idea.</p><p>The original implementation boils down to this definition (ε is the millisecond for JavaScript and microsecond for Python):<br>[<em>a</em> ≤ <em>x</em> ≤ <em>b</em> - ε], ε is smallest positive value “as possible”.</p><p>Whereas the corrected implementation is defined thus:<br>[<em>a</em> ≤ x &lt; <em>b</em>)</p><p>The cool thing is that if we take the limit of ε → 0⁺, the original definition <em>becomes</em> the latter one!</p><p>Thus, we can with good conscience state that our improved implementation is <em>provably</em> better than the original.</p><p><strong>Disclaimer</strong>: While I do like math, I’m no mathematician, so take my mathematical “proofs” with a grain of salt.</p><p>Helpful links:</p><ol><li><a href="https://en.wikipedia.org/wiki/Interval_(mathematics)">https://en.wikipedia.org/wiki/Interval_(mathematics)</a></li><li><a href="https://docs.python.org/3/library/datetime.html">https://docs.python.org/3/library/datetime.html</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date">https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bb01ef215d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[`$ pip freeze > requirements.txt` considered harmful]]></title>
            <link>https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895?source=rss-3ecc69a308f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0bce66cf895</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pip]]></category>
            <category><![CDATA[libraries]]></category>
            <category><![CDATA[requirements]]></category>
            <dc:creator><![CDATA[Tómas Árni Jónasson]]></dc:creator>
            <pubDate>Fri, 20 Apr 2018 20:27:43 GMT</pubDate>
            <atom:updated>2018-04-29T01:54:57.895Z</atom:updated>
            <content:encoded><![CDATA[<p>All over the interwebs I see people/companies suggesting:</p><pre>pip freeze &gt; requirements.txt</pre><p>as a way to update requirements file with latest and greatest after pip install-ing some new fancy packages.</p><h3>The problem</h3><p>There are few dangers to this approach, mainly that libraries have their own dependencies which will get surfaced via pip freeze.</p><p>These dependencies may then be deprecated as our libraries are updated, but will then be left in our requirements.txt file with no good reason, polluting our dependency list.</p><h3>Example</h3><p>When switching to Python 3 recently, we had futures package installed, which is a backport of some Python 3 functionality to Python 2. futures was brought in as a 2nd level dependency on some package (which one is very hard to know, because tracing it through the git log may result in seeing a single commit in which multiple packages are updated, and many added, as was the case for our futures dependency).</p><p>When I started to run our service on our hosts, I kept seeing very odd and confusing error messages related to some concurrency module. I did a bunch of googling, and testing (deploying the service again and again to servers), and finally, after 3 hours of digging in, I finally realized this was due to the futures package being in our requirements.txt file. There were other complications which made things worse — how our service deployment managed it’s virtual environment how the PYTHONPATH was managed and how/when it installed the packages. For some reason, this didn’t affect running our service locally, which didn’t make figuring this out any easier.</p><p><strong>Bottom line</strong>: We never directly depended on this futures package, and yet it managed to waste a lot of time and energy when we changed the environment in which our app ran.</p><p>Here’s how our requirements file progressed over time for this scenario:</p><pre>a_library_we_need=1.0<br>futures==1.0  # brought in by `a_library_we_need`</pre><p>Then, a_library_we_need was updated to a version that didn’t require futures aymore:</p><pre>a_library_we_need=2.0<br>futures==1.0  # not really required at all anymore</pre><h3>A Solution</h3><p>If we had hand-crafted our requirements.txt file, we would never have had this issue.</p><p>Manually updating the file has the benefit of having the history and reasoning behind all of our dependencies clearly stated in the repository history.</p><p>It also makes it much easier for developers to reason about when considering updating these dependencies, or re-evaluating their need.</p><p>It allows developers to add some sensible structure to the requirements.txt file — lines at top could be major libraries (say, Django) and closer to the bottom we could have utility libraries (say, requests).</p><p>Yet another benefit is that sometimes developers are testing things out locally, and may bring in some libraries that were part of experiments or some local setup, that have nothing to do with the production version of the service.</p><p>Using pip freeze is very simple. One has simply to do something like:</p><pre>$ pip install new_package==1.2.3<br>$ pip freeze &gt; requirements.txt<br>$ git add requirements.txt<br>$ git commit &quot;Update libs&quot;<br>$ git push</pre><p>But what I’d suggest is to rather go for an approach like this:</p><pre>$ pip install new_package==1.2.3<br>$ vim requirements.txt  # add &quot;new_package=1.2.3&quot; into file<br>$ git add requirements.txt<br>$ git commit &quot;Add library: &#39;new_package==1.2.3&#39;&quot;<br>$ git push</pre><p>This isn’t a complete solution however, as pointed out by my friend <a href="https://twitter.com/baldurthoremils">Baldur</a>, because as our dependencies often state their dependencies using &gt;=, it means that if two persons install the same hand-crafted requirements.txtfile at different time, those sub-dependencies may have been updated, resulting in a different final set of packages between the two persons. This could be amended by actually using pip freeze to set up another file, say, requirements-freeze.txt and exclude our direct dependencies from that resulting list.</p><p>One can then focus on updating the direct dependencies, but still keep an eye on how the sub-dependencies change over time, and keep it separate from the main requirements.</p><p><em>(Small side note on </em><em>git-usage: Try and get used to using </em><em>git add &lt;filepath&gt; or </em><em>git add -p/--patch rather than </em><em>git add -a. And use </em><em>git show and </em><em>git diff &lt;remote-branch&gt; every time before pushing your changes out. You never know if you accidentally left in some changes you didn’t mean to push.)</em></p><h3>Dependency hell</h3><p>Another problem with pip which I’m not really touching on is dependency conflict.</p><p>Say you require library p1 and p2 but that they both depend on another library, d . This is usually not a problem, unless they depend specifically on different versions of that package. If p1 depends on d==1.0 and p2 depends on d==2.0 , there will be conflicts when installing.</p><p>pip freeze will change a requirements file from:</p><pre>p1<br>p2</pre><p>into something like</p><pre>p1<br>p2<br>d==2.0</pre><p>This rarely happens, as usually dependencies are set up as &gt;=X.Y.Z and packages tend to “march” forward together over time, and it seems rare that people put &lt; or == in their dependency list.</p><p>I’m honestly not entirely sure how pip handles these — it may vary between Python 2 and Python 3, and may even vary between pip versions. I think I’ve seen the conflict surfaced during pip install in the form of a warning at the end.</p><p>Of course, wherever p1 is used in a way that uses parts of d==2.0 which are not understood by p1 , we may end up with some exceptions flying around.</p><p>There have been attempts at having good dependency resolution sitting on top of pip/PyPI, but most people just use vanilla pip, so I won’t get into those here. There’s also a new package tool, called <a href="https://docs.pipenv.org/">pipenv</a>, which may help solve some of these problems I raise.</p><p>The main point is that if you hand-craft the requirements.txt file, it’ll be easier to resolve the conflict, because the developer, looking at the requirements.txt file, will know for sure that d is a 2nd level dependency, and that the project doesn’t itself require a specific version of it.</p><p>Well — hope you gained something from this post, and that you will consider moving from pip freeze to hand-crafting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0bce66cf895" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>