<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Xavier Dubuc on Medium]]></title>
        <description><![CDATA[Stories by Xavier Dubuc on Medium]]></description>
        <link>https://medium.com/@xavier.dubuc?source=rss-d06b181e9a5c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sbBJ-UL2QvDCL_gRHRFkeQ.jpeg</url>
            <title>Stories by Xavier Dubuc on Medium</title>
            <link>https://medium.com/@xavier.dubuc?source=rss-d06b181e9a5c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:13:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xavier.dubuc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[“Migrations zero” or how to handle migrations on a large Django project]]></title>
            <link>https://medium.com/@xavier.dubuc/migrations-zero-or-how-to-handle-migrations-on-a-large-django-project-643627938449?source=rss-d06b181e9a5c------2</link>
            <guid isPermaLink="false">https://medium.com/p/643627938449</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[migration]]></category>
            <dc:creator><![CDATA[Xavier Dubuc]]></dc:creator>
            <pubDate>Tue, 16 Oct 2018 20:48:32 GMT</pubDate>
            <atom:updated>2024-03-09T21:34:12.918Z</atom:updated>
            <content:encoded><![CDATA[<p><em>(Update March 9th 2024 : someone has built some python package to ease the integration of this concept in CD/CI environment. I didn’t test it but it sure worths a look : </em><a href="https://pypi.org/project/django-migration-zero"><em>https://pypi.org/project/django-migration-zero</em></a><em>)</em></p><p>Working with Django is paired with working with migrations. Migrations are operations needed to be run on a database so that its state stays coherent with the code state. To maintain these migrations over time can be tricky and their execution can take a lot of time. “Migrations zero” is a solution to this. In this article, I’ll explain what it is, why it is needed and how it can be achieved.</p><h3>What is the problem with migrations in large projects ?</h3><p>First, let’s see how migrations can be such a pain. It resides in one sentence :</p><blockquote>“With large project comes big amount of migrations”.</blockquote><p>Also, there can be dependencies between migrations that Django needs to be aware of those dependencies in order to run Migration A before Migration B if B depends on A. To be able to do this, Django build a tree, the <strong>migrations tree, </strong>and the time needed to build it depends on the amount of migrations and dependencies between them. As i’m writing this, I’m working on a project who used to have ± 580 migrations and running migrate or makemigrations took like ± 20 minutes.</p><p>You see it coming, every time you run tests, every time you create a new migration, you loose 20 minutes and <strong>this is the problem</strong>.</p><h3>What is migrations zero ?</h3><p><strong>Migrations zero</strong> is the name given to a set of migrations that is needed and sufficient to reflect the database as it has to be (without any data, and without anything to migrate from). In that way of thinking, all other migrations depict the needed operations to transform the PRODUCTION database state to DEVELOPMENT database state. These migrations can contain data migration, new table creation, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*Trcw64dovrQu34dpYokc3Q.png" /><figcaption>Migrations zero is by definition a subset of (or equal to) the set of all existing migrations</figcaption></figure><p>The challenges of <strong>migrations zero</strong> pattern are:</p><ul><li>build the <strong>migrations zero</strong> set the first time</li><li>keep the <strong>migrations zero</strong> set coherent and minimal</li></ul><p>To achieve these challenges we need to be able to rebuild this migrations zero set easily and as frequently as possible so that the <strong>migrations tree</strong> does not grow too much. The ideal would be to rebuild the set as soon as the PRODUCTION database is updated but it will depend on your deployment policy. <em>(If you use an automated deployment system and you tend to use the migrations zero pattern, there are some concerns to think about, I’ll talk about it in a future article)</em></p><p><strong>Be careful</strong> : the building of migrations zero set will require to delete all migrations. You need to be sure that those migrations have already been executed on your PRODUCTION database and that they are no more required.</p><h3>How to build migrations zero set ?</h3><p>Now we know what it is and why it’s needed, it’s time to introduce the methodology used to build the migrations zero set. This section will give you all the keys you need to achieve it.</p><h4>Different types of migrations</h4><p>Before getting into details, let’s begin with some definitions. Each migration differs by the operations it runs. There are mainly 3 types of migrations :</p><ul><li><strong><em>Structural</em> migration</strong> : these migrations only contains table &amp; relation modifications generally automatically generated by Django.</li><li><strong><em>Data</em> migration</strong> : these migrations contains at least one RunPython operation and transfer data from a model to another or creates data based on other data; in other words, they need logic to create data.</li><li><strong><em>Fixture</em> migration</strong> : these migrations contains at least one RunPython operation and load data in database to allow the application to run, for example a list or countries, a list of user statuses … They <strong>don’t</strong> need logic to create data.</li></ul><p>(<em>They also can be a mix of two or all types above</em>)</p><h4>Building process</h4><p>For that section, I’ve been inspired by this article of Vitor Freitas.</p><p><a href="https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html">How to Reset Migrations</a></p><p>The building process is mainly composed of 2 steps :</p><ol><li>clear the actual migrations so that the application is not harmed</li><li>generate the initial migration state</li></ol><p>The second is quite simple but the first can be tricky as multiple things can happen and it will depend on the content of your migrations.</p><h4>Clear the actual migrations</h4><p>First, you have to be certain that your models state are strictly identical to the database state. To be sure of that, run python manage.py makemigrations.</p><ul><li>If the output is No changes detected then you can proceed</li><li>Otherwise, it will generates some migrations that need to be executed before the clearing. <strong>Be careful</strong> as at this point, these migrations may be also needed for other environments (PRODUCTION for example !) so it may be interesting to commit these migrations before going forward.</li></ul><p>Now it’s time to delete all migrations. The proper way would be to <em>fake</em> the unapplication of all migrations for each app but it can lead you in a deadlock of dependencies. You can try that way first and if you’re running into the deadlock, try the hard way.</p><p><em>The proper way</em> : for each app app of the project, run</p><pre>python manage.py migrate --fake app zero</pre><p><em>The hard way</em> : Clear the django_migrations table.</p><p>Now, the next step is to delete all migrations files <strong>BUT</strong> before that, you’ll want to check every migration to determine to which kind it belongs.</p><ul><li>If it’s a <strong><em>structural</em></strong> migration, you can consider it as <em>safe to delete </em>(its operations will be recreated in the second step)</li><li>It it’s a <strong><em>data</em></strong> migration, you can <strong>normally</strong> consider it as <em>safe to delete</em> without any further consideration (they only have to be executed once then they are useless). Be sure, by the way, that it does not contain some part of a “<em>fixture migration</em>”.</li><li>If it’s a <strong><em>fixture</em></strong> migration, you need to take care of it.</li></ul><p>As said before, <em>fixture</em> migrations contain data to load in the database so that the application can work. Deleting these migrations could be a problem as if you want to build a new system from scratch, you won’t be able to run the application. The solution to take care of these is to create a <a href="https://docs.djangoproject.com/en/2.1/howto/initial-data">fixture</a> to load these data. <em>(I could write an article about that) </em>Once the fixture is created, your data migration can be considered as <em>safe to delete</em>.</p><p>As soon as you checked all the migrations and handled them correctly, you can go ahead and remove all the migration files.</p><h4>Generate migrations zero set</h4><p>Now that you deleted all migrations, you can proceed to step 2 : generate the migrations zero set.</p><p>To achieve it, you simply need to run</p><pre> python manage.py makemigrations </pre><p>It will generate for you the migrations zero set. At this time migrations zero set is generated and you work is done. However, you still need to do something in order to be able to continue developing with migration. Indeed, you need to tell your database that all migrations from migrations zero is <em>already</em> executed.</p><p>For that just fake the migrations :</p><pre>python manage.py migrate --fake </pre><p>You can then run all the tests to be sure nothing has been broken. If all the tests passed, you can commit and push the new migrations zero set.</p><h3>How to work with migrations zero set ?</h3><p>Each time the migrations zero set is generated, on PRODUCTION (or even for each developer except the one responsible of the generation), it will be mandatory to handle the reception of the new set. For that, normally, you just have to clean the django_migrations table <em>(not mandatory)</em> and fake the migration zero set by running python manage.py migrate --fake.</p><p>But in some case, you could face a undesired situation : you just generated a migration based on the old state and someone just push the migration zero state. In that case, you cannot push your commits as your migration won’t be executable. That is something you really want to avoid, so it should be communicated to everyone that a given commit is rebuilding the migration zero set. But, if it happens anyway, follows the instructions below.</p><ol><li>If your generated migration is <em>structural </em>one:</li></ol><ul><li>delete it</li><li>run python manage.py migrate --fake to base your database on the new migrations zero set</li><li>run python manage.py makemigrations to recreate your migration</li><li>run python manage.py migrate if this migration has not been executed yet, add --fake option otherwise</li></ul><p>2. If it’s a <em>data migration</em>:</p><ul><li>delete the migration</li><li>copy the migration content (the method executed by the RunPython block)</li><li>run python manage.py migrate --fake to base your database on the new migrations zero set</li><li>run python manage.py makemigrations *app* --empty to recreate your migration</li><li>paste the previously copied content into the newly generated migration</li><li>run python manage.py migrate if this migration has not been executed yet, add --fake option otherwise</li></ul><p>3. If it’s a <em>fixture migration</em>:</p><ul><li>create the associated <strong>fixture</strong></li><li>delete the migration</li><li>run python manage.py migrate --fake to base your database on the new migrations zero set</li><li>Be also sure to load the created fixture.</li></ul><p>And that’s it, you are now a Migration zero professional !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=643627938449" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Django unit testing : the right way]]></title>
            <link>https://medium.com/@xavier.dubuc/django-unit-testing-the-right-way-ba465cf3f3c9?source=rss-d06b181e9a5c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba465cf3f3c9</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[unit-testing]]></category>
            <dc:creator><![CDATA[Xavier Dubuc]]></dc:creator>
            <pubDate>Sun, 07 Oct 2018 20:13:55 GMT</pubDate>
            <atom:updated>2019-06-12T17:46:07.405Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*24brg9lkgcH9OsjvVkYCSw.png" /><figcaption>unit_test_runner.py</figcaption></figure><h3>Django unit testing : the right way</h3><p>I’ve been working for years with Django and I really appreciate it. But one thing I never understood is why there is nothing “builtin” to run a suite of tests without any database handling. So I travelled the Internet to find a solution and I found it : I had to write my own <a href="https://docs.djangoproject.com/en/dev/topics/testing/advanced/#defining-a-test-runner">Test runner</a> … and that’s what I did.</p><h3>Unit tests Vs Integration Tests</h3><p>Understand me well, the provided Django test feature is very good and very useful but all run tests are <em>de facto</em> <strong>Integrations tests </strong>which can be quite annoying. What’s the difference you would say ? Well, if you trust <a href="https://www.atlassian.com/continuous-delivery/different-types-of-software-testing">Atlassian</a> :</p><ul><li><strong>Unit tests </strong>are very low level, close to the source of your application. They consist in testing individual methods and functions of the classes, components or modules used by your software. Unit tests are in general quite cheap to automate and can be run very quickly by a continuous integration server.</li><li><strong>Integration tests</strong> verify that different modules or services used by your application work well together. For example, it can be testing the interaction with the database or making sure that microservices work together as expected. These types of tests are more expensive to run as they require multiple parts of the application to be up and running.</li></ul><p>They have mainly two differences we’ll put the focus on :</p><ul><li>Unit tests <strong>do not need any database</strong></li><li>Unit tests are <strong>faster</strong></li></ul><p>For these two reasons, in a development cycle, we run unit tests far more often than integration ones. To be able to do that we need a way to determine if a given test is an integration one or a unit one.</p><h3>How Django handles that</h3><p>Django does, in fact, provide a way to distinct unit test from integration test. Indeed, it allows us to override two different classes depending on what we need.</p><p>We inherit from <strong>SimpleTestCase</strong> for a unit test :</p><pre><em>from </em>django.test <em>import </em>SimpleTestCase</pre><pre><br><em>class</em> <strong>MyUnitTestCase</strong>(SimpleTestCase):<br>    <em>pass</em></pre><p>SimpleTestCase don’t bother to use transactions and rollback any database modifications between tests. It is the most <em>basic</em> <strong>TestCase</strong> class provided by Django.</p><p>For an integration test, we inherit from <strong>TestCase</strong>:</p><pre><em>from </em>django.test <em>import </em>TestCase</pre><pre><br><em>class</em> <strong>MyIntegrationTestCase</strong>(TestCase):<br>    <em>pass</em></pre><p>Fine, but how do you run tests ? python manage.py test will work but all tests will be run including unit tests. We need a way to <strong>run only unit tests</strong> to get rid of database handling and gain a <em>(potentially)</em> huge amount of time <em>(think about the time needed just to create the test database if you have 500 migration files). </em>The way we need is to write a test runner.</p><h3>Writing a unit test Django test runner</h3><p>By default, Django use the <strong>DiscoverRunner</strong>, we are just going to inherit from it and change some of its behaviors.</p><p>Firstly, we are going to disable all database-linked stuff. For that, we override setup_databases and teardown_databases methods and make them do nothing.</p><pre><em>from </em>django.test.runner <em>import </em>DiscoverRunner</pre><pre><em><br>class </em><strong>UnitTestRunner</strong>(DiscoverRunner):<br>    <em>def </em>setup_databases(self, <em>**</em>kwargs):<br>        <em>pass</em></pre><pre><em>    def </em>teardown_databases(self, <em>old_config</em>, <em>**</em>kwargs):<br>        <em>pass</em></pre><p>Like that, it is already working… kind of. Indeed, all tests run by this runner will not trigger any databases creation. That’s great but as all tests will be run, all the integration ones will fail. At this point, we need to filter the list of tests to keep only unit tests. To achieve that we need to override another method.</p><p><strong>DiscoverRunner</strong> uses build_suite method to create a TestSuite object that contains all the tests to run. Overriding this method allows us to perform our filtering. As TestSuite object does not allow test removal, the solution is to create a new one with the filtered list.</p><pre><em>from </em>django.test <em>import </em>TransactionTestCase<br><em>from </em>django.test.runner <em>import </em>DiscoverRunner<br><em>from</em> unittest.suite <em>import</em> TestSuite</pre><pre><em><br>class </em><strong>UnitTestRunner</strong>(DiscoverRunner):<br>    <em>def </em>setup_databases(self, <em>**</em>kwargs):<br>        <em>pass</em></pre><pre><em>    def </em>teardown_databases(self, <em>old_config</em>, <em>**</em>kwargs):<br>        <em>pass</em></pre><pre><em>    def </em>build_suite(self, <em>**</em>kwargs):<br>        suite = super().build_suite(**<em>kwargs</em>)<br>        tests = [t <em>for </em>t <em>in </em>suite._tests <em>if self.is_unittest(t)</em>]<br>        <em>return </em>TestSuite(tests=tests)</pre><pre><em>    def </em>is_unittest(self, <em>test</em>):<br>        <em>return not </em>issubclass(<em>test</em>.__class__, TransactionTestCase)</pre><p>And that’s it, the runner is ready to be used !</p><h3>Using the Runner</h3><p>The runner is now defined, but how do we use it instead of the default one ? Well, it’s quite easy. The Django test command accepts --testrunner option in which you can provide the desired runner classname. Given our UnitTestRunner is defined in unit_test_runner.py we can use the following command.</p><pre>python manage.py test --testrunner=&quot;unit_test_runner.UnitTestRunner&quot;</pre><p>It is also possible to change the default Django test runner by giving a value to the setting <a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TEST_RUNNER">TEST_RUNNER</a> . In that case, you’ll have to provide the default one to the test command when you want to run integration tests.</p><h3>Going further</h3><p>The usage shown above can be tricky as we need to write a rather <em>long</em> command and we have to remember the path and the name of the runner. To simplify this, we can define a command to <em>“replace”</em> test . We inherit from base test command and we force the testrunner value to our test runner.</p><pre><em># unittest.py<br>from </em>django.core.management.commands.test <em>import </em>Command <em>as T</em>Command<br><br><br><em>class </em>Command(TCommand):<br><br>    <em>def </em>execute(self, <em>*</em>args, <em>**</em>options):<br>        test_class = &#39;oncokdm_api.runner.unittest.UnitTestRunner&#39;<br>        <em>options</em>[&#39;testrunner&#39;] = test_class<strong><br>        </strong>super().execute(*<em>args</em>, **<em>options</em>)</pre><p>We can then run unit tests by running : python manage.py unittest .</p><p>And that’s it, thanks for reading !</p><h3>Resources</h3><ul><li><a href="https://docs.djangoproject.com/en/dev/topics/testing/advanced/#defining-a-test-runner">Advanced testing topics | Django documentation | Django</a></li><li><a href="https://www.atlassian.com/continuous-delivery/different-types-of-software-testing">The different types of testing in Software | Continuous Delivery</a></li><li><a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-TEST_RUNNER">Settings | Django documentation | Django</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba465cf3f3c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>