<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Insight in Plain Sight on Medium]]></title>
        <description><![CDATA[Stories by Insight in Plain Sight on Medium]]></description>
        <link>https://medium.com/@insight-in-plain-sight?source=rss-a8a213c20e5b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qZ1MIKr-BHfsmTcKgR0_KQ.png</url>
            <title>Stories by Insight in Plain Sight on Medium</title>
            <link>https://medium.com/@insight-in-plain-sight?source=rss-a8a213c20e5b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:05:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@insight-in-plain-sight/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Transformers: Attention Mechanisms That No One Explains]]></title>
            <link>https://medium.com/@insight-in-plain-sight/transformers-attention-mechanisms-that-no-one-explains-1e360d8dc476?source=rss-a8a213c20e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e360d8dc476</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Insight in Plain Sight]]></dc:creator>
            <pubDate>Fri, 19 Jul 2024 18:43:23 GMT</pubDate>
            <atom:updated>2025-03-22T10:00:24.599Z</atom:updated>
            <content:encoded><![CDATA[<p>There are many articles out there that explain the scaled dot product attention from the <em>“Attention Is All You Need”</em> paper in great detail. Yet, I felt that some details are always left out without further explanation.</p><p>This article is for the reader who has read plenty of those but still have some lingering questions like:</p><ul><li>Why are positional encodings actually necessary? Without it where exactly is the positional information lost?</li><li>How does the look ahead mask work</li><li>Why can the attention handle arbitrary input sizes</li></ul><p>If you can answer these questions with confidence, then I commend your intellectual prowess. You may stop reading here. If you are not sure and felt that previous explanation are too vague, then I hope I can shed some light on that.</p><p>In actuality, these questions are not difficult, but without the proper visual help, it might be a little hard to imagine or explain.</p><h3>The Matrix Multiplication of Scaled Dot Product Attention</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*X1akDtK1SluVamiehZaMwQ.png" /><figcaption>Attention formula</figcaption></figure><p>We start a with list of tokens You can do it that is embedded in a 4 x 2 matrix with token length 4 and embedding dimension 2. Here I want to introduce a notation, that does not care about the actual values but the lineage of the values, so we can follow path of each token individually. To this end, we shorten values that depend a token from f(A1, A2) to simply A.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*P8UTISZUMTPyRi75-gmnsA.png" /><figcaption>x is the input to the attention layer</figcaption></figure><h4>K, V, Q</h4><p>To calculate K, V and Q, we feed x into a fully connected layer respectively. For simplicity we choose the channel size to be the same as the embedding size, but there are independent hyperparameters that may be different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*JleWPcuSc-7f5LBhs13W1w.png" /><figcaption>Linear (fully connected) layer to calculate K</figcaption></figure><p>One can observe that the fully connected layer does not cross pollinate the tokens, i.e. each token still stays in their own row. The values of course have changed but each row is still only dependent on their respective token.</p><p>The same procedure with K and V:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*JleWPcuSc-7f5LBhs13W1w.png" /><figcaption>Linear (fully connected) layer to calculate Q</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*-8t-sQ5IQiqpNVEneYunDg.png" /><figcaption>Linear (fully connected) layer to calculate V</figcaption></figure><h4>Attention Matrix</h4><p>The attention matrix is created with the multiplication of Q and K. This is the only place where the tokens speak to each other. We get the attention matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*fIkt8pyLYP8EdpgMNiWjzw.png" /></figure><p>Softmax is applied to QK but it does affect the lineage of the tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*xICP3y5kUC8tPfNA56lkaw.png" /></figure><p>In the resulting matrix each row has seen each token. This matrix can be fed into the same pipeline again. Just that it will not be the original tokens anymore, but higher level concepts that then again attend to each other.</p><p>Now we have the basics covered we are ready to inspect some properties of this operation and answer some of our initial questions.</p><h4>Why are positional encodings necessary?</h4><p>To answer why positional encodings are necessary, one must explain where the positional information is lost. Let us therefore compare two token sequences ABCD and BACD:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*hkeLVuhdjapjAqW2kY9JfQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*3DXuJr-LNOjAmZUABqy4rw.png" /></figure><p>One will first notice that that the same elements appear just in different positions. Because of the dot product the same values are just added together, and this is exactly where the positional information is lost:</p><p><em>A*A + AB*B + AC*C + AD*D = AB*B + A*A + AC*C + AD*D</em></p><p>In other words if the same values are switched around they will produce the same permuted matrix. To account for words appearing in different positions leading different values one must add positional encodings:</p><p><em>A*A*1 + AB*B*2 + AC*C*3+ AD*D*4 != AB*B*1 + A*A*2 + AC*C*3 + AD*D*4</em></p><h4>How does the look ahead mask work?</h4><p>To prevent tokens to look into the future, the look ahead mask is used. The mask is represented as a upper triangular matrix masking the values to zero in the attention matrix. When we follow the lineage of the tokens it becomes clear that the first token only sees itself but the last token can look up all past tokens through the masked attention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*ewuLiCqfM5s5prq36kkfTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*Z1ZfpLWQW3O5HxViuNONgw.png" /></figure><h4>Why can attention handle arbitrary input sizes?</h4><p>To be clear, this statement is only true for the batch size of one, which may be the case during inference if you are not processing multiple batches.</p><p>First important fact is that the learned parameters can only remain the same shape, as the weights are set in stone. But one observes that the learned parameters q, k, v can operate regardless of token length. Let us take the same q matrix but feed a 3x2 input. One can see the as long as the embedding dimensions are equal, any token length is compatible:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*WAV11RzpyUHuiKPPO7c5Hw.png" /></figure><p>The rest of the operations are not learned and adapt in similar ways to a different token size. The attention matrix QK itself does not contain learned parameters either:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*YzpaDtyaMah3FuQQGnf_sA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*qvzppU4Cz9XB38RYpQw_8A.png" /></figure><h3>Conclusion</h3><p>I hope this clears up some doubts about the attention mechanism and this article should serve as a reference for when you need to trace the path taken by each token.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e360d8dc476" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hydra Config Groups — Best Practices]]></title>
            <link>https://medium.com/@insight-in-plain-sight/hydra-config-groups-best-practices-892c5ac80a2d?source=rss-a8a213c20e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/892c5ac80a2d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hydra]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[best-practices]]></category>
            <dc:creator><![CDATA[Insight in Plain Sight]]></dc:creator>
            <pubDate>Mon, 29 May 2023 16:07:28 GMT</pubDate>
            <atom:updated>2023-05-30T11:36:58.220Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hydra Config Groups — Best Practices</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A4xjzQyeAaHBGWhpwXP83A.png" /><figcaption>Image created by Midjourney, fascinating times we live in</figcaption></figure><p>Our team has recently migrated our configuration management to Hydra and worked with it quite a while. In this article I will mainly talk about the Config Groups since I think they are the most important features. And I want to share the best practices for Config Groups.</p><p>If you are unfamiliar with Hydra, read their excellent article here:<br><a href="https://medium.com/pytorch/hydra-a-fresh-look-at-configuration-for-machine-learning-projects-50583186b710">https://medium.com/pytorch/hydra-a-fresh-look-at-configuration-for-machine-learning-projects-50583186b710</a></p><h3>Hydra Config Groups</h3><p>While hydra comes with a lot of quality of life features, there is a single feature that convinced me to migrate our codebase. The <a href="https://hydra.cc/docs/tutorials/basic/your_first_app/config_groups/">Config Groups</a>.</p><p>The idea is that not only code but also the configuration should be modular. Instead of having one monolith configuration, that covers all the cases, we can instead compose multiple smaller configurations.</p><p>To understand the benefit, let’s assume a simple example of the configuration of datasets. Imagine we have two diverse image datasets. One has additional lidar data and the other one only has pose data available. Without the concept of config groups you might come up with something like this:</p><pre>batch: 2<br>mode: train<br><br>dataset:<br>   name: image_lidar_dataset<br>   image_path: ./image_folder<br>   lidar_path: ./lidar_folder<br>   pose_path: None<br>   image_size: [30,30]<br>   lidar_size: 300</pre><pre>batch: 2<br>mode: train<br><br>dataset:<br>   name: image_pose_dataset<br>   image_path: ./image_folder<br>   lidar_path: None<br>   pose_path: ./pose_folder<br>   image_size: [30,30]<br>   lidar_size: None</pre><p>There is a lot of clutter but now imagine needing to support ten datasets, each one with their own little quirks and your configuration becomes quite unhandy. The other question you need to ask yourself is if you maintain a copy of each dataset configuration or you have one default and need to remember which lines to change to get the other configuration.</p><p>Hydra solves this by composing multiple smaller configurations. This is a great way to manage complexity. Now you can switch datasets with a single commandline argument datasets=image_pose_dataset or overwriting it in the yaml.</p><pre># Main configuration<br>defaults:<br>    # Or overwrite with image_lidar_dataset<br>    - dataset: image_pose_dataset <br><br>batch: 2<br>mode: train</pre><pre># dataset/image_lidar_dataset.yaml<br>name: image_lidar_dataset<br>image_path: ./image_folder<br>lidar_path: ./lidar_folder<br>image_size: [30,30]<br>lidar_size: 300</pre><pre># dataset/image_pose_dataset.yaml<br>name: image_pose_dataset<br>image_path: ./image_folder<br>pose_path: ./pose_folder<br>image_size: [30,30]</pre><p>Notice that each dataset configuration only contains fields that it is concerned with.</p><h3>Best Practices</h3><p>The way you setup config groups affects your software architecture. While in some cases it might be straightforward to identify Config Groups, often things are less obvious and there are many choices and you might ask yourself when or where you should introduce a configuration group. You will notice that it is similar to the problem of what responsibilities classes in OOP should have.</p><p>This is a set of core principles for effective configuration management.</p><h4>Strive for one class per configuration group</h4><p>A good rule of thumb is to mirror your code structure with your configuration. You will have dataset and ML model classes already in place, and you can simply add a configuration group for the abstract class. This makes testing also easier.</p><pre>class ImageLidarDataset:<br>    def __init__(self, image_path, lidar_path, image_size, lidar_size):<br>        pass</pre><pre># dataset/image_lidar_dataset.yaml<br>name: image_lidar_dataset<br>image_path: ./image_folder<br>lidar_path: ./lidar_folder<br>image_size: [30,30]<br>lidar_size: 300</pre><h4>Use factories to dynamically instantiate classes</h4><p>Once you have multiple configs, you will need some logic that maps the config to a class. Factories are the appropriate pattern here:</p><pre>class DatasetFactory:<br>    <br>    def create_dataset(config):<br>        if config.name == &quot;image_lidar_dataset&quot;:<br>            return ImageLidarDataset(image_path=config.image_path, ...)<br>        elif config.name == &quot;image_pose_dataset&quot;:<br>            return ImagePoseDataset(...)</pre><h4>Prefer independence over code sharing</h4><p>If you have few datasets, you might be inclined to write a single class, that just takes different arguments. After all, each dataset needs to load the image, so you would save some duplicate code and you do not need a factory:</p><pre>class GenericDataset:<br>    def __init__(self,<br>                 image_path,<br>                 lidar_path=None,<br>                 image_size=None,<br>                 lidar_size=None,<br>                 pose_path=None):<br>        pass</pre><p>This might work for two datasets. But with growing complexitiy, this design will fail. You basically circumvented Hydra’s benefit, by moving the global configuration into the constructor arguments. You introduce coupling between two unrelated datasets and changing one dataset could affect the other dataset. Which leads to our next point.</p><h4>Clearly distinguish configuration interface from output interface</h4><p>Once you introduce a factory, you will have two interfaces. One is the configuration interface. This is what is expected as input of the dataset factory and each individual dataset. There might be some fields like image_path that will be expected from every class. This is also part of the configuration interface.</p><p>The output interface is what is expected on the output of the Dataset class. This is usually a dictionary with certain fields or a list of tensors.</p><p>The important thing to realize is that configuration interface and output interface can be completely independent of each other. Realizing this, you can setup each of your datasets with whatever configuration schema you want, as long as each fulfills the output interface. This gives you great flexibility and no coupling between datasets.</p><h4>Inherit configurations to prevent configuration duplication</h4><p>One more nice thing about Hydra it in some way has inheritance implemented. This is great to prevent duplication of configurations.</p><p>For example, one common use case is the need to change multiple paths for a dataset:</p><pre># dataset/image_lidar_dataset.yaml<br>name: image_lidar_dataset<br>image_path: ./path/to/test_data<br>lidar_path: ./path/to/test_data<br>image_size: [10,10] # Reduced test data size<br>lidar_size: 300</pre><p>Changing between multiple fields is tedious busy work and sometimes error prone. The better way is to have one default configuration and overwrite only the changes in a new configuration:</p><pre># dataset/image_lidar_dataset.yaml<br>name: image_lidar_dataset<br>image_path: ./image_folder<br>lidar_path: ./lidar_folder<br>image_size: [30,30]<br>lidar_size: 300<br>hyperparameter: 42</pre><pre># dataset/test.yaml<br>defaults:<br>- image_lidar_dataset<br><br>image_path: ./path/to/test_data<br>lidar_path: ./path/to/test_data<br>image_size: [10,10]</pre><p>There are multiple benefits to this. First, now you can use a single line to switch between your actual and test data with datasets=test , compared to remember overwriting three fields manually. The second is, that test.yaml still tracks the non overwritten fields, so it is always up to date if other parameters are changing. This is a great for testing hyperparameter changes. It is also the better alternative to completely duplicating the configuration, since you would have two independent configs that you need to maintain.</p><p>One could go even further down the inheritance path and add a base configuration that documents the configuration interface.</p><pre># dataset/base.yml<br># Three question marks means required field<br>image_path: ???</pre><pre># dataset/test.yaml<br>defaults:<br>- base<br>- image_lidar_dataset<br><br>image_path: ./path/to/test_data<br>lidar_path: ./path/to/test_data<br>image_size: [10,10]</pre><p>This is similar to introducing an abstract class, because you have mandatory fields that are expected from the dataset. And not filling those fields will result in an error early at configuration time.</p><h4>If you need to change mulitple fields to achieve one type of behavior, it might indicate a need for a new configuration group.</h4><p>As seen in test.yaml, we were too lazy to change multiple fields by hand, so we introduced a new sub config. This is a general rule of thumb, where if you need to change multiple fields the same way at once, you might have a behavior that justifies its own configuration.</p><h4>Do not overwrite configurations during runtime and abuse it as global state</h4><p>This is a big anti pattern and should be self evident. It basically boils down to “don’t use global variables”. Some developers might be tempted to use the configuration object to hold some state. But usually the configuration object is passed around a lot. Not knowing where changes are made makes it very hard to track the state. And it is very discouraging that you cannot rely on what is written in the configuration, since any function might have already changed the configuration at runtime.</p><h3>Summary</h3><p>Hydra Config Groups are a powerful tool to manage complexity. Remember these best practices, when working with Config Groups</p><ul><li>Strive for one class per configuration group</li><li>Use factories to dynamically instantiate classes</li><li>Prefer independence over code sharing</li><li>Clearly distinguish configuration interface from output interface</li><li>Inherit configurations to prevent configuration duplication</li><li>If you need to change mulitple fields to achieve one type of behavior, it might indicate a need for a new configuration group.</li><li>Do not overwrite configurations during runtime and abuse it as global state</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=892c5ac80a2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Packaging with Setuptools]]></title>
            <link>https://itnext.io/python-packaging-12ef040c4ea0?source=rss-a8a213c20e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/12ef040c4ea0</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[packaging]]></category>
            <dc:creator><![CDATA[Insight in Plain Sight]]></dc:creator>
            <pubDate>Sun, 05 Mar 2023 22:42:59 GMT</pubDate>
            <atom:updated>2024-03-26T19:35:58.379Z</atom:updated>
            <content:encoded><![CDATA[<p>This article will explain the best practices for creating a Python package.</p><p>It will cover:</p><ul><li>Why we want to package our code</li><li>Evolution of Python packaging</li><li>PyScaffold</li><li>Useful configurations in setup.cfg</li><li>Build and publish</li><li>Alternative tools</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xeyqbBZg7P5kZjRK" /><figcaption>Photo by <a href="https://unsplash.com/@jiaweizhao?utm_source=medium&amp;utm_medium=referral">Jiawei Zhao</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Why do we want to package our code?</h3><p>One of the major strengths of Python is its vast package ecosystem. Everything is available at the fingertips of the developer with a simple pip install and import. Packages are a way to make code reusable and easy to share. Writing libraries instead of a collection scripts forces you to think about design and clear interfaces.</p><p>Too often, some code is stashed away in some repo. You need to clone it, install the requirements, and in general, it is not convenient to use. We should strive to do better.</p><h3>Evolution of Python Packaging</h3><h4>requirements.txt</h4><p>Most novices will start their Python journey with the requirements.txt and some loosely coupled scripts.</p><pre># Project structure<br>├── requirements.txt<br>├── script_1.py<br>└── script_2.py</pre><pre># requirements.txt<br>numpy<br>pandas==1.5.3</pre><pre># Installation<br>pip install -r requirements.txt</pre><p>Its simplicity is the main appeal, and it might be easy just to share those scripts. And it’s fine as long as these scripts work on their own. But it will be inconvenient to integrate them into another project. What then usually happens is that people will copy paste your script into their project, duplicate your code and make their own modifications.</p><p>If you want to make your code more reusable, you want to write a library, package and publish it. This way, everyone can just pip install and import your library.</p><p>Python packaging is notorious for not being well documented. There is no definitive way to do things. Even the official documentation (which has no central location) might contain outdated information. Over time, many different workflows and tools have evolved. I do have to say that the documentation has improved a lot in recent years. So how would you create a package?</p><h4>Distutils vs setuptools</h4><p>One might encounter these two names often in conjunction and it confused me a lot at the beginning. But it’s simply a historic thing, Python started out with distutils as a build-in library, It was a collection of tools for packaging, etc.. To improve on it, setuptools was created as a pip-installable package. Today, distutils is officially deprecated starting with Python version 3.10, as decided in <a href="https://peps.python.org/pep-0632/">PEP 632</a>.</p><h4>src-layout vs flat layout</h4><p>Before starting your project, you have to make a decision on your project structure. When I started with Python, figuring out the project structure was a major challenge. There are multiple ways to do it. It helped when I read this <a href="https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#src-layout">article</a>, which identified two major matterns. There is the src-layout and the flat-layout.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*mg-Ks7rV1c757xA1MTlB5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*9RQF_6yq4TXwRVbjKsW8wQ.png" /><figcaption>src vs flat-layout</figcaption></figure><p>The difference between the two is the location of the library code. In the src-layout, all the library code is stored in a src folder, separate from all the other boilerplate files, like tests and documentation. This clear separation is useful for automation and package discovery. E.g. you won’t accidentally ship the boilerplate code. The downside of this is that while developing, you need to do an editable install, because of the additional misdirection of the src folder, that would end up in your import path. (E.g. only import src.mypkg would work)</p><p>In the flat structure, you are able to directly execute the code (e.g. with python -m package.script), since everything is located in the top-level folder. But you need to be more careful with package discovery.</p><p>Both solutions are valid, for our following examples, we will exclusively use the src-layout.</p><h4>setup.py</h4><p>With setuptools, the minimal setup you can have is a single setup.py . We can move the dependencies in requirements.txt to the install_requires argument. Moreover, we need to pass some more parameters to account for the src-style layout. With packages and packages_dir, we remove src from the import path and specify that all packages are under src. For more details, you can read my article on <a href="https://medium.com/@insight-in-plain-sight/trial-and-error-package-discovery-with-setup-py-b2450dd52058">package discovery</a>.</p><pre># Project structure<br>├── setup.py<br>└── src<br>    └── tutorial<br>        ├── __init__.py<br>        ├── script_1.py<br>        └── script_2.py</pre><pre># setup.py<br>import setuptools<br>from setuptools import setup<br><br>setup(<br>    name=&quot;python_example&quot;,<br>    install_requires=[<br>        &quot;numpy&quot;,<br>        &quot;pandas==1.5.3&quot;,<br>    ],<br>    package_dir={&quot;&quot;: &quot;src&quot;},<br>    packages = setuptools.find_packages(<br>        where=&#39;src&#39;,<br>    ),<br>)</pre><pre># Editable install<br>pip install -e .</pre><p>This setup gets the job done. We can build and publish this package already. But … it is not the best practice. And we will find out why in the next sections.</p><h4>setup.py + setup.cfg</h4><p>One problem with setup.py is that it is executable. To read the metadata, setuptools need to execute this script. For automation tools it is too complicated and slow. For example, a dependency resolver would be much faster if it could read the dependencies from a machine readable configuration file than executing python code. This is why setup.cfg was introduced.</p><pre># Project structure<br>├── setup.cfg<br>├── setup.py<br>└── src<br>    └── tutorial<br>        ├── __init__.py<br>        ├── script_1.py<br>        └── script_2.py</pre><pre># setup.py<br>from setuptools import setup<br><br>setup()</pre><p>All arguments available in setup.py can be defined in setup.cfg. setup.py will automatically read the setup.cfg, when it is available.</p><pre># setup.cfg<br>[metadata]<br>name = python_example<br><br>[options]<br>packages = find_namespace:<br>include_package_data = True<br>package_dir =<br>    =src<br><br>[options.packages.find]<br>where = src</pre><p>The declarative setup.cfg is easier to parse automatically and it does not require to run Python code. It is considered to be bad practice to execute code in setup.py . My recommendation is, that you start out with setup.cfg and if you find that you reached its limitations, you can fallback to setup.py . pybind11 is an example where you have to rely on setup.py to build your C++ code.</p><h4>The build dependency issue</h4><p>There is one more problem, which again goes back to setup.py. It is the build dependency issue:</p><ul><li>You can’t run setup.py without knowing its dependencies</li><li>You can’t know the dependencies without running setup.py</li></ul><p>This is why PEP 518 was introduced, in their words:</p><blockquote>[…] when a project chooses to use setuptools, the use of an executable file like setup.py becomes an issue. You can’t execute a setup.py file without knowing its dependencies, but currently there is no standard way to know what those dependencies are in an automated fashion without executing the setup.py file where that information is stored. It’s a catch-22 of a file not being runnable without knowing its own contents which can’t be known programmatically unless you run the file.</blockquote><blockquote>Source: <a href="https://peps.python.org/pep-0518/">https://peps.python.org/pep-0518/#rationale</a></blockquote><p>The solution is a new file named pyproject.toml.</p><h4>setup.py + setup.cfg + pyproject.toml</h4><pre># Project structure<br>├── pyproject.toml<br>├── setup.cfg<br>├── setup.py<br>└── src<br>    └── tutorial<br>        ├── __init__.py<br>        ├── script_1.py<br>        └── script_2.py</pre><pre>[build-system]<br>requires = [&quot;setuptools&gt;=46.1.0&quot;, &quot;setuptools_scm[toml]&gt;=5&quot;]<br>build-backend = &quot;setuptools.build_meta&quot;</pre><p>In pyproject.toml we can specify the version of the build system. During the installation of the package, it will create an isolated environment and install the correct setuptools version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*1iIzHtLfQpsQzTnsntqyOQ.png" /><figcaption>pip install creates a isolated venv according to pyproject.toml</figcaption></figure><p>setuptools is only one out of many build backends. <a href="https://peps.python.org/pep-0518/">PEP 518</a> standardized the interface to the build backend, which enables the community to build alternative backends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*8vqThw7XLfb3vu4Bwg7J0w.png" /><figcaption>frontend and backend, source: <a href="https://peps.python.org/pep-0517/">https://peps.python.org/pep-0517/</a></figcaption></figure><h4>Use a Project Generator: PyScaffold</h4><p>Now we know that we need to create a bunch of files when starting a project. Luckily, we do not have to remember every detail. We can use a project generator like <a href="https://pyscaffold.org/en/stable/">PyScaffold</a>. It sets up the perfect project structure for you and includes the all best practices. The maintainers will stay up to date with the newest packaging developments and update the project templates accordingly.</p><p>You can simply install it with:</p><pre>pip install pyscaffold<br>putup python_example</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*gSGg7pwaJr_nZuSQ97N3Kg.png" /><figcaption>pyscaffold generated files</figcaption></figure><h3>Useful Configurations in setup.cfg</h3><p>Now that we have a proper project structure, we will discuss a few configurations that are regularly used.</p><h4>Package Name</h4><p>In setup.cfg you can give your package a name. This will determine the name that will appear under PyPI and you would install it with pip install &lt;name&gt; .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*XJt7F3FoYEfW6vrhwJgDvQ.png" /><figcaption>setup.cfg</figcaption></figure><p>We have to make a distinction here between the PyPI package and the Python package as in a collection of modules (Python files). The name you use to import the code is not related to the setup.cfg name, but is determined by your project structure under the src folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*7QlUFNV9i6-6-f2ncfpcSQ.png" /><figcaption>Project structure with multiple packages</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*BNKM8nzFl9a11LHsvcVO9A.png" /><figcaption>Import names</figcaption></figure><p>A prominent example is opencv. You would install it with pip install opencv-python but import cv2.</p><h4>Version</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*nP-workAhipgyGOJu_QFOw.png" /><figcaption>setup.cfg</figcaption></figure><p>The versioning follows the <a href="https://semver.org/">Semantic Versioning</a>. In short:</p><ul><li>MAJOR: incompatible API changes</li><li>MINOR: add functionality, backwards compatible</li><li>PATCH: backwards compatible bug fixes</li></ul><p>But most projects don’t follow these rules religiously and you should think about what makes sense from the perspective of the user.</p><p>A best practice is that you should upgrade the version on every change and should not overwrite already published packages. You might introduce breaking changes and other people’s build pipelines will break. But they will not be able to revert to a previous working version since you have overwritten it. This will block them until they adapt their codebase to your new interface. It would be easier for a 3rd party to revert to a previous version to fix their builds. Then, they can adapt their code to your changes with no pressure.</p><h4>Version - PyScaffold Pitfall</h4><p>When I use PyScaffold, there is one minor annoyance I always have to fix. The default project template will try to infer the version from your version control system. This assumes that you have one package per repo. If you have a monorepo with multiple packages you will encounter this error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iI_ZjugXjXgz8PktlUW1Jw.png" /><figcaption>Running: putup &lt;package&gt;</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IOOiIX6eOUdK3bjFIPRViw.png" /><figcaption>Running: pip install -e .</figcaption></figure><p>To fix it, follow these steps:</p><pre>putup &lt;package_name&gt; --force</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*ZEDhBv5lT3z0CX7t02uenA.png" /><figcaption>Remove the red parts</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*mH2ofYycZ65hG6MK-ljpWQ.png" /><figcaption>Remove the red parts</figcaption></figure><h4>install_requires</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*HhBxo-2nrV3Lo91WfNPu3Q.png" /><figcaption>setup.cfg</figcaption></figure><p>You can add your dependencies under install_requires. You can use &lt;=, ~=, &gt;=, == to specify version ranges.</p><p>A best practice is to add all direct dependencies i.e. the packages that you import in your code. For example, you should add numpy even though it comes with opencv, because if you decide to remove opencv from your project, your code will break because there is no numpy anymore.</p><p>Another best practice is to not be too strict on the dependency versions. You might feel inclined to use the newest version of a library, but with many dependencies, constraints might be hard to fulfill by users. Given enough dependencies the dependency resolver will not be able to find a constellation where all dependencies are compatible.</p><h4>extras_require</h4><p>Too many dependencies and you will quickly find yourself in dependency hell. Sometimes dependencies are optional to the core functionality. Like testing libraries that will not be required at runtime. With extras_requires you can define optional dependencies under a name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*C9WCsmTZ0Ef_7uL9DZwp_Q.png" /><figcaption>setup.cfg</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*qF9N4GsBQXB0fMtLQs_Kow.png" /><figcaption>Install extra dependencies</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/1*EX-ctQy4l-guJz9N1eBD0A.png" /><figcaption>Syntax for editable installs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*v8yWagPTvlEglzYY1QbOSg.png" /></figure><p>Horovod is a distributed learning framework that supports many deep learning frameworks. Installing all frameworks is quite impossible, so you are given the option to choose one with extras_require.</p><p>If you only use a single function from a package, you should consider implementing it yourself to reduce the number of dependencies.</p><h4>Namespace Packages</h4><p>Namespace packages are packages that do not contain code by themselves. They are a way to group your packages under one namespace. This is useful if you have multiple packages that you want to publish under the same name. It could be the name of your umbrella project or your organization. This way, you can also work on different packages in different repos but still publish under the same name. For example, Azure has a lot of packages that can be installed separately, but you can import them like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*etN2YdhGCwXHY4LqUxHkxw.png" /><figcaption>azure namespace package</figcaption></figure><p>PyScaffold already creates this behavior by default with the find_namespace directive. You just need to place your packages under a namespace folder. With find_namespace, __init__.py is not needed anymore, since empty folders will also be parsed as packages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*YxoFwDFHdTtgAdGxc53M5Q.png" /><figcaption>setup.cfg</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/1*LaNMR9rjH5XzqZSPbQoE8w.png" /><figcaption>Project structure</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*Y8gyrYDlQZhi_kphDk8-cQ.png" /><figcaption>Import path</figcaption></figure><h4>entry_points</h4><p>One more useful setting is entry_points. It will install an executable into your commandline. This is useful for creating CLIs. With PyScaffold you can simply uncomment the skeleton example to test this functionality. In combination with click, you can create your own CLIs in no time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*M3lbYBTx8rnfiMzTSWZJtg.png" /><figcaption>setup.cfg</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*Y-Faemne312PbHTaZjdOBg.png" /><figcaption>Script available in commandline</figcaption></figure><h3>Build and Publish</h3><p>When you are finished with your package, you should consider publishing it to the PyPI, either to the public index or consider a company hosted index for your inhouse solutions.</p><p>Use the build package for building your package:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*IYBrVLu8FHWtFIw2AcUB0Q.png" /><figcaption>Build command</figcaption></figure><p>The will create two artifacts: whl and sdist:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*243AcgPbqh8dLlHKIQAtnw.png" /><figcaption>Build artifacts</figcaption></figure><p>Wheel (whl) is the prefered format for packaging code. You can inspect the content with unzip -l &lt;package&gt;.whl. The wheel contains your files in a format that can be easily installed by unzipping the contents into the site-packages. More on wheels in <a href="https://peps.python.org/pep-0427/">PEP 427</a>.</p><p>The source distribution (sdist) is more like a copy of your project. It can also be installed, but it serves more of an archiving purpose.</p><p>You might run into some tutorials which still run setup.py manually. But this behavior is deprecated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*97MYLQOEIGU4HuAzqCXSyQ.png" /><figcaption>Deprecated way of build and install</figcaption></figure><p>The last step is to use twine to publish to a PyPI:</p><pre>pip install twine<br>twine upload dist/*</pre><h3>Alternative Tools</h3><p>What we covered previously was the basic packaging stack:</p><ul><li><strong>frontend:</strong> pip</li><li><strong>backend</strong>: setuptools</li><li><strong>build</strong>: build</li><li><strong>publish: twine</strong></li></ul><p>Due to the bumpy history of Python packaging many other tools have been created to offer a better experience. There are many, so I will only cover a few notable ones.</p><h4>Cookiecutter</h4><p>If you are not happy with the PyScaffold project template, you can use <a href="https://github.com/cookiecutter/cookiecutter">Cookiecutter</a>. Cookiecutter is a project where people can define their own reusable project templates. The idea is to create a cookiecutter and once you have it, you can use it to cut out new cookies (projects). You can either use other people’s templates or as an advanced user create your personal template.</p><h4>Poetry</h4><p><a href="https://python-poetry.org/">Poetry</a> is super popular right now and it is inspired by Rust’s Cargo package manager. It replaces the whole stack of pip, setuptools and etc. to offer a single commandline tool that can do everything.</p><h4>PDM</h4><p>Similar to poetry, pdm is inspired by another package manager, that is npm. One notable feature is that it supports <a href="https://peps.python.org/pep-0582/">PEP 582</a>, which removes the need of virtualenvs. This is an exciting new direction for Python. Maybe the concept of virtualenvs is not required at all.</p><p><strong>Update: </strong>PEP 582 was rejected. The rational being that there are already too many ways to affect the module search paths and having one more adds just more complexity compared to the small benefit for new users. Read in more detail here: <a href="https://discuss.python.org/t/pep-582-python-local-packages-directory/963/429">https://discuss.python.org/t/pep-582-python-local-packages-directory/963/429</a></p><h3>Final words</h3><p>There are many ways to do packaging. My personal recommendation is to use PyScaffold with the regular packaging stack — pip, setuptools, build, twine. If you know what you are doing consider cookiecutter. Or if you want to have a more streamlined experience and are willing to be a little experimental, go with Poetry and PDM. I say experimental but they are well established and are received well in the community. A dream would be that one day we would have a definitive way to do things, so we do not have to carefully compare different tools anymore.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12ef040c4ea0" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/python-packaging-12ef040c4ea0">Python Packaging with Setuptools</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trial and Error: Package discovery with setup.py]]></title>
            <link>https://medium.com/@insight-in-plain-sight/trial-and-error-package-discovery-with-setup-py-b2450dd52058?source=rss-a8a213c20e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2450dd52058</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[setuptools]]></category>
            <category><![CDATA[packaging]]></category>
            <dc:creator><![CDATA[Insight in Plain Sight]]></dc:creator>
            <pubDate>Sat, 03 Dec 2022 12:57:06 GMT</pubDate>
            <atom:updated>2022-12-22T16:00:09.526Z</atom:updated>
            <content:encoded><![CDATA[<h3>Preface</h3><p>This article is a accompanying guide to <a href="https://setuptools.pypa.io/en/latest/userguide/package_discovery.html">https://setuptools.pypa.io/en/latest/userguide/package_discovery.html</a>. While the documentation is by no means bad, there are some pitfalls. This article gives some examples.</p><h3>Our Goal</h3><p>We start with the bare minimum to build a package, that is an empty setup()and slowly add functionality until we understand why project generators like pyscaffold do things in a certain way.</p><h3>Setup</h3><p>For our initial folder structure, we will use the <a href="https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#src-layout">src layout</a>.</p><pre>├── src<br>│   ├── package1<br>│   │   └── main.py<br>│   └── package2<br>│       └── main.py<br>setup.py</pre><p>Install script</p><pre># clean up install<br>pip uninstall python_example --yes<br>rm -rf build dist<br><br># Build package and install, we use wheel builds to keep it clean<br>python -m build<br>pip install ./dist/*.whl<br><br># Show built files<br>unzip -l ./dist/*.whl</pre><h3>Bare-bones setup.py</h3><pre>setup(<br>    name=&quot;python_example&quot;,<br>)</pre><p>Surprisingly, the empty setup.py already builds the package in a proper way. It is not clear from the documentation what the default behavior is. You will be able to import the packages with import package1.main . Our job would be done here, but often you will find python packages in the wild with custom behavior and we need to understand how it works under the hood.</p><p>We will use the files inside of the wheel to evaluate how things are imported, because the folder structure is representative of how things are imported on the user side.</p><pre>  Length      Date    Time    Name<br>---------  ---------- -----   ----<br>        0  2022-12-02 15:12   package1/main.py<br>        0  2022-12-02 15:12   package2/main.py<br>     2182  2022-12-02 15:13   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 15:13   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 15:13   python_example-0.0.0.dist-info/WHEEL<br>       18  2022-12-02 15:13   python_example-0.0.0.dist-info/top_level.txt<br>      559  2022-12-02 15:13   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     2932                     7 files</pre><blockquote>Attention: Do not test the import in the project folder, otherwise you will import the local files and not the installed files.</blockquote><h3>Manually adding a single package</h3><pre>setup(<br>    name=&quot;python_example&quot;,<br>    packages = [&quot;src.package1&quot;],<br>)</pre><p>We can manually steer the package discovery, by passing packages to the packages argument. This way you can hand select, what ends up being delivered to the user, and choose to ignore packages. Since we use the src-layout, we need to add src to the name.</p><pre>Length      Date    Time    Name<br>---------  ---------- -----   ----<br>        0  2022-12-02 15:12   src/package1/main.py<br>     2182  2022-12-02 15:14   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 15:14   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 15:14   python_example-0.0.0.dist-info/WHEEL<br>       13  2022-12-02 15:14   python_example-0.0.0.dist-info/top_level.txt<br>      493  2022-12-02 15:14   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     2861                     6 files</pre><p><strong>The problem:</strong> The package can only be imported with import src.package1 . It is only available under the src namespace. But you actually want to import it like import package1 . package2 is not included and therefore cannot be imported (remember to execute python not in the project folder, otherwise you actually can import the local package2).</p><h3>Edge case: Passing only src</h3><pre>setup(<br>    name=&quot;python_example&quot;,<br>    packages = [&quot;src&quot;],<br>)</pre><p>We see that no src folder is created in the wheel. But, you still can import src . Why is that? There are no files named src that can be imported.</p><pre>  Length      Date    Time    Name<br>---------  ---------- -----   ----<br>     2182  2022-12-02 15:17   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 15:17   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 15:17   python_example-0.0.0.dist-info/WHEEL<br>        4  2022-12-02 15:17   python_example-0.0.0.dist-info/top_level.txt<br>      418  2022-12-02 15:17   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     2777                     5 files</pre><p>Turns out src is added to this file and therefore is available:</p><pre># In ./dist/python_example-0.0.0.dist-info/top_level.txt<br>src</pre><h3>Two subpackages</h3><pre>setup(<br>    name=&quot;python_example&quot;,<br>    packages = [&quot;src.package1&quot;, &quot;src.package2&quot;],<br>)</pre><p>As expected you will have both packages available. The problem with src.&lt;package&gt; remains.</p><pre>  Length      Date    Time    Name<br>---------  ---------- -----   ----<br>       25  2022-12-02 15:14   src/package1/main.py<br>       25  2022-12-02 15:14   src/package2/main.py<br>     2182  2022-12-02 15:25   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 15:25   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 15:25   python_example-0.0.0.dist-info/WHEEL<br>       26  2022-12-02 15:25   python_example-0.0.0.dist-info/top_level.txt<br>      569  2022-12-02 15:25   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     3000                     7 files</pre><h3>package_dir — Unexpected behavior</h3><p>The package_dir argument lets you rename your project packages, so they will appear under a new path/name on the user side. Following the examples, you might have tried this (at least I did it):</p><pre># This is NOT how to do it.<br>setup(<br>    name=&quot;python_example&quot;,<br>    packages = [&quot;src.package1&quot;, &quot;src.package2&quot;],<br>    package_dir={&quot;src&quot;: &quot;name&quot;},<br>)</pre><p>You will be surprised that name is nowhere to be found and your packages are still available under src . Also import name actually works, but import name.package1 does not.</p><pre>  Length      Date    Time    Name<br>---------  ---------- -----   ----<br>       25  2022-12-02 15:14   src/package1/main.py<br>       25  2022-12-02 15:14   src/package2/main.py<br>     2182  2022-12-02 15:38   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 15:38   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 15:38   python_example-0.0.0.dist-info/WHEEL<br>       26  2022-12-02 15:38   python_example-0.0.0.dist-info/top_level.txt<br>      569  2022-12-02 15:38   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     3000                     7 files</pre><p>The explanation is simple, change in package_dir are applied first, so the name.package1 is available during the execution of setup.py . But in the end packages dictate what is distributed and the old src.packages are still available.</p><h3>Correct way of using package_dir</h3><pre>setup(<br>    name=&quot;python_example&quot;,<br>    package_dir={<br>        &quot;name&quot;: &quot;src&quot;,<br>        &quot;name/package1&quot;: &quot;src/package1&quot;,<br>        &quot;name/package2&quot;: &quot;src/package2&quot;,<br>    },<br>    packages = [&quot;name.package1&quot;, &quot;name.package2&quot;],<br>)</pre><p>Learning from our previous mistake, we now successfully renamed our packages. So remember, what ever is in packages gets shipped.</p><pre>Archive:  ./dist/python_example-0.0.0-py3-none-any.whl<br>  Length      Date    Time    Name<br>---------  ---------- -----   ----<br>       25  2022-12-02 15:14   name/package1/main.py<br>       25  2022-12-02 15:14   name/package2/main.py<br>     2182  2022-12-02 16:38   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 16:38   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 16:38   python_example-0.0.0.dist-info/WHEEL<br>        5  2022-12-02 16:38   python_example-0.0.0.dist-info/top_level.txt<br>      570  2022-12-02 16:38   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     2980                     7 files</pre><h3>Edge case: A convoluted example</h3><pre>setup(<br>    name=&quot;python_example&quot;,<br>    package_dir={<br>        &quot;name&quot;: &quot;src&quot;,<br>        &quot;name/package1&quot;: &quot;src/package1&quot;,<br>        &quot;other_name/package2&quot;: &quot;src/package2&quot;,<br>    },<br>    packages = [&quot;src.package1&quot;, &quot;name.package2&quot;],<br>)</pre><p>We can construct a convoluted example. Can you predict what is importable?</p><pre>Archive:  ./dist/python_example-0.0.0-py3-none-any.whl<br>  Length      Date    Time    Name<br>---------  ---------- -----   ----<br>       25  2022-12-02 15:14   name/package2/main.py<br>       25  2022-12-02 15:14   src/package1/main.py<br>     2182  2022-12-02 16:32   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 16:32   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 16:32   python_example-0.0.0.dist-info/WHEEL<br>        9  2022-12-02 16:32   python_example-0.0.0.dist-info/top_level.txt<br>      569  2022-12-02 16:32   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     2983                     7 files</pre><p>Available imports:</p><p>import src, name, src.package1, name.package2</p><p>Not available:</p><p>import other_name, src.package2, name.package1</p><h3>package_dir — remove src from import</h3><pre>setup(<br>    name=&quot;python_example&quot;,<br>    package_dir={<br>        &quot;package1&quot;: &quot;src/package1&quot;,<br>        &quot;package2&quot;: &quot;src/package2&quot;,<br>    },<br>    packages = [&quot;package1&quot;, &quot;package2&quot;],<br>)</pre><p>Now that we understand the details of package_dir , we can finally use it in a practical manner. The examples with name.package were not practical. What we actually want to do was to remove the src from the import.</p><pre>Length      Date    Time    Name<br>---------  ---------- -----   ----<br>       25  2022-12-02 15:14   package1/main.py<br>       25  2022-12-02 15:14   package2/main.py<br>     2182  2022-12-02 16:42   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 16:42   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 16:42   python_example-0.0.0.dist-info/WHEEL<br>       18  2022-12-02 16:42   python_example-0.0.0.dist-info/top_level.txt<br>      561  2022-12-02 16:42   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     2984                     7 files</pre><p>One remaining problem is that we need to do it manually. This is where the helper functions of setuptools come in.</p><h3>Canonical way</h3><pre>setup(<br>    name=&quot;python_example&quot;,<br>    package_dir={&quot;&quot;: &quot;src&quot;},<br>    packages = setuptools.find_packages(<br>        where=&#39;src&#39;,<br>    ),<br>)</pre><p>We can use the special syntax {&quot;&quot;: &quot;src&quot;} to remove src from all the names. Additionally, we use find_packages() to automatically find packages.</p><p>But it fails …</p><pre>  Length      Date    Time    Name<br>---------  ---------- -----   ----<br>     2182  2022-12-02 17:00   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 17:00   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 17:00   python_example-0.0.0.dist-info/WHEEL<br>        1  2022-12-02 17:00   python_example-0.0.0.dist-info/top_level.txt<br>      418  2022-12-02 17:00   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     2774                     5 files</pre><p>One detail is that find_packages() only find packages with a __init__.py . There are two options here. One is to add the files:</p><pre>├── src<br>│   ├── package1<br>|   │   ├──__init__.py<br>│   │   └── main.py<br>│   └── package2<br>|   │   ├──__init__.py<br>│   |   └── main.py</pre><p>Or we use find_namespace_packages() which accepts packages without a __init__.py . This also explains the confusion of whether to add __init__.py to packages. Depending on your package discovery settings it might or might not matter. In my opinion, the second option is more elegant, which is also the preferred setting of pyscaffold .</p><pre># __init__.py not required<br>setup(<br>    name=&quot;python_example&quot;,<br>    package_dir={&quot;&quot;: &quot;src&quot;},<br>    packages=find_namespace_packages(where=&#39;src&#39;),<br>)</pre><p>The result is this wheel structure:</p><pre>  Length      Date    Time    Name<br>---------  ---------- -----   ----<br>        0  2022-12-02 17:02   package1/__init__.py<br>       25  2022-12-02 15:14   package1/main.py<br>        0  2022-12-02 17:02   package2/__init__.py<br>       25  2022-12-02 15:14   package2/main.py<br>     2182  2022-12-02 17:04   python_example-0.0.0.dist-info/LICENSE<br>       81  2022-12-02 17:04   python_example-0.0.0.dist-info/METADATA<br>       92  2022-12-02 17:04   python_example-0.0.0.dist-info/WHEEL<br>       18  2022-12-02 17:04   python_example-0.0.0.dist-info/top_level.txt<br>      709  2022-12-02 17:04   python_example-0.0.0.dist-info/RECORD<br>---------                     -------<br>     3132                     9 files</pre><h3>setup.cfg from pyscaffold</h3><p>The Python community is moving away from setup.py and is to setup.cfg and pyproject.toml (for more info check PEP 517, 518). We can now check how pyscaffold does packaging, which can be considered as industry standard.</p><pre># setup.py<br>setup(<br>    name=&quot;python_example&quot;<br>    ...<br>)</pre><pre># setup.cfg<br>[options]<br>zip_safe = False<br>packages = find_namespace:<br>include_package_data = True<br>package_dir =<br>    =src<br><br># Function arguments passed like this<br>[options.packages.find]<br>where = src<br>exclude =<br>    tests</pre><p>For every option in setup.py there is an equivalent API in setup.cfg . While arguments are mapped one to one, for functions there is a special syntax, that might not be intuitive at first glance.</p><p>find_packages() is mapped to find:and find_namespace_packages() is mapped to find_namespace: . Arguments for these functions are passed in a separate [options.packages.find] section.</p><h3>Conclusion</h3><p>We have examined various examples for package discovery. While it is good to know how to do it yourself, which helps you debug packages in the wild, I would not recommend creating these files yourself.</p><p>The best way for package is to use project generators like Cookiecutter, poetry or pyscaffold and follow a common project structures like src-layout.</p><h3>Resources</h3><ul><li><a href="https://setuptools.pypa.io/en/latest/userguide/declarative_config.html">Configuring setuptools using setup.cfg files - setuptools 65.6.3.post20221124 documentation</a></li><li><a href="https://setuptools.pypa.io/en/latest/references/keywords.html">Keywords - setuptools 65.6.3.post20221124 documentation</a></li><li><a href="https://setuptools.pypa.io/en/latest/userguide/package_discovery.html">Package Discovery and Namespace Packages - setuptools 67.1.0.post20230201 documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2450dd52058" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Estimating the Homography Matrix with the Direct Linear Transform (DLT)]]></title>
            <link>https://medium.com/@insight-in-plain-sight/estimating-the-homography-matrix-with-the-direct-linear-transform-dlt-ec6bbb82ee2b?source=rss-a8a213c20e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec6bbb82ee2b</guid>
            <category><![CDATA[direct-linear-transform]]></category>
            <category><![CDATA[numerical-computation]]></category>
            <category><![CDATA[svd]]></category>
            <category><![CDATA[homography]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Insight in Plain Sight]]></dc:creator>
            <pubDate>Wed, 06 Apr 2022 22:04:44 GMT</pubDate>
            <atom:updated>2022-04-06T22:04:44.050Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Yu5b4Hl5lakfCC-TN0VHsA.gif" /></figure><h3>Application of the Homography Matrix</h3><p>Given the homography matrix, one can transform an unfavorable view of a plane into a more desirable one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xhyMiTE_7rZY4Joilc_rrA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TprhChxM4Aa9xX-56eillg.jpeg" /><figcaption>Left: Source, Right: Warped with homography</figcaption></figure><p>While the chosen plane will be transformed with high accuracy, every other point in the image will suffer a wrong transformation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qluN-vNUv7-S6y2px3xb9A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N_lZ3t-BR2qe3FzGt7YhYQ.jpeg" /><figcaption>Left: Real view, Right: Artificially warped view</figcaption></figure><h3>Direct Linear Transform</h3><p>The Direct Linear Transform (DLT) is an algorithm that solves a homogeneous system. In case of estimating a homography, it takes the following form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/1*kr3XyBZOCw4hTZ6W_Of8kw.png" /><figcaption>A homogenous system, (3/4 infinite sign means proportional)</figcaption></figure><p>The distinguishing factor to a ordinary linear system of equations is that the solution is fixed up to scale.</p><h4>Algorithm summary</h4><ol><li>Get at least 4 point correspondences between two views</li><li>For each point correspondence, setup a partial <strong>A_i</strong> matrix of the form 2x9</li><li>Assemble <strong>A</strong> matrix to the form of 8x9 or 2nx9</li><li>Apply Singular Value Decomposition to the <strong>A</strong> matrix, to get <strong>U</strong>, <strong>D</strong> and <strong>V</strong> transpose</li><li>Assuming the entries in <strong>D</strong> are sorted in descending order, the last column of <strong>V</strong> is the solution to <strong>h</strong>. Reshape <strong>h</strong> 9x1 into <strong>H</strong> of the form 3x3</li></ol><h4>Derivation</h4><p>We first reformulate the problem with a cross product. For each point correspondence <em>i</em>, we get a set of equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/1*KiVGl53m6IMCxd4Q3aHVag.png" /><figcaption>Using a cross product to set up an error metric, also gets rid of the scaling factor</figcaption></figure><p>The homography <strong><em>h</em></strong> is arranged as vector:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*3zeNaQJGdg6wq3k1MM2-bA.png" /><figcaption>Homography as a 9x1 vector</figcaption></figure><p>Rearranging the cross product, so <strong><em>h</em></strong> is on the left hand side:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*HqV-gl5A1xxfXLz6gd81LQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*1vJwyjCEvBvh30PafvItAQ.png" /><figcaption>A_i is a 2x9 matrix, the 3rd row is omitted since only two rows are linearly independent</figcaption></figure><p>Assemble the partial <em>A_i</em> for each point correspondence to the full <em>A</em> matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/96/1*rDj2giZodsU9c5WRLjsNRg.png" /><figcaption>Full matrix is at least 8x9 or 2nx9</figcaption></figure><p>If there are more than 4 point correspondences, the system is usually over-determined and an exact solution is not possible. Instead, we solve a least squares problem for <strong><em>Ah:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*B_PPrR-kwl8nH1F9v4WjqA.png" /><figcaption>If the system is overdetermined we solve a least square. We set a constraint to avoid h=0 as obvious solution</figcaption></figure><p>Anyway, it makes no difference to the SVD, since it will solve the least square and return an exact solution, if n=4 (under non-degenerate conditions).</p><h4>Why the last column of V is the solution</h4><p>One might ask, why the last column must be the solution to the problem. It seems to be a rather arbitrary choice. Below are some excellent slides that derive this fact.</p><p>In essence, assuming that the diagonal matrix is sorted in descending order, the column in <strong><em>V </em></strong>corresponds to the smallest singular value. Since the singular value also determines the weight of the error, all the weight is placed onto the smallest singular value and therefore we choose the last column of <strong><em>V</em></strong> as solution. During the derivation we especially use the fact that U and V are orthonormal and therefore norm preserving.</p><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//www.cs.cmu.edu/%7E16385/s17/Slides/11.5_SVD.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b94e0c524873cfd7127bef3f7a79626/href">https://medium.com/media/4b94e0c524873cfd7127bef3f7a79626/href</a></iframe><h4>Examination of other entries in V</h4><p>One might be curious about the other entries of V. Since they explicitly do not minimize the problem, the transformation is indeed wrong. We can also observe that with bigger singular values the transformation error increases, until the transformation becomes numerically degenerate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N_lZ3t-BR2qe3FzGt7YhYQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jf1YGITQ8HWlGVNuG2mdmg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TBFsuzTWA8a0Ku6h5409bA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dVBorJX8jhZfP--wUace8Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hEeswwQqvJPvlncpA4cELA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1uvB0uOk_7VpdpKBoNuBVg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BsRe-ZMcjC9Wx4YjMD1aXQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vqgQwa6LQlcJO3RJGN3QFg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vqgQwa6LQlcJO3RJGN3QFg.jpeg" /><figcaption>From left to right and top to bottom, from column in V corresponding to the lowest to the highest singular value</figcaption></figure><p>Indeed, if you calculate the L1 error for <strong><em>|Ah|</em></strong>, you can see that they increase for each column in <strong><em>V</em></strong>.</p><p>Error |Ah| for column 9: 0.0<br>Error |Ah| for column 8: 0.4667<br>Error |Ah| for column 7: 284.903<br>Error |Ah| for column 6: 596.1555<br>Error |Ah| for column 5: 1159.5317<br>Error |Ah| for column 4: 2497.2716<br>Error |Ah| for column 3: 5652.094<br>Error |Ah| for column 2: 1316581.706<br>Error |Ah| for column 1: 7635426.2101</p><h3>Implementation of DLT in Python</h3><p>This function has the same signature as OpenCV’s <em>cv2.findHomography().</em> You can just replace it with this custom function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70f0fc62ba4ac13b201bf69cff5ffa71/href">https://medium.com/media/70f0fc62ba4ac13b201bf69cff5ffa71/href</a></iframe><h3>Literature</h3><p>Multiple view geometry in computer vision, R. Hartley, A, Zisserman</p><p>SVD for Total Least Squares, Carnegie Mellon University and Kris Kitani: <br><a href="http://www.cs.cmu.edu/~16385/s17/Slides/11.5_SVD.pdf">http://www.cs.cmu.edu/~16385/s17/Slides/11.5_SVD.pdf</a></p><h3>Related</h3><ul><li><a href="https://medium.com/@insight-in-plain-sight/deconstructing-the-homography-matrix-35989ecc0b2">Deconstructing the Homography Matrix</a></li><li><a href="https://towardsdatascience.com/simple-svd-algorithms-13291ad2eef2">Simple SVD algorithms</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec6bbb82ee2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deconstructing the Homography Matrix]]></title>
            <link>https://medium.com/@insight-in-plain-sight/deconstructing-the-homography-matrix-35989ecc0b2?source=rss-a8a213c20e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/35989ecc0b2</guid>
            <category><![CDATA[photogrammetry]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[homography]]></category>
            <dc:creator><![CDATA[Insight in Plain Sight]]></dc:creator>
            <pubDate>Tue, 29 Mar 2022 21:36:11 GMT</pubDate>
            <atom:updated>2022-03-30T23:02:39.810Z</atom:updated>
            <content:encoded><![CDATA[<p>The homography is a core concept in computer vision and multiple view geometry. It describes the mapping between two images that observe the same plane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*gjgmAuPFm0HQqMnApPMJbQ.png" /><figcaption>A homography relates a plane seen from two cameras by a linear mapping</figcaption></figure><p>Using homogeneous coordinates one can describe this mapping by a 3x3 matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/1*LasHxy8dmiOr3v1vQfGLYQ.png" /><figcaption>Homography matrix for projective 2-space</figcaption></figure><p>It might be intimidating to interpret the effects of a matrix with 9 parameters and 8 degrees of freedom at first. But we can decompose the matrix into separate parts, each with less DoF and easier to understand.</p><h3>The Transformation Hierarchy</h3><p>Therefore, we have to understand the hierarchy of transformations first. Each is more powerful than the former:</p><ul><li>Euclidean</li><li>Similarity</li><li>Affine</li><li>Projective</li></ul><p>We will cover each transform and step by step uncover the homography matrix.</p><h4>Euclidean Transform</h4><p>Euclidean geometry is very natural to us. It basically describes rigid object movement. A euclidean transform consists of a rotation and translation. It has 3 DoF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*4OYzpTcrB8ednUFXzHdnZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*gFC0y7bmg20l7_6XzVghdw.gif" /><figcaption>Example of euclidean transform, in this case pure rotation</figcaption></figure><h4>Similarity Transform</h4><p>Almost the same as Euclidean, except we scale the space in addition. It has 4 DoF. In photogrammetry, we can usually reconstruct a scene up to a similarity transform. This is the case when we have only images and no metric measurements in the outside world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*ygYBTd__fWPe0FKArK_-kQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*BYHnase0saIZ3UhU_4IuXQ.gif" /><figcaption>Example of similarity transform with increasing scale</figcaption></figure><h4>Affine Transform</h4><p>The affine transformation is already more general. It includes a similarity transform. But it can also stretch and shear the space. We can also describe a transformation by its invariants, by asking the question: “<em>What does not change</em>?”. For euclidean transform angles and sizes do not change. For similarity transform, relative angles do not change. And for affine transform parallel lines stay parallel. The affine transform has 6 DoF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*u4gXVwRTF_3nIOuHpi3YGg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*Y1AqeKidg0ghRWrntYjE_Q.gif" /><figcaption>Example of affine transform with increasing skew factors</figcaption></figure><h4>Projective Transform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/1*0ZX615gtsHrMM2hMhYcfmw.png" /></figure><p>Finally, we arrived at the projective transform. The projective transform is special, because the other transforms can be described in euclidean space. A projective transform only makes sense in projective space. So, what power does this transform hold? The most unique ability of a projective transform is to warp points at infinity. For example, parallel lines will intersect at a finite point after transformation. This especially happens if you observe an image from an other view point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*leQJD_jphpe6npld7o2QHA.gif" /><figcaption>Example of projective transform with only projective elements (affine part is identity matrix)</figcaption></figure><h3>The Chain Decomposition</h3><p>One can go one step further and isolate the specific effects of the different transforms. Remember, we are dealing with a transformation hierarchy. It means that the affine transformation includes all similarity transforms. And a projective transform includes <em>all</em> other transforms. In fact, every general projective transform can be decomposed into three parts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*tCtBXbgFOSiXGuu7sYBNlw.png" /><figcaption>Chain decomposition of a general homography matrix</figcaption></figure><p>We can see that what distinguishes a projective transform from affine is a matrix multiplication with two DoF in the <strong>v </strong>vector. With these additional two parameters we gain the “ability” to affect points at infinity. Remember that vectors with the last entry equals to 0 are ideal points, that means intersections of two parallel lines at infinity.</p><p>The affine transformation cannot modify ideal points:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*TFsvuAGOJZ7wQ4vl1y9iyg.png" /><figcaption>z-coordinate remains zero</figcaption></figure><p>The isolated projective part does just that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*ktTFOUH2SN3Rl-VzsW6ovg.png" /><figcaption>Ideal point is transformed into finite point</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMJWHViJWSQt5urbc3BJww.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZiQ-8aV3-qK4ToMDqwRNwQ.jpeg" /><figcaption>Right: Pencils are parallel, Left: Lines intersect slightly outside of the image</figcaption></figure><p>Similarly, the affine decomposition does not contain a translation vector <strong>t</strong>, since it is already covered by the similarity transform. One also observes, that similarity and euclidean transform are represented by one matrix.</p><h3>Summary</h3><p>We saw that a homography follows a hierarchy of transformation. From euclidean to similarity, affine and projective, each transformation adds a bit of functionality. Instead of understanding the matrix H as a whole, we can decompose H into a chain of transformations and isolate the specific effects.</p><h3>Code</h3><p>You can find the code for the transformations here:<br><a href="https://github.com/hq-jiang/deconstruction-of-homography-matrix">github.com/hq-jiang/deconstruction-of-homography-matrix</a></p><h3>Literature</h3><p>Multiple view geometry in computer vision, R. Hartley, A, Zisserman</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35989ecc0b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>