<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohit Tyagi on Medium]]></title>
        <description><![CDATA[Stories by Mohit Tyagi on Medium]]></description>
        <link>https://medium.com/@mohit2501tyagi?source=rss-1242a2a6ad28------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Mohit Tyagi on Medium</title>
            <link>https://medium.com/@mohit2501tyagi?source=rss-1242a2a6ad28------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:09:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohit2501tyagi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Let’s walk through Podman]]></title>
            <link>https://medium.com/@mohit2501tyagi/lets-walk-through-podman-37636cf223c5?source=rss-1242a2a6ad28------2</link>
            <guid isPermaLink="false">https://medium.com/p/37636cf223c5</guid>
            <category><![CDATA[podman]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Mohit Tyagi]]></dc:creator>
            <pubDate>Fri, 28 Aug 2020 11:06:25 GMT</pubDate>
            <atom:updated>2020-08-28T11:06:25.969Z</atom:updated>
            <content:encoded><![CDATA[<h4>Things I learnt as part of my GSoC 2020 journey</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o0V9IKa-pmoLx3TyGXkaOw.jpeg" /><figcaption>Containers without daemon : Podman</figcaption></figure><p>We all know how useful containers are in today’s world. It has been a long journey from Virtual Machines to Linux containers to Docker and other container runtimes. In this post I won’t go through the benefits of container technologies, there are excellent post on internet today for that :) . Instead we will look at Podman, an open source project developed by Red Hat which allow us to run containers without a privileged daemon. We won’t be discussing Podman vs other container runtimes. We will look at the usage of podman and some details on how it works internally and the libraries it uses.</p><h3>What is Podman?</h3><p>As stated in the official podman website, Podman is a daemonless container engine for running and managing OCI compliant containers and container images on Linux systems. It can run containers both as a root and a non-privileged user. The best part is that the command line is compatible with Docker and the images can be build using Dockerfiles as well. You can use all the commands of Docker replacing ‘docker’ with ‘podman’. More details on installation and usage are available <a href="https://podman.io/">here</a>.</p><p>Let’s get into the details of how podman works.</p><h3>Where are the images stored?</h3><p>When you run podman images, podman looks for the images already pulled on the system. The images for podman and docker are stored in different locations. Hence, podman images won’t list the images previously pulled by docker. The images for podman are stored in /var/lib/containers for root users and ~/.local/share/containers for non-privileged users. The storage location is different from docker ( /var/lib/docker ) because it’s storage structure is based on the OCI standards. The images for docker and podman are compatible, but they are stored differently on the system. We will get into the structure of the storage in the post later.</p><h3>Libpod : Home of Podman</h3><p>Podman uses <a href="https://github.com/containers/storage">containers/storage</a> as it’s storage backend and <a href="https://github.com/containers/image">containers/image</a> to manipulate images.</p><p>The core object in containers/storage is the <a href="https://github.com/containers/storage/blob/bc034cc69a6171938c42ff7868be48cb3d7293a5/store.go#L567">Store object</a>. It encapsulates the storage structure and stores all the images and layers present on the system along with other relevant metadata.</p><p>The <a href="https://github.com/containers/storage/blob/bc034cc69a6171938c42ff7868be48cb3d7293a5/images.go#L31">image object</a> and <a href="https://github.com/containers/storage/blob/bc034cc69a6171938c42ff7868be48cb3d7293a5/layers.go#L42">layer object</a> in the storage library encapsulates all the information related to an image and a layer respectively.</p><h3>Storage</h3><p>We will discuss the storage structure for overlay driver only in this post. Let’s look at an example structure of /var/lib/containers/storage (the digest values are shown only upto 6 hexadecimal digits for compactness). The following store contains the ubuntu:latest image pulled from docker hub.</p><p>/var/lib/containers/storage<br>├── libpod<br>│ └── bolt_state.db<br>├── mounts<br>├──<strong> overlay</strong><br>│ ├── 2ce3c1<br>│ │ ├── diff<br>│ │ ├── empty<br>│ │ ├── link<br>│ │ ├── merged<br>│ │ └── work<br>│ ├── 4cc8a6<br>│ │ ├── diff<br>│ │ ├── link<br>│ │ ├── lower<br>│ │ ├── merged<br>│ │ └── work<br>│ ├── 720c33<br>│ │ ├── diff<br>│ │ ├── link<br>│ │ ├── lower<br>│ │ ├── merged<br>│ │ └── work<br>│ ├── compat214010866<br>│ │ ├── lower1<br>│ │ ├── lower2<br>│ │ ├── merged<br>│ │ ├── upper<br>│ │ └── work<br>│ ├── compat567238191<br>│ │ ├── lower1<br>│ │ ├── lower2<br>│ │ ├── merged<br>│ │ ├── upper<br>│ │ └── work<br>│ ├── da0180<br>│ │ ├── diff<br>│ │ ├── link<br>│ │ ├── lower<br>│ │ ├── merged<br>│ │ └── work<br>│ └── l<br>│ │ ├── DCA3F65L7EAFZTJ55MTGPQMDSM -&gt; ../4cc8a6/diff<br>│ │ ├── JJG53NDRFWK4UKBVGWDMYNB3L3 -&gt; ../da0180/diff<br>│ │ ├── NQZ7WILONDZ22PEBUGEZ3MOFID -&gt; ../720c33/diff<br>│ │ └── VAEMYZJDRVJZDW6ZIJO76NEVCQ -&gt; ../2ce3c1/diff<br>├── <strong>overlay-containers</strong><br>│ ├── containers.json<br>│ └── containers.lock<br>├── <strong>overlay-images</strong><br>│ ├── 4e2eef<br>│ │ ├── =bWFY3YTU=<br>│ │ ├── =bWFZTU=<br>│ │ ├── =c2hM2M=<br>│ │ ├── =c2lZTU=<br>│ │ └── manifest<br>│ ├── images.json<br>│ └── images.lock<br>├──<strong> overlay-layers</strong><br>│ ├── 2ce3c1.tar-split.gz<br>│ ├── 4cc8a6.tar-split.gz<br>│ ├── 720c33.tar-split.gz<br>│ ├── da0180.tar-split.gz<br>│ ├── layers.json<br>│ └── layers.lock<br>├── storage.lock<br>├── tmp<br>├── userns.lock<br>└── volumes</p><p>We will discuss the role of directories marked bold in the above directory tree.</p><p><strong>overlay dir: </strong>The unpacked content of layers are stored in this directory. For each layer, it’s exploded rootfs is store in overlay/&lt;digest&gt;/diff, where the &lt;digest&gt; is the sha256 digest of uncompressed content of the layer. For each layer, a random id is stored in overlay/&lt;digest&gt;/link file.</p><p>The overlay/l dir contains the symlinks to the exploded rootfs of layers in overlay/&lt;digest&gt;/diff . The name of the symlink is same as the random id of the layer stored in the corresponding overlay/&lt;digest&gt;/link file.</p><p>The overlay/&lt;digest&gt;/lower file stores the random id of the parent layer (and their parent layer’s random id) separated by ‘:’</p><p>The purpose of storing the symlinks in overlay/l and random id of parents in overlay/&lt;digest&gt;/lower files will be explained later.</p><p>The overlay/&lt;digest&gt;/(merge/upper/work) are meant for overlay driver to use as work and merge and upper dir.</p><p><strong>overlay-containers dir: </strong>This stores the ‘write-able’ layers of the running container.</p><p><strong>overlay-images dir: </strong>This stores the relevant metadata about images in the local system. For each image in the local store, it stores various configuration and manifest files inside overlay-images/&lt;digest&gt;/* ,where &lt;digest&gt; is the sha256 sum of image.</p><p>Apart from this, there is a <strong>images.json</strong> file. This stores the relevant metadata about all the images in the corresponding store as json objects.</p><p><strong>overlay-layers dir: </strong>It contains a <strong>layers.json</strong> file, which stores the relevant metadata of all the layers in the corresponding store as json objects.</p><p>I’ll explain the content of images.json and layers.json file later in the post.</p><h3>How Podman uses the store</h3><p>Let’s see a few example commands to learn how different files are used by podman:</p><p>When we run podman images, podman extracts information from images.json file in overlay-images dir. It prints the images listed in this file.</p><p>When we run podman run ubuntu:latest it searches for ubuntu image in images.json file. If it not present, it pulls the image from the registry. Once the image is present in the local system, it retrieves the relevant metadata from images.json and layers.json files. Then it mounts all the layers and adds a write-able layer on top. For mounting the images, it looks for the top most layer of the image (stored in images.json). It retrieves the random id of this layer from the overlay/&lt;digestoftoplayer&gt;/link file and of all the parent layers from the overlay/&lt;digestoftoplayer&gt;/lower file. It then mounts these layers from overlay/l/&lt;randomidoflayer&gt;</p><p>When we run podman ps command, it looks into the overlay-containers/containers.json file and list all running containers in it.</p><h3>Images.json and Layers.json:</h3><p>Finally, let’s have a look at the content of images.json and layers.json files. Each store, whether read-write or read-only(see next section) has their own images.json and layers.json file.For each image, images.json stores the following metadata as json object.<br> 1. Image id<br> 2. Name<br> 3. Top most Layer id<br> 4. Creation time</p><p>For each layer in the store, layers.json stores the following metadata as json object:<br> 1. Layer id<br> 2. Creation time<br> 3. Compressed digest<br> 4. Compressed size<br> 5. Uncompressed digest<br> 6. Uncompressed size<br> 7. Parent layer</p><h3>Concept of Additional Stores</h3><p>Earlier I stated that the images are stored in /var/lib/containers/storage This is the (only) read-write image store of podman. There can be (any number of) read-only image stores, called additional stores. To use an additional store, add the absolute path to the configuration file of podman, which is located at /etc/containers/storage.conf.</p><p>The reason I explained the content of only a few directories present in var/lib/containers/storage is that, we only need these directories and files to be necessarily present in a directory for it to be a valid additional store. It may contain all the files that are present in read-write store, but they are not necessarily required.</p><p>Hence an additional store containing the ubuntu image would look like:</p><p>/path/to/additional/store<br>├──<strong> overlay</strong><br>│ ├── 2ce3c1<br>│ │ ├── diff<br>│ │ ├── link<br>│ ├── 4cc8a6<br>│ │ ├── diff<br>│ │ ├── link<br>│ │ ├── lower<br>│ ├── 720c33<br>│ │ ├── diff<br>│ │ ├── link<br>│ │ ├── lower<br>│ ├── da0180<br>│ │ ├── diff<br>│ │ ├── link<br>│ │ ├── lower<br>│ └── l<br>│ │ ├── DCA3F65L7EAFZTJ55MTGPQMDSM -&gt; ../4cc8a6/diff<br>│ │ ├── JJG53NDRFWK4UKBVGWDMYNB3L3 -&gt; ../da0180/diff<br>│ │ ├── NQZ7WILONDZ22PEBUGEZ3MOFID -&gt; ../720c33/diff<br>│ │ └── VAEMYZJDRVJZDW6ZIJO76NEVCQ -&gt; ../2ce3c1/diff<br>├── <strong>overlay-images</strong><br>│ ├── 4e2eef<br>│ │ ├── =bWFY3YTU=<br>│ │ ├── =bWFZTU=<br>│ │ ├── =c2hM2M=<br>│ │ ├── =c2lZTU=<br>│ │ └── manifest<br>│ ├── images.json<br>│ └── images.lock<br>├──<strong> overlay-layers</strong><br>│ ├── layers.json<br>│ └── layers.lock</p><p>Note the following changes from the read-write store: We don’t have upper, merge and work directories in overlay/&lt;digest&gt;/ directory any more. This is because it is a read only store, hence we can’t use them with overlay. Instead podman mounts these layers in the read-write store’s overlay directory.</p><p>Additional stores is an awesome feature of podman. Imagine you setup a read-only image store. You can then use the store over a NFS, to run images without having to pull the images into local system. As part of my GSoC (2020) project, I created an additional store in <a href="https://cvmfs.readthedocs.io/en/stable/">CVMFS</a> repository from which we can run containers without pulling them to local system.</p><p>Thank you for reading. I hope this article helps :)</p><h3>Additional Resources:</h3><ol><li><a href="https://www.redhat.com/en/blog/working-container-storage-library-and-tools-red-hat-enterprise-linux">Working with containers/storage library</a></li><li><a href="https://www.redhat.com/sysadmin/image-stores-podman">Working with images stores in podman</a></li><li><a href="https://podman.io/new/2020/03/03/new.html">Behind the scenes of rootless podman containers</a></li><li><a href="https://podman.io/new/2019/11/05/new.html">Migrating from docker to podman</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37636cf223c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>