<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohiuddin Bilwal on Medium]]></title>
        <description><![CDATA[Stories by Mohiuddin Bilwal on Medium]]></description>
        <link>https://medium.com/@mohiuddinbilwalayon?source=rss-ae71648026a8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jUZ2QVv_0ukyAN91V5qw7Q.jpeg</url>
            <title>Stories by Mohiuddin Bilwal on Medium</title>
            <link>https://medium.com/@mohiuddinbilwalayon?source=rss-ae71648026a8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:54:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohiuddinbilwalayon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How a Rails App Handles Real Traffic: Puma, Threads, Connection Pools and Sidekiq]]></title>
            <link>https://medium.com/@mohiuddinbilwalayon/how-a-rails-app-handles-real-traffic-puma-threads-connection-pools-and-sidekiq-e5b101384bd5?source=rss-ae71648026a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5b101384bd5</guid>
            <category><![CDATA[threads]]></category>
            <category><![CDATA[connection-pool]]></category>
            <category><![CDATA[workers]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Mohiuddin Bilwal]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 05:02:10 GMT</pubDate>
            <atom:updated>2026-10-20T05:02:10.338Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been working as a Ruby on Rails backend engineer for about a year and a half. For the first stretch, I treated things like indatabase.yml Sidekiqconcurrency as magic numbers I copied from the last project. Then production started throwing errors likeActiveRecord::ConnectionTimeoutError, and I had to learn what those numbers actually do.</p><p>This article is what I wish someone had told me earlier. I’ll explain it through Rails, because that’s what I work with, but none of it is Rails-only. Django with Gunicorn, Laravel with PHP-FPM, and Node with cluster: they all have the same pieces under different names. If you can reason about these pieces, you can answer system design questions for any backend, which is something interviews ask a lot these days.</p><h3>Start with Requirements, not With Config</h3><p>System design always starts with requirements, and it’s the habit that separates “I set threads to 5 because the template said so” from “I set threads to 3 because the numbers say so.”</p><p>Take a made-up requirement:</p><ul><li>50,000 daily active users</li><li>Each user makes about 40 requests a day</li><li>Peak traffic is about 5 times the average</li><li>Average response time is around 100 ms, and 95% of requests are under 300 ms</li><li>Slow things like emails and PDF exports must not block users</li></ul><p>The first step is converting that into requests per second (RPS):</p><pre>requests per day  = 50,000 × 40            = 2,000,000<br>average RPS       = 2,000,000 / 86,400     ≈ 23<br>peak RPS          = 23 × 5                 ≈ 115 (round up to ~120)</pre><p>Now you have a number to design against. Keep it in mind, because we’ll come back to it after covering the building blocks.</p><h3>What Happens When a Request Hits a Rails App?</h3><ol><li>The browser sends a request to a <strong>load balancer</strong> (nginx, AWS ALB, etc.).</li><li>The load balancer passes it to one of your app servers.</li><li>On that server, <strong>Puma</strong> (the default Rails web server) hands it to a free <strong>thread</strong>.</li><li>The thread runs your controller. When it queries the database, it borrows a <strong>connection</strong> from ActiveRecord’s <strong>connection pool</strong>.</li><li>If you use Rails.cache or sessions backed by Redis, it borrows a Redis connection the same way.</li><li>If something slow needs to happen, the controller calls SomeJob.perform_later and moves on. A separate <strong>Sidekiq</strong> process picks the job up.</li><li>The response goes back to the user.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Ds4CDly7xmVSLahySVHS4Q.png" /><figcaption><strong>Request Travelling from Browser to Load Balancer, Puma, DB, Redis and Sidekiq</strong></figcaption></figure><p>Each step has a limit, and most production incidents are one of those limits being hit. Let’s go through them.</p><h3>Puma Workers: Using all Your CPU Cores</h3><p>A <strong>process</strong> is a running program with its own private memory. One Ruby process (MRI, the standard Ruby) can only run Ruby code on one CPU core at a time. If your server has 4 cores and you run one Puma process, three cores sit idle.</p><p>So Puma can start several copies of your app. In config/puma.rb:</p><pre>workers ENV.fetch(&quot;WEB_CONCURRENCY&quot;) { 4 }<br>preload_app!</pre><p>Each of these copies is a <strong>worker</strong>. A <strong>master</strong> process starts them by <strong>forking</strong> (copying itself) and replaces any worker that crashes. The master does not serve requests.</p><p>What this means in practice:</p><ul><li><strong>Real parallelism.</strong> Four workers can run Ruby code on four cores at the same moment.</li><li><strong>Isolation.</strong> If one worker crashes or leaks memory, the others keep serving.</li><li><strong>Memory cost.</strong> Each worker loads your whole app. A typical Rails worker uses 300 to 500 MB. Eight workers can eat 3 to 4 GB of RAM.</li></ul><p>preload_app! The app is loaded once in the master before forking. Thanks to copy-on-write, workers share the unchanged memory, which saves RAM. The catch: a network connection can&#39;t be shared across processes, so each worker must open its own database and Redis connections after the fork. Modern Rails handles the database side for you, but in older apps you&#39;ll still see this:</p><pre>on_worker_boot do<br>  ActiveRecord::Base.establish_connection<br>end</pre><p><strong>How many workers?</strong> Check the two limits and take the smaller. About one worker per CPU core, and workers × memory per worker must fit comfortably in RAM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*W3P3KPp8UxA3mN9g61xmKQ.png" /><figcaption><strong>Puma master forking several workers</strong></figcaption></figure><h3>Threads: Using the Time Spent Waiting</h3><p>A <strong>thread</strong> is a line of work inside a process. Threads in the same process share memory, and they are much cheaper than processes. In Puma:</p><pre>max_threads_count = ENV.fetch(&quot;RAILS_MAX_THREADS&quot;) { 3 }<br>threads max_threads_count, max_threads_count</pre><p>One thread handles one request at a time. A worker with 3 threads can work on 3 requests at once.</p><p>Why help at all? Because a typical Rails request spends very little of its time running Ruby. Most of it is spent waiting for PostgreSQL, for Redis, and for a payment API. While one thread waits, another can use the CPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*NaJ5bvv-e9eydfpMUb1kvg.png" /><figcaption><strong>One Thread vs Two Threads, Showing How Waiting Time Gets Reused</strong></figcaption></figure><h3>The GVL</h3><p>MRI has a <strong>Global VM Lock (GVL)</strong>. Inside one process, only one thread can run Ruby code at any instant, even if you have 8 cores. Threads take turns.</p><p>This sounds like it defeats the purpose, but the GVL is released while a thread waits on I/O (database, network). Since Rails apps mostly wait, threads still help a lot. The rule I keep in my head:</p><ul><li><strong>More CPU cores in use → add workers.</strong></li><li><strong>More waiting to overlap → add threads.</strong></li></ul><p>(JRuby has no GVL, and Go, Java, and C# threads run truly in parallel. The principle is the same: keep the CPU busy while requests wait.</p><h3>Thread Safety</h3><p>Since threads share memory, two threads can modify the same data at once. Typical pitfalls:</p><ul><li>Class variables (@@count) and mutable constants that get changed at runtime.</li><li>Memoising at the class level (def self.config; @config ||= ...; end) when the value differs per request.</li><li>Using Thread.current a global state to store the current user. Use ActiveSupport::CurrentAttributes instead, which Rails resets for each request.</li></ul><p>A database connection is also not thread-safe. Two threads must never use the same one at the same time, which is why ActiveRecord uses a pool.</p><p><strong>How many threads?</strong> Common values are 3 to 10. More threads only help while there’s waiting time to fill. Past that, you get extra memory use and contention, not speed. And every thread may need its own database connection, which leads to the next topic.</p><h3>Workers vs Threads, in Short</h3><ul><li><strong>Worker:</strong> separate memory, runs on its own core, expensive in RAM, a crash affects only that worker.</li><li><strong>Thread:</strong> shared memory, takes turns under the GVL, cheap in RAM, and a bad crash can take the whole worker with it.</li></ul><p><strong>Capacity of one server = workers × threads.</strong> Four workers with 3 threads each means 12 requests can be in progress at once. I call these <strong>slots</strong>.</p><h3>ActiveRecord Connection Pool</h3><p>Opening a database connection is slow. It involves a network handshake, authentication, and often encryption, and PostgreSQL even starts a dedicated process for each connection. That setup can take longer than the query. So Rails doesn’t open one per query. It keeps a <strong>connection pool set</strong> in database.yml:</p><pre>production:<br>  adapter: postgresql<br>  pool: &lt;%= ENV.fetch(&quot;RAILS_MAX_THREADS&quot;) { 3 } %&gt;<br>  checkout_timeout: 5</pre><p>The flow:</p><ol><li>A thread runs its first query. ActiveRecord does a <strong>checkout</strong>, giving it a connection from the pool.</li><li>The thread uses it. Nobody else can until it’s returned.</li><li>At the end of the request, Rails does a <strong>check-in</strong>. The connection stays open.</li><li>The next thread that needs one reuses it.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*zYbjLlti50XQXDX07ffZRg.png" /><figcaption><strong>Threads Borrowing Connections From The Pool</strong></figcaption></figure><p>If all connections are in use, the next thread waits. If it waits longer than checkout_timeout 5 seconds by default, you get:</p><pre>ActiveRecord::ConnectionTimeoutError: could not obtain a connection<br>from the pool within 5.000 seconds</pre><p>This is probably the most common Rails scaling error. It means either the pool is smaller than the number of threads, or something is holding connections for too long.</p><p>You can check pool health from the console:</p><pre>ActiveRecord::Base.connection_pool.stat<br># =&gt; { size: 3, connections: 3, busy: 3, idle: 0, waiting: 2, ... }</pre><p>waiting: 2 means two threads are stuck waiting for a connection right now.</p><h3>Rule 1: Pool ≥ Threads</h3><p>Set pool to at least RAILS_MAX_THREADS. That&#39;s why the template reads it from the same environment variable. If you set 5 threads and leave the pool at 3, two threads will keep waiting for nothing.</p><h3>Rule 2: One Pool Per Process, So Count Everything.</h3><p>A pool lives inside one process. Worker 1 can’t borrow from Worker 2’s pool. So every Puma worker has its own pool, and every Sidekiq process has its own too.</p><p>PostgreSQL only sees connections, and it has a hard limit (often 100 by default). All pools from all servers add up against it:</p><pre>total = (servers × workers × pool)<br>      + (Sidekiq processes × Sidekiq pool)<br>      + spare for rails console, migrations, monitoring</pre><p>If this goes over the limit, you’ll see PG::ConnectionBad or FATAL: remaining connection slots are reserved. This tends to show up the day someone adds another server or bumps Sidekiq concurrency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*-HbdKQQAQZm1BM3pbhz6gQ.png" /><figcaption><strong>Adding Up DB Connections Across Servers and Job Workers</strong></figcaption></figure><p>Two things worth knowing:</p><ul><li>Rails opens connections lazily, which pool: 5 means &quot;up to 5&quot;, not &quot;5 right now&quot;. But under load you should plan for the worst case, because that&#39;s when it hurts.</li><li>A bigger pool is not a faster app. Each connection costs memory on the database, and the database only has so many CPU cores. Hundreds of simultaneous queries don’t finish faster than dozens.</li></ul><p>When you really need more connections than Postgres can handle, put a <strong>connection pooler</strong> like <strong>PgBouncer</strong> between the app and the database. Many cheap app connections get shared over a few real ones. In transaction pooling mode (the most efficient), turn off prepared statements in database.yml:</p><pre>prepared_statements: false<br>advisory_locks: false</pre><h3>Redis: Cache, Sessions and Queues</h3><p><strong>Redis</strong> is a very fast in-memory data store. In a Rails app, it typically powers:</p><ul><li>Rails.cache (caching expensive results)</li><li>Sessions (if you don’t use cookie sessions)</li><li>Sidekiq’s job queue</li><li>Rate limiting, for example, with Rack::Attack</li></ul><p>Connections work like the database: borrowed and returned from a pool, one pool per process, with a pool size at least equal to threads. Redis answers in well under a millisecond, so this rarely becomes your bottleneck. A cache setup looks like this:</p><pre>config.cache_store = :redis_cache_store, {<br>  url: ENV[&quot;REDIS_CACHE_URL&quot;],<br>  connect_timeout: 1,<br>  read_timeout: 1,<br>  write_timeout: 1<br>}</pre><p>And you use it with:</p><pre>Rails.cache.fetch(&quot;dashboard/#{user.id}&quot;, expires_in: 10.minutes) do<br>  expensive_calculation(user)<br>end</pre><p>Good habits I’ve learned the hard way:</p><ul><li><strong>Use separate Redis instances for cache and Sidekiq.</strong> If they share one, flushing the cache can wipe queued jobs.</li><li><strong>Different eviction policies.</strong> The cache instance should use it allkeys-lru so it can drop old keys when full. The Sidekiq instance should usenoeviction, because silently deleting a job is much worse than an error.</li><li><strong>Keep timeouts short.</strong> One second is fine. A cache that’s down should look like a cache miss, not a frozen app. redis_cache_store swallows connection errors and returns nil by default, which gives you that behaviour.</li></ul><h3>Background Jobs with Sidekiq</h3><p>Some work shouldn’t happen inside a request: sending emails, generating PDFs, resizing images, calling flaky third-party APIs, big imports, and nightly reports. If a web thread does it, the user waits, and that thread can’t serve anyone else.</p><p>Rails has <strong>Active Job</strong> as a common interface, and <strong>Sidekiq</strong> (with Redis) is the most popular real worker behind it.</p><pre>class WelcomeEmailJob &lt; ApplicationJob<br>  queue_as :default<br>  retry_on Net::OpenTimeout, wait: :polynomially_longer, attempts: 5</pre><pre>  def perform(user_id)<br>    user = User.find(user_id)<br>    return if user.welcome_email_sent?</pre><pre>    UserMailer.welcome(user).deliver_now<br>    user.update!(welcome_email_sent: true)<br>  end<br>end</pre><pre># in the controller or model<br>WelcomeEmailJob.perform_later(user.id)</pre><p>How it flows:</p><ol><li><strong>Enqueue:</strong> perform_later Writes a small job description into a Redis list. It takes about a millisecond, and the user gets their response.</li><li><strong>Run.</strong> A separate Sidekiq process watches the queue. One of its threads takes the job and runs it.</li><li><strong>Retry.</strong> If it raises an error, the job goes back with a growing delay, called <strong>exponential backoff</strong>. Raw Sidekiq retries 25 times over roughly three weeks by default.</li><li><strong>Dead set.</strong> After too many failures, the job lands in the <strong>Dead</strong> set, visible in the Sidekiq web UI, where you can retry it by hand.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*XbsrG6hRHy2K5LnmGq1ozA.png" /><figcaption><strong>Enqueue, Queue, Job Worker, Retry, Dead jobs</strong></figcaption></figure><p>Sidekiq is just another process, so everything above applies. It has its own <strong>threads</strong>, set by concurrency (default 5 in Sidekiq 7, 10 in older versions), and its own database pool. The same rule holds: <strong>the Sidekiq process&#39;s DB pool must be at least its concurrency.</strong> Forgetting this is a classic cause of issues ConnectionTimeoutError in jobs.</p><pre># config/sidekiq.yml<br>:concurrency: 10<br>:queues:<br>  - [critical, 5]<br>  - [default, 2]<br>  - [low, 1]</pre><p>(Rails 8 also ships <strong>Solid Queue</strong>, which stores jobs in the database instead of Redis. It’s convenient, but your job workers then use database connections for queue work too, so count them in the same math.)</p><h3>Rules for Writing Good Jobs</h3><ul><li><strong>Pass IDs, not objects.</strong> Send user.id and call User.find inside the job. The record may have changed between enqueue and run.</li><li><strong>Make jobs idempotent.</strong> That means safe to run twice with the same outcome. Queues deliver “at least once”, so a job can run again after a crash or retry. Notice the return if user.welcome_email_sent? check above.</li><li><strong>Enqueue after the commit.</strong> If you enqueue inside a transaction, the job can run before the data is saved and find fails. Use after_commit callbacks (newer Rails versions can also delay Active Job enqueues until after commit).</li><li><strong>Keep jobs small.</strong> Split a huge import into many small jobs that can run in parallel and retry independently.</li><li><strong>Separate queues for priority.</strong> Password-reset emails shouldn’t wait behind 10,000 report jobs.</li><li><strong>Watch queue latency.</strong> If it keeps growing, jobs arrive faster than they finish.</li></ul><h3>Why a Rails App Slows Down Under Load</h3><p>When all slots are busy, new requests wait in a line in front of Puma. A user’s wait is:</p><pre>response time = time waiting in line + time being served</pre><p>With few users, the line is empty. Once slots are full, each extra user just lengthens the line. Throughput flattens, and latency shoots up. On a graph, it looks like a hockey stick.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*ifKS6v6WvDk93p7Ma_plTw.png" /><figcaption><strong>All Slots Busy, New Requests Waiting in Line</strong></figcaption></figure><p>There are two ceilings on one server, and the lower one wins:</p><ul><li><strong>Slot ceiling</strong> = slots ÷ average request time. With 12 slots and 0.1 s per request, that&#39;s 120 RPS.</li><li><strong>CPU ceiling</strong> = cores in use ÷ CPU time per request. With 4 cores and 0.03 s of pure CPU time per request, that&#39;s about 133 RPS.</li></ul><p>Adding threads raises the slot ceiling only. It can’t give you more CPU. This is the reason “just add more threads” doesn’t always help.</p><p>What to do depends on which resource is the bottleneck:</p><ul><li><strong>CPU at 100%:</strong> more Puma workers (if cores are free), more servers, less Ruby work per request.</li><li><strong>CPU low but requests slow:</strong> threads are probably waiting. More threads, with a matching pool, or move the slow part into a job.</li><li><strong>Database is the bottleneck:</strong> add indexes, kill <strong>N+1 queries</strong> with includes (the bullet gem helps find them), cache, or add a read replica.</li><li><strong>Slow external calls inside requests:</strong> background jobs.</li><li><strong>Spiky traffic:</strong> more servers, plus timeouts so overload fails fast.</li></ul><h3>Putting It Together: Designing For the Requirement</h3><p>Back to our requirement: <strong>about 120 RPS at peak, 100 ms average response, 300 ms at p95</strong>. Let’s say profiling shows that of those 100 ms, roughly 30 ms is Ruby CPU work and 70 ms is waiting on the database and Redis. Infrastructure available: servers with 4 cores and 8 GB RAM, and a Postgres instance max_connections = 100.</p><p><strong>Step 1: How many requests are in progress at once?</strong> There’s a handy rule for it, <strong>Little’s Law</strong>: concurrent requests = RPS × average time per request.</p><pre>120 × 0.1 = 12 requests in progress on average at peak</pre><p>You don’t want to run at 100% of capacity, because bursts and slow requests will queue. Aim for roughly 2× headroom, so about <strong>24 to 36 slots in total</strong>.</p><p><strong>Step 2: How much CPU do we need?</strong></p><pre>120 RPS × 0.03 s = 3.6 cores busy at peak</pre><p>At about 60% target CPU utilization, that’s roughly 6 cores. Servers with 4 cores each mean <strong>2 servers (8 cores)</strong> work, and gives us redundancy so one server dying doesn’t take the site down.</p><p><strong>Step 3: Workers and threads per server.</strong> 4 Puma workers per server (one per core). Memory check: 4 × 400 MB = 1.6 GB, fine on 8 GB. Slots needed per server are about 18, and the requests mostly wait, so <strong>3 threads per worker</strong> gives 12 slots per server, 24 in total. That’s inside our 24 to 36 target. (5 threads would give 40 slots and more DB connections for no real benefit.)</p><p><strong>Step 4: Pools.</strong> Web pool = 3 (matches threads). One Sidekiq process at concurrency 10, pool 10. Some jobs are heavy, so run 2 of them.</p><p><strong>Step 5: Add up database connections.</strong></p><pre>Web:    2 servers × 4 workers × 3  = 24<br>Jobs:   2 processes × 10           = 20<br>Spare (console, migrations, etc.)  ≈ 10<br>Total                              ≈ 54   (limit is 100)</pre><p>Comfortable. Now the growth check. If traffic doubles and we go to 4 servers: web becomes 48, total about 78. Still fine. At 6 servers we’d be near 100, which is the moment to add PgBouncer or a read replica, not after the incident. Knowing this in advance is the entire point of the exercise.</p><p><strong>Step 6: Measure.</strong> Numbers on paper are estimates. Load test it (k6, Apache Bench or Locust), then watch CPU, connection_pool.stat, p95 latency and Sidekiq queue latency. If CPU is the limit, you&#39;ve hit the CPU ceiling. If CPU is low and you see pool timeouts, the pool is wrong. If the database is busy, fix queries before adding hardware.</p><h3>Other System Design Tools, in Rails Terms</h3><p>Once the basics are solid, these are the usual levers when requirements grow:</p><ul><li><strong>Caching:</strong> low-level (Rails.cache.fetch), fragment caching in views, HTTP caching with stale?/ETags, and a CDN for assets.</li><li><strong>Database:</strong> indexes, pagination instead of loading everything, counter caches, and <strong>read replicas</strong> (Rails supports multiple databases with connects_to and automatic read/write role switching).</li><li><strong>Stateless app servers:</strong> keep sessions in cookies or Redis, never in the server’s local memory or disk, so you can add servers freely behind the load balancer.</li><li><strong>Rate limiting:</strong> Rack::Attack Protects you from abusive clients and from your own bugs.</li><li><strong>Idempotency keys:</strong> for payment or webhook endpoints, so a retried request doesn’t charge someone twice.</li><li><strong>Pagination and batching:</strong> use find_each and in_batches for large tables instead of .all.each.</li></ul><h3>A Note on AI features</h3><p>If you add an LLM call to your app, everything above becomes more important. An AI API call can take 5 to 30 seconds, which is 50 to 300 times longer than a normal request. If you make that call inside a controller:</p><ul><li>That thread is held the whole time, and with 24 slots, ten users generating text at once can use up almost half of your capacity.</li><li>If you also hold a database connection (for example, by wrapping the call inside a transaction), the pool empties much faster than you’d expect.</li></ul><p>What I’d do instead:</p><ul><li>Run the call in a <strong>background job</strong>, store the result, and show it to the user via polling or Turbo Streams / Action Cable.</li><li>Never call an external API inside a database transaction.</li><li>Handle rate limits (HTTP 429) with retry_on and backoff, and make the job idempotent so a retry doesn&#39;t double-bill you.</li><li>Cache repeated prompts and results. It saves both time and money.</li><li>Use a separate Sidekiq queue (and possibly a separate process) for AI work, so slow jobs don’t delay password-reset emails.</li><li>If you stream tokens to the browser withActionController::Live, remember each stream occupies a thread for its entire duration. Plan slots for that or use a separate server for it.</li></ul><h3>Timeouts: The Safety Net</h3><p>Without timeouts, one slow dependency can freeze every thread. Each layer needs one:</p><ul><li><strong>Postgres </strong><strong>statement_timeout:</strong> a query that runs too long gets cancelled. Set it in database.yml:</li></ul><pre>variables:<br>  statement_timeout: 5000   # milliseconds</pre><ul><li><strong>checkout_timeout:</strong> How long a thread waits for a pooled connection.</li><li><strong>Redis </strong><strong>connect_timeout / </strong><strong>read_timeout:</strong> Keep them to 1 to 2 seconds.</li><li><strong>HTTP clients:</strong> Set open and read timeouts on every outgoing call (Faraday, Net::HTTP). The Ruby defaults can be very generous.</li><li><strong>Request timeout:</strong> rack-timeout Can abort a request that runs too long.</li><li><strong>Load balancer timeout:</strong> How long nginx or the ALB waits for Rails.</li></ul><p>The rule: I<strong>nner timeouts must be shorter than outer ones.</strong> The query should give up before the request does, and the request before the load balancer gives up on the app. Otherwise, an outer layer cuts the connection while the inner layer is still working, and you get confusing errors and wasted resources.</p><h3>Common Problems and What They Usually Mean</h3><ul><li><strong>ActiveRecord::ConnectionTimeoutError:</strong> pool smaller than threads, or connections held too long (slow API call inside a transaction). Make pool ≥ threads and move slow calls out of transactions.</li><li><strong>FATAL: remaining connection slots are reserved / too many connections:</strong> all pools together exceed max_connections. Redo the sum, lower pools, or add PgBouncer.</li><li><strong>Latency climbs, CPU at 100%:</strong> out of CPU. Add workers or servers, or cut work per request.</li><li><strong>Latency climbs, CPU low, DB busy:</strong> slow queries or N+1. Fix those first.</li><li><strong>Latency climbs, CPU low, DB idle:</strong> threads stuck on something external. Add threads (and pool) or move that call to a job.</li><li><strong>Memory keeps growing until workers restart:</strong> a leak or bloat. Find it, and meanwhile restart workers periodically.</li><li><strong>Sidekiq queue keeps growing:</strong> add processes, raise concurrency (with the pool), or speed up the jobs.</li><li><strong>A job ran twice:</strong> normal at-least-once delivery. Make it idempotent.</li></ul><h3>Checklist I Use Before a Deploy that Touches Scaling</h3><ul><li>Workers are limited by CPU cores and by RAM.</li><li>RAILS_MAX_THREADS is modest (3 to 10).</li><li>DB pool ≥ threads for every Puma and Sidekiq process.</li><li>Sum of all pools + spare &lt; max_connections.</li><li>Cache Redis and Sidekiq Redis are separate, with different eviction policies.</li><li>Slow work (emails, PDFs, external and AI APIs) runs in background jobs.</li><li>Jobs take IDs, are idempotent, and enqueue after commit.</li><li>Every layer has a timeout, inner shorter than outer.</li><li>I know my current RPS, p95 latency, pool usage and queue latency.</li></ul><h3>Terms to Remember</h3><ul><li><strong>Process:</strong> a running program with its own private memory.</li><li><strong>Worker:</strong> a Puma process that serves requests (or a Sidekiq process that runs jobs).</li><li><strong>Master:</strong> the Puma process that forks workers and replaces failed ones.</li><li><strong>Thread:</strong> a line of work inside a process, sharing its memory.</li><li><strong>GVL:</strong> Ruby’s global lock; one thread runs Ruby code at a time per process.</li><li><strong>Slot:</strong> capacity for one request in progress; slots = workers × threads.</li><li><strong>Connection pool:</strong> open connections that threads borrow (check out) and return (check in).</li><li><strong>Idempotent:</strong> safe to run more than once with the same result.</li><li><strong>Exponential backoff:</strong> waiting longer after every failed retry.</li><li><strong>Little’s Law:</strong> concurrent requests = RPS × average response time.</li><li><strong>Throughput:</strong> requests completed per second.</li><li><strong>Latency (p95):</strong> how long a request takes; p95 means 95% finish faster than this.</li><li><strong>N+1 query:</strong> one query to load a list, then one more query per item.</li></ul><h3>Wrapping Up</h3><p>When a Rails app feels slow or starts throwing errors, it almost always comes down to a few things: too few workers, a bad thread count, a pool that’s too small or too big, or slow work sitting inside a request. Start from the requirement, turn it into RPS, work out slots, CPU, and connections on paper, and then measure. That habit works the same in any backend, and it’s also exactly how you’d walk through a system design interview.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5b101384bd5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the API Documentation: What I Learned While Integrating SSLCommerz in Ruby on Rails]]></title>
            <link>https://medium.com/@mohiuddinbilwalayon/beyond-the-api-documentation-what-i-learned-while-integrating-sslcommerz-in-ruby-on-rails-2fbdbde23b96?source=rss-ae71648026a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fbdbde23b96</guid>
            <dc:creator><![CDATA[Mohiuddin Bilwal]]></dc:creator>
            <pubDate>Sat, 06 Jun 2026 07:46:29 GMT</pubDate>
            <atom:updated>2026-06-06T07:46:29.088Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past few days, I have been working with Ruby on Rails to integrate a payment gateway for a SaaS service. While researching and integrating SSLCommerz into the billing workflow, I learned a lot more about payment architecture, transaction validation, webhooks, and subscription lifecycle management than I was anticipating. Before beginning the work, I assumed payment processing would be simple, such as redirecting the user to a payment site, obtaining a successful answer, and updating the subscription. However, after reading documentation, testing various scenarios, and learning how payment systems operate, I found there is a lot more happening behind the scenes.</p><p>In this article, I would like to share what I have learned from this experience. This is not a project-specific instructional or a detailed implementation manual. Rather, it is a compilation of ideas, difficulties, and experiences that helped me understand how payment gateways work and what programmers should consider when integrating them into an application.</p><h4>Difference Between a Payment Gateway and a Payment Processor</h4><p>Since I was dealing with SSLCommerz APIs, my initial focus was solely on generating payment sessions and processing payment responses. However, after reading the documentation and trying various scenarios, I found that the gateway is just one aspect of a much larger payment ecosystem. One of the first things I discovered was that people frequently use the word <strong>“payment gateway”</strong> to refer to the full payment system. In reality, various elements operate together.</p><p>Payment gateways are in charge of securely collecting payment information and sending payment requests. Behind the gateway, banks, card networks, mobile financial services, and payment processors conduct the actual transactions.</p><p>As engineers, we usually interact with the payment gateway APIs, but understanding the complete payment flow makes the implementation much easier.</p><p>A typical payment flow looks like this:<br>1. User initiates a payment.<br>2. The application creates a payment session.<br>3. The user is redirected to the payment gateway.<br>4. The payment is completed through the selected payment method.<br>5. The gateway processes the transaction.<br>6. The application receives payment confirmation.<br>7. The subscription, invoice, or purchased service is activated.</p><p>At first glance, this seems simple, but each of these steps introduces edge cases that need to be handled properly.</p><h3>The Biggest Mistake: Trusting the Redirect Response</h3><p>While testing SSLCommerz’s success, failure, and cancellation URLs, I thought the success URL would be the final source of truth. Later, after understanding more about IPN handling and server-to-server communication, I understood why most production systems use backend notifications rather than browser redirects.</p><p>For example, after completing a payment, the user is redirected to:</p><pre>https://example.com/payment/success</pre><p>Initially, this feels like the perfect place to update the payment status. However, this approach is unreliable.</p><p>A user can close the browser before the redirect occurs. The internet connection might drop. The redirect request might fail. In some cases, users may even manually visit the success URL.</p><p>This is where I learned the importance of server-to-server payment notifications.</p><p>Most modern payment gateways provide a mechanism such as:</p><ul><li>IPN (Instant Payment Notification)</li><li>Webhooks</li><li>Callback APIs</li></ul><p>These notifications are routed directly from the payment source to the application’s backend server. Because they are not dependent on the user’s browser, they are regarded as the most dependable source of payment confirmation. One of the most important lessons I learned was:</p><p><strong>Never consider a payment successful only because the user landed on the success page.</strong></p><h4>Understanding Payment States</h4><p>Before working on payment systems, I assumed a payment could only be successful or fail. In reality, there are usually multiple states, like:</p><ul><li>Pending</li><li>Processing</li><li>Successful</li><li>Failed</li><li>Cancelled</li><li>Refunded</li><li>Expired</li></ul><p>This indicates that payment status should be viewed as a state machine rather than a simple boolean value. During implementation planning, I found it helpful to consider how a payment travels from one stage to the next and what actions should be authorized in each step.</p><h4>Why Transaction Validation Matters</h4><p>Another concept that took me some time to fully understand was transaction validation. Most payment providers, including SSLCommerz, offer transaction validation APIs that enable applications to verify payment information straight from the gateway before modifying internal records. Receiving an IPN or webhook does not necessarily imply that the payment should be trusted. A malicious actor may attempt to submit fake requests to an endpoint. For this reason, payment gateways usually provide a validation API.</p><p>The general process looks like this:<br>1. Receive the payment notification.<br>2. Extract the transaction identifier.<br>3. Call the gateway validation API.<br>4. Verify the transaction details.<br>5. Update the payment status only after successful verification.</p><p>This extra validation step significantly increases the reliability of the payment workflow.</p><h4>The Importance of Idempotency</h4><p>While reading documentation and development discussions, I repeatedly came across the term <strong>“idempotency.”</strong> At first, the concept seemed inappropriately theoretical, but payment systems quickly demonstrated why it was important. In a Rails application, this usually means designing APIs and database activities so that duplicate webhook or IPN deliveries do not result in duplicate undesirable outcomes.</p><p>A payment gateway may:</p><ul><li>Retry an IPN multiple times.</li><li>Retry a webhook delivery.</li><li>Resend notifications after temporary failures.</li></ul><p>Without proper safeguards, the same transaction could be processed multiple times. Like imagine:</p><ul><li>Subscription activated twice.</li><li>Credits added twice.</li><li>The invoice was generated multiple times.</li></ul><p>To avoid these issues, every payment should have a unique transaction identifier, and processing logic should ensure the same transaction cannot be applied more than once. This became one of the most important architectural lessons I learned from payment systems.</p><h4>What Happens If the Backend Is Down?</h4><p>One question that kept coming up in my head was:<br>“<strong>What happens if the user successfully pays, but the backend server is down?”</strong></p><p>Interestingly, the money still moves. The payment provider completes the transaction regardless of whether the application server is available. The challenge is that the application may not immediately know about the payment. Fortunately, most payment gateways retry <strong>IPN</strong> or <strong>webhook</strong> delivery for a certain period. However, relying entirely on retries can be risky. Temporary outages, network issues, or unexpected failures may still cause notifications to be missed.</p><p>This is why many production systems implement reconciliation mechanisms, where transactions can be verified later using validation APIs if notifications were not received successfully. It is also common to use background jobs to periodically verify pending transactions and synchronize payment statuses when necessary.</p><p>Learning this made me appreciate how much thought goes into designing reliable payment systems.</p><h4>Subscription Billing Introduced New Concepts</h4><p>While researching subscription-based payments, I encountered several concepts that are less common in one-time payment systems.</p><p>These include:</p><ul><li>Billing cycles</li><li>Renewal dates</li><li>Trial periods</li><li>Grace periods</li><li>Subscription upgrades</li><li>Subscription downgrades</li><li>Proration</li><li>Failed renewals</li></ul><p>Initially, I thought recurring billing was simply repeating the same payment every month. In reality, subscription management often becomes more complex than payment collection itself. Even understanding these concepts helped me design cleaner billing workflows.</p><h4>Things We Should Always Consider</h4><p>After spending time working with payment integrations, these are the areas I believe deserve special attention:</p><p>1. <strong>Security<br></strong>Never trust client-side payment status. Always verify transactions through the gateway.</p><p><strong>2. Logging<br></strong>Payment issues are difficult to debug without proper logs. Track transaction identifiers, validation requests, and notification events.</p><p><strong>3. Retry Handling<br></strong>Assume external systems can fail temporarily. Design endpoints to safely handle retries.</p><h4>Database Consistency</h4><p>Payment records should remain consistent, even in the event of unexpected failures during processing.</p><h4>Monitoring</h4><p>Payments directly affect revenue. Monitoring and alerting become much more important than in many other application features.</p><h4>Resources That Helped Me Learn</h4><p>During my research, I found the following topics particularly valuable:<br>1. Payment gateway architecture<br>2. Webhooks and IPN systems<br>3. Transaction validation<br>4. Idempotency<br>5. Subscription billing systems<br>6. Event driven workflows<br>7. Payment reconciliation</p><p>Understanding these concepts often provided more value than focusing on a specific payment gateway implementation.</p><h4>Final Thoughts</h4><p>Before starting this journey, I viewed payment integration as just another API integration task. After spending time researching and implementing different payment workflows, I realized payment systems are fundamentally about reliability, consistency, and trust. The actual API calls are usually the easiest part.</p><p>The primary challenge is dealing with network failures, duplicate notifications, validation, retries, subscription lifecycle events, and numerous edge circumstances that arise in real-world applications. There are multiple strategies for creating payment systems, and different gateways provide unique features. The ideas presented here are just the lessons I gained when researching payment integrations and understanding how modern billing systems work.</p><p>Looking back, implementing the payment flow was only a small part of the task. The majority of my learning came from knowing what occurs when things go wrong and how payment systems are built to manage those circumstances consistently. I am still learning, and there’s a lot more to discover, but I hope that sharing my experience may benefit other developers and engineers who are starting their own experiments into payment systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fbdbde23b96" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Conceptual Guide to Begin with Docker — Part 4]]></title>
            <link>https://medium.com/@mohiuddinbilwalayon/a-conceptual-guide-to-begin-with-docker-part-4-35c3b60d8a03?source=rss-ae71648026a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/35c3b60d8a03</guid>
            <category><![CDATA[docker-ecosystem]]></category>
            <category><![CDATA[kernel]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-engine]]></category>
            <category><![CDATA[docker-cli]]></category>
            <dc:creator><![CDATA[Mohiuddin Bilwal]]></dc:creator>
            <pubDate>Fri, 03 Apr 2026 13:38:45 GMT</pubDate>
            <atom:updated>2026-04-03T13:38:45.415Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous part, I explained the concepts of Virtual Machines, Hypervisors, and Memory Ballooning, what they are, how they work, and why they are important for understanding Docker. In this article, I will introduce the Docker ecosystem. To understand the Docker platform in depth, it is important to know how the workflow operates when a Docker command is executed in the terminal. By the end of this article, you should have a clear high-level understanding of how the Docker ecosystem works behind the scenes.</p><p>Before we begin, we need to understand the ecosystem. In general, an ecosystem refers to a group of different components that interact and work together within a common environment to perform a particular function. Each component has its own role, but the overall system works only when these components cooperate with each other. In technology, the term ecosystem is often used to describe a collection of tools, services, and platforms that operate together to support a larger system. Understanding how these components interact helps us see how the entire platform functions as a whole.</p><p>In the context of Docker, the ecosystem refers to the set of tools, services, and components that work together to build, manage, and run containers efficiently. As mentioned earlier, Docker can be viewed as a platform that consists of several interconnected components. These components interact with each other to execute Docker commands and manage containerized applications smoothly. The main parts of the Docker ecosystem include:</p><ol><li>Docker Client</li><li>Docker Engine</li><li>Docker Desktop</li><li>Docker Images</li><li>Docker Hub</li><li>Docker Compose</li></ol><p>To understand the workflow of the Docker ecosystem, let’s consider a simple example. Imagine running the command <strong>docker run hello-world</strong> in your Linux terminal. Within a few moments, several lines of output appear in the terminal. From the user’s perspective, this process appears straightforward. However, behind the scenes, multiple components of the Docker ecosystem work together to execute this command. Let’s take a closer look at how this process actually works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WuHyT0ONR1AYmiR1E4V8RQ.png" /></figure><p>When a user runs the command docker run hello-world, the request is first received by the <strong>Docker CLI (Docker Client)</strong>. The Docker CLI then sends this request to the <strong>Docker Engine</strong> through a REST API.</p><p>A <strong>REST API</strong> (Representational State Transfer Application Programming Interface) is a communication mechanism that allows two software systems to interact with each other over a network.</p><p>The Docker Engine itself consists of several important components: <strong>dockerd (Docker Daemon)</strong>, <strong>containerd</strong>, and <strong>runc</strong>. Once the request reaches the Docker Engine, it is handled by <strong>dockerd</strong>, which is responsible for managing Docker objects such as images, containers, and networks.</p><p>After receiving the request to run the hello-world container, <strong>dockerd</strong> instructs <strong>containerd</strong> to check whether the required image already exists on the local system. If the image is not available locally, containerd requests the image from <strong>Docker Hub</strong>.</p><p>Docker Hub is a public registry that stores and distributes Docker images. If the hello-world image is available there, it is downloaded to the local system.</p><p>Once the image is available, <strong>containerd</strong> passes the task to <strong>runc</strong>, which is responsible for actually creating and running the container. The runc tool interacts with the <strong>Linux kernel</strong>, using features such as <strong>cgroups</strong> and <strong>namespaces</strong>, to create an isolated container environment.</p><p>If the kernel is of a different operating system, <strong>Docker Desktop</strong> creates a virtual environment like Linux where Docker Engine can work.</p><p>Inside this container environment, the <strong>hello-world</strong> program runs as a process. After executing, it generates an output message. This output is then sent back through the Docker Engine to the <strong>Docker CLI</strong>, which finally displays the response in the terminal for the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*EcourfVXglGh6RK7ot6SAg.png" /></figure><p>If you run the command docker run hello-world in your terminal, you will see output similar to the lines shown above. If you look closely, you will notice that Docker is attempting to pull the image from library/hello-world. The library namespace refers to the official images hosted on Docker Hub.</p><p>Since the image is not available on the local system during the first execution, Docker downloads it from Docker Hub. Once the image is downloaded, it is stored locally on the system by the Docker engine.</p><p>After the image is available locally, containerd uses it to create and run the container. If you run the same command again, Docker will not download the image a second time because it already exists locally. Instead, it will directly use the stored image to create the container, which makes the workflow much faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*rKTCOmS2rWFH8VhRuOpV7g.png" /></figure><p>Through this example, we can see how different components of the Docker ecosystem work together to execute a simple command. What appears to be a straightforward operation in the terminal actually involves several processes happening behind the scenes. The Docker CLI sends the request, the Docker Engine processes it, containerd manages the image, and runc interacts with the Linux kernel to create and run the container. This coordinated workflow allows Docker to run applications in isolated environments efficiently. Understanding this ecosystem is important because it provides a clear picture of how Docker manages containers and simplifies application deployment in modern computing environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35c3b60d8a03" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Conceptual Guide to Begin with Docker — Part 3]]></title>
            <link>https://medium.com/@mohiuddinbilwalayon/a-conceptual-guide-to-begin-with-docker-part-3-fe35565ac4a5?source=rss-ae71648026a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe35565ac4a5</guid>
            <category><![CDATA[virtual-machine]]></category>
            <category><![CDATA[hypervisors]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[virtualization]]></category>
            <dc:creator><![CDATA[Mohiuddin Bilwal]]></dc:creator>
            <pubDate>Sat, 03 Jan 2026 16:45:44 GMT</pubDate>
            <atom:updated>2026-01-03T16:45:44.096Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Conceptual Guide to Begin with Docker — Part 3</h3><p>In the previous part, I explained the Operating System Kernel, how it works, and why it is so important for understanding Docker. In this article, I will focus on another important concept, the <strong>Virtual Machine</strong>. We will explore what a Virtual Machine is, how it works, and why it played a significant role before container based technologies like Docker became popular. By the end of this part, you should have a clear high level understanding of Virtual Machines and their place in modern computing.</p><p>Before understanding Virtual Machines, let’s recall how a computer normally works. A physical machine consists of hardware components like CPU, RAM, and storage. On top of this hardware, we install an Operating System such as Linux or Windows. This operating system allows us to install applications and perform daily tasks. In earlier days, if we wanted to run multiple operating systems, we needed multiple physical machines. This was due to hardware limitations, bootloader restrictions, and the lack of advanced virtualization technology. As computers became more powerful, a software based solution was developed that allowed multiple operating systems to run on a single physical machine. This solution is known as <strong>virtualization</strong>.</p><p>A <strong>Hypervisor</strong> is a special software that enables virtualization. It allows a single physical machine to host multiple Virtual Machines (VMs). Each Virtual Machine behaves like a separate computer, even though it runs on the same physical hardware. A <strong>Virtual Machine</strong> is a software based computer that has its own virtual CPU, memory, storage, and network. Inside each VM, a full operating system runs, which is known as the Guest Operating System. From the perspective of the guest OS, it feels like it is running on a real physical machine. Let’s see the overall picture below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C3rEz_zyJygNBjvS5qxvXA.png" /></figure><p>Assume that the hypervisor is an application whose code is stored on the hard disk. When the system starts, the operating system schedules the hypervisor like any other process, and the kernel executes it. The hypervisor then creates virtual versions of hardware components such as CPU, RAM, and storage, and maps them to the underlying physical resources.</p><p>When the hypervisor is running, the operating system does not need to understand the internal logic of virtualization. It simply executes the hypervisor’s instructions like any other program. Similarly, the CPU only executes machine instructions provided by the operating system and does not need to know whether those instructions belong to a normal application or a hypervisor. Through this mechanism, the hypervisor is able to manage and run multiple virtual machines on top of a single physical system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5l31IicqovojqF48utivcw.png" /></figure><p>Let’s assume a physical machine has a 6 core CPU, 10 GB RAM, and 100 GB storage. A hypervisor can create multiple virtual machines and assign virtual resources to each of them. For example, one VM might be assigned 4 CPU cores, 4 GB RAM, and 50 GB storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXoFMX02ljffN8dHD3G5Ag.png" /></figure><p>Although the total virtual resources assigned to VMs may appear to exceed the physical limits, the hypervisor manages this intelligently. It schedules CPU time, allocates memory dynamically, and ensures that all virtual machines can run efficiently on the same physical system. One common technique used by the hypervisor to manage memory efficiently is called <strong>memory ballooning</strong>. In this approach, the hypervisor can temporarily reclaim unused memory from one virtual machine and provide it to another virtual machine that needs more memory. This allows multiple virtual machines to run smoothly on the same physical system, even when total assigned memory appears to exceed the physical RAM.</p><p>Virtual Machines were a major breakthrough because they allowed better hardware utilization, isolation between systems, and flexibility in deployment. However, each VM requires its own full operating system, which makes them relatively heavy in terms of resource usage. Understanding this limitation helps us later understand why Docker and containers took a different and more lightweight approach.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe35565ac4a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Conceptual Guide to Begin with Docker — Part 2]]></title>
            <link>https://medium.com/@mohiuddinbilwalayon/a-conceptual-guide-to-begin-with-docker-part-2-edf415443ad5?source=rss-ae71648026a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/edf415443ad5</guid>
            <category><![CDATA[kernel]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Mohiuddin Bilwal]]></dc:creator>
            <pubDate>Sat, 27 Dec 2025 19:00:31 GMT</pubDate>
            <atom:updated>2025-12-27T19:00:31.686Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Conceptual Guide to Begin with Docker — Part 2</h3><p>In the previous part, I gave an overall idea of what Docker actually is and how it works at a high level. However, to truly understand the Docker platform and why it is so powerful, we need to look a bit deeper into how a computer works internally. To get that overall feeling, it is important to understand some basic computer architecture concepts and how the system performs tasks behind the scenes. That is why, in this part, I will focus on a very important concept: the <strong>Kernel</strong>. This part plays a key role in understanding how Docker works so efficiently and how it is different from traditional approaches. Once the kernel is clear, the core idea of Docker becomes much easier to grasp.</p><p>If we look at a computer system at a basic level, it mainly consists of three parts: <strong>CPU (Central Processing Unit), RAM (Random Access Memory), and HDD (Hard Disk Drive)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AP3KsbImc6yRSk8EQCQy7A.png" /></figure><p>I will explain the kernel with a simple example. Suppose we install a Linux operating system on our device, as you can see in the picture above. Initially, all the operating system codes are stored on the hard disk. When we turn on the PC, the required parts of these codes are loaded from the hard disk into the RAM. As we can see, a portion of the RAM is now occupied by the operating system. For this example, let’s assume a CPU with multiple cores and threads. Each core contains a processing unit and a set of registers. The processing unit performs the actual operations, while the registers temporarily hold data, instructions, and memory addresses needed for immediate execution. The CPU starts executing the operating system instructions step by step, and this is how we see the operating system interface on our screen. The part of RAM where these core operating system codes run is called the <strong>Kernel Space</strong>. Now, when we want to run other applications or processes, such as a browser or a text editor, they are loaded into the remaining part of the RAM, which is known as the <strong>User Space</strong>. These applications do not get direct access to the entire system. The operating system has full control over the CPU and system resources, and the most critical part of the operating system is the <strong>Kernel</strong>. The kernel manages the system at the lowest level and controls how hardware and software interact. That is why the kernel is often called the <strong>brain of the operating system</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D2XnF682ZgSWoS4gj18Teg.png" /></figure><p>Now, let’s understand the work of the kernel more simply. Before that, we need to know that a CPU operates in two different modes: <strong>User Mode</strong> and <strong>Kernel Mode</strong>. When the CPU runs normal applications or processes, such as a browser, editor, or any user program, it runs in <strong>User Mode</strong>. These processes live in the <strong>User Space</strong> of the RAM. On the other hand, when the CPU runs core operating system tasks, it switches to <strong>Kernel Mode</strong>, where the kernel operates inside the <strong>Kernel Space</strong>. As shown in the image above, multiple applications run in the user space. If any of these applications need to access system resources, such as reading a file from the hard disk, they cannot do it directly. Direct access is restricted for safety and security reasons. Instead, the application makes a <strong>System Call</strong> to the kernel. A system call is a controlled request asking the kernel to operate on behalf of the application. The kernel receives this request and checks whether the requested file exists on the hard disk. If it does, the kernel then verifies whether the requesting process has the necessary permissions.</p><p>If all conditions are satisfied, the kernel allows the operation, such as granting <strong>read access</strong>, and returns the result to the application. In this way, the kernel acts as a <strong>middle communicator</strong> between user applications and hardware resources like the hard disk. This mechanism is extremely important. Without the kernel enforcing these rules, one application could access or modify another application’s files, leading to serious security and stability issues. That is why this structure is critical in any device or server where multiple applications run at the same time.</p><p>Kernel not only does the communicator part, it also does the following activities:</p><p>(i) <strong>Process Management</strong> — Responsible for controlling the life cycle of processes like process creation, scheduling, execution, context switching, etc.<br>(ii) <strong>Memory Management —</strong> Responsible for managing RAM efficiently and safely, like memory allocation, deallocation, virtual memory management, memory protection, pagination, and segmentation, etc.<strong><br></strong>(iii) <strong>Device Management — </strong>Responsible for handling hardware devices through drivers like device driver management and I/O operations.<br>(iv) <strong>File System Management — </strong>Responsible for organizing and securing files and directories, like file creation, deletion, and read and write operations.<br>(v) <strong>System Call — </strong>Responsible for providing a safe interface between user space and kernel space, which has been described above. <br>and so on. The kernel plays a crucial role in Docker because Docker does not create a new operating system for each application. Instead, it uses the host machine’s kernel to isolate processes, manage memory, control file access, and handle system calls. This is what makes containers lightweight, fast, and efficient compared to traditional virtual machines.</p><p>In this part, I gave an overall idea about what a kernel is, how it works, and how it makes an impact on the Docker platform. In the next part, I will give a conceptual understanding of Virtual Machine and its roles in the Docker system. Till then, keep grinding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edf415443ad5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Conceptual Guide to Begin with Docker—Part 1]]></title>
            <link>https://medium.com/@mohiuddinbilwalayon/a-conceptual-guide-to-begin-with-docker-part-1-d2f0ddde7cfe?source=rss-ae71648026a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2f0ddde7cfe</guid>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Mohiuddin Bilwal]]></dc:creator>
            <pubDate>Sat, 29 Nov 2025 08:38:59 GMT</pubDate>
            <atom:updated>2025-11-29T08:38:59.709Z</atom:updated>
            <content:encoded><![CDATA[<p>As software engineers or any other technical person, we often come to know about Docker, right? But while studying the concepts of Docker initially, everything looks very messy. At least, I faced it a lot, honestly. But is it really hard to understand? The knowledge I gathered till now, and with the practical implementation I did, I found it a really interesting concept to work with, and it’s really fun to get into the depth of it. So, let’s move into the concept quickly.</p><p>So, what is Docker? It’s a common question to have before getting started with Docker. If you look carefully at the term ‘Docker’, the prefix of this term is ‘Dock’, which has come from the concept of the port of a ship. And Docker is actually the person who loads or unloads the goods from the ship at the dock. Really confusing, right? If you look at the figure below, you can get some idea of what I am talking about. Look at how people are loading goods in the port of the ship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*_YYqbQWoDkvm150-nLmXhA.jpeg" /></figure><p>But according to the technical perspective, Docker is a platform that loads or unloads something. Let me simplify this. Imagine you’re developing an application on your Linux computer, but when you try to run the same application on someone else’s Windows machine, it fails. Maybe a library is missing, maybe the configuration is different, or maybe the versions don’t match. These “works on my machine” problems used to be very common in software development. Docker solves this problem by packaging your application together with everything it needs, like its libraries, tools, and configuration, inside something called a <strong>container</strong>. A container is something that shares the host system’s kernel, the software, and the dependencies your application actually requires. This makes containers lightweight, portable, and able to run consistently on any machine that has Docker installed, regardless of whether the host is Linux, Windows, or macOS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*SbRN_YdMwvAWBp7Tnwj8rQ.png" /></figure><p>So, while knowing about ‘container,’ many questions can arise. What happens when there are multiple containers? How does Docker handle the containers? Does any container hamper other containers while running? Well, Docker never runs containers on its own. Each container has a different ID, name, file system, and runtime environment. When you give a command on your terminal, <strong>docker run application_name</strong>, then Docker just follows your instruction. Inside Docker, there is a system named <strong>Docker Engine,</strong> which includes <strong>Docker Daemon</strong>, which decides which container will be made from which image and which container will run according to the command or instruction you give to the terminal. At this point, it can be really confusing to know about all these Docker terms; questions can arise about what these are and how they work. Let’s get deep into these concepts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9F8_dnaTP-BLN1Gt-beXUg.png" /></figure><p>Suppose there are four containers in a Docker platform on PC 1. Now, you just want to run the Container 2 application on another device, PC 2. What will you do? Just take a screenshot of Container 2 from PC 1, which will be an image or a copy of that container. Now pass the image to PC 2, run it, and it will create a new container on PC 2, which will actually be the same Container 2 that was on PC 1. A <strong>Docker Image</strong> is like the packaged version of your application, containing the app code, libraries, and dependencies. Now, if we change anything inside container 2 of PC 1, it won’t be changed on PC 2, because we didn’t provide the updated copy of the container on PC 2, so it will work similarly it was working before the update was made on that container. Another interesting part of PC 1 is that no container will hamper the other containers, because no container knows what is going on in the other containers. Each container is isolated, with its own file system, processes, network, and unique ID, so it runs independently from other containers.</p><p>Now, an important question. What is the difference between a <strong>Docker Container</strong> and a <strong>Docker Image</strong>? A Docker image is more like a blueprint or a set of instructions for your application. It is a static file where your application code, libraries, and other required parts are set up. On the other hand, a container is a created running application from that image, which is actually the running environment. You can create as many containers from an image as you want.</p><p>Earlier, I mentioned the command <strong>docker run application_name</strong>.<br>But what actually happens when you run this command? Before understanding what happens when you type the command, you need to know the main characters involved in this process. The first character is the <strong>Docker CLI,</strong> which is actually the command line tool where you type Docker commands like ‘<strong>docker run’</strong>.<strong> </strong>It only works by sending commands. The second one is the <strong>Docker Daemon</strong>, which is like Docker’s main brain that runs in the background. Whatever command you give through the CLI, the Docker Daemon receives it and decides what to do next. Both of these together are part of a bigger system called the <strong>Docker Engine</strong>, the core technology that allows Docker to build images, run containers, and manage all the resources underneath.</p><p>Now, Docker also works with two other important concepts: <strong>Docker Images</strong> and <strong>Docker Containers</strong>, which I’ve already explained before. A <strong>Docker Image</strong> is like a blueprint containing your application code, libraries, and dependencies. A <strong>Docker Container</strong> is the running instance of that image. Every container runs in its <strong>own isolated environment</strong>, which means it has its own file system, processes, network settings, and unique ID (You can find it as a Process ID). Think of it like each container living in its own flat inside your computer. It has its own “room” with its files, its own tasks running, its own network connections, and its own name tag. Because of this isolation, containers run independently and do not interfere with each other, even if multiple containers are running on the same machine at the same time.</p><p>Docker also connects to a huge online library called <strong>Docker Hub</strong>, where official and community made images are stored. You can think of it like a library where different domains of books can be found if searched. If Docker needs an image and it’s not found on your local computer, it automatically looks for it on <strong>Docker Hub</strong> and downloads it. This ensures that your container can be created and run anywhere, without missing any required files or dependencies.</p><p>With these characters introduced, the story becomes clear. When you type ‘<strong>docker run application_name’ </strong>in the CLI, the <strong>Docker Daemon</strong> inside the <strong>Docker Engine</strong> starts working step by step. <strong>Docker Daemon, </strong>which is also known as<strong> dockerd, </strong>is the brain of Docker, which always runs on your computer and executes all of the Docker work. When the command is given to the <strong>Docker CLI,</strong> the CLI doesn’t control the daemon directly; instead, it converts the command into an API request. The <strong>Docker Daemon</strong> exposes a <strong>REST API,</strong> usually over a Unix socket or TCP, where the CLI sends the request. This request contains all the instructions, like which image to use, container name, ports, environment variables, etc. The <strong>daemon</strong> reads the API request, decides what needs to happen, and executes it. First, it checks if the required image is available on your local device. If it’s not, the daemon pulls the image from <strong>Docker Hub</strong>. Once the image is ready, the <strong>Docker Daemon</strong> creates a new container from that image. Then it starts the container, giving it its own isolated environment so the application can run smoothly. After completing all these tasks, the daemon sends the results back to the CLI, such as the container ID, success or error messages, logs, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*m3z9L-C-9lKv7mzejPzGfg.png" /></figure><p>By understanding all these concepts together, Docker CLI, Daemon, Engine, Image, Container, and Docker Hub, I believe you will get a complete picture of how Docker works behind the scenes. Now, when you see the simple command like <strong>docker run, </strong>you can visualize the entire sequence happening step by step, just like a story where every character has a role. That’s all in this part. In the next part, I will discuss more about Docker methodologies with some new commands and behind the scene scenarios. Until then, be well and keep learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2f0ddde7cfe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Converting 360-Degree Video Frames to Equirectangular Format on Linux]]></title>
            <link>https://medium.com/@mohiuddinbilwalayon/converting-360-degree-video-frames-to-equirectangular-format-on-linux-23c16b7622b7?source=rss-ae71648026a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/23c16b7622b7</guid>
            <dc:creator><![CDATA[Mohiuddin Bilwal]]></dc:creator>
            <pubDate>Sun, 18 May 2025 16:41:38 GMT</pubDate>
            <atom:updated>2025-05-22T16:52:48.581Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to convert 360-degree video frame-wise images to equirectangular format on Linux</h3><p>Over the past few days, I’ve been working on extracting metadata from 360-degree videos. While it’s true that modern GoPro cameras make this possible by embedding rich metadata through advanced built-in sensors, that’s not the main focus of my article.</p><p>Instead, I want to highlight an issue I’ve encountered while working with 360-degree videos on Linux. If you’ve ever tried playing these videos using a standard media player on a Linux system, you might have noticed something unusual: the video often appears as a flat, stitched-together view of multiple angles. Though it is easier in the Windows operating system, as GoPro Player can convert the video to an equirectangular format, which is unavailable on Linux. This flattened projection doesn’t give the real experience you would expect from a true 360-degree video.</p><p>More importantly, this affects frame-by-frame image extraction. When you extract individual frames from such videos, they often look like two separate images merged side by side, rather than a seamless spherical view. This can be problematic for anyone trying to process or analyze 360-degree content in its intended form. So I want to discuss how I have approached and solved this issue to an extent. I have used the Python programming language throughout all of these steps, so I am assuming whoever reads the article has a basic understanding of Python.</p><p>First of all, you have to make sure that Python is installed in your Linux Operating System. It’s always great to work on a virtual environment for the dependency issues. After creating a virtual environment, make sure to activate it. In that environment, run the following commands on the terminal:</p><pre>pip install opencv-python Pillow<br>pip install exifread hachoir <br>pip install PyExifTool</pre><p>OpenCV is an open-source computer vision library that will allow you to read, write, and process video and image files. It is necessary to extract frame-by-frame images from the 360-degree video. Pillow is the Python Imaging Library Fork to handle image files and do the post-processing on extracted frames. ExifRead is used for extracting metadata from images, and hachoir is a library for parsing binary files, which will allow you to analyze and extract metadata directly from the video file. PyExifTool is a Python wrapper for the command-line tool, which will allow you to extract metadata from both 360-degree videos and images. After installing all of these, run the following commands in your virtual environment:</p><blockquote>sudo apt update &amp;&amp; sudo apt install ffmpeg -y<br>pip install gopro2gpx<br>sudo ln -s $(which ffmpeg) ffmpeg<br>sudo ln -s $(which ffprobe) ffprobe</blockquote><p>To continue with the setup, we need to install <em>ffmpeg</em>, a powerful multimedia framework that handles video, audio, and other multimedia files and streams. This is essential for running <em>gopro2gpx</em>, a tool used to extract GPS data from GoPro videos. After updating the system and installing <em>ffmpeg</em>, we install <em>gopro2gpx</em> using pip. Since <em>gopro2gpx </em>expects <em>ffmpeg </em>and <em>ffprobe </em>to be located in the local directory, we create symbolic links to them using <em>ln -s</em>. This setup ensures that <em>gopro2gpx </em>can function correctly and extract GPS metadata from 360-degree GoPro videos. The <em>gopro2gpx </em>commands can be run like below:</p><pre>python3 -m gopro2gpx INPUT_VIDEO.360 video360.gpx</pre><p>It will create a .csv file where you can get the timestamp of each frame. If you extract image frames using the timestamp, you will see the images have been stitched, and they are not converted to 360-degree images. They need to be converted into equirectangular format on Linux to make them proper 360-degree images. If you get an error while running this command, try running the following command and then try the above command:</p><pre>pip install git+https://github.com/juanmcasillas/gopro2gpx</pre><p>To make the images equirectangular, you have run the following commands on your terminal of virtual environment:</p><blockquote>git clone <a href="https://github.com/trek-view/max2sphere">https://github.com/trek-view/max2sphere</a> <br>cd max2sphere <br>sudo apt update &amp;&amp; sudo apt install g++ make <br>make clean <br>make -f Makefile <br>cd .. <br>mkdir track0 track5 <br>ffmpeg -i INPUT_VIDEO.360 -map 0:0 -r 1 -q:v 1 track0/img%d.jpg -map 0:5 -r 1 -q:v 1 track5/img%d.jpg <br>mv max2sphere max2sphere_files <br>cd max2sphere_files <br>./max2sphere -n first_image_number -m last_image_number ../track%d/img%d.jpg <br>cd ..</blockquote><p>After extracting GPS data and timestamps using <em>gopro2gpx</em>, the next step is to properly extract and convert stitched frames into usable 360-degree images. To do this, we start by cloning the <em>max2sphere</em> repository from GitHub, which provides a tool to reproject GoPro MAX stitched frames into an equirectangular format. We enter the cloned directory, update the system, and install the necessary build tools (<em>g++</em> and <em>make</em>) to compile the project. The commands <em>make clean </em>and <em>make -f Makefile </em>clean any previous builds and compile the tool using the provided Makefile. Once compiled, the binary file <em>max2sphere </em>is generated inside the <em>max2sphere </em>directory.</p><p>To ensure the tool can be run correctly, especially when calling <em>./max2sphere </em>from within the compiled folder, we rename the entire directory to <em>max2sphere_files </em>using <em>mv max2sphere max2sphere_files</em>. This step ensures we can later navigate to the correct folder and execute the binary without path issues. We then create two folders, <em>track0 </em>and <em>track5</em>, to store frame extractions. Though folders with different names can be created but their name should be the same with different numbers.</p><p>Using <em>ffmpeg</em>, we extract one frame per second (<em>-r 1</em>) from both video tracks (<em>track0</em> and <em>track5</em>) of the input <em>.360 </em>file (can be modified according to the requirements) and save them as JPEG images in the respective folders. To run this command of <em>ffmpeg</em>, you have to make you have two video streams available in your 360 video. Here, <em>map 0:0 </em>is the Stream 0 of my video, which contains the frames of the front camera, and <em>map 0:5 </em>is the Stream 5 of my video, which contains the frames of the back camera. Basically, I have joined these two frames with <em>max2sphere</em> at last, to make images equirectangular. To check if your video contains two video streams or not, run the following command:</p><blockquote>ffmpeg -i INPUT_VIDEO.360</blockquote><p>Finally, we enter the renamed folder and run the command <strong><em>./max2sphere -n first_image_number -m last_image_number ../track%d/img%d.jpg</em></strong>, which reprojects the paired images into proper 360-degree equirectangular frames. The JPEG images having names like <strong><em>img%d_sphere.jpg</em></strong> will be created in the <em>track0 </em>folder, which are the converted equirectangular images. This completes the conversion of stitched GoPro MAX footage into panoramic frames ready for further processing or visualization. It can be felt that you can do it only by <em>max2sphere</em>, but you can use other commands, libraries, and packages, and make your projects ready in your own ways. I just wanted to share my experience and approach while implementing my project, which I haven’t shared so much in this article. Hope my article helps in developing your individual skills.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23c16b7622b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>