<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mahesh Saravanan  on Medium]]></title>
        <description><![CDATA[Stories by Mahesh Saravanan  on Medium]]></description>
        <link>https://medium.com/@mahesh.saravanan.3493?source=rss-60728a7a241e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*cLdfZ2jF38OiXnjp</url>
            <title>Stories by Mahesh Saravanan  on Medium</title>
            <link>https://medium.com/@mahesh.saravanan.3493?source=rss-60728a7a241e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:58:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mahesh.saravanan.3493/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The 3-Pixel Problem: How Reinforcement Learning Fixed Our Robotic Camera]]></title>
            <link>https://medium.com/iceapple-tech-talks/the-3-pixel-problem-how-reinforcement-learning-fixed-our-robotic-camera-0c6492716ce6?source=rss-60728a7a241e------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c6492716ce6</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[simulation]]></category>
            <dc:creator><![CDATA[Mahesh Saravanan ]]></dc:creator>
            <pubDate>Sat, 02 May 2026 06:02:51 GMT</pubDate>
            <atom:updated>2026-05-02T06:02:51.717Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*odzi3r6TZ9i5dhQynMdkbA.png" /></figure><p><em>This Blog is about how we used Reinforcement Learning to solve the messy, real-world physics of robotic camera alignment for industrial display inspections.</em></p><p>Picture this: you need to inspect massive, ultra-high-definition displays to catch microscopic dead pixels or subtle color shifts. Instead of straining human eyes, you mount a high-resolution digital camera onto a robotic base, turning it into an untiring, robotic eye that methodically scans the screen section by section.</p><p>For an “apples-to-apples” comparison across different sections of the screen, the camera’s position relative to the display must be exactly identical every single time. Even a millimeter of translation or a fraction of a degree in rotation can alter the captured data, rendering the quality check useless and incomparable.</p><p>Setting this up manually is tedious and highly error-prone. We needed an automated solution.</p><p>Our approach? Place 8 ArUco markers on a white display wall and teach an autonomous robotic base to align its camera perfectly with them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t1-bIoxxq_FoBJqAJu_3sQ.png" /><figcaption>Fig1: Real live camera feedback measuring spatial alignment with the screen.</figcaption></figure><h3><strong>The Messy Reality of Real-World Kinematics</strong></h3><p>If you’re wondering, <em>“Why not just calculate the geometric distance to the markers and drive the robot straight there?” </em>— welcome to the real world.</p><p>The real world is noisy.</p><p>A mobile robotic base’s motion model is never perfect. If you command a robot to move exactly 20 cm to the right, factors like floor friction, wheel slippage, and non-symmetrical weight distribution mean it might travel 15 cm, 25 cm, or perhaps slightly curve its path.</p><p>Because we needed pixel-perfect precision (under <strong>3 pixels of error</strong>) despite severe mechanical noise, simple hardcoded navigation logic falls apart. We needed a system capable of adapting, learning, and finding the optimal sequence of actions through trial and error.</p><p>We required Reinforcement Learning (RL). This method goes beyond simply learning a sequence of actions; it also learns to model and understand the inherent uncertainties accompanied with it.</p><p><strong>What is Reinforcement Learning (in this context)?</strong></p><p>Reinforcement Learning is a paradigm of machine learning where an <strong>Agent </strong>(our robot) learns to make decisions by interacting with an <strong>Environment </strong>(the room and display panel) to maximize a <strong>Reward </strong>(perfect camera alignment).</p><p>Think of training a dog: you give it a treat for doing a trick right and penalize it when it messes up. Over time, the dog figures out the sequence of behaviors that guarantees treats.</p><p><strong>Structuring Our RL Problem</strong></p><p>To teach our robot to align, we had to define four core components: the Environment, the State Space, the Action Space, and the Reward Function.</p><p><strong>1. The Environment</strong></p><p>We built a simulated room with the 8 ArUco markers placed on a mock white display wall, aligned with the global coordinate system. The bottom edge of the display runs perfectly along the <strong>X-axis</strong>, while the vertical span of the screen extends along the <strong>Z-axis</strong>. The center point of the display’s bottom edge sits exactly at the <strong>origin (0,0,0)</strong>. The grid resolution of this environment is 5 centimeters per pixel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*ssFVsqu95HVnOe6DzqYIxA.png" /><figcaption>Fig. 2: Simulated wall with the global coordinate axes shown.</figcaption></figure><p>The robot navigates within this space, starting from a randomized position. Each attempt (an <strong>episode</strong>) lasts a maximum of 200 steps before resetting.</p><p><strong>2. The Action Space (a): How the Robot Moves</strong></p><p>Our robot operates in a discrete action space of 8 distinct commands.</p><ul><li><strong>Linear Movements: </strong>Forward, Backward, Left, Right (5 cm steps in the XY plane)</li><li><strong>Vertical Movements: </strong>Up, Down (5 cm steps)</li><li><strong>Rotations: </strong>Clockwise, Counter-Clockwise (2-degree rotations on the Z-axis)</li></ul><p>To bridge the gap between simulation and reality, we intentionally injected <strong>Gaussian noise (μ:0, σ: 5) </strong>into every action taken to replicate the wheel slippage of the physical rover.</p><p><strong>3. The State Space (S): What the Robot “Sees”</strong></p><p>Unlike complex vision-based RL models that ingest raw pixels, we dramatically simplified the state space using the ArUco markers.</p><p>The state is a clean <strong>24-dimensional vector</strong>:</p><ul><li>16 dimensions: The X and Y pixel distances between the center of each of the 8 markers and a reference target box in the camera feed.</li><li>8 dimensions: Boolean flags indicating the visibility of each marker from the camera’s point of view (1 if visible, 0 if occluded or out of frame).</li></ul><p><strong>4. The Reward function (R): The Carrot and the Stick</strong></p><p>This is how the magic happens. A good reward function guides the agent smoothly to the goal.</p><ul><li><strong>+0.1 Reward (Encouragement): </strong>For any action that reduces the total distance between the current marker positions and the target reference box.</li><li><strong>-0.1 Penalty (Correction): </strong>For any action that moves the camera further away.</li><li><strong>+10 Bounty (Success): </strong>If the robot achieves a 95% Intersection over Union (IoU) with the target box <em>and </em>all marker center distances are under 3 pixels.</li><li><strong>-10 Penalty (Failure): </strong>If the robot drives out of simulation bounds or fails to align within the maximum step count.</li><li><strong>-0.05 (Step cost): </strong>A small penalty is added to encourge agent to converge quickly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*jJ61ACVZmqqifsS3mF2qyg.png" /><figcaption>Fig. 3: Reinforcement learning workflow.</figcaption></figure><p><strong>Deep Q-Learning: The Brain of the Operation</strong></p><p>To solve this, we used <strong>Deep Q-Learning (DQN)</strong>.</p><p>At its core, Q-learning attempts to learn the “Quality” (Q-value) of taking a specific action a while in a specific state s. The goal is to maximize the expected future rewards.</p><p><strong>The Network Architecture</strong></p><p>Because our state space is a lightweight 24-dimensional vector, we didn’t need heavy Convolutional Neural Networks (CNNs). We used a straightforward Multi-Layer Perceptron (MLP):</p><p>Input (<strong>24 </strong>nodes) → Dense (<strong>128</strong>, ReLU, BatchNorm) → Dense (<strong>128</strong>, ReLU, BatchNorm) → Output (<strong>8 </strong>nodes)</p><p>The network takes the 24-dim state as input and spits out 8 Q-values, one for each possible action. The robot simply executes an argmax on the output to pick the action with the highest Q-value.</p><p><strong>The Math: The Bellman Equation</strong></p><p>The core update rule of our network relies on the Bellman Equation. We want to update our current Q-value prediction to be closer to a “target” value:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*W8pEwRNu7iPUyMl2e_Boxg.png" /></figure><p>Where:</p><ul><li>R is the immediate reward received (+1, -1, etc.).</li><li>𝛾 is the discount factor (how much we care about future rewards vs immediate ones).</li><li>max (a′) Q(s′,a′) is the highest predicted Q-value for the <em>next </em>state.</li></ul><p><strong>Stabilizing the Training: The Secret Sauce of DQN</strong></p><p>If you just hook up a neural network to the Bellman equation and let the robot drive, it will likely fail catastrophically. The learning will oscillate and diverge.</p><p>Why? Because in robotics, consecutive states are highly correlated. Taking a step forward at 𝑡=1 is almost identical to the state at 𝑡=2. A standard neural network will immediately overfit to its most recent trajectory, forgetting everything else it learned.</p><p>To stabilize training, we implemented two crucial DQN techniques:</p><p><strong>1. Experience Replay Buffer</strong></p><p>Instead of learning from states sequentially, the robot stores every experience (𝑠,𝑎,𝑟,𝑠′,𝑑𝑜𝑛𝑒) — into a massive memory bank called the Replay Buffer. During training, the network randomly samples batches (e.g., 64 experiences) from this buffer. This random sampling breaks the correlation between consecutive steps, stabilizes the gradient updates, and vastly improves generalization.</p><p><strong>2. Dual Networks (Target vs. Online)</strong></p><p>If you calculate your Target Q-value using the <em>same </em>network that is currently predicting the Q-value, your network is essentially a dog chasing its own tail. The target keeps moving while you try to hit it.</p><p>We solve this by using two identical networks:</p><ol><li><strong>Online Network: </strong>This is the network actively training and choosing actions.</li><li><strong>Target Network: </strong>A frozen copy of the Online Network used <em>only </em>to compute the target Q-values in the Bellman equation.</li></ol><p>Every 𝑁 steps, we update the Target Network</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*rbG9pnT3EIhB2rD1JdqJyQ.png" /></figure><p>Where 𝜏 is a small parameter (e.g., 0.005). This gives the Online network a fixed, stable reference point to learn against.</p><p><strong>How We Actually Trained It</strong></p><p>With the architecture built, we ran the simulation for <strong>10,000 episodes</strong>. But we couldn’t just drop the robot in and expect magic. We used two key training strategies:</p><p><strong>Epsilon-Greedy Policy</strong></p><p>At the beginning of training, the robot’s neural network is entirely randomized. If it only chose the “best” action, it would get stuck doing the wrong thing forever.</p><p>We use an 𝜀 (epsilon) value. With a probability of 𝜀, the robot takes a completely random action (Exploration). With a probability of 1−𝜀, it takes the best action according to the network (Exploitation). We start 𝜀 near 1.0 (100% random) and slowly decay it over the 10,000 episodes as the robot gets smarter.</p><p><strong>Curriculum Learning</strong></p><p>Just like humans don’t learn calculus before algebra, robots shouldn’t start with the hardest alignment tasks. We implemented a 3-level curriculum.</p><ul><li><strong>Level 1: </strong>The robot spawns very close to the perfect alignment spot, facing the right direction. It learns the final micro-adjustments.</li><li><strong>Level 2: </strong>The robot spawns further away, requiring broader navigation.</li><li><strong>Level 3: </strong>Full randomization of distance and orientation.</li></ul><p>By slowly increasing the difficulty, we dramatically improved the convergence speed and stability of the model.</p><p><strong>Conclusion</strong></p><p>By framing a complex mechanical kinematics problem as a Reinforcement Learning game, we bypassed the need to perfectly model wheel slippage and floor friction. The agent simply learned how to behave optimally in the presence of noise.</p><p>Through the combination of a well-crafted reward function, the stability of Deep Q-Learning (with replay buffers and target networks), and a structured curriculum, our robotic base successfully learned to align its camera to the ArUco markers with pixel-perfect precision.</p><p>The next time you look at a massive, flawless industrial display, there’s a good chance an AI learned how to stare at it first.</p><p><strong>References</strong></p><ol><li>Sutton, R. S., &amp; Barto, A. G. (2018). <em>Reinforcement Learning: An Introduction</em>. 2nd ed. MIT Press.</li><li>Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., &amp; Hassabis, D. (2015). Human-level control through deep reinforcement learning. <em>Nature</em>, 518(7540), 529–533.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c6492716ce6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/iceapple-tech-talks/the-3-pixel-problem-how-reinforcement-learning-fixed-our-robotic-camera-0c6492716ce6">The 3-Pixel Problem: How Reinforcement Learning Fixed Our Robotic Camera</a> was originally published in <a href="https://medium.com/iceapple-tech-talks">IceApple DevLogs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>