<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sepideh Shamsizadeh on Medium]]></title>
        <description><![CDATA[Stories by Sepideh Shamsizadeh on Medium]]></description>
        <link>https://medium.com/@sepideh.92sh?source=rss-747261c080c0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XT4dfgGCzE0QW36sypo9DA.jpeg</url>
            <title>Stories by Sepideh Shamsizadeh on Medium</title>
            <link>https://medium.com/@sepideh.92sh?source=rss-747261c080c0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:12:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sepideh.92sh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part III — Composing Rotations: Euler Angles and Roll–Pitch–Yaw]]></title>
            <link>https://medium.com/@sepideh.92sh/part-iii-composing-rotations-euler-angles-and-roll-pitch-yaw-38aa816a5bcd?source=rss-747261c080c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/38aa816a5bcd</guid>
            <dc:creator><![CDATA[Sepideh Shamsizadeh]]></dc:creator>
            <pubDate>Sun, 14 Dec 2025 14:13:10 GMT</pubDate>
            <atom:updated>2025-12-14T14:13:10.589Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part III — Composing Rotations: Euler Angles and Roll–Pitch–Yaw</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ZNxbyUW7uKFzzExbxc3tw.jpeg" /></figure><p>In the first two parts of this series, we built a solid foundation for understanding rotations in 3D. We started with the idea of attaching coordinate frames to robots and tools, then derived the rotation matrices around the x, y, and z axes from simple geometric arguments. We saw how these matrices transform coordinates between frames (P = R·P′) and how they describe the active rotation of vectors in space.</p><p>However, real robots rarely rotate around just one axis at a time. A robotic arm, a camera gimbal, or a drone’s body usually ends up in some arbitrary orientation that comes from several rotations composed together. Writing every orientation as a 3×3 matrix works, but it doesn’t tell us much about how that orientation was reached or how to command it.</p><p>This is where three-angle representations come in.</p><p>In this part, we will:</p><ul><li>use Euler’s rotation theorem to understand why any 3D rotation can be built from a sequence of simpler ones,</li><li>introduce Euler angle conventions, focusing on ZYZ rotations as a classic example,</li><li>connect this to the more intuitive roll–pitch–yaw (RPY) angles used in robotics and aerospace, and</li><li>look at the inverse problem: given a rotation matrix R, how can we recover the corresponding angles?</li></ul><p>By the end of this article, you will be able to move comfortably between “matrix language” and “angle language”: composing rotations with products of matrices, interpreting them as sequences of rotations around specific axes, and extracting angles back from a given rotation matrix.</p><h3>Composition of Rotation Matrices</h3><p>To work with real robots, we rarely move between just two frames. Instead, we have a whole chain of frames: base, joints, end-effector, sensors, and so on. Rotation matrices let us move along this chain in a clean, systematic way.</p><p>We will use the following notation:</p><ul><li>Pⁱ = coordinates of the same geometric point P <strong>expressed in frame i</strong></li><li><strong>Rⱼⁱ</strong> = denotes the rotation matrix of F<strong>rame j with respect to Frame i</strong></li></ul><p>With this convention,</p><p>Pⁱ = <strong>Rⱼⁱ </strong>Pʲ</p><p>means: <em>“take the coordinates of P in frame j and express them in frame i.”</em></p><h3>1. Chaining Frames (Current-Frame View)</h3><p>Consider three frames that share the same origin:</p><ul><li>frame 0 with axes (x₀, y₀, z₀)</li><li>frame 1 with axes (x₁, y₁, z₁)</li><li>frame 2 with axes (x₂, y₂, z₂)</li></ul><p>Let P², P¹ and P⁰ be the coordinates of the same point P in frames 2, 1 and 0:</p><p>By definition of Rⱼⁱ​:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*E2L3PXbyM-votdK2_ROhPQ.png" /></figure><p>If we substitute the first relation into the second, we get</p><p>P⁰ = R₁⁰ (R₂¹P²) = (R₁⁰ R₂¹)P²</p><p>So the <strong>direct</strong> rotation matrix that converts coordinates from frame 2 to frame 0 is</p><p>R₂⁰ = R₁⁰ R₂¹</p><p>In words:</p><p><em>When each rotation is defined with respect to the </em><strong><em>current frame</em></strong><em>, the overall rotation is obtained by </em><strong><em>post-multiplying</em></strong><em> the individual rotation matrices as we walk along the chain of frames.</em></p><p>If we add more frames, the pattern is the same. For example, with frames 0–1–2–3:</p><p>R₃⁰ = R₂⁰ R₃² = R₁⁰ R₂¹ R₃²</p><p>and in general we accumulate rotations by multiplying new matrices on the <strong>right</strong>.</p><p>This is exactly the viewpoint used in forward kinematics: starting at the base frame and successively applying rotations (and later translations) to reach the end-effector frame.</p><h3>2. Successive Rotations of a Vector (Fixed-Frame View)</h3><p>There is another common situation: instead of chaining <strong>frames</strong>, we keep one frame fixed and apply several rotations to the <strong>vector</strong> itself.</p><p>Think of a vector P in a fixed frame O–xyz.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*nLdHPkXIPUGBfjMn1c3_xA.png" /></figure><p>We first rotate it by an angle α around the z-axis:</p><p>P′=Rz(α) P</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*tpqq1XjPPRhb0-_SrDLUFw.png" /></figure><p>Then we apply a second rotation by an angle β\betaβ around the <strong>same fixed axis</strong>:</p><p>P′′=Rz(β) P′</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*zbVRxqk9cCON9YYDP6tFrg.png" /></figure><p>Substituting the first relation into the second gives</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*mp-gV1h54mCA6rH7Zd-vHg.png" /></figure><p>P′′=Rz(β)(Rz(α) P)=(Rz(β)Rz(α)) P</p><p>The matrix</p><p>Rtot=Rz(β)Rz(α)</p><p>is the rotation matrix that represents the <strong>composition</strong> of these two successive rotations with respect to the same fixed frame.</p><p>Here the order is important:</p><blockquote><em>When we perform a sequence of rotations </em><strong><em>about a fixed frame</em></strong><em>, the overall rotation matrix is obtained by </em><strong><em>pre-multiplying</em></strong><em> the individual rotation matrices in the same order as the rotations occur.</em></blockquote><p>We can apply the same reasoning in 3D with rotations about different axes. For example, if we rotate first by α around z, then by β around y, then by γ around x, all with respect to the same fixed frame, the total rotation is</p><p>Rtot=Rx(γ) Ry(β) Rz(α)</p><h3>3. Why the Order of Rotations Matters</h3><p>A crucial property of 3D rotations is:</p><blockquote><em>In general, rotations do </em><strong><em>not</em></strong><em> commute.<br> Doing rotation A then rotation B is not the same as doing B then A.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*Q3Yh1YTUbLGIW0DchaVyMQ.png" /></figure><p>The picture shows this with a simple example using two 90° rotations.</p><ul><li>We start from the same initial frame (X, Y, Z).</li><li><strong>Top row:</strong></li><li>Rotate by π/2 around the X-axis.</li><li>Then rotate by π/2 around the Y-axis.<br> In matrix notation this sequence is<br> R_total = Ry(π/2) Rx(π/2).</li><li><strong>Bottom row:</strong></li><li>Rotate by π/2 around the Y-axis.</li><li>Then rotate by π/2 around the X-axis.<br> Now the sequence is<br> R_total = Rx(π/2) Ry(π/2).</li></ul><p>Even though we use the same angles and the same axes, the final frames in the two rows of the picture are clearly different: the labels X, Y, Z end up pointing in different directions.</p><p>We can also see this directly from the matrices. Using the standard rotation matrices around X and Y:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*_yAd0avYW5h83SHAP1ynbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*d-9pVkr34G2uTyNGlwqwaA.png" /></figure><p>If we first rotate around X and then around Y, the total rotation is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*OYWxffa6-FJqLCpt1naUrw.png" /></figure><p>If we swap the order, first Y then X, we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*dQbTSlVhRIyAgamr7Luh8A.png" /></figure><p>These two matrices are not equal, and they correspond exactly to the two different final frames shown in the figure.</p><p>This is why, when we write something like</p><p>Rtot = Rx(γ) Ry(β) Rz(α)</p><p>the <strong>order matters</strong>. It encodes an exact sequence of rotations:</p><ol><li>first a rotation by α around the z-axis,</li><li>then by β around the y-axis,</li><li>then by γ around the x-axis,</li></ol><p>all with respect to the chosen frame (fixed-frame view in this case). Changing the order of the matrices means changing the physical motion and gives a different final orientation of the robot, camera, or drone.</p><h3>4. From Rotation Matrices to Three Angles</h3><p>Rotation matrices are powerful, but they are also <strong>redundant</strong>:</p><ul><li>A 3D rotation matrix has 9 numbers.</li><li>Because the columns (and rows) must be orthonormal, there are 6 independent constraints.</li><li>Only <strong>3 real degrees of freedom</strong> remain.</li></ul><p>So, although we like matrices for computation, we know that <strong>3 independent parameters are enough</strong> to describe any orientation of a rigid body in space. In group-theory language this is the fact that SO(3), the space of all 3D rotations, is 3-dimensional; in general SO(m) needs m(m−1)/2 parameters.</p><p>This motivates the idea of a <strong>minimal representation</strong>:</p><blockquote><em>A minimal representation of orientation is any set of three independent parameters that uniquely (up to singular cases) describes a 3D rotation.</em></blockquote><p>Three-angle descriptions, like Euler angles and roll–pitch–yaw, are examples of such minimal representations.</p><h3>5. Euler’s Rotation Theorem</h3><p>Euler’s rotation theorem gives us the conceptual foundation:</p><blockquote><strong><em>Euler’s rotation theorem</em></strong><em><br> Any two independent orthonormal coordinate frames can be aligned by a sequence of </em><strong><em>no more than three</em></strong><em> rotations about coordinate axes, where no two successive rotations are about the same axis.</em></blockquote><p>This tells us that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*hEZp1683zt4huTrAIJ657A.png" /></figure><ul><li>We can always go from one frame to another using at most three elementary rotations.</li><li>There are many possible ways to choose these three axes and angles.</li></ul><p>If we restrict ourselves to rotations about the x, y, z axes, and require that no two consecutive rotations use the same axis, we get <strong>12 admissible triplets</strong> (like ZYZ, ZXZ, ZYX, etc.). Each triplet defines a particular <strong>Euler angle convention</strong>.</p><p>In robotics and aerospace, two sets are especially common:</p><ul><li><strong>ZYZ Euler angles</strong></li><li><strong>ZYX angles (roll–pitch–yaw)</strong></li></ul><p>We’ll start with ZYZ, using the “current-frame” view you have already seen.</p><h3>6. ZYZ Euler Angles (Current-Frame View)</h3><p>Imagine you have an initial frame O–XYZ and you want to reach some final frame O–X‴Y‴Z‴. The ZYZ convention uses three rotations:</p><ol><li>Rotate by an angle φ around the original Z axis.</li><li>Rotate by an angle θ around the <strong>new Y axis</strong> (often written Y′).</li><li>Rotate by an angle ψ around the <strong>new Z axis after the second rotation</strong> (often written Z″).</li></ol><p>You can show these three steps with three images:</p><ul><li>Step 1: frame after rotation φ around Z.</li><li>Step 2: frame after rotation θ around Y′.</li><li>Step 3: frame after rotation ψ around Z″, giving the final orientation.</li></ul><p>Because each rotation is defined <strong>with respect to the current frame</strong>, the total rotation matrix is obtained by <strong>post-multiplication</strong> of the elementary rotations:</p><p>R = Rz(φ) Ry(θ) Rz(ψ)</p><p>Here:</p><ul><li>Rz(φ) = rotation about the z-axis by angle φ</li><li>Ry(θ) = rotation about the <em>current</em> y-axis by angle θ</li><li>Rz(ψ) = rotation about the <em>current</em> z-axis by angle ψ</li></ul><p>If you like a compact notation, you can abbreviate</p><ul><li>cφ = cosφ, sφ = sinφ</li><li>cθ = cosθ, sθ = sinθ</li><li>cψ = cosψ, sψ = sinψ</li><li>Then the full matrix R can be written as</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*2_lNXmDgwrc_0T0f0yRTrw.png" /></figure><p>The important thing is not to memorise this matrix, but to <strong>understand its structure</strong>:</p><ul><li>It is just the matrix product of three simple rotations.</li><li>The order matches the physical sequence “rotate around Z, then around Y′, then around Z″”.</li><li>Changing that order would give a different matrix and a different final orientation.</li></ul><p>In practice, ZYZ angles are often used in robot manipulators because they map nicely to spherical wrists and to problems where the end-effector must reach a given orientation with some symmetry around the tool axis.</p><h3>7. ZYZ Angles: The Inverse Problem</h3><p>So far, we have gone from the three ZYZ angles (φ,θ,ψ) <strong>to</strong> the rotation matrix R.<br> In practice we often face the opposite problem:</p><blockquote><em>Given a rotation matrix R, how can we recover the angles θ, φ, ψ?</em></blockquote><p>This is very similar in spirit to inverse kinematics: instead of computing the pose from angles, we want to compute the angles from the pose.</p><p>Assume that R has the form</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*wRmstUudpsRyS78v9WHJJA.png" /></figure><p>and that it was generated by</p><p>R = Rz(φ) Ry(θ) Rz(ψ)</p><p>with the compact notation<br> cφ = cosφ, sφ = sinφ, cθ = cosθ, sθ = sinθ, cψ = cosψ, sψ = sinψ.</p><p>From the explicit matrix expression (right-hand side of the equation above) we read in particular:</p><p>r₁₃ = cφ sθ</p><p>r₂₃ = sφ sθ</p><p>r₃₃ = cθ</p><p>r₃₁ = −sθ cψ</p><p>r₃₂ = sθ sψ</p><p>These entries involve only one or two of the angles, which makes them perfect to invert.</p><h3>7.1 Generic case: sθ ≠ 0 (θ ∈ (0, π))</h3><p>Let us first assume that sθ≠0sθ ≠ 0sθ=0, i.e. θ is not 0 or π. This is the “regular” case where there is no singularity.</p><ol><li><strong>Recover φ from the (1,3) and (2,3) entries</strong></li></ol><p>From<br> r₁₃ = cφ sθ and r₂₃ = sφ sθ<br> we get</p><p>tanφ = r₂₃ / r₁₃</p><p>To handle all four quadrants correctly we use the two-argument arctangent:</p><p>φ = Atan2(r₂₃, r₁₃)</p><p>Atan2(y, x) returns the angle whose cosine–sine pair has components (x, y), keeping track of the signs.)</p><p><strong>2. Recover θ from the same column</strong></p><p>Still from r₁₃ = cφ sθ, r₂₃ = sφ sθ and r₃₃ = cθ we get</p><p>r₁₃² + r₂₃² = (cφ² + sφ²) sθ² = sθ²</p><p>so</p><p>sθ = √(r₁₃² + r₂₃²) cθ = r₃₃</p><p>Again using Atan2, with θ chosen in (0, π):</p><p>θ = Atan2( √(r₁₃² + r₂₃²), r₃₃ )</p><p><strong>3. Recover ψ from the bottom row</strong></p><p>From r₃₁ = −sθ cψ and r₃₂ = sθ sψ we obtain</p><ul><li>tanψ = r₃₂ / (−r₃₁)</li></ul><p>so</p><p>ψ = Atan2( r₃₂, −r₃₁ )</p><p>Putting everything together, in the <strong>generic case</strong> sθ≠0 and θ ∈ (0, π):</p><p>φ = Atan2( r₂₃, r₁₃ )<br>θ = Atan2( √(r₁₃² + r₂₃²), r₃₃ ) with θ ∈ (0, π)<br>ψ = Atan2( r₃₂, −r₃₁ )</p><p>These formulas take a rotation matrix R that is known to come from ZYZ angles and return one consistent triplet (φ, θ, ψ).</p><h3>7.2 Second solution branch</h3><p>Because trigonometric functions are periodic and not one-to-one, there is typically more than one valid triplet of angles that corresponds to the same matrix R.</p><p>A second branch can be obtained by taking θ in (−π, 0) instead of (0, π). A convenient way to write it is:</p><p>φ = Atan2( −r₂₃, −r₁₃ )<br>θ = Atan2( −√(r₁₃² + r₂₃²), r₃₃ ) with θ ∈ (−π, 0)<br>ψ = Atan2( −r₃₂, r₃₁ )</p><p>You can check by substitution that both branches produce the same matrix R when plugged into R = Rz(φ) Ry(θ) Rz(ψ).</p><p>In applications (e.g. inverse kinematics of a manipulator wrist) we often use additional criteria to select which branch is preferable: angle limits, continuity with the previous solution, or minimal motion.</p><h3>7.3 Singular cases: θ = 0 or θ = π (gimbal-like degeneracy)</h3><p>The formulas above assumed sθ≠0. If sθ=0, then θ is 0 or π, and several of the key entries become zero:</p><p>r₁₃ = cφ sθ = 0<br>r₂₃ = sφ sθ = 0<br>r₃₁ = −sθ cψ = 0<br>r₃₂ = sθ sψ = 0<br>r₃₃ = cθ = ±1</p><p>The matrix R simplifies to</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*mvL3HGwEnsaDV5NSgyukwA.png" /></figure><p>In this situation, φ and ψ only appear in the <strong>sum</strong> (or difference) of their angles, so they are no longer uniquely determined: many combinations of (φ, ψ) give the same matrix R.</p><p>This is a typical <strong>singularity</strong> of ZYZ Euler angles (similar to gimbal lock): when the middle angle θ is 0 or π, the first and third rotations become indistinguishable around the same axis. In practice, one usually:</p><p>sets θ = 0 (or π) from r₃₃ = cθ,</p><p>chooses φ = 0 and lets ψ absorb the whole rotation, or vice versa,</p><p>or uses additional information from the application to pick a convenient pair (φ, ψ).</p><p>With these formulas, you now have both directions:</p><ul><li><strong>forward problem</strong>: from (φ, θ, ψ) to R via R = Rz(φ) Ry(θ) Rz(ψ)</li><li><strong>inverse problem</strong>: from R back to (φ, θ, ψ), handling both generic and singular cases.</li></ul><h3>8. Roll–Pitch–Yaw (ZYX) Angles</h3><p>In robotics and aerospace, orientations are very often described using <strong>roll–pitch–yaw (RPY)</strong> angles. These are simply a specific <strong>ZYX</strong> convention:</p><blockquote><em>Roll–Pitch–Yaw angles are </em><strong><em>ZYX angles</em></strong><em> that represent rotations defined with respect to a </em><strong><em>fixed frame</em></strong><em> (pre-multiplication).</em></blockquote><p>Using the usual naming:</p><ul><li>ψ = roll (rotation about the x axis)</li><li>θ = pitch (rotation about the y axis)</li><li>φ = yaw (rotation about the z axis)</li></ul><p>and the <strong>fixed-frame view</strong>, the sequence is:</p><ol><li>Rotate by ψ around the <strong>fixed x axis</strong> → Rx(ψ)</li><li>Rotate by θ around the <strong>fixed y axis</strong> → Ry(θ)</li><li>Rotate by φ around the <strong>fixed z axis</strong> → Rz(φ)</li></ol><p>Since rotations are defined with respect to the fixed frame, the total rotation matrix is</p><p>R = Rz(φ) Ry(θ) Rx(ψ)</p><p>If we use the shorthand cφ = cosφ, sφ = sinφ, cθ = cosθ, sθ = sinθ, cψ = cosψ, sψ = sinψ, the full matrix is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*as6vJwCHmnARpIegkii9sw.png" /></figure><p>Again, the key idea is the same as for ZYZ:</p><ul><li>this matrix is just the product of three simple rotations,</li><li>the order encodes the exact RPY sequence around z, y, x,</li><li>changing the order changes the resulting orientation.</li></ul><h3>9. Inverse Problem for Roll–Pitch–Yaw</h3><p>Just like with ZYZ angles, in practice we often know <strong>R</strong> and want to recover <strong>(φ, θ, ψ)</strong>.</p><p>Write</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*wRmstUudpsRyS78v9WHJJA.png" /></figure><p>and assume it comes from RPY angles via R = Rz(φ) Ry(θ) Rx(ψ).</p><p>From the explicit expression above we can read some useful entries:</p><p>r₃₁ = −sθ<br>r₃₂ = cθ sψ<br>r₃₃ = cθ cψ<br>r₂₁ = sφ cθ<br>r₁₁ = cφ cθ</p><h4>9.1 Generic case: cθ ≠ 0</h4><p>If cθ ≠ 0 (i.e. θ is not ±π/2), the extraction is straightforward:</p><p>φ = Atan2( r₂₁, r₁₁ )<br>θ = Atan2( −r₃₁, √(r₃₂² + r₃₃²) ) with θ ∈ (−π/2, π/2)<br>ψ = Atan2( r₃₂, r₃₃ )</p><p>This gives one consistent triplet (φ, θ, ψ) for a given rotation matrix R.</p><p>Because trigonometric functions are periodic, a second valid branch exists; one convenient form is</p><p>φ = Atan2( −r₂₁, −r₁₁ )<br>θ = Atan2( −r₃₁, −√(r₃₂² + r₃₃²) ) with θ ∈ (π/2, 3π/2)<br>ψ = Atan2( −r₃₂, −r₃₃ )</p><p>Both branches correspond to the same R when plugged back into R = Rz(φ) Ry(θ) Rx(ψ); in applications we pick the one that best fits joint limits or continuity.</p><h4>9.2 Singular case: cθ = 0 (gimbal-like degeneracy)</h4><p>When cθ = 0, pitch θ is ±π/2 and the expressions above become singular: only the <strong>sum or difference</strong> of φ and ψ is uniquely determined.</p><p>Geometrically, this is the same “gimbal lock” phenomenon you saw for ZYZ: two of the rotation axes align and some information is lost from the angle triplet. In practice, one typically:</p><ul><li>sets θ = ±π/2 from r₃₁,</li><li>merges φ and ψ into a single effective yaw/roll angle,</li><li>or switches to a different representation (e.g. quaternions) near these singularities.</li></ul><h3>Conclusion</h3><p>In this third part of the series, we moved from single rotations to <strong>full 3D orientation descriptions</strong> built from multiple rotations and three angles.</p><p>First, we saw how rotation matrices <strong>compose</strong>:</p><ul><li>In the <strong>current-frame view</strong>, we chained frames using<br> Pⁱ = Rʲⁱ Pʲ and products like R²⁰ = R¹⁰ R²¹.<br> This is the language of forward kinematics along a robot arm: we walk from base to end-effector by post-multiplying the rotations between successive frames.</li><li>In the <strong>fixed-frame view</strong>, we kept one frame still and applied several rotations to the <em>same</em> vector, with products such as<br> P″ = Rz(β) Rz(α) P.<br> Here, rotations are pre-multiplied in the order they are applied.</li></ul><p>Using a simple example with Rx(π/2) and Ry(π/2), we made it clear that rotations <strong>do not commute</strong>: changing the order changes the final orientation. This is why the exact order in expressions like<br> R_tot = Rx(γ) Ry(β) Rz(α)<br> carries real geometric meaning.</p><p>Next, we introduced <strong>three-angle representations</strong>:</p><ul><li><strong>ZYZ Euler angles (φ, θ, ψ)</strong>, built as<br> R = Rz(φ) Ry(θ) Rz(ψ) in the current-frame view.<br> We wrote the full matrix and then solved the <strong>inverse problem</strong>, extracting (φ, θ, ψ) back from the entries of R, including regular and singular (gimbal-like) cases.</li><li><strong>Roll–Pitch–Yaw (RPY) angles</strong>, interpreted as a <strong>ZYX</strong> convention in the fixed-frame view:<br> R = Rz(φ) Ry(θ) Rx(ψ).<br> Again, we derived the forward matrix and the inverse formulas, and saw how singularities appear when pitch θ reaches ±π/2.</li></ul><p>Across all of this, a few key ideas keep repeating:</p><ul><li>A 3D orientation has only <strong>three degrees of freedom</strong>, no matter whether we describe it by a matrix or by angles.</li><li>Rotation matrices are great for computation and composition; angle triplets (Euler, RPY) are great for <strong>human intuition</strong> and for specifying commands.</li><li>The choice of convention (ZYZ, ZYX, current frame, fixed frame) and the <strong>order of factors</strong> in the product are critical — they encode the actual physical motion.</li></ul><p>With Parts I–III, you now have a complete toolkit to think about rotations in robotics:</p><ul><li>frames and rotation matrices,</li><li>coordinate transformations and vector rotations,</li><li>composition of rotations,</li><li>and two of the most common three-angle parametrizations with their inverse mappings.</li></ul><p>In later work you can build on this foundation to explore <strong>quaternions</strong>, full <strong>homogeneous transformations</strong> (rotation + translation), and ultimately the complete forward and inverse kinematics of real robot arms.</p><p>If you’d like to dive deeper into the math and formal proofs behind these ideas, a classic reference is:<br> <strong>B. Siciliano, L. Sciavicco, L. Villani, G. Oriolo</strong>, <em>Robotics: Modelling, Planning and Control</em>, Springer, 2009.<br> Chapters on rotation matrices and Euler angles expand on the material in this article with full derivations and more examples.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38aa816a5bcd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part II — Inside Rotation Matrices: Axes, Frames, and Coordinate Transformations]]></title>
            <link>https://medium.com/@sepideh.92sh/part-ii-inside-rotation-matrices-axes-frames-and-coordinate-transformations-f6d0810a804a?source=rss-747261c080c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6d0810a804a</guid>
            <dc:creator><![CDATA[Sepideh Shamsizadeh]]></dc:creator>
            <pubDate>Thu, 27 Nov 2025 15:40:15 GMT</pubDate>
            <atom:updated>2025-11-27T15:40:15.372Z</atom:updated>
            <content:encoded><![CDATA[<h2>Part II — Inside Rotation Matrices: Axes, Frames, and Coordinate Transformations</h2><p>In the first part of this series, we focused on orientation: how robots use rotation matrices and coordinate frames to describe how a rigid body is turned in space while preserving its shape and size. We saw that attaching a frame to a robot, camera, or tool gives us a clean way to answer the question: <em>“Which way is it facing?”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*mjf_LMiFYy6jlegHCQESHw@2x.jpeg" /></figure><p>In this second part, we stay with <strong>pure rotation</strong>, but go much deeper into the mathematics behind it. Instead of just writing down the standard formulas, we derive the rotation matrices around the x, y, and z axes from simple geometric pictures: projections, right triangles, and sine–cosine relationships. This lets us see every entry of the matrix as a concrete quantity: a component of a rotated axis along an original axis.</p><p>Once we have these matrices, we interpret them in two complementary ways:</p><ul><li>as <strong>changes of coordinates</strong> between two frames that share the same origin (P = R · P′), and</li><li>as <strong>active rotations</strong> of vectors in a fixed frame (P = Rz(α) · P′).</li></ul><p>By the end of this article, rotation matrices should feel much less mysterious: just compact summaries of geometry that relate axes, frames, and vector coordinates in 3D space. In the next paper, we will build on this foundation to compose rotations and explore three-angle representations of orientation, including Euler angles (ZYZ), roll–pitch–yaw, and the inverse problem of recovering angles from a given rotation matrix.</p><h3>Rotation Around the z-Axis: Seeing the Matrix in the Picture</h3><p>To really understand where rotation matrices come from, let’s start with the simplest case: a rotation around the z-axis. In this situation, everything happens in the x–y plane and the z coordinate never changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*e1JYHLPDKF1icqJ4qUgpRA.png" /></figure><p>In the figure above:</p><ul><li>(x, y) is the original frame.</li><li>(x′, y′) is the frame rotated by an angle α around the z-axis.</li><li>The vector x′ is the new x-axis. Its projections on the old axes are labeled x′ₓ (projection on x) and x′ᵧ (projection on y).</li><li>Similarly, y′ has projections y′ₓ and y′ᵧ.</li></ul><p>Consider x′ is a unit vector, the right triangle formed by |x′|, x′ₓ, and x′ᵧ has hypotenuse 1. From basic trigonometry:</p><ul><li>cosα = x′ₓ/|x′| = x′ₓ</li><li>sinα = x′ᵧ/|x′| = x′ᵧ</li></ul><p>So the components of the rotated x-axis in the original frame are</p><ul><li>x′ = [ x′ₓ, x′ᵧ, x′𝑧 ]ᵀ = [ cosα, sinα, 0 ]ᵀ</li></ul><p>Now look at the y-axis. The vector y′ is also a unit vector, orthogonal to x′, and rotated by the same angle α. Relative to the x-axis, its angle is 90° + α, which gives</p><ul><li>y′ₓ = cos(90° + α) = −sinα</li><li>y′ᵧ = sin(90° + α) = cosα</li><li>y′𝑧 = 0</li></ul><p>So</p><ul><li>y′ = [ −sinα, cosα, 0 ]ᵀ</li><li>z′ = [ 0, 0, 1 ]ᵀ</li></ul><p>From <a href="https://medium.com/@sepideh.92sh/part-i-how-robots-understand-space-kinematics-and-the-power-of-rotation-matrices-6b2ba5bc07be">Part I</a> we know that the columns of the rotation matrix are just the body axes written in the world frame. Stacking x′, y′, and z′ as columns gives the rotation matrix for a rotation by α around the z-axis:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*N3b_Jld3sObpalDBpEi9YA.png" /></figure><p>You can literally read each entry of this matrix from the picture: every little projection (x′ₓ, x′ᵧ, y′ₓ, y′ᵧ) becomes one element of Rz(α).</p><h3>Rotation Around the y-Axis (Using the Picture)</h3><p>Now consider a rotation by an angle β around the y-axis.</p><p>In this situation:</p><ul><li>The y-axis does not move, so y′ = y.</li><li>The x and z axes rotate inside the x–z plane.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*mIIKYAwHgyC_4_kcjIzVjw.png" /></figure><p>In the figure:</p><ul><li>(x, z) is the original pair of axes in that plane.</li><li>(x′, z′) is the rotated pair after turning by β around y.</li><li>The components of x′ are labelled x′ₓ (projection on x) and x′𝑧 (projection on z).</li><li>The components of z′ are labelled z′ₓ and z′𝑧.</li></ul><p>Because x′ is a unit vector, the right triangle formed by |x′|, x′ₓ and x′𝑧 has hypotenuse 1. From the geometry we read:</p><ul><li>cosβ = x′ₓ</li><li>|sinβ| = |x′𝑧|</li></ul><p>With the usual right-hand rule (positive rotation around y takes the x-axis towards <strong>negative</strong> z), x′𝑧 must be negative, so</p><ul><li>x′ₓ = cosβ</li><li>x′𝑧 = −sinβ</li></ul><p>and therefore</p><ul><li>x′ = [ cosβ, 0, −sinβ ]ᵀ</li></ul><p>The y-axis is unchanged:</p><ul><li>y′ = [ 0, 1, 0 ]ᵀ</li></ul><p>The z′ axis is also a unit vector, orthogonal to x′ and y′, and rotated by the same angle β. Its angle with respect to the original x-axis is 90° + β, which gives</p><ul><li>z′ₓ = sinβ</li><li>z′𝑧 = cosβ</li></ul><p>so</p><ul><li>z′ = [ sinβ, 0, cosβ ]ᵀ</li></ul><p>Putting x′, y′ and z′ as columns, we obtain the rotation matrix around the y-axis:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*Uk7gOf89YCn79UCOTC99-g.png" /></figure><p>Again, every entry of this matrix comes directly from the little projection labels in your picture: x′ₓ, x′𝑧, z′ₓ, z′𝑧.</p><h3>Rotation Around the x-Axis</h3><p>Now we look at a rotation by an angle γ around the x-axis. In this case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*0Ws6QO_V8SfLpSsuGidx8Q.png" /></figure><p>I’d correct your sentence as:<br> “Here is the image for rotation around the x-axis.”</p><p>Let’s write the matching explanation in the same Unicode style as before, using your figure.</p><ul><li>The x-axis does not move, so x′ = x.</li><li>The y and z axes rotate inside the y–z plane.</li></ul><p>In your figure:</p><ul><li>(y, z) is the original pair of axes in that plane.</li><li>(y′, z′) is the rotated pair after turning by γ around x.</li><li>The components of y′ are labelled y′ᵧ (projection on y) and y′𝑧 (projection on z).</li><li>The components of z′ are labelled z′ᵧ and z′𝑧.</li></ul><p>Because y′ is a unit vector, the right triangle formed by |y′|, y′ᵧ, and y′𝑧 has hypotenuse 1. From the geometry:</p><ul><li>cosγ = y′ᵧ</li><li>sinγ = y′𝑧</li></ul><p>So the components of the rotated y-axis in the original frame are</p><ul><li>y′ = [ 0, y′ᵧ, y′𝑧 ]ᵀ = [ 0, cosγ, sinγ ]ᵀ</li></ul><p>The z′ axis is also a unit vector, orthogonal to y′ and rotated by the same angle γ. Its angle with respect to the original y-axis is 90° + γ, so</p><ul><li>z′ᵧ = cos(90° + γ) = −sinγ</li><li>z′𝑧 = sin(90° + γ) = cosγ</li></ul><p>and therefore</p><ul><li>z′ = [ 0, −sinγ, cosγ ]ᵀ</li></ul><p>The x-axis is unchanged by this rotation:</p><ul><li>x′ = [ 1, 0, 0 ]ᵀ</li></ul><p>Putting x′, y′ and z′ as columns, we obtain the rotation matrix around the x-axis:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*YccSdoHV9iiq8bICTA43Fg.png" /></figure><p>Again, every entry of this matrix comes directly from the little projection labels in your picture: y′ᵧ, y′𝑧, z′ᵧ, z′𝑧.</p><h3>The Three Building Blocks</h3><p>We now have explicit forms for rotations about each coordinate axis:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*YccSdoHV9iiq8bICTA43Fg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*Uk7gOf89YCn79UCOTC99-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*N3b_Jld3sObpalDBpEi9YA.png" /></figure><p>Each one was built from the same simple recipe:</p><ol><li>Choose the axis of rotation.</li><li>Look at the plane that rotates (x–y, x–z, or y–z).</li><li>Use sine and cosine in that plane to find the projections of the rotated axes.</li><li>Use those projections as the entries of the rotation matrix.</li></ol><h3>Switching Between Frames: P and P′</h3><p>So far we have focused on how a rotation matrix describes the <strong>orientation</strong> between two frames. Now let’s connect that to the coordinates of an actual point in space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*z_pMSQnShJmEs8wkhB6egQ.png" /></figure><p>Look at the figure above:</p><ul><li>Frame <strong>O</strong> has axes (x, y, z).</li><li>Frame <strong>O′</strong> has axes (x′, y′, z′).</li><li>Both frames share the same origin O (for now we ignore translation).</li><li>The black point <strong>P</strong> is a fixed point in space.</li></ul><p>The key idea is:</p><p><em>P is a single point in 3D space, but it can have </em><strong><em>different coordinates</em></strong><em> depending on which frame we use.</em></p><ul><li>In frame O we write its coordinates as<br> P= [ pₓ, pᵧ, p𝑧 ]ᵀ</li><li>In frame O′ we write its coordinates as<br> P<strong>′</strong> = [ p′ₓ, p′ᵧ, p′𝑧 ]ᵀ</li></ul><p>Both vectors P and P<strong>′</strong> describe the same geometric point P, just expressed in different bases.</p><h4>Writing P using the axes of O′</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*XCYdpq-VJJqz76XadMjAMA.png" /></figure><p>P = p′ₓ · x′ + p′ᵧ · y′ + p′𝑧 · z′<br>In frame O′, the point P is a combination of the basis vectors x′, y′, z′:</p><p>If we collect the basis vectors of O′ as the columns of one matrix</p><p>[ x′ y′ z′ ]</p><p>P′ = [ p′ₓ p′ᵧ p′𝑧 ]ᵀ</p><p>and the coordinates in O′ as a column vector</p><p>then the same expression can be written compactly as</p><p>P (expressed in frame O) = [ x′ y′ z′ ] · P′</p><p>By definition, the matrix that has x′, y′, z′ as its columns <strong>is</strong> the rotation matrix R from O′ to O:</p><p>R = [ x′ y′ z′ ]</p><p>So the relation becomes</p><p>P = R · P′</p><p>where</p><ul><li><strong>P′</strong> are the coordinates of P in frame O′,</li><li>P are the coordinates of P in frame O,</li><li><strong>R</strong> is the rotation matrix that maps O′ onto O.</li></ul><p>R transforms the coordinates of a vector from frame O′ into the coordinates of the <strong>same</strong> vector in frame O.</p><h3>Further Meaning: Rotating a Vector</h3><p>So far we have seen that</p><p>P = R · P′</p><p>relates the coordinates of the <strong>same</strong> point P in two frames. We can also read this formula in another way: as a description of how a vector rotates in a fixed frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*LMU8yYNtuDOvMQbmYkVOHg.png" /></figure><p>Consider the 2D picture in the x–y plane (z is coming out of the page):</p><ul><li>The original vector is <strong>P′</strong>, with coordinates<br> P′ = [ p′ₓ, p′ᵧ, p′𝑧 ]ᵀ.</li><li>We rotate this vector by an angle α around the z-axis and obtain <strong>P</strong>, with coordinates<br> P = [ pₓ, pᵧ, p𝑧 ]ᵀ.</li></ul><p>Let β be the angle between P′ and the x-axis. After the rotation by α, the new vector P makes an angle β + α with the x-axis.</p><p>The length of the vector (call it |P|) does not change during rotation. Using basic trigonometry:</p><p>pₓ = |P| · cos(β + α)<br>pᵧ = |P| · sin(β + α)<br>p𝑧 = p′𝑧 (rotation around z does not change z)</p><p>Recall the angle–addition formulas:</p><p>cos(β + α) = cosβ · cosα − sinβ · sinα<br>sin(β + α) = sinβ · cosα + cosβ · sinα</p><p>Now, from the original vector P′ we also have</p><p>p′ₓ = |P’| · cosβ<br>p′ᵧ = |P’| · sinβ</p><p>Substituting these into the expressions for pₓ and pᵧ:</p><p>pₓ = |P| · (cosβ · cosα − sinβ · sinα)<br> = p′ₓ · cosα − p′ᵧ · sinα</p><p>pᵧ = |P| · (sinβ · cosα + cosβ · sinα)<br> = p′ₓ · sinα + p′ᵧ · cosα<br>p𝑧 = p′𝑧</p><p>In vector form:</p><p>pₓ = cosα · p′ₓ − sinα · p′ᵧ<br>pᵧ = sinα · p′ₓ + cosα · p′ᵧ<br>p𝑧 = p′𝑧</p><p>or, as a matrix multiplication:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*9Gdp8OlJkPS2DOYRAs281g.png" /></figure><p>This is exactly</p><p>P = Rz(α) · P′</p><p>So the same rotation matrix Rz(α) that we built from the geometry of the axes also describes how a vector’s coordinates change when we actively rotate the vector by α around the z-axis.</p><h3>Conclusion</h3><p>In Part I, we saw how robots describe orientation using rotation matrices and coordinate frames. In this part, we went one level deeper and uncovered the geometry and algebra behind those matrices.</p><p>We started with simple pictures in 2D and then extended them to 3D:</p><ul><li>By looking at the projections of the rotated axes, we built the three basic rotation matrices around the coordinate axes:<br> Rx(γ), Ry(β) and Rz(α).<br> Each entry in these matrices is nothing more than a cosine or sine of the angle between an axis and its projection.</li><li>We then interpreted a rotation matrix as a <strong>change of basis</strong> between two frames that share the same origin.<br> The equation<br> P = R · P′<br> means: “take the coordinates of P in frame O′ and express the same point in frame O.”</li><li>Finally, we looked at the “active” view of rotation: keeping the frame fixed and physically rotating the vector itself. Using the angle–addition formulas for sine and cosine, we showed that the coordinates of a rotated vector are transformed by the same matrix Rz(α).<br> In other words, <strong>rotating the frame</strong> and <strong>rotating the vector</strong> are two sides of the same mathematical coin.</li></ul><p>By now, rotation matrices should feel less like mysterious 3×3 grids of numbers and more like what they really are: compact summaries of geometry. Every column is just a rotated axis; every entry is a projection; every multiplication is either a change of coordinates or a rotation in space.</p><p>In the next part of this series, we will stay with pure rotation and go a step further: we will compose multiple rotation matrices, introduce three-angle representations of orientation, and explore how they relate to each other. We will look at Euler’s rotation theorem, ZYZ Euler angles, roll–pitch–yaw angles, and the inverse problem of recovering angles from a given rotation matrix.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6d0810a804a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part I — How Robots Understand Space: Kinematics and the Power of Rotation Matrices]]></title>
            <link>https://medium.com/@sepideh.92sh/part-i-how-robots-understand-space-kinematics-and-the-power-of-rotation-matrices-6b2ba5bc07be?source=rss-747261c080c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b2ba5bc07be</guid>
            <dc:creator><![CDATA[Sepideh Shamsizadeh]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 20:29:38 GMT</pubDate>
            <atom:updated>2025-12-14T14:14:23.313Z</atom:updated>
            <content:encoded><![CDATA[<h2>Part I — How Robots Understand Space: Kinematics and the Power of Rotation Matrices</h2><p><strong>Introduction</strong><br>In robotics and computer graphics, understanding how objects move and orient themselves in space is essential. From robotic arms positioning tools with precision to drones stabilizing in flight or 3D models rotating on screen, all rely on the same mathematical foundation — kinematics and transformations.<br> Kinematics describes motion without considering forces, defining how components of a system relate through position, velocity, and acceleration. These relationships are expressed through transformation matrices, which combine translation and rotation. Among them, the rotation matrix is key — it determines how an object’s orientation changes in 2D or 3D space while preserving its shape and scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCIHa8KkCHEluHbkxcIUwQ.png" /><figcaption>Kinematics</figcaption></figure><h3>Where Is a Rigid Body Located in Space?</h3><p>The first important question we need to address is: <strong>Where is a rigid body located in space?</strong><br> A <em>rigid body</em> is a solid object in which the distance between any two points remains constant over time, regardless of external forces. Such a body is fully described by its <strong>pose</strong>, which consists of its <strong>position</strong> and <strong>orientation</strong> relative to a chosen reference frame:</p><p><strong><em>Pose = Position + Orientation</em></strong></p><h3>Before We Begin: A Quick Review of Vectors and Trigonometry</h3><p>Before diving into kinematics, it helps to refresh two tools we’ll use constantly: <strong>vectors</strong> and <strong>trigonometric functions</strong> (sine and cosine).</p><h3>Vectors (quick refresher)</h3><p>A <strong>vector</strong> has both <strong>magnitude</strong> (length) and <strong>direction</strong>. In robotics, vectors represent positions, velocities, accelerations, and forces—anything that says <em>how much</em> and <em>in which direction</em>.</p><p>Let a point <strong>P</strong> have coordinates Px, Py, Pz in a Cartesian frame with unit vectors i, j, k along the x, y, and z axes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*dbZJpIfkUHzt7tJGRaI6BA.png" /></figure><p>Its position vector from the origin is:</p><p>P = Px · i + Py · j + Pz · k</p><p>Equivalently, as a column vector:</p><p>P = [ Px<br> Py<br> Pz ]</p><p>Each component is just a projection (dot product) of P onto the axis unit vectors:</p><p>Px = P · i<br>Py = P · j<br>Pz = P · k</p><h3>Unit Vectors</h3><p>A unit vector has a magnitude of 1 and indicates only direction. They form the basis of the coordinate frame and are used to describe orientations and transformations in space.</p><p>Below is a simple illustration of the three unit vectors in a 3D Cartesian coordinate system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*SoCjf1iJ3rZUdd5QfOBkxQ.png" /><figcaption>Unit Vectors</figcaption></figure><p>Each unit vector defines a direction:</p><p>i → along the x-axis</p><p>j → along the y-axis</p><p>k → along the z-axis</p><p>Together, they define the 3D reference frame.</p><h3>Sine and Cosine Refresher</h3><p>The sine (sin) and cosine (cos) functions describe the relationship between an angle and the sides of a right triangle. They are fundamental for expressing rotations and projections in space. For an angle θ:</p><p>sin(θ) = opposite / hypotenuse</p><p>cos(θ) = adjacent / hypotenuse</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4VfmtDL1vtPi-eEhnMhEEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JIpxxSFnwG38w65wNN6ww.png" /><figcaption>Sin &amp; Cos</figcaption></figure><p>These functions help determine how much a point moves or rotates along each axis when an object turns — a key concept behind <strong>rotation matrices</strong>.</p><h3>Describing Orientation and Position in Space</h3><p>To describe how a rigid body is oriented in space, we use <strong>reference frames</strong>.<br> A <strong>reference frame</strong> defines both the <strong>origin</strong> and the <strong>directions</strong> from which we measure positions and rotations.</p><p>In the figure below, two <strong>orthonormal coordinate frames</strong> are shown:</p><ul><li><strong>O–xyz:</strong> the <strong>world (or inertial) frame</strong>, fixed in space.</li><li><strong>O′–x′y′z′:</strong> the <strong>body frame</strong>, which moves and rotates together with the rigid body.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0yHwc74NtnshE2IrOa0-Q.png" /><figcaption>World frame O–xyz and body-attached frame O′–x′y′z′</figcaption></figure><p>Here, the origin <strong>O′</strong> of the body frame has shifted relative to <strong>O</strong>, and its axes (<strong>i′</strong>, <strong>j′</strong>, <strong>k′</strong>) are rotated with respect to the world axes (<strong>i</strong>, <strong>j</strong>, <strong>k</strong>).</p><p>However, <strong>orientation alone</strong> doesn’t tell us where the body is located in space.<br> To fully describe the body’s pose, we also need its <strong>position</strong>, defined by the location of <strong>O′</strong> relative to the world origin <strong>O</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*KtwGDsmo8uOItl7UE0dUmw.png" /><figcaption><em>World frame O–xyz and body-attached frame O′–x′y′z′. The red vector represents the position of O′ relative to O.</em></figcaption></figure><h3>Position of the Body Frame Origin</h3><p>The position of the body frame’s origin <strong>O′</strong>, expressed in <strong>world coordinates</strong>, is represented by a <strong>position vector</strong>.<br> Each component defines how far <strong>O′</strong> lies along the corresponding world axis (<strong>x</strong>, <strong>y</strong>, <strong>z</strong>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/76/0*NXez_AITv9qkAIOT" /></figure><p>or equivalently, using unit vectors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/0*_m0n52TY4WvSeD9D" /></figure><p>This vector defines the <strong>translation</strong> of the body frame relative to the world frame and serves as the second part of the rigid body’s <strong>pose</strong>, alongside its orientation.</p><h3>Expressing the Orientation of the Body Frame</h3><p>To describe how the <strong>body frame O′–x′y′z′</strong> is oriented relative to the <strong>world frame O–xyz</strong>, we must express each of the <strong>body frame’s unit vectors</strong> (<strong>i′</strong>, <strong>j′</strong>, <strong>k′</strong>) in terms of the <strong>world frame’s unit vectors</strong> (<strong>i</strong>, <strong>j</strong>, <strong>k</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*SF9CDbWrKhDvaV2VLT5C-w.png" /><figcaption><em>The unit vector i′ of the body frame expressed in the world frame O–xyz.</em></figcaption></figure><p>Each axis of the body frame can be represented as a linear combination of the world axes.<br> For example, the unit vector <strong>i′</strong> (the x′ direction of the body frame) can be written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/66/0*mUnz24Ud1f77z5AG" /></figure><p>or equivalently, in vector form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*bXVBr7dLFb3kvDWp" /></figure><p>Similarly, the y′-axis unit vector j′ can be expressed in the same way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/0*bgu-milSVE69kvn9" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*ietnq5R_S3cvIW2MbUF5EQ@2x.jpeg" /><figcaption>The unit vector j′ of the body frame expressed in the world frame O — xyz.</figcaption></figure><p>Here, the red arrow represents j′, projected onto the world axes.</p><p>Its components j′x, j′y, and j′z define how the body’s y′-axis is oriented in space relative to the world coordinate system.</p><p>Finally, the same relationship holds for the z′-axis unit vector k′, which can also be expressed in terms of the world frame axes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/0*vNOhYAOoCRAHeQ-x" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*azRwmnVyfZdSdQCwj-7UqQ@2x.jpeg" /><figcaption><em>The unit vector k′ of the body frame expressed in the world frame O–xyz.</em></figcaption></figure><p>Each of the three body-frame unit vectors (i′, j′, k′) is therefore described by its direction cosines relative to the world frame.</p><p>Together, these relationships define the complete rotation matrix, which represents the orientation of the body frame O′ — x′y′z′ with respect to the world frame O — xyz.</p><h4>The Rotation Matrix Representation</h4><p>The rotation matrix R is a 3×3 matrix composed of the direction cosine coefficients of the three body axes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/0*B_kglnP-MO7Ud-HO" /></figure><p>Each column of R corresponds to one of the body frame’s axes expressed in the world frame:</p><p>•	1st column → i′ (x′-axis)</p><p>•	2nd column → j′ (y′-axis)</p><p>•	3rd column → k′ (z′-axis)</p><p>Thus, the matrix R provides a complete description of the rigid body’s orientation in space.</p><h4>Transforming Between Frames</h4><p>Once R is known, we can easily transform coordinates between the body and world frames.</p><p>If a point has coordinates p′ in the body frame, its coordinates p in the world frame are given by:</p><p>p = R · p′</p><p>Conversely, to express a world vector in the body frame:</p><p>p′ = Rᵀ · p</p><p>Because rotation matrices are orthonormal, their inverse equals their transpose:</p><p>R⁻¹ = Rᵀ</p><p>This property makes rotation transformations simple, stable, and computationally efficient — which is why they are the foundation of robot kinematics and computer graphics transformations.</p><h4>Showing RᵀR=I explicitly</h4><p>Let the body axes expressed in the world frame be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*UeCdkV_zEgrGPceZsT6O5w.png" /></figure><p>Then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*fGMmv9DQCRAiWbOFg_ENCw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*8at05iX36dAfZoV2JyY-kw.png" /></figure><p>Recognize each sum as a dot product of <strong>body axes</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*soo8Hi2rhqJhHd5Q0NcZvg.png" /></figure><p>because i′,j′,k′ are unit and mutually orthogonal.</p><p>Therefore,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*tTh8OhKHA6db1vcc6la7bw.png" /></figure><h4>Conclusion</h4><p>In this first part, we built the foundation for how robots “understand” space. We clarified the role of reference frames, reviewed vectors and unit vectors, and showed how a body’s <strong>pose</strong> combines position and orientation. Most importantly, we expressed orientation with the <strong>rotation matrix</strong> R, whose columns are the body axes in world coordinates, and we proved its orthonormality via RᵀR=I. This property makes rotations numerically stable, easily invertible, and central to nearly every robotics and graphics pipeline.</p><p>With these ideas in place, we’re ready to move from pure orientation to full spatial motion. Next, we’ll combine rotation with translation to represent rigid motions compactly and compose them cleanly.</p><p><strong>Coming next:</strong></p><ul><li><a href="https://medium.com/@sepideh.92sh/part-ii-inside-rotation-matrices-axes-frames-and-coordinate-transformations-f6d0810a804a"><strong>Part II — Inside Rotation Matrices: Axes, Frames, and Coordinate Transformations</strong></a></li><li><a href="https://medium.com/p/38aa816a5bcd"><strong>Part III — Composing Rotations: Euler Angles and Roll–Pitch–Yaw</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b2ba5bc07be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup and Troubleshooting of Raspberry Pi Camera Module v2.1 (IMX219) on Ubuntu 24.04 LTS]]></title>
            <link>https://medium.com/@sepideh.92sh/setup-and-troubleshooting-of-raspberry-pi-camera-module-v2-1-imx219-on-ubuntu-24-04-lts-fb518f4576c0?source=rss-747261c080c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb518f4576c0</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[raspberry-pi-4]]></category>
            <category><![CDATA[robotics]]></category>
            <dc:creator><![CDATA[Sepideh Shamsizadeh]]></dc:creator>
            <pubDate>Sat, 25 Oct 2025 17:50:36 GMT</pubDate>
            <atom:updated>2025-10-25T17:50:36.853Z</atom:updated>
            <content:encoded><![CDATA[<h3>Abstract</h3><p>This document presents a complete guide for installing, configuring, and troubleshooting the Raspberry Pi Camera Module v2.1 (IMX219) under Ubuntu 24.04 LTS on a Raspberry Pi board. It includes detailed instructions for building the libcamera and rpicam utilities from source, resolving library-loading issues, and diagnosing hardware communication errors such as <em>“No cameras available!”</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OthWDBldtoIDxTzi8x25RA.png" /></figure><h3>1. Introduction</h3><p>The Raspberry Pi Camera Module v2.1 uses the Sony IMX219 sensor, which requires the libcamera framework and Raspberry Pi–specific rpicam utilities for image capture and video streaming.<br> Unlike Raspberry Pi OS, Ubuntu 24.04 does not include these tools by default; they must be built from source.<br> This paper describes the installation procedure and provides systematic error-handling strategies for each common failure encountered during setup.</p><h3>2. System Requirements</h3><ul><li>Raspberry Pi 4 Model B or newer</li><li>Ubuntu 24.04 LTS (64-bit, <em>Noble Numbat</em>)</li><li>Internet connection</li><li>Camera Module v2.1 (IMX219)</li><li>Correctly seated CSI ribbon cable (contacts toward HDMI ports)</li></ul><h3>3. Installation Procedure</h3><h3>3.1. Update and Install Dependencies</h3><pre>sudo apt update<br>sudo apt full-upgrade -y<br>sudo apt install -y git cmake meson ninja-build build-essential \<br>python3-pip python3-yaml python3-ply libboost-dev libgnutls28-dev \<br>openssl libjpeg-dev libtiff5-dev libpng-dev libdrm-dev libexpat1-dev \<br>libcamera-dev v4l-utils</pre><h3>3.2. Build and Install libcamera</h3><pre>cd ~<br>git clone https://github.com/raspberrypi/libcamera.git<br>cd libcamera<br>meson setup build<br>ninja -C build<br>sudo ninja -C build install</pre><h3>3.3. Build and Install rpicam Applications</h3><pre>cd ~<br>git clone https://github.com/raspberrypi/libcamera-apps.git<br>cd libcamera-apps<br>meson setup build --buildtype=release<br>ninja -C build<br>sudo ninja -C build install</pre><p>After compilation, confirm that binaries such as rpicam-hello, rpicam-jpeg, rpicam-still, and rpicam-vid are present in /usr/local/bin.</p><h3>4. Common Errors and Solutions</h3><h3>4.1. libcamera-hello: command not found</h3><p><strong>Cause:</strong> libcamera utilities not installed on Ubuntu.<br> <strong>Solution:</strong> Install or build libcamera as in Section 3.2.<br> Use rpicam-* instead of the deprecated libcamera-* tools.</p><h3>4.2. E: Unable to locate package libcamera-apps</h3><p><strong>Cause:</strong> Ubuntu repositories lack Raspberry Pi–specific packages.<br> <strong>Solution:</strong> Build libcamera-apps manually as shown in Section 3.3.</p><h3>4.3. libcamera-hello: command not found after building</h3><p><strong>Cause:</strong> Binaries were built but not installed in the system PATH.<br> <strong>Solution:</strong></p><pre>sudo ninja -C build install</pre><p>or temporarily:</p><pre>export PATH=&quot;$HOME/libcamera-apps/build:$PATH&quot;</pre><h3>4.4. rpicam-hello: error while loading shared libraries: librpicam_app.so.1</h3><p><strong>Cause:</strong> The dynamic linker cannot find the installed library.<br> <strong>Solution:</strong></p><pre>echo &quot;/usr/local/lib/aarch64-linux-gnu&quot; | sudo tee /etc/ld.so.conf.d/rpicam.conf<br>sudo ldconfig</pre><p>Then verify:</p><pre>ldconfig -p | grep librpicam_app</pre><p>The output should include /usr/local/lib/aarch64-linux-gnu/librpicam_app.so.1.</p><h3>4.5. No cameras available!</h3><p><strong>Cause:</strong> The IMX219 driver is loaded, but the camera is not communicating through the CSI interface.</p><p><strong>Diagnostic and Handling Steps:</strong></p><ol><li><strong>Verify Physical Connection</strong></li></ol><ul><li>Power off the Raspberry Pi.</li><li>Reseat the ribbon cable (contacts facing HDMI ports).</li><li>Reboot the system.</li></ul><p><strong>2. Check I²C Detection</strong></p><pre>sudo apt install -y i2c-tools<br>sudo i2cdetect -y 10</pre><p>The grid should show an address (e.g., 0x10).<br> If all cells are --, the cable or module is misconnected.</p><p><strong>3. Confirm Driver Modules</strong></p><pre>lsmod | grep imx219 <br>lsmod | grep unicam</pre><p>If bcm2835_unicam is missing:</p><pre>sudo modprobe bcm2835_unicam</pre><p><strong>4. Inspect Kernel Messages</strong></p><pre>dmesg | grep -E &quot;imx|unicam&quot;b</pre><p>Successful output:</p><pre>imx219 10-0010: Device found, sensor ID = 0x219<br>bcm2835_unicam fe801000.csi: Device registered as /dev/video0<br></pre><p>Failure messages such as probe failed indicate a hardware or overlay issue.</p><p><strong>5. Verify Overlay Configuration</strong><br> Edit /boot/firmware/config.txt:</p><pre>start_x=1 gpu_mem=128 dtparam=i2c_arm=on dtoverlay=imx219</pre><p>Save and reboot:</p><pre>sudo reboot</pre><h3>5. Verification</h3><h3>5.1. List Cameras</h3><pre>rpicam-hello --list-cameras</pre><p>Expected output:</p><pre>Available cameras:<br>0 : imx219 [3280x2464 10-bit RGGB] (/base/soc/i2c0mux/i2c@1/imx219@10)</pre><h3>5.2. Capture a Test Image</h3><pre>rpicam-jpeg -o test.jpg -t 2000</pre><h3>5.3. Record a Short Video</h3><pre>rpicam-vid -t 5000 -o test.h264<br>ffmpeg -i test.h264 -c copy test.mp4</pre><h3>6. Conclusion</h3><p>By following the procedures outlined in this document, the Raspberry Pi Camera Module v2.1 (IMX219) operates reliably on Ubuntu 24.04 LTS.<br> Key issues — such as missing binaries, library path errors, and camera detection failures — can be systematically resolved through build verification, dynamic linker configuration, and driver-level diagnostics.</p><h3>7. References</h3><ul><li>Raspberry Pi Ltd., <em>libcamera-apps Repository</em>, GitHub, 2024</li><li>Raspberry Pi Documentation, <em>Camera Module Hardware Guides</em></li><li>Ubuntu 24.04 LTS (Noble Numbat) Technical Notes</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb518f4576c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic Segmentation: A TensorFlow Exploration of FCN, and Transfer Learning]]></title>
            <link>https://medium.com/@sepideh.92sh/semantic-segmentation-a-tensorflow-exploration-of-fcn-and-transfer-learning-ecca31c624c3?source=rss-747261c080c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecca31c624c3</guid>
            <category><![CDATA[vgg16]]></category>
            <category><![CDATA[fcn]]></category>
            <category><![CDATA[image-segmentation]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[iou]]></category>
            <dc:creator><![CDATA[Sepideh Shamsizadeh]]></dc:creator>
            <pubDate>Thu, 02 Nov 2023 08:12:07 GMT</pubDate>
            <atom:updated>2023-11-02T08:25:18.991Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the world of computer vision, where computers learn to see and understand images. Picture this: a computer not only recognizes objects but also draws precise lines around them — this is the power of image segmentation.</p><p>In our tech-driven age, TensorFlow, a tool that helps machines learn, has become essential. Pair that with the simplicity of Python, and we’re set to demystify image segmentation through a fun project.</p><p>This article is your guide, whether you’re a newbie or a coding pro. Before we dive into the hands-on part, let’s take a moment to understand the different types of image segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4qQPzyL2v8Fx1saBh-6l3Q.png" /><figcaption>Implementation of semantic segmentation(FCN-8) on <a href="https://colab.research.google.com/corgiredirector?site=http%3A%2F%2Fmi.eng.cam.ac.uk%2Fresearch%2Fprojects%2FVideoRec%2FCamVid%2F">CamVid</a> dataset.</figcaption></figure><h3>Types of Image Segmentation: Beyond the Basics</h3><ol><li><strong>Semantic Segmentation:</strong> Classifying each pixel in an image into predefined categories or classes. For example, in a street scene, it would label every pixel as part of a car, pedestrian, road, or building. The goal is to understand the general context of the scene without distinguishing between individual instances.</li><li><strong>Instance Segmentation: </strong>takes it a step further by not only assigning a class to each pixel but also by precisely identifying and outlining the boundaries of each distinct object instance within that class. In a crowded image, it excels at distinguishing between different individuals, even if they belong to the same category, effectively recognizing and outlining each person in a group with a high level of precision.</li><li><strong>Panoptic Segmentation:</strong> A fusion of semantic and instance segmentation. It not only provides a detailed semantic understanding of the scene but also separates each instance within those categories. Additionally, it designates a special category for the background, capturing a holistic view of the entire visual environment. It’s akin to understanding the full story of a scene, including the main characters and the backdrop they’re placed against.</li></ol><p>Now that we’ve brushed over the basics, let’s delve into the practical side. We’ll focus on semantic segmentation, an exciting technique that provides a detailed understanding of images at the pixel level.</p><p>By the end, you won’t just get the theory; you’ll be ready to build your own semantic segmentation model using TensorFlow and Python.</p><p>Get ready for a journey through the exciting world of computer vision. We’re here to turn pixels into meaning and make coding concepts a breeze. Let’s dive in!</p><h3>High-Level Architecture of Image Segmentation: Breaking it Down</h3><p>In the realm of image segmentation, we follow a high-level architecture known as an encoder-decoder structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*iY9ljRqZ6yHPq2qjCQiWfw.png" /><figcaption>Encoder-decoder structure.</figcaption></figure><h4>Encoder — Unveiling Features:</h4><p>The encoder acts as a feature extractor, usually implemented with a Convolutional Neural Network (CNN). This part extracts features from the image, forming what we call a “feature map.” The initial layers focus on low-level features like lines, gradually aggregating them into more complex features like eyes and ears. This aggregation is facilitated by downsampling, akin to reducing pixel representation — like when your video chat gets a bit grainy but still allows you to recognize your friend through those key features.</p><h4>Decoder — Crafting Predictions:</h4><p>Now, let’s move to the decoder. Its role is to take these extracted features and generate the model’s output or prediction. Like the encoder, the decoder is also a convolutional neural network. It assigns intermediate class labels to each pixel in the downsized feature map. Then, it begins the intricate task of upsampling, gradually reintroducing the fine-grained details of the original image. This process continues until the image is restored to its original dimensions.</p><h4>Final Touch — Pixel-Wise Labels:</h4><p>The ultimate result is a pixel-wise labeled map. Each pixel is assigned a final class label, capturing the essence of what that pixel represents in the broader context of the image.</p><p>In simpler terms, think of the encoder as an investigator, diligently extracting clues (features) from the image. The decoder is the storyteller, piecing together these clues to reveal the complete narrative — pixel by pixel. This architecture, with its encoder and decoder dance, forms the backbone of image segmentation algorithms, providing us with detailed, pixel-level insights into visual data.</p><p>In this article, we dissect two powerhouse architectures for semantic segmentation: <strong>Fully Convolutional Networks (FCN) </strong>and<strong> U-Net</strong>. Dive into the core of these frameworks with us as we implement and demystify them using Python and TensorFlow. We’ll provide clear, concise code snippets accompanied by insightful explanations, catering to both beginners and seasoned practitioners. Get ready for a hands-on exploration that bridges theory with practice, empowering you to leverage FCN and U-Net effectively in your projects. Whether you’re a curious enthusiast or a seasoned developer, this article is your guide to mastering semantic segmentation.</p><h3><strong>Fully Convolutional Networks (FCN)</strong></h3><p>In traditional convolutional neural networks (CNNs), the final layers are often fully connected, meaning they connect every neuron to every other neuron. This is fine for tasks like image classification where the spatial information is not as critical.</p><p>However, for tasks like semantic segmentation, where the goal is to classify each pixel in an image into a specific class (e.g., person, car, background), preserving spatial information is essential. Fully Convolutional Neural Networks (FCNs) were introduced by Long, Shelhamer, and Darrell in their paper titled “<a href="https://arxiv.org/abs/1411.4038">Fully Convolutional Neural Networks for Semantic Segmentation</a>”. FCN addresses this by replacing the fully connected layers with convolutional layers. This allows the network to take input of any size and produce a corresponding output of the same spatial dimensions.</p><p>In semantic segmentation, FCNs enable pixel-level predictions by retaining spatial information throughout the network. The network learns to associate each pixel in the input image with a particular class, leading to a detailed segmentation map. This is particularly useful in computer vision applications such as object detection, scene understanding, and image segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gQhOSfDULDKZB8Z44oDWJg.png" /><figcaption>Fully convolutional networks <a href="https://arxiv.org/abs/1411.4038">[1]</a>.</figcaption></figure><p>Continuing with our exploration of Fully Convolutional Networks (FCNs), let’s dive deeper into the variants known as FCN-8, FCN-16, and FCN-32. These are different configurations of the FCN architecture, each with its own unique characteristics.</p><h3>FCN-8, FCN-16, and FCN-32: Unraveling the Configurations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2ryb1cw-OIytBcqZn_phw.png" /><figcaption>Detail the architecture of the adapted nets as regards dense prediction<a href="https://arxiv.org/abs/1411.4038">[1]</a>.</figcaption></figure><h4>FCN-32</h4><p>FCN-32 employs a 32-pixel stride in the final layer, resulting in an output resolution at 1/32nd of the input. This configuration prioritizes computational efficiency over detailed spatial information. FCN-32 is faster in terms of processing, making it suitable for scenarios where speed is a critical factor, and fine-grained segmentation details are less crucial.</p><h4>FCN-8</h4><p>FCN-8 refers to the fully convolutional network with an 8-pixel stride in the final layer. In other words, the final layer produces an output at 1/8th of the input resolution. This configuration allows for more detailed predictions, capturing finer nuances in the segmentation map. The decoder in FCN-8 involves fusing predictions from different layers, including the ones with coarser spatial information. This fusion process enhances the overall segmentation quality.</p><h4>FCN-16</h4><p>Moving on to FCN-16 involves a 16-pixel stride in the final layer. This means the output resolution is 1/16th of the input resolution. While FCN-16 sacrifices a bit of spatial detail compared to FCN-8, it still achieves a good balance between resolution and computation efficiency. The intermediate layers contribute to the final segmentation map, providing a compromise between fine details and computational resources.</p><h3>Choosing the Right Configuration</h3><p>FCN-8 allows detailed predictions with an 8-pixel stride, capturing finer nuances. FCN-16 strikes a balance between resolution and efficiency with a 16-pixel stride, while FCN-32 prioritizes computational efficiency with a 32-pixel stride.</p><p>The choice between these configurations depends on project requirements. In upcoming sections, we’ll implement each using TensorFlow and Python, unlocking the potential of FCN variants in semantic segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*-gSba77_puDFuHVs6aYbrQ.png" /><figcaption>Refining fully convolutional nets by fusing information from layers with different strides improves segmentation detail. The first three images show the output from our 32, 16, and 8-pixel stride nets. <a href="https://arxiv.org/abs/1411.4038">[1]</a></figcaption></figure><p>But there’s more — let’s supercharge our FCN models by integrating transfer learning, leveraging pre-trained encoders like VGG16 or ResNet. This game-changer accelerates training and enhances accuracy, especially with limited labeled data.</p><p>Stay tuned for hands-on implementation, where we’ll provide clear code snippets and insightful explanations, elevating your understanding and proficiency in computer vision. Unleash the full potential of FCN-8, FCN-16, and FCN-32 with the added strength of transfer learning!</p><h4>FCN-Encoder: VGG16 transfer learning</h4><p>VGG16, or Visual Geometry Group 16, is a renowned convolutional neural network known for its simplicity and effectiveness in image classification. With a uniform structure and 16 layers, it excels in learning hierarchical features. In our FCN-Encoder, we’ll leverage VGG16 for transfer learning, tapping into its pre-trained weights to enhance feature extraction in semantic segmentation tasks. Here is the <a href="https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5">link</a> to get pre-trained weights for VGG-16.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PcfZSxvlz9jXaTzp1TF2CA.png" /><figcaption><a href="https://iopscience.iop.org/article/10.1088/1757-899X/245/1/012004/meta">VGG16 Architecture [3].</a></figcaption></figure><p>Following that, we utilize the VGG16 function to create the VGG16 network based on the image mentioned above (just convolutional parts conv1, conv2, conv3, conv4, and conv5). We proceed by loading the downloaded weights and obtaining the essential layers required for the decoder part of the FCN.</p><pre>def VGG_16(image_input):<br>    x = tf.keras.layers.Conv2D(filters=64, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv1-1&#39;)(image_input)<br>    x = tf.keras.layers.Conv2D(filters=64, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv1-2&#39;)(x)<br>    x = tf.keras.layers.MaxPooling2D(pool_size=(2,2), strides=(2,2), name=&#39;max1&#39;)(x)<br>    p1 = x<br><br>    x = tf.keras.layers.Conv2D(filters=128, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv2-1&#39;)(x)<br>    x = tf.keras.layers.Conv2D(filters=128, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv2-2&#39;)(x)<br>    x = tf.keras.layers.MaxPooling2D(pool_size=(2,2), strides=(2,2), name=&#39;max2&#39;)(x)<br>    p2 = x<br><br>    x = tf.keras.layers.Conv2D(filters=256, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv3-1&#39;)(x)<br>    x = tf.keras.layers.Conv2D(filters=256, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv3-2&#39;)(x)<br>    x = tf.keras.layers.Conv2D(filters=256, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv3-3&#39;)(x)<br>    x = tf.keras.layers.MaxPooling2D(pool_size=(2,2), strides=(2,2), name=&#39;max3&#39;)(x)<br>    p3 = x<br><br>    x = tf.keras.layers.Conv2D(filters=512, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv4-1&#39;)(x)<br>    x = tf.keras.layers.Conv2D(filters=512, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv4-2&#39;)(x)<br>    x = tf.keras.layers.Conv2D(filters=512, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv4-3&#39;)(x)<br>    x = tf.keras.layers.MaxPooling2D(pool_size=(2,2), strides=(2,2), name=&#39;max4&#39;)(x)<br>    p4 = x<br><br>    x = tf.keras.layers.Conv2D(filters=512, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv5-1&#39;)(x)<br>    x = tf.keras.layers.Conv2D(filters=512, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv5-2&#39;)(x)<br>    x = tf.keras.layers.Conv2D(filters=512, kernel_size=(3,3), activation=&#39;relu&#39;, padding=&#39;same&#39;, name=&#39;conv5-3&#39;)(x)<br>    x = tf.keras.layers.MaxPooling2D(pool_size=(2,2), strides=(2,2), name=&#39;max5&#39;)(x)<br>    p5 = x<br><br>    vgg = tf.keras.Model(image_input, p5)<br><br>    vgg.load_weights(&#39;vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5&#39;)<br><br>    return (p1, p2, p3, p4, p5)</pre><h4>FCN-8-Decoder: Upsampling</h4><p>The fcn8_decoder function employs Conv2DTranspose for upsampling. This operation is essential for increasing the spatial resolution of the feature maps. Unlike traditional convolution, which involves a stride to reduce resolution, Conv2DTranspose uses a stride to increase resolution.</p><p>In simple terms, it “fills in” the gaps between existing pixels, effectively expanding the image size. This is crucial in semantic segmentation as it helps recover finer details lost during downsampling in the encoder.</p><p>The convolutional transpose operation in o = tf.keras.layers.Conv2DTranspose(n_classes , kernel_size=(4,4) , strides=(2,2) , use_bias=False)(f5) doubles the spatial dimensions, and subsequent operations further refine the segmentation map.</p><p>In summary, Conv2DTranspose is the key to reconstructing detailed information in semantic segmentation by expanding the resolution of feature maps.</p><pre>def fcn8_decoder(convs, n_classes):<br>    f1, f2, f3, f4, p5 = convs<br>    n = 4096<br><br>    c6 = tf.keras.layers.Conv2D( n , ( 7 , 7 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;, name=&quot;conv6&quot;)(p5)<br>    c7 = tf.keras.layers.Conv2D( n , ( 1 , 1 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;, name=&quot;conv7&quot;)(c6)<br><br>    f5 = c7<br>    # upsample the output of the encoder then crop extra pixels that were introduced<br>    o = tf.keras.layers.Conv2DTranspose(n_classes , kernel_size=(4,4) ,  strides=(2,2) , use_bias=False )(f5)<br>    o = tf.keras.layers.Cropping2D(cropping=(1,1))(o)<br><br>    # load the pool 4 prediction and do a 1x1 convolution to reshape it to the same shape of `o` above<br>    o2 = f4<br>    o2 = ( tf.keras.layers.Conv2D(n_classes , ( 1 , 1 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;))(o2)<br><br>    # add the results of the upsampling and pool 4 prediction<br>    o = tf.keras.layers.Add()([o, o2])<br>    print(o.shape)<br>    # upsample the resulting tensor of the operation you just did<br>    o = (tf.keras.layers.Conv2DTranspose( n_classes , kernel_size=(4,4) ,  strides=(2,2) , use_bias=False ))(o)<br>    o = tf.keras.layers.Cropping2D(cropping=(1, 1))(o)<br><br>    # load the pool 3 prediction and do a 1x1 convolution to reshape it to the same shape of `o` above<br>    o2 = f3<br>    o2 = ( tf.keras.layers.Conv2D(n_classes , ( 1 , 1 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;))(o2)<br><br>    # add the results of the upsampling and pool 3 prediction<br>    o = tf.keras.layers.Add()([o, o2])<br>    <br>    # upsample up to the size of the original image<br>    o = tf.keras.layers.Conv2DTranspose(n_classes , kernel_size=(8,8) ,  strides=(8,8) , use_bias=False )(o)<br><br>    # append a softmax to get the class probabilities<br>    o = (tf.keras.layers.Activation(&#39;softmax&#39;))(o)<br><br>    return o </pre><h4>FCN-16-Decoder: Upsampling</h4><p>The FCN-16s architecture combines predictions from the final layer and the pool4 layer with a stride of 16. This enables the network to predict finer details by leveraging high-resolution information from pool4, while still retaining the high-level semantic information from the final layer. It’s a balance between detailed information and semantic context.</p><pre>def fcn16_decoder(convs, n_classes):<br>    f1, f2, f3, f4, p5 = convs<br>    n = 4096<br><br>    c6 = tf.keras.layers.Conv2D( n , ( 7 , 7 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;, name=&quot;conv6&quot;)(p5)<br>    c7 = tf.keras.layers.Conv2D( n , ( 1 , 1 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;, name=&quot;conv7&quot;)(c6)<br><br>    f5 = c7<br>    # upsample the output of the encoder then crop extra pixels that were introduced<br>    o = tf.keras.layers.Conv2DTranspose(n_classes , kernel_size=(4,4) ,  strides=(2,2) , use_bias=False )(f5)<br>    o = tf.keras.layers.Cropping2D(cropping=(1,1))(o)<br><br>    # load the pool 4 prediction and do a 1x1 convolution to reshape it to the same shape of `o` above<br>    o2 = f4<br>    o2 = ( tf.keras.layers.Conv2D(n_classes , ( 1 , 1 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;))(o2)<br><br>    # add the results of the upsampling and pool 4 prediction<br>    o = tf.keras.layers.Add()([o, o2])<br>    <br>    # upsample up to the size of the original image<br>    o = tf.keras.layers.Conv2DTranspose(n_classes , kernel_size=(8,8) ,  strides=(16,16) , use_bias=False )(o)<br><br>    # append a softmax to get the class probabilities<br>    o = (tf.keras.layers.Activation(&#39;softmax&#39;))(o)<br><br>    return o</pre><h4>FCN-32-Decoder: Upsampling</h4><p>In FCN-32s, the single-stream network directly upsamples predictions with a stride of 32 back to pixel level in a single step. This approach simplifies the upsampling process and provides an overall prediction for each pixel.</p><pre>def fcn32_decoder(convs, n_classes):<br>    f1, f2, f3, f4, p5 = convs<br>    n = 4096<br><br>    c6 = tf.keras.layers.Conv2D( n , ( 7 , 7 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;, name=&quot;conv6&quot;)(p5)<br>    c7 = tf.keras.layers.Conv2D( n , ( 1 , 1 ) , activation=&#39;relu&#39; , padding=&#39;same&#39;, name=&quot;conv7&quot;)(c6)<br><br>    f5 = c7<br>    # upsample up to the size of the original image<br>    o = tf.keras.layers.Conv2DTranspose(n_classes , kernel_size=(8,8) ,  strides=(32,32) , use_bias=False )(f5)<br><br>    # append a softmax to get the class probabilities<br>    o = (tf.keras.layers.Activation(&#39;softmax&#39;))(o)<br><br>    return o</pre><p>In the context of upsampling, the stride is crucial because it determines the spacing between the sampled points, influencing the output size. The kernel size, on the other hand, affects the filter size and the information each unit gathers, but it doesn’t directly impact the output spatial resolution during upsampling. The choice of stride is key for controlling how finely the network predicts at the pixel level. You can find all codes of implementation <a href="https://github.com/sepideh-shamsizadeh/Image-Segmentation">here</a>.</p><h4>Calculating Metrics for Mask Evaluation</h4><p>This function computes two essential metrics — Intersection over Union (IOU) and Dice Score — between prediction and ground truth masks. These metrics are expressed by the formulas:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*q4KenFq6BkKs6IvMgz7ddA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXSBMhzdB2Ops65K-doa0w.png" /></figure><p>The provided code automates this calculation for your convenience. Additionally, a slight smoothening factor has been incorporated into the denominators to prevent potential division by zero scenarios.</p><pre>def IOU_diceScore(y_true, y_pred):<br>    class_wise_iou = []<br>    class_wise_dice_score = []<br><br>    smoothening_factor = 0.00001<br><br>    for i in range(12):<br>        intersection = np.sum((y_pred==i) * (y_true==i))<br>        y_true_area = np.sum((y_true==1))<br>        y_pred_area = np.sum((y_pred==i))<br>        combined_area = y_true_area + y_pred_area<br><br>        iou = (intersection + smoothening_factor) / (combined_area-intersection+smoothening_factor)<br>        class_wise_iou.append(iou)<br><br>        dice_score = 2 * ((intersection+smoothening_factor)) / (combined_area + smoothening_factor)<br>        class_wise_dice_score.append(dice_score)<br><br>    return class_wise_iou, class_wise_dice_score</pre><h3>Conclusion</h3><p>In concluding our exploration of semantic segmentation with TensorFlow, we’ve covered the basics, decoded image segmentation types, and delved into the high-level architecture of FCN. The article dissected FCN variants, emphasizing the strategic choice between FCN-8, FCN-16, and FCN-32 based on project needs.</p><p>The integration of transfer learning, notably with VGG16 as an encoder, enhances our FCN models, boosting training efficiency and accuracy, especially in scenarios with limited labeled data.</p><p>Our detailed dive into FCN-8, FCN-16, and FCN-32 decoders, coupled with metrics like IOU and Dice Score for model evaluation, equips readers to implement these architectures effectively.</p><p>Whether you’re a novice or an experienced developer, this article provides a concise guide, complete with clear code snippets and hands-on implementation. To access all the implementation codes and data preprocessing details, visit <a href="https://github.com/sepideh-shamsizadeh/Image-Segmentation/tree/main">this link</a>. Now armed with the tools to turn pixels into meaning, you’re ready to unleash the full potential of semantic segmentation using TensorFlow and Python. Let’s dive in and master the art of computer vision!</p><h3>References</h3><p>[1] <a href="https://arxiv.org/abs/1411.4038">Fully Convolutional Networks for Semantic Segmentation</a></p><p>[2] <a href="https://arxiv.org/abs/1409.1556">VGG16 original paper (Very Deep Convolutional Networks for Large-Scale Image Recognition)</a></p><p>[3] <a href="https://iopscience.iop.org/article/10.1088/1757-899X/245/1/012004/meta">VGG16 Architecture</a></p><p><a href="https://github.com/sepideh-shamsizadeh/Image-Segmentation/tree/main">Codes on Github</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecca31c624c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding PointNet: A Practical Guide to 3D Segmentation with Python and PyTorch]]></title>
            <link>https://medium.com/@sepideh.92sh/decoding-pointnet-a-practical-guide-to-3d-segmentation-with-python-and-pytorch-7a037fecb8a7?source=rss-747261c080c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a037fecb8a7</guid>
            <category><![CDATA[pointnet]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[3d-vision]]></category>
            <category><![CDATA[3d-segmentation]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Sepideh Shamsizadeh]]></dc:creator>
            <pubDate>Mon, 16 Oct 2023 13:43:10 GMT</pubDate>
            <atom:updated>2023-10-16T13:43:10.318Z</atom:updated>
            <content:encoded><![CDATA[<p>Get ready to explore the world of 3D segmentation as we take a journey through PointNet, a super cool way to understand 3D shapes. PointNet is like a smart tool for computers to look at 3D things, especially groups of points floating in space. It’s different from other methods because it works directly with these points, without forcing them into grids or pictures.</p><p>In this article, we’re going to make PointNet easy to understand. We’ll go from the big ideas to actually writing code in Python and PyTorch to do 3D segmentation. But before we jump into the fun part, let’s first get what PointNet is all about — how it became a big deal for figuring out 3D stuff like objects and their parts.</p><p>So, join us as we look at the summary of the PointNet paper. We’ll talk about its design, the cool theories behind it, and how it showed off in real experiments. We’ll keep it simple as we explore the world of random points, special functions, and how PointNet handles different 3D tasks. Ready to discover the power of 3D segmentation with PointNet? Let’s get started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GzFBdJCZu4Zu0-VJlGcDbg.png" /></figure><h4>Understanding PointNet: The Core Concepts</h4><p>Now that we’ve set the stage, let’s break down PointNet into bite-sized pieces. PointNet is like a special tool that helps computers understand 3D things, especially those tricky point clouds. But what makes it so cool? Unlike other methods that organize data neatly, PointNet takes points as they are — no need for grids or pictures. This makes it stand out in the world of 3D.</p><p><strong>The Basics of Point Sets:</strong></p><p>Imagine a bunch of points floating around in 3D space. These points don’t follow a specific order, and they interact with each other. PointNet deals with this randomness by being friendly to changes like spinning or moving around. It doesn’t get confused when the points switch places.</p><p><strong>Symmetry Magic:</strong></p><p>PointNet has a special power called symmetry. Imagine you have a bag of points, and no matter how you shuffle them, PointNet can still understand what’s inside. It’s like magic for dealing with points that don’t follow a particular order.</p><p><strong>Gathering Local and Global Info:</strong></p><p>PointNet is smart at gathering information. It looks at the big picture (global) and the details (local) of the points. This helps it do tasks like figuring out the shape of objects and their parts.</p><p><strong>Alignment Trick:</strong></p><p>PointNet is also good at dealing with changes. If you spin or move the points, PointNet can adjust and understand things just fine. It’s like a robot that aligns things so it can see them clearly.</p><h4><strong>Theoretical Magic:</strong></h4><p>Now, let’s talk about some big ideas behind PointNet. There are two special theorems that say PointNet is not just cool in practice, but it’s also a smart choice in theory.</p><p><strong>1. Universal Approximation:</strong><br> — PointNet can learn to understand any 3D shape really well. It’s like saying PointNet is a superhero that can tackle any shape you throw at it.</p><p><strong>2. Bottleneck Dimension and Stability:</strong><br> — PointNet is tough. Even if you add some extra points or mess with the ones it already has, it doesn’t get confused. It sticks to its job and stays stable.</p><p>These big ideas make PointNet a trustworthy tool for understanding 3D shapes. Stay tuned as we move from theory to practice in the next section, where we’ll get our hands dirty with some coding in Python and PyTorch.</p><h4>PointNet Architecture Overview:</h4><p>The PointNet architecture is composed of two main components: the classification network and an extended segmentation network. The classification network takes n input points, applies input and feature transformations, and aggregates point features through max pooling. It produces classification scores for k classes. The segmentation network is a natural extension of the classification network, combining global and local features to generate per-point scores. The term “mlp” denotes a multi-layer perceptron, with layer sizes specified in square brackets. Batch normalization is consistently applied to all layers, accompanied by ReLU activation. Additionally, dropout layers are strategically incorporated in the final mlp of the classification network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*OU2tozHPZjL-qxKD14qhSg.png" /><figcaption>PointNet Architecture.</figcaption></figure><p>In the provided code snippet, the MLP_CONV class encapsulates the application of ReLU activation on the output of a batch-normalized convolutional layer. This mirrors the convolutional and mlp layers described in the architectural figure. Let&#39;s take a closer look at the code:</p><pre># Multi Layer Perceptron<br>class MLP_CONV(nn.Module):<br>   def __init__(self, input_size, output_size):<br>     super().__init__()<br>     self.input_size   = input_size<br>     self.output_size  = output_size<br>     self.conv  = nn.Conv1d(self.input_size, self.output_size, 1)<br>     self.bn    = nn.BatchNorm1d(self.output_size)<br><br>   def forward(self, input):<br>     return F.relu(self.bn(self.conv(input)))</pre><p>This class definition corresponds to the architecture’s building blocks, where convolutional layers, batch normalization, and ReLU activation are combined to achieve the desired feature transformations. Additionally, the FC_BN class, described below, complements this architecture when fully connected layers are employed.</p><pre># Fully Connected with Batch Normalization<br>class FC_BN(nn.Module):<br>   def __init__(self, input_size, output_size):<br>     super().__init__()<br>     self.input_size   = input_size<br>     self.output_size  = output_size<br>     self.lin  = nn.Linear(self.input_size, self.output_size)<br>     self.bn = nn.BatchNorm1d(self.output_size)<br><br>   def forward(self, input):<br>     return F.relu(self.bn(self.lin(input)))</pre><p>This class further illustrates the integration of fully connected layers with batch normalization and ReLU activation, emphasizing the consistent application of these techniques in the PointNet architecture.</p><h4>Input and Feature Transformations:</h4><p>The input transformation network, a mini-PointNet denoted as TNet, is instrumental in processing raw point clouds. It aims to regress to a 3 × 3 matrix through a series of operations. The network architecture is defined by a shared MLP(64, 128, 1024) applied to each point, followed by max pooling across points and two fully connected layers with output sizes 512 and 256. The resulting matrix is initialized as an identity matrix. Each layer, excluding the last, incorporates ReLU activation and batch normalization. The second transformation network mirrors the architecture of the first but outputs a 64 × 64 matrix, similarly initialized as an identity matrix. A regularization loss, with a weight of 0.001, is added to the softmax classification loss to promote orthogonality.</p><p>The TNet class is employed to create transformation networks based on the specifications provided in the paper:</p><pre># Transformation Network (TNet) class<br>class TNet(nn.Module):<br>   def __init__(self, k=3):<br>      super().__init__()<br>      self.k = k<br><br>      self.mlp1 = MLP_CONV(self.k, 64)<br>      self.mlp2 = MLP_CONV(64, 128)<br>      self.mlp3 = MLP_CONV(128, 1024)<br><br>      self.fc_bn1 = FC_BN(1024, 512)<br>      self.fc_bn2 = FC_BN(512, 256)<br><br>      self.fc3 = nn.Linear(256, k*k)<br><br>   def forward(self, input):<br>      # input.shape == (batch_size, n, 3)<br><br>      bs = input.size(0)<br>      xb = self.mlp1(input)<br>      xb = self.mlp2(xb)<br>      xb = self.mlp3(xb)<br><br>      pool = nn.MaxPool1d(xb.size(-1))(xb)<br>      flat = nn.Flatten(1)(pool)<br><br>      xb = self.fc_bn1(flat)<br>      xb = self.fc_bn2(xb)<br><br>      # initialize as identity<br>      init = torch.eye(self.k, requires_grad=True).repeat(bs, 1, 1)<br>      if xb.is_cuda:<br>        init = init.cuda()<br>      matrix = self.fc3(xb).view(-1, self.k, self.k) + init<br>      return matrix</pre><p>This TNet class encapsulates the process of transforming input point clouds into 3 × 3 or 64 × 64 matrices, utilizing shared MLPs, max pooling, and fully connected layers with batch normalization.</p><h4>PointNet Network:</h4><p>The PointNet network, encapsulated in the PointNet class, follows the design principles laid out in the PointNet architecture image:</p><pre>class PointNet(nn.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.input_transform = TNet(k=3)<br>        self.feature_transform = TNet(k=64)<br>        self.mlp1 = MLP_CONV(3, 64)<br>        self.mlp2 = MLP_CONV(64, 128)<br>        <br>        # 1D convolutional layer with kernel size 1<br>        self.conv = nn.Conv1d(128, 1024, 1)<br>        <br>        # Batch normalization for stability and faster training<br>        self.bn = nn.BatchNorm1d(1024)<br><br>    def forward(self, input):<br>        n_pts = input.size()[2]<br>        matrix3x3 = self.input_transform(input)<br>        input_transform_output = torch.bmm(torch.transpose(input, 1, 2), matrix3x3).transpose(1, 2)<br>        x = self.mlp1(input_transform_output)<br>        matrix64x64 = self.feature_transform(x)<br>        feature_transform_output = torch.bmm(torch.transpose(x, 1, 2), matrix64x64).transpose(1, 2)<br>        x = self.mlp2(feature_transform_output)<br>        x = self.bn(self.conv(x))<br>        global_feature = nn.MaxPool1d(x.size(-1))(x)<br>        global_feature_repeated = nn.Flatten(1)(global_feature).repeat(n_pts, 1, 1).transpose(0, 2).transpose(0, 1)<br><br>        return [feature_transform_output, global_feature_repeated], matrix3x3, matrix64x64</pre><p>This PointNet implementation seamlessly integrates the TNet transformation networks, multi-layer perceptrons (MLP_CONV), and 1D convolutional layers with batch normalization. The forward pass handles input and feature transformations, followed by the extraction of global features. The resulting tensors, along with the transformation matrices, are returned as outputs.</p><h4>PointNet Segmentation Network:</h4><p>The segmentation network extends the classification PointNet. Local point features from the second transformation network and global features from max pooling are concatenated for each point. No dropout is used in the segmentation network, and training parameters are consistent with the classification network.</p><p>For shape part segmentation, modifications include the addition of a one-hot vector indicating the class of the input, concatenated with the max pooling layer’s output. Neurons in some layers are increased, skip links are added to collect local point features in different layers, and they are concatenated to form the point feature input to the segmentation network.</p><pre>class PointNetSeg(nn.Module):<br>    def __init__(self, classes=3):<br>        super().__init__()<br>        self.pointnet = PointNet()<br>        self.mlp1 = MLP_CONV(1088, 512)<br>        self.mlp2 = MLP_CONV(512, 256)<br>        self.mlp3 = MLP_CONV(256, 128)<br>        self.conv = nn.Conv1d(128, classes, 1)<br>        self.logsoftmax = nn.LogSoftmax(dim=1)<br>    <br>    def forward(self, input):<br>        inputs, matrix3x3, matrix64x64 = self.pointnet(input)<br>        stack = torch.cat(inputs, 1)<br>        x = self.mlp1(stack)<br>        x = self.mlp2(x)<br>        x = self.mlp3(x)<br>        output = self.conv(x)<br>        return self.logsoftmax(output), matrix3x3, matrix64x64</pre><p>In the PointNetSeg class, the forward pass incorporates the features obtained from the PointNet, concatenates them, and then passes them through a series of multi-layer perceptrons (MLP_CONV) and a convolutional layer. The final output is obtained after applying the LogSoftmax activation.</p><h3>Training and Testing the PointNet Model</h3><p>In our model training journey, we harnessed the power of PointNet by leveraging point clouds from the renowned Semantic-Kitti dataset. This influential dataset, capturing various urban scenes, originally encompasses around 30 labels. However, for our purposes, we judiciously remapped them into three categories:</p><ol><li><strong>Traversable:</strong> Encompassing road, parking, sidewalk, and more.</li><li><strong>Not-Traversable:</strong> Including cars, trucks, fences, trees, people, and various objects.</li><li><strong>Unknown:</strong> Reserved for outliers.</li></ol><p>The remapping process involves the use of a key-value dictionary to transform original labels into their reduced counterparts. To visualize the colored point clouds, we utilized the Open3D Python package. The left image showcases the Semantic-Kitti original color scheme, while the right reveals the remapped color scheme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GmqvhMXVJxKEr5arkXy59w.png" /></figure><p>You can find the code for loading and visualizing data <a href="https://github.com/sepideh-shamsizadeh/3DP-Point-Cloud-Segmentation/blob/1d3a874919988c2c508ac64934566fa02f1060ce/data_processing.py">here</a>.</p><h4>Data Transformation</h4><p>A crucial step in preparing our data involved normalization and tensor conversion through custom transformations. Two primary transformations were employed:</p><p><strong>Normalize: </strong>This operation centers the point cloud by subtracting its mean and scales it to ensure the maximum norm is unity.</p><pre>class Normalize(object):<br>    def __call__(self, pointcloud):<br>        assert len(pointcloud.shape)==2<br><br>        norm_pointcloud = pointcloud - np.mean(pointcloud, axis=0)<br>        norm_pointcloud /= np.max(np.linalg.norm(norm_pointcloud, axis=1))<br><br>        return  norm_pointcloud</pre><p><strong>ToTensor: </strong>This transformation converts the point cloud into a PyTorch tensor.</p><pre>class ToTensor(object):<br>    def __call__(self, pointcloud):<br>        assert len(pointcloud.shape)==2<br><br>        return torch.from_numpy(pointcloud)</pre><p>The combination of these transformations was encapsulated in the default_transforms() function.</p><h4><strong>PointCloud Dataset</strong></h4><p>We then crafted a custom dataset, PointCloudData, extending PyTorch&#39;s Dataset class. This dataset represents a collection of point clouds for training and testing. The structure involves:</p><ul><li>Initializing with dataset details and an optional transformation function.</li><li>Defining the length of the dataset.</li><li>Retrieving an item, and applying transformations if specified.</li></ul><pre>class PointCloudData(Dataset):<br>    def __init__(self, dataset_path, transform=default_transforms(), start=0, end=1000):<br>        &quot;&quot;&quot;<br>          INPUT<br>              dataset_path: path to the dataset folder<br>              transform   : transform function to apply to point cloud<br>              start       : index of the first file that belongs to dataset<br>              end         : index of the first file that do not belong to dataset<br>        &quot;&quot;&quot;<br>        self.dataset_path = dataset_path<br>        self.transforms = transform<br><br>        self.pc_path = os.path.join(self.dataset_path, &quot;sequences&quot;, &quot;00&quot;, &quot;velodyne&quot;)<br>        self.lb_path = os.path.join(self.dataset_path, &quot;sequences&quot;, &quot;00&quot;, &quot;labels&quot;)<br><br>        self.pc_paths = os.listdir(self.pc_path)<br>        self.lb_paths = os.listdir(self.lb_path)<br>        assert(len(self.pc_paths) == len(self.lb_paths))<br><br>        self.start = start<br>        self.end   = end<br><br>        # clip paths according to the start and end ranges provided in input<br>        self.pc_paths = self.pc_paths[start: end]<br>        self.lb_paths = self.lb_paths[start: end]<br><br>    def __len__(self):<br>        return len(self.pc_paths)<br><br>    def __getitem__(self, idx):<br>      item_name = str(idx + self.start).zfill(6)<br>      pcpath = os.path.join(self.pc_path, item_name + &quot;.bin&quot;)<br>      lbpath = os.path.join(self.lb_path, item_name + &quot;.label&quot;)<br><br>      # load points and labels<br>      pointcloud, labels = readpc(pcpath, lbpath)<br><br>      # transform<br>      torch_pointcloud  = torch.from_numpy(pointcloud)<br>      torch_labels      = torch.from_numpy(labels)<br><br>      return torch_pointcloud, torch_labels</pre><h4>Dataset Creation</h4><p>With the dataset class in place, we instantiated training, validation, and test datasets. This not only provides a structured organization but also sets the stage for the efficient use of PyTorch’s DataLoader module.</p><pre>train_ds = PointCloudData(dataset_path, start=0, end=100)<br>val_ds = PointCloudData(dataset_path, start=100, end=120)<br>test_ds = PointCloudData(dataset_path, start=120, end=150)</pre><p>DataLoader Utilization</p><p>Harnessing the capabilities of PyTorch’s DataLoader, we unlocked features like batching, shuffling, and parallel loading.</p><pre>train_loader = DataLoader(dataset=train_ds, batch_size=5, shuffle=True)<br>val_loader = DataLoader(dataset=val_ds, batch_size=5, shuffle=False)<br>test_loader = DataLoader(dataset=test_ds, batch_size=1, shuffle=False)</pre><p>This meticulous approach to dataset creation and loading is not only beneficial for basic problems but becomes indispensable as the complexity of the dataset and training procedures increases. It lays the groundwork for efficient, scalable, and parallelized data processing during training and testing.</p><h4>Loss Function</h4><p>In the realm of neural network training, the loss function plays a pivotal role in guiding the model’s parameter updates. Our PointNet model employs a carefully designed loss function, influenced by the insights provided in the paper:</p><p>“A regularization loss (with weight 0.001) is added to the softmax classification loss to make the matrix close to orthogonal.”</p><p>The loss function is expressed in code as follows:</p><pre>def pointNetLoss(outputs, labels, m3x3, m64x64, alpha=0.0001):<br>    criterion = torch.nn.NLLLoss()<br>    bs =  outputs.size(0)<br>    id3x3 = torch.eye(3, requires_grad=True).repeat(bs, 1, 1)<br>    id64x64 = torch.eye(64, requires_grad=True).repeat(bs, 1, 1)<br>    <br>    # Check if outputs are on CUDA<br>    if outputs.is_cuda:<br>        id3x3 = id3x3.cuda()<br>        id64x64 = id64x64.cuda()<br>    <br>    # Calculate matrix differences<br>    diff3x3 = id3x3 - torch.bmm(m3x3, m3x3.transpose(1, 2))<br>    diff64x64 = id64x64 - torch.bmm(m64x64, m64x64.transpose(1, 2))<br>    <br>    # Compute the loss<br>    return criterion(outputs, labels) + alpha * (torch.norm(diff3x3) + torch.norm(diff64x64)) / float(bs)</pre><p>Breaking down the components:</p><ul><li>outputs: Model predictions.</li><li>labels: Ground truth labels.</li><li>m3x3 and m64x64: Matrices from PointNet&#39;s transformation networks.</li><li>alpha: Weight for the regularization term.</li></ul><p>This loss function combines the standard Negative Log Likelihood (NLL) loss for softmax classification with a regularization term. The regularization term penalizes deviations of the transformation matrices from orthogonality, aligning with the paper’s emphasis on achieving orthogonality.</p><p>This meticulous crafting ensures that our PointNet model not only excels in classification precision but also adheres to structural constraints, enhancing its robustness and generalization during the training process.</p><h4>Training Loop</h4><p>The training loop is a procedural sequence that iteratively updates the PointNet model’s weights. It consists of a set number of epochs, each encompassing a training phase and an optional validation phase. The model alternates between training and evaluation states during these phases.</p><pre>def train(pointnet, optimizer, train_loader, val_loader=None, epochs=15, save=True):<br>    best_val_acc = -1.0<br><br>    for epoch in range(epochs):<br>        pointnet.train()<br>        running_loss = 0.0<br><br>        # Training phase<br>        for i, data in enumerate(train_loader, 0):<br>            inputs, labels = data<br>            inputs = inputs.to(device).float()<br>            labels = labels.to(device)<br>            optimizer.zero_grad()<br><br>            outputs, m3x3, m64x64 = pointnet(inputs.transpose(1, 2))<br>            loss = pointNetLoss(outputs, labels, m3x3, m64x64)<br>            loss.backward()<br>            optimizer.step()<br><br>            running_loss += loss.item()<br>            if i % 10 == 9 or True:<br>                print(&#39;[%d, %5d] loss: %.3f&#39; % (epoch + 1, i + 1, running_loss / 10))<br>                running_loss = 0.0<br><br>        # Validation phase<br>        pointnet.eval()<br>        correct = total = 0<br><br>        with torch.no_grad():<br>            for data in val_loader:<br>                inputs, labels = data<br>                inputs = inputs.to(device).float()<br>                labels = labels.to(device)<br>                outputs, __, __ = pointnet(inputs.transpose(1, 2))<br>                _, predicted = torch.max(outputs.data, 1)<br><br>                total += labels.size(0) * labels.size(1)<br>                correct += (predicted == labels).sum().item()<br><br>        print(&quot;correct&quot;, correct, &quot;/&quot;, total)<br>        val_acc = 100.0 * correct / total<br>        print(&#39;Valid accuracy: %d %%&#39; % val_acc)<br><br>        # Save the model if current validation accuracy surpasses the best<br>        if save and val_acc &gt; best_val_acc:<br>            best_val_acc = val_acc<br>            path = os.path.join(&#39;&#39;, &quot;MyDrive&quot;, &quot;pointnetmodel.yml&quot;)<br>            print(&quot;best_val_acc:&quot;, val_acc, &quot;saving model at&quot;, path)<br>            torch.save(pointnet.state_dict(), path)<br><br># Initialize the optimizer<br>optimizer = torch.optim.Adam(pointnet.parameters(), lr=0.005)<br><br># Commence the training<br>train(pointnet, optimizer, train_loader, val_loader, save=True)</pre><p>This loop serves as a systematic framework for updating the model parameters, monitoring loss, and evaluating performance over multiple iterations.</p><h4>Test</h4><p>The compute_stats function is designed to analyze the performance of your model during the testing phase. It counts the occurrences of different classes in true labels (unk, trav, nontrav), computes the total number of predictions, and counts the number of correct predictions. The results are returned as a tuple (correct, total_predictions).</p><pre>def compute_stats(true_labels, pred_labels):<br>  unk     = np.count_nonzero(true_labels == 0)<br>  trav    = np.count_nonzero(true_labels == 1)<br>  nontrav = np.count_nonzero(true_labels == 2)<br><br>  total_predictions = labels.shape[1]*labels.shape[0]<br>  correct = (true_labels == pred_labels).sum().item()<br><br>  return correct, total_predictions</pre><h3>Conclusion</h3><p>PointNet emerges as a groundbreaking tool for 3D segmentation, overcoming challenges posed by unordered point sets. Its theoretical foundations, architectural design, and practical implementation showcase its versatility and reliability. By combining theoretical prowess with practical implementation, we’ve demystified the journey of understanding and utilizing PointNet for 3D segmentation. The integration of PyTorch and Python provides a practical framework for exploring the potential of PointNet in real-world applications. For all codes check my <a href="https://github.com/sepideh-shamsizadeh/3DP-Point-Cloud-Segmentation/tree/main">GitHub</a>.</p><p>I must note that this project was a pivotal part of my 3D vision courses during my master’s degree at the University of Padova. The majority of the code was generously provided by Professor Alberto Pretto, the instructor of the course. I undertook the task of completing the code and added explanations to simplify the process for those interested in implementing the PointNet network for 3D segmentation. I sincerely hope you find this guide insightful and enjoyable.</p><h3>References</h3><p><a href="https://arxiv.org/pdf/1612.00593.pdf">The original paper of PointNet</a></p><p><a href="https://sepideh-shamsizadeh.github.io/blog-post-2/">A summary of the PointNet paper</a></p><p>Coding assistance: <a href="https://github.com/nikitakaraevv/pointnet/tree/master">nikitakaraevv</a> and <a href="https://github.com/yunxiaoshi/pointnet-pytorch/tree/master">yunxiaoshi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a037fecb8a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker container with pre-installed Ubuntu20.04 and ROS Noetic: A Comprehensive Guide for Beginners]]></title>
            <link>https://medium.com/@sepideh.92sh/how-docker-revolutionizes-application-development-a-comprehensive-guide-for-beginners-fc2d3e53eb31?source=rss-747261c080c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc2d3e53eb31</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[ros-noetic]]></category>
            <dc:creator><![CDATA[Sepideh Shamsizadeh]]></dc:creator>
            <pubDate>Tue, 10 Oct 2023 12:47:46 GMT</pubDate>
            <atom:updated>2023-11-24T09:34:23.846Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJbbyUKHKcYPambS0qM1LQ.png" /><figcaption>Docker.</figcaption></figure><p>Welcome to the exciting world of Docker, where containerization opens up new possibilities in software development and deployment. In this journey, we will unravel the mysteries of Docker, guiding you step-by-step through the fundamentals. But that’s not all — brace yourself for a hands-on experience as we delve into installing Docker alongside ROS Noetic on Ubuntu 20.04.</p><p>By the end of this guide, you’ll not only grasp the core concepts of Docker but also seamlessly integrate it with the Robot Operating System (ROS) Noetic, bringing together the power of containerization and robotics development. Let’s embark on this adventure, unlocking the potential of Docker in your software development endeavors.</p><h3>What is Docker?</h3><p>Docker is a platform designed to make it easier to develop, deploy, and run applications using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package. This ensures that the application runs reliably across various computing environments.</p><h3>Key Docker Concepts:</h3><h4><strong><em>1. Containerization:</em></strong></h4><p>Utilizes container technology to encapsulate applications and dependencies, ensuring consistency across environments.</p><h4><strong><em>2. Images and Containers:</em></strong></h4><p>Images are lightweight, standalone packages containing everything needed to run the software, while containers are runtime instances of these images, ensuring isolation.</p><h4><strong><em>3. Dockerfile:</em></strong></h4><p>Dockerfiles are the blueprints for images, providing instructions for building an image, specifying dependencies, and configuring the environment.</p><h4><strong><em>4. Portability:</em></strong></h4><p>Docker containers run seamlessly on any machine with Docker installed, offering a consistent environment for development and deployment.</p><h4><strong><em>5. Docker Hub:</em></strong></h4><p>A cloud-based registry service, Docker Hub centralizes Docker images, simplifying the sharing and distribution of containerized applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w68dON-sdkDkXsld-2Fywg.png" /></figure><h3>Benefits of Using Docker:</h3><ol><li><strong>Consistency Across Environments:</strong> Ensures uniformity from development to testing and production.</li><li><strong>Portability:</strong> Runs on any machine, simplifying deployment across diverse environments.</li><li><strong>Isolation: </strong>Provides process isolation, preventing conflicts between dependencies.</li><li><strong>Resource Efficiency: </strong>Lightweight containers use fewer resources and scale quickly.</li><li><strong>Rapid Deployment: </strong>Facilitates fast and consistent deployment with pre-built images.</li><li><strong>Scalability:</strong> Easily scales applications horizontally for optimal resource utilization.</li><li><strong>Version Control: </strong>Enables version-controlled management of dependencies and configurations.</li><li><strong>Dependency Management: </strong>Simplifies handling of dependencies within encapsulated containers.</li><li><strong>DevOps Integration: </strong>Key tool in DevOps practices, streamlining CI/CD workflows.</li><li><strong>Microservices Support: </strong>Supports microservices architecture for independent development and scaling.</li><li><strong>Vibrant Ecosystem:</strong> The thriving community and Docker Hub offer a rich repository of pre-built images.</li></ol><p>If your computer has a different version of Ubuntu or you’re using Windows and don’t want to switch to Ubuntu 20 for working with ROS Noetic, this tutorial is here to help. It’s all about getting your development environment ready without the need to change your operating system. So, let’s kick things off by installing Docker.</p><h3>Installing Docker on Ubuntu:</h3><h4>OS requirements</h4><p>To install Docker Engine, you need the 64-bit version of one of these Ubuntu versions:</p><ul><li>Ubuntu Lunar 23.04</li><li>Ubuntu Kinetic 22.10</li><li>Ubuntu Jammy 22.04 (LTS)</li><li>Ubuntu Focal 20.04 (LTS)</li></ul><p>For more details check the <a href="https://docs.docker.com/engine/install/ubuntu/">Docker</a> website.</p><h4>Update Ubuntu:</h4><p>Open a terminal and run:</p><pre>sudo apt update &amp;&amp; sudo apt upgrade</pre><h4>Install Docker Dependencies:</h4><pre>sudo apt install -y \<br>    apt-transport-https \<br>    ca-certificates \<br>    curl \<br>    gnupg \<br>    lsb-release</pre><h4>Add Docker GPG Key:</h4><pre>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg</pre><h4>Set Up Docker Repository:bashCopy code</h4><pre>echo \<br>  &quot;deb [arch=&quot;$(dpkg --print-architecture)&quot; signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \<br>  &quot;$(. /etc/os-release &amp;&amp; echo &quot;$VERSION_CODENAME&quot;)&quot; stable&quot; | \<br>  sudo tee /etc/apt/sources.list.d/docker.list &gt; /dev/null</pre><h4>Install Docker:</h4><pre>sudo apt update<br>sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin</pre><h4>Run Docker without sudo (Optional):</h4><pre>sudo usermod -aG docker $USER</pre><p>Log out and log back in or run su - $USER to apply the group changes.</p><h4><strong>Verify Installation:</strong></h4><pre>docker --version</pre><h4>Also, test with:</h4><pre>docker run hello-world</pre><h3>Installing Docker on Windows:</h3><ol><li><strong>Download Docker Desktop:</strong> Visit <a href="https://www.docker.com/products/docker-desktop">Docker Desktop for Windows</a> and download the installer.</li><li><strong>Install Docker Desktop: </strong>Follow the installation instructions in the Docker Desktop installer.</li><li><strong>Run Docker Desktop: </strong>Once installed, the Docker Desktop should start automatically. Look for the Docker icon in the system tray.</li><li><strong>Verify Installation: </strong>Open PowerShell and run:</li></ol><pre>docker --version</pre><ol><li>Also, check Docker Compose:</li></ol><pre>docker-compose --version</pre><p>Now you’re all set! Docker is installed on both Ubuntu 20 and Windows, ready for your ROS Noetic adventures.</p><h3>Building a Docker Container for ROS Development</h3><p>In the realm of Robotics and ROS (Robot Operating System) development, creating a Docker container provides a streamlined and reproducible environment. Let’s embark on a journey to build a Docker container tailored for Ubuntu 20.04 with ROS Noetic and essential modules.</p><h4>Build the Dockerfile:</h4><pre># Create a new folder for Docker<br>cd ~<br>mkdir ~/Docker<br><br># Move into the Docker folder and create a Dockerfile<br>cd ~/Docker<br>touch Dockerfile</pre><p>This sets up the directory structure and creates an empty Dockerfile for your container.</p><h4>Fill Dockerfile with Installation Instructions:</h4><pre>echo &#39;FROM osrf/ros:noetic-desktop-full&#39; &gt; Dockerfile &amp;&amp; echo &#39;ARG USER=user&#39; &gt;&gt; Dockerfile &amp;&amp; echo &#39;ARG DEBIAN_FRONTEND=noninteractive&#39; &gt;&gt; Dockerfile &amp;&amp; echo &#39;RUN apt-get update &amp;&amp; apt-get install -y ros-noetic-moveit ros-noetic-ros-controllers ros-noetic-gazebo-ros-control ros-noetic-rosserial ros-noetic-rosserial-arduino ros-noetic-roboticsgroup-upatras-gazebo-plugins ros-noetic-actionlib-tools terminator python3-pip &amp;&amp; rm -rf /var/lib/apt/lists/*&#39; &gt;&gt; Dockerfile &amp;&amp; echo &#39;RUN pip install flask flask-ask-sdk ask-sdk&#39; &gt;&gt; Dockerfile &amp;&amp; echo &#39;WORKDIR /home/${USER}&#39; &gt;&gt; Dockerfile</pre><p>This Dockerfile starts from the official ROS Noetic desktop-full image and adds additional packages and dependencies necessary for your robotics development. It also installs the Terminator terminal emulator and Flask for web development.</p><h4>Build the Docker Container:</h4><pre># Build the container and name it ros-noetic-container<br>docker build ~/Docker -t ros-noetic-container</pre><p>This command uses the Dockerfile to build a container image named “ROS-Noetic-container.”</p><h4>Create Shared Volumes:</h4><pre># Create a folder called Workspace for sharing between PC and Docker container<br>mkdir ~/Volumes</pre><p>This creates a directory on your host machine that will be shared with the Docker container. Sharing a folder between your PC and the Docker container is essential for seamless collaboration and data persistence. This shared volume acts as a bridge, enabling the exchange of files and data between your local machine and the container. It ensures that any changes made within the container, such as code modifications or project updates, are reflected on your PC, providing a synchronized development environment. This shared folder approach eliminates the risk of losing crucial work and fosters a fluid development process, allowing you to harness the benefits of Docker containerization while maintaining the flexibility and accessibility of your local file system.</p><h4>Run the Docker Container:</h4><pre># Run the ros-noetic-container and share Volumes folder and network configuration<br>docker run -it --rm --user=$(id -u $USER):$(id -g $USER) --env=&quot;DISPLAY&quot; --volume=&quot;/etc/group:/etc/group:ro&quot; --volume=&quot;/etc/passwd:/etc/passwd:ro&quot; --volume=&quot;/etc/shadow:/etc/shadow:ro&quot; --volume=&quot;/etc/sudoers.d:/etc/sudoers.d:ro&quot; --net host -v /home:/home -v ~/Volumes:/home/usr/ ros-noetic-container</pre><p>This command starts the Docker container in interactive mode, sharing your host user and group IDs to avoid permission issues. It also shares the necessary volumes and network configuration.</p><h4>Start Terminator Application:</h4><pre># Start the terminator application for better terminal window management<br>terminator -u</pre><p>Terminator is a terminal emulator that provides a more advanced terminal window management experience.</p><h4>Verify Container Environment:</h4><pre># Verify that you&#39;re in a Docker container running Ubuntu 20.04<br>lsb_release -a</pre><p>This command confirms that you are indeed inside a Docker container with Ubuntu 20.04.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*0fNpK2MCx1YBhJSwI_WiBw.png" /></figure><h4>Verify ROS Installation:</h4><pre># Verify that ROS is correctly installed in the container<br>roscore</pre><p>Running roscore checks if the Robot Operating System is functioning properly within the Docker container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bxwmT0DVOwLeUvIdKcD4TA.png" /></figure><h4>Exiting Docker:</h4><p>To exit the Docker container, you can use the `exit` command or press `Ctrl + D`. This will terminate the interactive session inside the container, and you’ll be back in your host machine’s terminal.</p><h4>Starting Docker Container:</h4><p>Whenever want to work with this built container you should run the below command in a terminal.</p><pre>docker run -it --rm --user=$(id -u $USER):$(id -g $USER) --env=&quot;DISPLAY&quot; --volume=&quot;/etc/group:/etc/group:ro&quot; --volume=&quot;/etc/passwd:/etc/passwd:ro&quot; --volume=&quot;/etc/shadow:/etc/shadow:ro&quot; --volume=&quot;/etc/sudoers.d:/etc/sudoers.d:ro&quot; --net host -v /home:/home -v ~/Volumes:/home/usr/ ros-noetic-container</pre><p>Once your Docker container is up and running, seamlessly connect Visual Studio Code to it. Download Visual Studio Code <a href="https://code.visualstudio.com/">here</a> and install it. After installation, head to the “Extensions” tab and add the following extensions for C++ and Python, the go-to languages in ROS development:</p><p>- ms-vscode.cpptools<br>- twxs.cmake<br>- ms-vscode.cmake-tools<br>- ms-python.python<br>- redhat.vscode-xml<br>- ms-vscode-remote.remote-containers</p><p>Now, navigate to the “Remote Explorer” tab in Visual Studio Code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*qc4bT_b9Z5YQ6cH85UJleg.png" /></figure><p>Right-click on the running ros-noetic-container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*08RvSaxMN-ihqqfcM0S2Eg.png" /></figure><p>Select “Attach to Container.”</p><p>Voila! You now have Visual Studio Code seamlessly integrated with your ros-noetic-container, ready for development in Ubuntu 20.04 and ROS Noetic.</p><h4>Save Docker Container:</h4><p>If you’ve made modifications to the container, like installing new packages, tools, or libraries that you want to preserve for future use, follow these commands to capture those changes effectively.</p><h4>List Running Containers:</h4><pre>docker ps</pre><p>This command shows the list of running containers, and you need to note the CONTAINER ID or NAME of the container you want to commit.</p><h4>Commit Changes:</h4><pre>docker commit CONTAINER_ID_OR_NAME new_image_name</pre><p>Replace CONTAINER_ID_OR_NAME with the actual ID or name of your container and new_image_name with the desired name for the new image.</p><h4>Example:</h4><pre>docker commit my_container ROS-Noetic-Updated</pre><p>This command creates a new image named ROS-Noetic-Updated with the changes made in the specified container.</p><p>Now, you can start a new container based on this updated image whenever you want to resume your work with the saved changes.</p><p>Remember, it’s good practice to use version control systems (e.g., Git) for managing your code changes, and Docker images can capture the environment and dependencies, providing reproducibility.</p><h3>Docker Hub: Home of Container Magic</h3><p><a href="https://hub.docker.com/">Docker Hub</a> is where container wonders happen. It’s the ultimate space for Docker images, making sharing and finding containers a breeze. Need an image? Docker Hub has it. It’s like a digital library for containers, making your software adventures smoother and more fun!</p><p>Embarking on this Docker adventure has been quite a journey, unlocking the vast potential of containerization in software development. We’ve demystified Docker, explored its key concepts, and seamlessly integrated it with the Robot Operating System (ROS) Noetic on Ubuntu 20.04. From the consistency of environments to the agility of deployment, Docker has proven to be a game-changer.</p><p>As we bid farewell to this guide, remember that Docker is not just a tool; it’s a gateway to a vibrant ecosystem. Whether you’re ensuring consistency across environments, isolating processes, or scaling applications horizontally, Docker is your ally. The journey doesn’t end here; it extends to the realms of microservices, DevOps integration, and a rich repository of pre-built images on Docker Hub.</p><p>For those on different versions of Ubuntu or Windows, fear not — this guide stands as a testament that you can harness the power of Docker without changing your operating system. As you navigate the world of containers, don’t forget to explore Docker Hub, the home of container magic. It’s your digital library for containers, making software adventures smoother and more enjoyable.</p><p>So, here’s to Docker, to ROS Noetic, and to the endless possibilities of containerized development. Happy coding! 🚀</p><h3>References</h3><ul><li><a href="https://docs.docker.com/engine/install/ubuntu/"></a></li><li><a href="https://hub.docker.com/">Docker Hub Container Image Library | App Containerization</a></li><li><a href="https://wiki.ros.org/noetic/Installation/Ubuntu">Making sure you&#39;re not a bot!</a></li><li><a href="https://code.visualstudio.com/">Visual Studio Code - Build what comes next</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc2d3e53eb31" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling the Magic of 3D Reconstruction: A Journey through Structure from Motion (SfM) in C++]]></title>
            <link>https://medium.com/@sepideh.92sh/unveiling-the-magic-of-3d-reconstruction-a-journey-through-structure-from-motion-sfm-in-c-d66bc1d01a96?source=rss-747261c080c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d66bc1d01a96</guid>
            <category><![CDATA[sfm]]></category>
            <category><![CDATA[structure-from-motion]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[3d-reconstruction]]></category>
            <category><![CDATA[3d-vision]]></category>
            <dc:creator><![CDATA[Sepideh Shamsizadeh]]></dc:creator>
            <pubDate>Fri, 06 Oct 2023 17:29:45 GMT</pubDate>
            <atom:updated>2023-10-09T09:38:49.583Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s explore Structure from Motion by figuring out the 3D layout of a small scene through a series of smartphone images. Choose nearby objects and take pictures with some overlap in what your camera captures. For accurate results, use a checkerboard to calibrate your camera — take a few shots from different angles. This calibration step provides vital info for the entire process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hAaMkJvKK4TIlCa1QEC3xw.jpeg" /><figcaption>Our selected objects for SfM.</figcaption></figure><h3>1. Introduction</h3><p>Structure from Motion (SfM) is a computer vision and photogrammetry technique that reconstructs the 3D structure of a scene and determines camera poses from a collection of 2D images. By identifying common features and solving mathematical equations, it transforms a set of images into a cohesive 3D representation, applicable in fields like 3D modeling, augmented reality, and historical landscape reconstruction.</p><p>Here’s a concise breakdown of the steps involved in solving Structure from Motion (SfM):</p><h3><strong>2. Feature Extraction</strong></h3><p>Feature extraction is the process of identifying and extracting distinctive and repeatable patterns or points from an image. These features serve as key points that can be matched across multiple images, forming the basis for establishing correspondences and ultimately reconstructing 3D structures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*sjlJ9G8Ru9J-q5yXdw3mfg.png" /><figcaption>Sample of extracted features for the object</figcaption></figure><h4>2.1. SIFT (Scale-Invariant Feature Transform):</h4><p>SIFT is a powerful algorithm for feature extraction developed by David Lowe. It’s particularly robust to changes in scale, rotation, and illumination, making it ideal for various computer vision tasks.</p><pre>void extractFeatures()<br>{<br>    features_.resize(images_names_.size());<br>    descriptors_.resize(images_names_.size());<br>    feats_colors_.resize(images_names_.size());<br><br>    for( int i = 0; i &lt; images_names_.size(); i++  )<br>    {<br>        std::cout&lt;&lt;&quot;Computing descriptors for image &quot;&lt;&lt;i&lt;&lt;std::endl;<br><br>        // Read the image<br>        cv::Mat img = readUndistortedImage(images_names_[i]);<br><br>        // Create a SIFT detector<br>        cv::Ptr&lt;cv::SiftFeatureDetector&gt; detector = cv::SiftFeatureDetector::create();<br><br>        // Create a SIFT descriptor extractor<br>        cv::Mat descriptor;<br>        cv::DescriptorExtractor* extractor = new cv::SiftDescriptorExtractor();<br><br>        // Detect SIFT features in the image<br>        detector-&gt;detect(img, features_[i]);<br><br>        // Compute SIFT descriptors for the detected features<br>        extractor-&gt;compute(img, features_[i], descriptor);<br><br>        // Store the descriptors for later use<br>        descriptors_[i].push_back(descriptor);<br><br>        // Extract colors for visualization (optional)<br>        for(int k=0; k&lt;features_[i].size(); k++) {<br>            cv::Vec3b color = img.at&lt;cv::Vec3b&gt;(features_[i][k].pt);<br>            feats_colors_[i].push_back(color);<br>        }<br>    }<br>}</pre><h3><strong>3. Feature Matching</strong></h3><p>In the context of Structure from Motion (SfM), once features are extracted from multiple images, the next step is to match these features across different images. This process aims to establish correspondences between points in different views, forming the basis for reconstructing the 3D structure of the scene.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*Khp4B6nObQR2jxn1bDVpUA.png" /><figcaption>Feature matching</figcaption></figure><p>In the pursuit of aligning different images and establishing correspondences between their features, we employed theircv::FlannBasedMatcher interface. This interface taps into the Clustering and Search in Multi-Dimensional Spaces module, offering a rapid and efficient mechanism for matching features.</p><p>To enhance the robustness of our matches, we implemented a distance ratio test. This test evaluates the distance ratio between the two nearest matches of a given keypoint. A match is deemed reliable if this ratio falls below a certain threshold. This clever approach aids in differentiating between potentially ambiguous matches, where the distance ratio is close to one, and well-discriminated matches.</p><p>Moreover, we introduced a crucial condition to the matching process. If the number of inlier matches is deemed insufficient (let’s say, less than or equal to 10 matches), we refrain from establishing matches between the two images. This pragmatic check prevents the inclusion of unreliable matches, contributing to the overall accuracy of the subsequent reconstruction process.</p><pre>void exhaustiveMatching()<br>{<br>    for (int i = 0; i &lt; images_names_.size() - 1; i++)<br>    {<br>        for (int j = i + 1; j &lt; images_names_.size(); j++)<br>        {<br>            std::cout &lt;&lt; &quot;Matching image &quot; &lt;&lt; i &lt;&lt; &quot; with image &quot; &lt;&lt; j &lt;&lt; std::endl;<br><br>            // Read undistorted images<br>            cv::Mat img1 = readUndistortedImage(images_names_[i]);<br>            cv::Mat img2 = readUndistortedImage(images_names_[j]);<br><br>            // Create a FLANN-based descriptor matcher<br>            cv::Ptr&lt;cv::DescriptorMatcher&gt; matcher = cv::DescriptorMatcher::create(cv::DescriptorMatcher::FLANNBASED);<br><br>            // Perform K-nearest neighbors matching<br>            std::vector&lt;std::vector&lt;cv::DMatch&gt;&gt; knn_matches;<br>            matcher-&gt;knnMatch(descriptors_[i], descriptors_[j], knn_matches, 2);<br><br>            // Filter matches using the Lowe&#39;s ratio test<br>            const float ratio_thresh = 0.5f;<br>            std::vector&lt;cv::DMatch&gt; inlier_matches;<br>            for (size_t k = 0; k &lt; knn_matches.size(); k++)<br>            {<br>                if (knn_matches[k][0].distance &lt; ratio_thresh * knn_matches[k][1].distance)<br>                {<br>                    inlier_matches.push_back(knn_matches[k][0]);<br>                }<br>            }<br><br>            // Check if the number of inliers is sufficient<br>            if (inlier_matches.size() &lt;= 10)<br>                continue;<br><br>            // Set intrinsics matrix and establish matches<br>            intrinsics_matrix_ = new_intrinsics_matrix_;<br>            setMatches(i, j, inlier_matches);<br>        }<br>    }<br>}</pre><h3>4. Triangulation: Revealing 3D Secrets</h3><p>As we navigate through the intricacies of Structure from Motion (SFM), let’s delve into the process of triangulation — a fundamental aspect in unraveling the 3D shape of our scene. This process unfolds with the Essential (E) and Homography (H) matrices, each playing a unique role in our narrative.</p><h4>4.1. Essential Matrix and Locating 3D Points:</h4><p>In scenes filled with diverse elements, extending beyond mere flat surfaces, the Essential matrix takes the spotlight. It conceals the intricacies of how cameras are positioned relative to each other, establishing the foundation for discovering 3D points. Utilizing OpenCV’s cv::findEssentialMat(), we peer into the realm of matching features, breathing life into 3D points.</p><pre>cv::Mat E = cv::findEssentialMat(points0, points1, intrinsics_matrix, cv::RANSAC, 0.99, 1, inlier_mask_E);</pre><h4>4.2. Homography Matrix: Unveiling 3D Secrets on Flat Surfaces</h4><p>However, when our scene primarily features flat elements, the Homography matrix takes center stage. Through the use of cv::findHomography(), we streamline the process, facilitating the identification of 3D points on these flat surfaces.</p><pre>cv::Mat H = cv::findHomography(points0, points1, inlier_mask_H, cv::RANSAC);</pre><pre>int e = cv::sum(inlier_mask_E)[0];<br>        int h = cv::sum(inlier_mask_H)[0];<br>        if( e&gt;h){<br>            cv::recoverPose(E, points0, points1, intrinsics_matrix, init_r_mat,init_t, inlier_mask_E);<br>            seed_found = true;<br>        }</pre><p>These matrices play a crucial role in the decision-making process, operating discreetly in the background. The output mask, a sequence of numbers, meticulously identifies which points hold significant importance (set to 1) and which are of lesser significance (set to 0). In this realm of crucial and less crucial points, a pivotal decision arises.</p><p>We assess the popularity of Essential (E) over Homography (H) by comparing the number of crucial points. If Essential (E) has more supporters (more crucial points), we employ it to breathe life into our 3D world, illuminating the “seed found” flag.</p><p>Simply put our journey through Structure from Motion (SFM) resembles a captivating show. We carefully select the right tools (matrices), akin to magic spells, tailored to our scene, ensuring our 3D world comes to life in perfect harmony.</p><h3>5. Unveiling the Harmony: Bundle Adjustment and Ceres Solver in the 3D Reconstruction</h3><p>In our journey through Structure from Motion (SfM), we’ve covered feature extraction, matching, and the enchanting world of matrices. Now, let’s shine a spotlight on the dynamic duo: Bundle Adjustment and Ceres Solver.</p><h4>5.1. Bundle Adjustment: Precision Refinement</h4><p>Imagine capturing a series of photos for a 3D scene. Tiny errors, like slight misalignments, can sneak in. Bundle Adjustment steps up to the plate, refining camera poses and 3D structures, ensuring a more accurate reconstruction.</p><pre>// Bundle Adjustment code snippet<br>void bundleAdjustmentIter(int new_cam_idx)<br>{<br>    ceres::Solver::Options options;<br>    options.linear_solver_type = ceres::SPARSE_SCHUR;<br>    options.minimizer_progress_to_stdout = true;<br>    options.num_threads = 4;<br>    options.max_num_iterations = 200;<br><br>    std::vector&lt;double&gt; bck_parameters;<br>    bool keep_optimize = true;<br><br>    // Global optimization<br>    while (keep_optimize)<br>    {<br>        bck_parameters = parameters_;<br><br>        ceres::Problem problem;<br><br>        // For each observation....<br>        for (int i_obs = 0; i_obs &lt; num_observations_; i_obs++)<br>        {<br>            // Check if this observation has been registered<br>            if (pose_optim_iter_[pose_index_[i_obs]] &gt; 0 &amp;&amp; pts_optim_iter_[point_index_[i_obs]] &gt; 0)<br>            {<br>                // Extract camera, point, and observation data<br>                double *camera = (parameters_.data()) + (pose_index_[i_obs] * camera_block_size_),<br>                       *point = (parameters_.data()) + (num_poses_ * camera_block_size_ + point_index_[i_obs] * point_block_size_),<br>                       *observation = observations_.data() + (i_obs * 2);<br><br>                ceres::CostFunction *cost_function =<br>                    ReprojectionError::Create(observation[0], observation[1]);<br><br>                problem.AddResidualBlock(cost_function,<br>                                         new ceres::CauchyLoss(2 * max_reproj_err_),<br>                                         camera,<br>                                         point);<br>            }<br>        }<br><br>        ceres::Solver::Summary summary;<br>        Solve(options, &amp;problem, &amp;summary);<br><br>        // Handle violations and outliers...<br>    }<br><br>    // Update counts and print pose if a new camera is added<br>    // printPose(new_cam_idx);<br>}</pre><h4>5.2. Ceres Solver: Precision Guide in Optimization</h4><p>Ceres Solver, our numerical optimization guide, partners seamlessly with Bundle Adjustment. It ensures precise adjustments to parameters, paving the way for an accurate 3D reconstruction.</p><pre>// Ceres Solver code snippet (within Bundle Adjustment)<br>ceres::Solver::Summary summary;<br>Solve(options, &amp;problem, &amp;summary);</pre><h4>5.3. Residual Blocks &amp; Auto-Differentiable Cost Functions: The Precision</h4><p>Now, let’s unravel the magic behind the scenes. Residual Blocks and Auto-Differentiable Cost Functions are the unsung heroes. A Residual Block is a mathematical representation of the difference between observed and predicted values. In simpler terms, it helps quantify the “error” that needs correction.</p><pre>struct ReprojectionError<br>{<br>    ReprojectionError(double observed_x, double observed_y)<br>            : observed_x(observed_x), observed_y(observed_y) {}<br><br>    template &lt;typename T&gt;<br>    bool operator()(const T* const camera,<br>                    const T* const point,<br>                    T* residuals) const {<br>        // camera[0,1,2] are the angle-axis rotation.<br>        T p[3];<br>        ceres::AngleAxisRotatePoint(camera, point, p);<br>        // camera[3,4,5] are the translation.<br>        p[0] += camera[3]; p[1] += camera[4]; p[2] += camera[5];<br><br>        // Compute final projected point position.<br>        const T predicted_x =  p[0] / p[2];<br>        const T predicted_y =  p[1] / p[2];<br><br>        // The error is the difference between the predicted and observed position.<br>        residuals[0] = predicted_x - T(observed_x);<br>        residuals[1] = predicted_y - T(observed_y);<br>        return true;<br>    }<br>    // Factory to hide the construction of the CostFunction object from<br>    // the client code.<br>    static ceres::CostFunction* Create(const double observed_x, const double observed_y) {<br>        return (new ceres::AutoDiffCostFunction&lt;ReprojectionError, 2, 6, 3&gt;(<br>                new ReprojectionError(observed_x, observed_y)));<br>    }<br>    double observed_x;<br>    double observed_y;<br>    <br>};</pre><p>In a nutshell, Residual Blocks, powered by Auto-Differentiable Cost Functions, quantify the correction needed in our optimization process. They transform the intricate dance of errors into numerical steps that Ceres Solver elegantly executes.</p><p>Next time you witness a stunning 3D reconstruction, remember the behind-the-scenes maestros: Bundle Adjustment, Ceres Solver, Residual Blocks, and Auto-Differentiable Cost Functions. They’re the precision architects making Structure from Motion a visual marvel.</p><p>The outcomes are visually manifested in point-cloud snapshots. Utilize MeshLab to render and explore the generated .ply file for a comprehensive visualization experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*UkAh4eNQhbcu0EZC4DDqwA.png" /><figcaption>The resulting image in MeshLab.</figcaption></figure><p>And there you have it! We’ve reached the end of this exploration into the fascinating world of 3D reconstruction using Structure from Motion in C++. I hope you’ve enjoyed this journey through feature extraction, matching, and the intricate dance of matrices.</p><p>If you found this article both informative and captivating, stay tuned for more deep dives into the realms of computer vision and photogrammetry. Follow me for future articles where I’ll continue to demystify complex concepts, turning them into digestible insights.<br>For an in-depth exploration and access to the full code, refer to [<a href="https://github.com/sepideh-shamsizadeh/3D_structure_from_motion/tree/main">here</a>]. Dive into the mechanics that make 3D reconstruction both precise and visually impressive!</p><p>Until next time, goodbye 👋 and thanks for embarking on this 3D adventure with me. 🚀 Keep exploring, keep learning, and stay awesome! 🌟</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d66bc1d01a96" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>