<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Logos on Medium]]></title>
        <description><![CDATA[Stories by Logos on Medium]]></description>
        <link>https://medium.com/@d.d.tananaev?source=rss-a6caf4f5f26e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Tapb_RIUNFW4oo1ncatxvQ.jpeg</url>
            <title>Stories by Logos on Medium</title>
            <link>https://medium.com/@d.d.tananaev?source=rss-a6caf4f5f26e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:16:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@d.d.tananaev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Motion Modeling in Videos: Part 3 — Scene Flow]]></title>
            <link>https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-3-scene-flow-b63006a2ffb2?source=rss-a6caf4f5f26e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b63006a2ffb2</guid>
            <dc:creator><![CDATA[Logos]]></dc:creator>
            <pubDate>Sun, 21 Jul 2024 08:24:49 GMT</pubDate>
            <atom:updated>2024-07-28T11:11:50.700Z</atom:updated>
            <content:encoded><![CDATA[<h3>Motion Modeling in Videos: Part 3 — Scene Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UKTFEk6sFeunqw55aYhfMA.gif" /><figcaption>Left — motion instance segmentation; Right — optical flow (scene flow projection to image)</figcaption></figure><p>In the previous parts we investigated possibilities of the motion modeling in videos with optical flow and optical expansion and concluded that the only way to correctly model the dynamic environment is a scene flow.</p><p>Note this paper is a part of a series:</p><ol><li>Motion Modeling in Videos: Part 1 — Optical Flow (<a href="https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-1-optical-flow-68901cf01435">link</a>)</li><li>Motion Modeling in Videos: Part 2 — Optical Expansion (<a href="https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-2-optical-expansion-816abe1e92d9">link</a>)</li><li>Motion Modeling in Videos: Part 3 — Scene Flow</li></ol><p><strong>What is scene flow?</strong></p><p>Scene flow is 3d motion <em>S(Sx, Sy, Sz) </em>of the 3d point <em>P(Px, Py, Pz)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*1FuuNMOtfXTnnGoEDphhwA.png" /><figcaption>Scene flow example: The scene flow S(Sx, Sy, Sz) is a 3d motion of the point P to P’ position. The projection of the scene flow f(fx, fy) in 2d image is optical flow. Image source [1]</figcaption></figure><p>The rigid motion can be represented:</p><p><em>P’ = R*P + T</em></p><p>than scene flow</p><p><em>S = P’ — P = R*P -P +T = (R-I)*P +T</em></p><p>Here R — 3x3 rotation matrix, I -3x3 identity matrix and T — translation vector.</p><p>The rotation matrix<em> R</em> represented as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bP3bIFvwKyysgPxw4eP-lQ.png" /><figcaption>image source [2]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*vNV7xrB7vYRISvQ5fB4EUQ.png" /></figure><p>Lets substitute this matrix into our scene flow equation and get final scene flow relation of motion in 3D:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*qN62Opy-BMxunzgHjuZpTg.png" /></figure><p><strong>The relation between scene flow and pinhole camera model</strong></p><p>The pinhole camera model is simplified model of the projection of the 3d point to 2d plane with assumption that there are no lenses (for more information see tutorial <a href="https://youtu.be/uHApDqH-8UE?si=OEj9xkibrur0-Os1">here</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*8vyZWnmIDFAcpBIR.png" /></figure><p>The schematic representation of the pinhole projection: the example of projection of the 3d point X to the image plane ends up at pixels coordinate x (on the left). The example of relations between 3d point X the Y coordinate and focal length f (on the right). The equations of the projection of 3D point X(X, Y, Z) to pixels coordinate (x, y) in image (bottom of the image)</p><p>The projection of the 3D point <strong>X</strong> with coordinates <strong>(X, Y, Z)</strong> to the image plane with pixels coordinates <strong>(x, y)</strong> can be represented as equation in the picture above(where <strong>X , Y, Z</strong> — 3d coordinates, <strong>x, y </strong>— 2d coordinates in the image plane and <strong>f</strong>-focal length). It is important to note that after projection with this equation we will have the pixels coordinates origin <strong>(0, 0)</strong> in the principle point <strong>p(cx, cy)</strong> (around the center of the image). If we want to transfer it to the image coordinates with origin at top left corner we should shift all pixels coordinates by <strong>p(cx, cy)</strong>.</p><p>We can derive the dependencies to the scene flow as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjtSeNDZ2D21FNuAvn_6vw.png" /></figure><p>Note: here <em>flow = (u, v) </em>— optical flow in x direction and y direction.</p><p>We can observe that scene flow can be estimated from optical flow, motion in depth and depth for current image.</p><p><strong>Deep learning based motion estimation from scene flow</strong></p><p>In the paper [3], the authors first employ three separate networks to compute depth from monocular images, optical flow, and motion in depth. Additionally, they compute four cost tensors: homography error, Sampson error, plane plus parallax, and depth contrast error (see the appendix of paper [3] for details).</p><p>The authors also estimate camera motion by calculating the essential matrix and remove camera rotation from the scene flow, which is reconstructed from depth and optical flow, as this simplifies the task of rigid body masking.</p><p>They then use the network to predict a binary mask distinguishing between static and dynamic environments, as well as identifying instances of moving objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kV1ixHKdGfQtdpmLzjV2Gw.png" /></figure><p>Finally they use optimization to refine resulted rigid body scene flow.</p><p>As result the approach can generalize over out of distribution domains and reliably detect instances of moving objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*8PUH2AJA7OtHqTm6R2gdyQ.gif" /><figcaption>Left — moving object instances; Right — optical flow</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wMGrflwUq7AoenPv_pnBZg.gif" /><figcaption>Left — moving object instances; Right — optical flow</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*7J9qseWRtwQpvGKDxAyEQA.gif" /><figcaption>Left — moving object instances; Right — optical flow</figcaption></figure><h3>Literature</h3><p>[1] Self-Supervised Monocular Scene Flow Estimation <a href="https://openaccess.thecvf.com/content_CVPR_2020/papers/Hur_Self-Supervised_Monocular_Scene_Flow_Estimation_CVPR_2020_paper.pdf">https://openaccess.thecvf.com/content_CVPR_2020/papers/Hur_Self-Supervised_Monocular_Scene_Flow_Estimation_CVPR_2020_paper.pdf</a></p><p>[2] <a href="https://en.wikipedia.org/wiki/Rotation_matrix">https://en.wikipedia.org/wiki/Rotation_matrix</a></p><p>[3] Learning to Segment Rigid Motions from Two Frames. <a href="https://arxiv.org/abs/2101.03694">https://arxiv.org/abs/2101.03694</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b63006a2ffb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Motion Modeling in Videos: Part 2 — Optical Expansion]]></title>
            <link>https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-2-optical-expansion-816abe1e92d9?source=rss-a6caf4f5f26e------2</link>
            <guid isPermaLink="false">https://medium.com/p/816abe1e92d9</guid>
            <dc:creator><![CDATA[Logos]]></dc:creator>
            <pubDate>Sat, 11 May 2024 10:39:44 GMT</pubDate>
            <atom:updated>2024-07-21T08:29:53.158Z</atom:updated>
            <content:encoded><![CDATA[<h3>Motion Modeling in Videos: Part 2 — Optical Expansion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Xm3nkP2SjqxbHDVcCbFew.gif" /></figure><p>From the<a href="https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-1-optical-flow-68901cf01435"> part 1</a> we learned that optical flow is not enough to do explicit motion estimation. Can we do something better than only optical flow?</p><p>Note this paper is a part of a series:</p><ol><li>Motion Modeling in Videos: Part 1 — Optical Flow (<a href="https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-1-optical-flow-68901cf01435">link</a>)</li><li>Motion Modeling in Videos: Part 2 — Optical Expansion</li><li>Motion Modeling in Videos: Part 3 — Scene Flow (<a href="https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-3-scene-flow-b63006a2ffb2">link</a>)</li></ol><p>When driving a car, we perceive the motion of the vehicle in front of us by observing changes in its size. There are three possible scenarios:</p><p>1 . If the car in front of us appears larger, it is likely driving slower, and the distance between us is decreasing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hb-iN3nbDvhrhRZi4JWpJw.gif" /><figcaption>An example of the decreasing distance between an observer and a car</figcaption></figure><p>2. If the car in front of us becomes smaller, it drives faster, and the distance increases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CuaIDDXQu2bVOYVwT3HUmg.gif" /><figcaption>An example of the increase in distance between the observer and the car.</figcaption></figure><p>3. If the car doesn’t change its size, it means we have the same speed and a constant distance between vehicles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RyBBR6w0rMSqeH8yD_O5hg.png" /><figcaption>An example of no change in distance between the observer and the car.</figcaption></figure><p>Through simple observation of how the scale of an object changes, we can intuitively understand its relative motion and velocity with respect to us.</p><p>This effect can be modeled in computer vision as optical expansion or scale change.</p><blockquote><strong>The optical expansion (or scale change) is key biological vision feature for velocity perception and navigation in dynamic environment.</strong></blockquote><p><strong>What is optical expansion?</strong></p><p>Optical expansion/contraction is a scale change of the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6GF7MWYChUyonclUauF-Q.png" /><figcaption>Optical flow provides the position change (u, v) for the given pixel (see (a-b)). Optical expansion provides the scale change s, which can be measured as square root of the ratio between two areas covered by blue rectangles it give us (s), where s =l′ /l. In the case of one side of rectangle (1D case). Image source [1]</figcaption></figure><p>he optical flow provides us with information about how objects move along the <em>x</em> and <em>y</em> directions (<em>u, v</em>) in the image. The scale change (<em>s</em>) gives us insight into how the object moves in the depth (<em>z</em>) direction. Let’s develop an intuition with a simple 1D case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HL731hcUhsjkpHdvTB3jCQ.png" /></figure><p>As you can see from the image above, the scale change in the 1D case can be computed as the ratio between the pixel width of the car in the image at time <em>t+1</em> and the pixel width of the car in the image at time <em>t</em>. This ratio is metric invariant (pixels/pixels cancel out), which means it is independent of any camera parameters or real-world object sizes.</p><p>Of course, we are also interested in computing the scale change in the 2D case. In this scenario, it is equivalent to the square root of the ratio between the areas in pixels covered by the blue car at time <em>t+1</em> and at time <em>t</em>.</p><p>One way to achieve this is through instance segmentation and tracking, which allows us to obtain masks of the object in consecutive frames and compute their respective areas. Another option is to compute the scale change directly from optical flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GLZaY9F8QuXvVFfUWIeEfw.png" /></figure><p>The pytorch code for scale change estimation from optical flow:</p><pre>import torch<br>import torch.nn.functional as F<br><br>def affine(pref: torch.Tensor, flow: torch.Tensor, pw:int =1)-&gt; Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:<br>    &quot;&quot;&quot;Find affine transform.<br>    <br>    Note: original code from https://github.com/gengshan-y/expansion/blob/master/models/VCN_exp.py<br><br>    Args:<br>        pref: reference coordinates of the shape [B, 2, H, W]<br>        flow: optical flow of the shape [B, 2, H, W]<br><br>    Returns:<br>        exp: optical expansion of the shape [B, 1, H, W]<br>        error: error of the affine transform [B, 1, H, W]<br><br>    &quot;&quot;&quot;<br><br>    b,_,lh,lw = flow.shape<br><br>    ptar = pref + flow<br>    pref = F.unfold(pref, (pw*2+1,pw*2+1), padding=(pw)).view(b,2,(pw*2+1)**2,lh,lw)-pref[:,:,np.newaxis]<br>    ptar = F.unfold(ptar, (pw*2+1,pw*2+1), padding=(pw)).view(b,2,(pw*2+1)**2,lh,lw)-ptar[:,:,np.newaxis] # b, 2,9,h,w<br>    pref = pref.permute(0,3,4,1,2).reshape(b*lh*lw,2,(pw*2+1)**2)<br>    ptar = ptar.permute(0,3,4,1,2).reshape(b*lh*lw,2,(pw*2+1)**2)<br>    # Find first inverse of pref by using The Moore–Penrose inverse<br><br>    prefprefT = pref.matmul(pref.permute(0,2,1))<br>    ppdet = prefprefT[:,0,0]*prefprefT[:,1,1]-prefprefT[:,1,0]*prefprefT[:,0,1]<br>    ppinv = torch.cat((prefprefT[:,1,1:],-prefprefT[:,0,1:], -prefprefT[:,1:,0], prefprefT[:,0:1,0]),1).view(-1,2,2)/ppdet.clamp(1e-10,np.inf)[:,np.newaxis,np.newaxis]<br><br>    Affine = ptar.matmul(pref.permute(0,2,1)).matmul(ppinv)<br>    error = (Affine.matmul(pref)-ptar).norm(2,1).mean(1).view(b,1,lh,lw)<br><br>    Avol = (Affine[:,0,0]*Affine[:,1,1]-Affine[:,1,0]*Affine[:,0,1]).view(b,1,lh,lw).abs().clamp(1e-10,np.inf)<br>    exp = Avol.sqrt()<br><br>    return exp, error</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oIAQ9m1aTq_vJiOmLad8-w.gif" /><figcaption>The example of computation of the scale change from optical flow: white — increase scale, black -decrease, gray — scale =1.0</figcaption></figure><p>Now that we know how to compute scale change, we need to understand its relation to real motion in 3D (motion in depth) in order to make use of it.</p><p><strong>How to relate optical expansion (scale change) to the motion in depth?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1RLeeVSmUB2xmhh06crScA.png" /></figure><p>Let&#39;s examine an example to see how it works. Consider three cases:</p><ol><li>If the car in front of the camera moves forward, it decreases the depth (distance to the camera), and the motion in depth ratio is smaller than 1.0.</li><li>If the car moves backward, the motion in depth increases and becomes greater than 1.0.</li><li>If the car stays in the same place, the motion in depth equals 1.0</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*7BPd2z0nYO4l0J2OPx33Tg.png" /></figure><p>You may also note that, as the motion in depth is determined by the ratio between depth values, it does not have an inherent metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uQOrL0WQFsYAdNXZsjxneQ.png" /><figcaption>Side-by-side comparison of the scale change and motion in depth</figcaption></figure><p>If we place our examples of scale change and motion in depth side by side, we can observe an interesting relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*IT30OAG_tr6xhORmtjpayw.png" /></figure><p>Is this always true? Not necessarily.</p><p>In the examples above, we consider a specific scenario:</p><ul><li>The car does not have rotational motion.</li><li>We also assume that the distance to all parts of the car can be measured as the closest point of the car to the camera. This condition is equivalent to modeling the projection to the camera with a scaled orthographic projection camera model.</li></ul><p><strong>What is scaled orthographic camera model projection?</strong></p><p>The way how most of standard cameras projects 3d <em>X(X, Y, Z)</em> point to 2d point <em>x(x,y)</em> in image can be explained by pinhole projection model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGH3QlXK3H6zwdlE9O1A0A.png" /><figcaption>Pinhole camera model with focal lenght f: example of projection of 3d point X (X,Y,Z) to the image coordinates (x,y)</figcaption></figure><p>Orthographic projection model assumes that all 3d points of the world located on single plane, therefore they all have the same depth. In order to project the 3d points to 2d image we can ignore depth (since all world has same Z) and project to the image by following equations <em>x = X, y = Y.</em></p><p>Here x,y — pixels coordinates, X, Y — 3d coordinates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*eCU-W_dqSEg8ixAMtbI3Fw.png" /><figcaption>The orthographic camera model: projection equation x = X, y = Y, where x,y — 2d coordinates and X, Y -3d coordinates</figcaption></figure><p>The orthographic projection could be reasonable approximation of the 3d world in case if the objects appear at far distances (e.g. in satellites images).</p><p>The scaled orthographic projection is a combination of the pinhole camera model and orthographic model. It is also called weak prospective model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kja3rnH6Qgrfz_7Z4ZFD_w.png" /><figcaption>The scaled orthographic model is a combination of the orthographic and pinhole projection</figcaption></figure><p>First, all the points of the objects projected to the single plane in 3d with <em>Zconst</em> distance to the camera, and than pinhole camera model is used to project this points to the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*OhTMu_uZ6apvm14_k0cHhg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*ot6tvuSQCr8TjlsuX9kMUA.png" /><figcaption>The example of scaled orthographic projection assumption in real world: we assume that all car located in front of the camera with constant Z distance 25 meters. This assumption introduces the errors in real depth</figcaption></figure><p>In the image above you may see how scaled orthographic assumption introduces the distance errors.</p><blockquote>Note: The depth constancy assumption works only in the patches where we apply affine transformation for scale change estimation. It will be different for different patches.</blockquote><p><strong>The derivation of the relation between scale change and motion in depth</strong></p><p>Lets first consider the case where we have motion of the car without rotation (see the image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-vSWVHbvj0l5X-vLnO1Zsw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*9wy5v4AHwy71u88r-3-JMQ.png" /></figure><p>We get exact inverse relation as in intuition example.</p><p>Now let’s consider 2d case with rotation (we assume top view world with only rotation <em>theta</em> angle.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aiMB-BNF9-RZlPtJ0ICK0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*4m0_dV53MXOyBrUsOMyK6A.png" /></figure><p>We may see that when we have rotation even in simple scenario, where we model only one angle <em>theta</em>, the relation becomes non-linear with respect to rotation. However, in case of small rotation angles it can be neglected.</p><p>This could be the case for the cameras with high frame rate (e.g. 30 fps).</p><p>Therefore, we will assume for such cameras the relation between motion in depth and scale change is simplified to inverse dependency.</p><p><strong>Application of the motion in depth for time to collision estimation</strong></p><p>One important application of the motion in depth is estimation of the time before collision.</p><p>The time to collision can be formulated as distance to the object divided by the relative velocity between it and the camera.</p><p>Let’s find out how time to collision (TTC) related to the motion in depth:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*qceKiYut4X_CDLNK5J1Ktw.png" /></figure><p>As application we can use cameras to estimate time to collision and replace ultrasound sensors for autonomous driving, or make automatic collision avoidance system for UAV drones, robots etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zrg7Q47Q2eF6KeYR5qJaOA.gif" /><figcaption>Time to collision prediction to the obstacle, source [2], full <a href="https://youtu.be/glX75vj2csI">video</a></figcaption></figure><p><strong>Motion modeling with normalized scene flow</strong></p><p>We already learned that for motion modeling we need scene flow.</p><p>Let’s find out how motion in depth related to the scene flow and how can we use it to estimate the motion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*slWzc0L_G9CmxuRacUWtCg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SZj9YAih398cQJiW74xlVQ.png" /><figcaption>Optical flow contains high degree of ambiguity for 3d motion, optical expansion (normalized scene flow) resolves most of ambiguities except scale [1]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GarZ2Ctoa1JbxI-DDX--yQ.png" /><figcaption>Given depth we can get full scene flow from normalized scene flow [2]</figcaption></figure><p>From the images above, we can observe that we can resolve ambiguities in motion directions from normalized scene flow by utilizing motion in depth (see equations above). If we know the motion direction of the ego vehicle (or can compute it from essential matrix estimation), we can apply rotational homography to de-rotate normalized scene flow vectors. Here’s the code:</p><pre><br>def get_derotated_normalized_sene_flow(forward_flow: np.ndarray, motion_in_depth: np.ndarray, camera2_R_camera1: np.ndarray, intrinsic: np.ndarray)-&gt; np.ndarray:<br>    &quot;&quot;&quot;Get derotated normalized scene flow.<br>    <br>    Args:<br>    forward_flow: forward optical flow from t to t+1 of the shape [height, width, 2]<br>    motion_in_depth: motion in depth Z2&#39; / Z1 of the shape [height, width]<br>    camera2_R_camera1: 3x3 rotation matrix from camera1(t) to camera2 (t+1)<br>    intrinsic: 3x3 intrinsic matrix form camera (we assume camera1 is the same as camera2)<br>    <br>    Return:<br>        norm_scene_flow: normalized derotated scene flow of the shape [height*width, 3]<br>    &quot;&quot;&quot;<br><br>    height, width, _ = forward_flow.shape<br>    xx, yy = np.meshgrid(range(width), range(height), indexing=&#39;xy&#39;)<br>    coords1 = np.asarray(np.stack((xx, yy), axis=-1))<br>    coords2 = coords1 + forward_flow<br><br>    # Make homogeneous coordinates<br>    ones = np.ones((height, width, 1))<br>    coords1 = np.concatenate((coords1, ones), axis=-1).reshape(-1, 3).T<br>    coords2 = np.concatenate((coords2, ones), axis=-1).reshape(-1, 3).T<br><br>    H01 = intrinsic.dot(camera2_R_camera1).dot(np.linalg.inv(intrinsic))<br>    # Remove rotation<br>    coords1 = (H01 @ coords1)<br>    norm_scene_flow = np.linalg.inv(intrinsic).dot(motion_in_depth.flatten()[np.newaxis] * coords2 - coords1)<br>    return norm_scene_flow.T</pre><p>and then we can estimate angle between ego motion translation vector and all normalized scene flow vectors as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*Qb3jmPAFO-QyG_2BlALCrw.png" /></figure><pre><br>def compute_plane_plus_parallax(norm_scene_flow: np.ndarray, camera2_t_camera1: np.ndarray)-&gt; np.ndarray:<br>    &quot;&quot;&quot;Compute plane plus parallax distance.<br>    <br>    Args:<br>        norm_scene_flow: normalized derotated scene flow of the shape [height*width, 3]<br>        camera2_t_camera1: translation vector from camera1(t) to camera2(t+1)<br><br>    Returns:<br>        c3d: plane plus parallax error of the shape [height * width]<br>    &quot;&quot;&quot;<br><br>    norm = np.linalg.norm(norm_scene_flow, axis=-1) * np.linalg.norm(-camera2_t_camera1)<br>    dot_product = np.sum(norm_scene_flow *  -camera2_t_camera1[None], axis=-1)<br><br>    cos_theta = dot_product / (norm + 1e-8)<br>    sin_theta = np.sqrt(1.0  -  cos_theta**2)<br>    c3d = np.linalg.norm(norm_scene_flow, axis=-1) * sin_theta<br><br>    return c3d</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQntJRyL-Qm3pFYYFhhW6g.gif" /><figcaption>The resulting motion estimation is derived from derotated normalized scene flow, along with the camera motion. This method enables the distinction of all 3D motion directions that are not collinear with the camera motion. However, detecting collinear motion is not possible with this approach. To resolve this last motion ambiguity, we require full scene flow. Dataset: Virtual KITTI [3].</figcaption></figure><p>As a result, we can model all possible motions with normalized scene flow except for collinear (same direction as the camera) motion. To resolve this final motion ambiguity, depth information is needed, indicating that we still require the estimation of full 3D scene flow.</p><p><strong>Literature:</strong></p><p>[1] Upgrading Optical Flow to 3D Scene Flow through Optical Expansion, <a href="https://openaccess.thecvf.com/content_CVPR_2020/papers/Yang_Upgrading_Optical_Flow_to_3D_Scene_Flow_Through_Optical_Expansion_CVPR_2020_paper.pdf">https://openaccess.thecvf.com/content_CVPR_2020/papers/Yang_Upgrading_Optical_Flow_to_3D_Scene_Flow_Through_Optical_Expansion_CVPR_2020_paper.pdf</a></p><p>[2] TTC4MCP: Monocular Collision Prediction Based on Self-Supervised TTC Estimation, <a href="https://www.researchgate.net/publication/376502431_TTC4MCP_Monocular_Collision_Prediction_Based_on_Self-Supervised_TTC_Estimation">https://www.researchgate.net/publication/376502431_TTC4MCP_Monocular_Collision_Prediction_Based_on_Self-Supervised_TTC_Estimation</a></p><p>[3] Virtual kitti dataset: <a href="https://europe.naverlabs.com/research-old2/computer-vision/proxy-virtual-worlds-vkitti-2/">https://europe.naverlabs.com/research-old2/computer-vision/proxy-virtual-worlds-vkitti-2/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=816abe1e92d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Motion Modeling in Videos: Part 1 — Optical Flow]]></title>
            <link>https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-1-optical-flow-68901cf01435?source=rss-a6caf4f5f26e------2</link>
            <guid isPermaLink="false">https://medium.com/p/68901cf01435</guid>
            <category><![CDATA[motion-modelling]]></category>
            <category><![CDATA[optical-flow]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[epipolar-constraints]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Logos]]></dc:creator>
            <pubDate>Fri, 05 Apr 2024 16:33:18 GMT</pubDate>
            <atom:updated>2024-07-21T08:27:55.877Z</atom:updated>
            <content:encoded><![CDATA[<h3>Motion Modeling in Videos: Part 1 — Optical Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6px8sfSUcQvAtaIixbNYbQ.gif" /><figcaption>Image with flow vectors (left); Optical flow (middle); Motion segmentation (right)</figcaption></figure><p>Motion modeling seems to be an overlooked problem in computer vision, especially given all the progress with NeRF and implicit representation.</p><p>If we can combine motion modeling from 3D geometry with the current state-of-the-art NeRF, we may solve 3D reconstruction in dynamic environments for self-driving, AR/VR, etc.</p><p>Can we actually incorporate moving objects inside the 3D reconstruction framework and make it possible to directly reconstruct both static and dynamic environments within the same framework? Let’s find out!</p><p>Note this paper is a part of a series:</p><ol><li>Motion Modeling in Videos: Part 1 — Optical Flow</li><li>Motion Modeling in Videos: Part 2 — Optical Expansion (<a href="https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-2-optical-expansion-816abe1e92d9">link</a>)</li><li>Motion Modeling in Videos: Part 3 — Scene Flow (<a href="https://medium.com/@d.d.tananaev/motion-modeling-in-videos-part-3-scene-flow-b63006a2ffb2">link</a>)</li></ol><p><strong>Background subtraction for motion segmentation</strong></p><p>The simple way to model motion is through background subtraction. We can find the difference between the current and next images (to reduce noise, apply Gaussian smoothing to the images before subtraction). If the difference is greater than a certain threshold value, the pixel corresponds to a moving object; otherwise, it corresponds to static.</p><p>We consider two scenarios:</p><ol><li>With static camera</li><li>With moving camera</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*oSt5KA7vbrhnRX4QUPvfwg.gif" /><figcaption>1. Static camera scenario: green — moving pixels</figcaption></figure><p>In the first scenario, the results seem reasonable since we can achieve motion segmentation on the edges of the objects. However, we can’t fully segment them because homogeneous areas without textures will yield zero difference. Additionally, you may observe problems with shadows cast by cars, as well as reflections and illumination changes, which can lead to incorrect results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Xmlhz0GPeXMpA4fScHxhDQ.gif" /><figcaption>2.Dynamic camera scenario: green — moving pixels</figcaption></figure><p>In the second scenario, the approach fails. Things become more complex in the case of general motion. We need a better way to model motion. This leads us to the concept of optical flow.</p><p><strong>What is optical flow?</strong></p><p>The optical flow represents the displacement of pixels (delta_x, delta_y) between the current and next images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Hmyyb3IX6pxMvmvdrGY6g.png" /><figcaption>Optical flow example: forward flow (on the right) from image 1 to image 2</figcaption></figure><p>We can establish a direct mapping between pixels of image 1 and image 2 by employing optical flow (delta_x, delta_y):</p><p>image2(x2, y2) = image1(x1 + delta_x, y1 + delta_y)</p><p>Optical flow can be visualized with arrows or colors corresponding to directions on a color wheel (see the image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xl7isBVm675j_5WV4IS2Eg.png" /><figcaption>The optical flow visualized with special colors which corresponds to different directions (image courtesy prof. T. Brox)</figcaption></figure><p>Optical flow can represent various types of motions, and sometimes, by observing it, we can understand how the camera moves (see image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBrtMRkepzuTokKRma1rtg.png" /><figcaption>Different types of motion represented as optical flow: rotation around y-axis (top); rotation over z-axis (middle); forward translation (bottom) [1]</figcaption></figure><p>In the case of pure translation, there is a point where all flow vectors diverge. This point is called the focus of expansion (FoE) or contraction, depending on the motion. At this point, the optical flow equals zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_XGB57kHMH_1QGIYgAF9cw.png" /><figcaption>The FoE example given different motion: a — forward motion; b -backward motion</figcaption></figure><p>The FoE is the projection of the center of the camera from the next frame to the previous frame. It is also called the epipole. In the case of pure translation, its image coordinates are easy to find from the translation vector T:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*Kgr1vDrxSYHjN6MaFCP4MA.png" /></figure><p>The FoE shows the direction of the motion of the camera.</p><blockquote>if you want to know more about camera geometry check tutorial video <a href="https://youtu.be/ND2fa08vxkY?si=T8MX1qb_-R2t2WJN">here</a>.</blockquote><p>When dealing with general motion, accurately interpreting the optical flow arrows to assess the motion solely through observation can be challenging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKyWDBmCElfQJIQ6Y_P78A.png" /><figcaption>The example of the general motion optical flow [1]</figcaption></figure><p>The complexity of the flow vectors arises from the combined effects of translation and rotation, involving both static and dynamic objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9s3OB-btVwgWwo6XS7q_g.png" /><figcaption>We can decompose optical flow vectors by the translation and rotational part, resulted vectors is just sum of this components [1]</figcaption></figure><p>The overall optical flow vector comprises the following components:</p><ol><li>For static objects: camera translation + camera rotation.</li><li>For dynamic objects: camera translation + camera rotation + dynamic object translation + dynamic object rotation.</li></ol><p>Optical flow modeling encompasses all potential motion patterns and can serve as a viable option for motion segmentation</p><p><strong>How to compute optical flow?</strong></p><p>I will focus on state-of-the-art deep learning-based approaches, as they are both simple to understand and provide the best performance.</p><blockquote>If you are more interested in classical optical flow formulation you may check video tutorial <a href="https://youtu.be/lnXFcmLB7sM?si=mwYadDxFB-ihGknE">here</a>.</blockquote><p>Optical flow can be formulated as a supervised learning problem. A convolutional neural network (CNN) takes two consecutive images as input and outputs the respective optical flow. The ground truth optical flow can be generated from simulators (e.g., Carla [2]). The L2 distance loss function is typically used to measure the distance between the predicted flow and the ground truth.</p><p>The first supervised deep learning networks for optical flow were FlowNet [3] and FlowNet2 [4]. These works showed that it is possible to achieve state-of-the-art optical flow performance and<strong> generalize to real data while being trained only on simulated images.</strong></p><p><strong>PWCNet — efficient optical flow implementation</strong></p><p>The first optimized version of optical flow is PWCNet [5]. It is 17 times smaller in size, 2 times faster, and easier to train than FlowNet2.</p><p>These improvements were achieved through the <strong>careful incorporation of best practices from classical approaches into the CNN architecture </strong>(see the image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xTh7k4LhEuGwxffC-c1img.png" /><figcaption>The best practices from classical optical flow (on the left) transferred to the PWCNet architecture of the decoder (on the right)</figcaption></figure><p>In classical optical flow computation, the images are down-sampled to several different scales. The optical flow calculation begins at the coarsest scale. The resulting optical flow is then upscaled to the next level, and Image 2 is warped using this optical flow. Afterward, the flow is computed again.</p><p>The warping procedure enables the movement of pixels from Image 2 closer to Image 1, thereby reducing the size of the search window needed to find corresponding pixels. PWCNet applies the same workflow on features extracted from the encoder.</p><p><strong>RAFT — optical flow with global search</strong></p><p>RAFT: Recurrent All-Pairs Field Transforms [6] is the first work to utilize global matching search for correspondences with neural networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lUUMhhkX4a5okmLN4sBh1g.png" /><figcaption>The correlation volume for whole image 1 lead to 4D correlation volume of the shape HxWxHxW and down-sampled versions after applying pooling operation</figcaption></figure><p>Global matching entails searching over all pixels in Image 2 for every pixel in Image 1. This results in quadratic complexity costs relative to the height and width of the images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1NZ-DFLuGN4dIKEZJrOXNQ.png" /><figcaption>The RAFT architecture. Images processed by feature encoder and after dot product 4D correlation volume is constructed. Recurrent unit uses this correlation volume (and its down-sampled versions) and also context features of image1 to iteratively compute optical flow</figcaption></figure><p>The RAFT utilizes recurrent GRU unit to iteratively compute optical flow (see the image above).</p><p>Although the approach achieves state-of-the-art results (as of 2020), it exhibits quadratically growing memory demands relative to resolution.</p><p>As a result, its usage is limited, even as an offline solution. Furthermore, the PWCNet performs competitively with RAFT given careful tuning and training [7].</p><p><strong>Unimatch: Unifying Flow, Stereo and Depth Estimation</strong></p><p>Finally, the current state-of-the art (as for beginning 2024) unimatch [8] neural network, which incorporates transformers and enables the integration of tasks such as optical flow, stereo disparity, and multi-view depth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EuGc2eUAwMVREFQvDo5feQ.png" /><figcaption>Unimatch network architecture: As input it takes two images, features extracted with CNN encoders than several transformers with self-attention followed by cross-attention computes discriminative features for matching. After that non-learnable matching happened, followed by refinement self-attention.</figcaption></figure><p>This architecture boasts several advantages: it is simple, fast for inference, and integrates a shifted local window attention strategy from the Swin Transformer [9] for efficiency. It significantly outperforms RAFT and can be applied to both online and offline applications.</p><p><strong>Self-supervised optical flow</strong></p><p>For completeness, it’s worth mentioning that optical flow can be computed from raw videos without the need for any labels.</p><p>Given two images, optical flow can be computed and used to warp Image 2 to align with Image 1. The difference between the original and warped images is then measured using photometric loss. There is a comprehensive survey of these techniques [10], along with a state-of-the-art method as of 2021 [11].</p><p>Supervised optical flow trained with simulated data has shown excellent performance on real images, while self-supervised techniques can be valuable for fine-tuning models for further improvements.</p><p><strong>Motion modeling with optical flow</strong></p><p>Let’s consider the first simple use case where the camera is stationary and only surrounding objects are moving. In this scenario, the background optical flow should be zero, while optical flow for all moving objects should be non-zero. This straightforward observation can lead to reliable motion segmentation when high-quality optical flow is available.</p><p>In practice, it may be necessary to use a threshold value and estimate the uncertainty of the optical flow (see [12]) to filter out noisy flow vectors. Although there are works focused on single-object segmentation with a slightly moving camera, our primary interest lies in general motion segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hwi_YVPamhxYEHbwFl4ebg.gif" /><figcaption>Motion segmentation from optical flow with stand still camera: all non-zero flow marked as moving objects</figcaption></figure><p>For general motion segmentation, this simple approach would not suffice since optical flow exists both in static and dynamic objects. However, we can assume that static objects should exhibit consistent motion. If we could leverage this observation, we might be able to distinguish the static background from dynamic objects.</p><p>First, let’s examine the flow vectors in the image. Ideally, the static environment should have same flow vectors, while dynamic objects should have varying flow vectors. However, this assumption does not hold true.</p><p>Upon observation, we may notice that flow vectors for the same static environment differ, both in magnitude and direction. Therefore, a simple clustering approach is not suitable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*Z515hlIRHycmgOEiLAf7LA.png" /><figcaption>Same static object can have different optical flow vectors depending on pixels location</figcaption></figure><p>Why this happens?</p><p>The reason for this discrepancy lies in the fact that we are not directly observing the real 3D motion of the objects or the static environment, but rather their 2D perspective projection onto the image plane.</p><p>This projection effect leads to different flow vectors depending on the pixel positions to which they are projected. Consequently, we must incorporate 3D geometry to effectively distinguish between static and dynamic objects.</p><p><strong>Epipolar constraints for motion modeling in optical flow</strong></p><p>To utilize 3D geometry, it’s essential to have knowledge of camera calibration parameters. The intrinsic camera calibration parameters include:</p><ul><li>Focal length for the width and height dimensions (<em>fx, fy</em>)</li><li>Principal point (the projection of the camera center onto the image plane) pixel coordinates (<em>cx, cy</em>). (For a perfect camera, <em>cx = width/2.0, cy = height/2.0</em>.)</li></ul><p>These parameters allow us to construct the calibration matrix<em> K.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x2BTr4qbPK_DvlpjEiaeiw.png" /></figure><p>This matrix can be received with calibration process (if you are interested how to do calibration check <a href="https://youtu.be/-9He7Nu3u8s?si=lFldXWhH497tFXC_">this</a> tutorial.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7W9sgUFM6dBaQA-HIsPZZg.png" /><figcaption>Given two cameras, O1 and O2, a static 3D point P corresponds to pixel positions p in camera O1 and p’ in camera O2. Connecting the centers of cameras O1 and O2 with point P forms a plane (gray triangle). The intersection of this plane with each camera’s image creates epipolar lines (blue lines). The projections of the camera centers onto each other’s planes, denoted as e and e’, are called epipoles or focus of expansion (FoE). All epipoles intersect at these points. [13]</figcaption></figure><p>Given the camera center position <em>O1</em> and a static 3D point <em>P</em>, the observed pixel position of this 3D point in camera <em>O1</em> image corresponds to pixel <em>p</em>. When the camera is moved to position<em> O2</em>, the new corresponding pixel position of point <em>P</em> is <em>p’ = p + optical_flow.</em></p><p>Ideally, if we know the camera motion rotation matrix <em>R</em>, translation vector <em>t</em>, and depth<em> Z</em> for each pixel, we can compute the static optical flow from the camera’s 3D motion using the following matrix multiplication equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhZ14xbJOcJPR3Bgvns7qw.png" /></figure><p>Given static flow we can subtract from full optical flow and theoretically it will give us moving objects.</p><p>The approach demands depth <em>Z</em> estimation which is not trivial task. Can we do something simpler?</p><p><strong>Fundamental matrix estimation and Sampson error</strong></p><p>The answer is yes!</p><p>We can utilize the assumption that correspondences for all static objects lie in a plane (see the image above). By connecting the camera centers <em>O1</em> and <em>O2</em> with the 3D point <em>P</em>, we define a plane (grey triangle).</p><p>The projection of this plane onto the image results in a line. Leveraging this fact, we can compute the relative motion between cameras.</p><p>This observation is known as the coplanarity constraint</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*7E4A1Hpc7xco8vgKVUObuw.png" /></figure><p>The re-projection between corresponded point p and p’ depends on a special 3x3 matrix F which called fundamental matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*amWZav-jhMhNDXYadDemjQ.png" /></figure><p>We can estimate fundamental matrix from optical flow as follows (see <a href="https://youtu.be/gYYumFSDsvA?si=yBlPTjkaRM3roIPz">this</a> tutorial for more details):</p><pre>import numpy as np<br>import cv2<br>def compute_fundamental_matrix(flow):<br>    &quot;&quot;&quot;Compute fundamental matrix.<br>    <br>    Args:<br>        flow: optical flow of the shape [height, width, 2 (dx, dy)]<br>    <br>    Returns:<br>        F:  3x3 fundamental matrix<br>    &quot;&quot;&quot;<br><br>    # Get pixels coordinates of the image1<br>    height, width, _ = flow.shape<br>    xx, yy = np.meshgrid(range(width), range(height), indexing=&#39;xy&#39;)<br>    pts1 = np.asarray(np.stack((xx, yy), axis=-1), dtype=np.float32)<br><br>    # Compute corresponded coordinates in the image2<br>    pts2 = pts1 + flow<br><br>    # Compute fundamental matrix by using 8 point algorithm and ransac<br>    pts1  = np.reshape(pts1, (-1, 2))<br>    pts2  = np.reshape(pts2, (-1, 2))<br>    inlier_tr = 3.0<br>    ransac_prob = 0.999<br>    # Here output F fundamental matrix<br>    F, valid_mask = cv2.findFundamentalMat(pts1, pts2, cv2.FM_RANSAC, inlier_tr, ransac_prob)<br>    return F</pre><p>The fundamental matrix allows to estimate coplanarity constraints for every optical flow points correspondence.</p><p>We assume that points which do not satisfy coplanarity (i.e., are not located in the same plane) are moving objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_2qOkFc9YaxkalVlaNyadQ.png" /></figure><p>The distance from epipolar plane (or in image line) to the given point correspondence can be formulated as Sampson error[14]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*79MOCwkQ0CHHFdyuKdMkQg.png" /></figure><p>the python example of the error computation:</p><pre>import numpy as np<br>import cv2<br>def sampson_error(F, pts1, pts2):<br>    &quot;&quot;&quot;Estimate sampson error from coplanarity constraints.<br>    <br>    Args:<br>        F: fundamental matrix<br>        pts1: points on the first image of the shape [num_points, 2]<br>        pts2: corresponded points on the second image of the shape [num_points, 2]<br>    <br>    Returns:<br>        error: the coplanarity constraint error for each correspondence<br>        Note: for coplanar correspondence it should be zero (or near so)<br>    &quot;&quot;&quot;<br><br>    num_points, _ = pts1.shape<br>    ones = np.ones((num_points, 1))<br><br>    hom_pts1 = np.concatenate((pts1, ones), axis=-1).T<br>    hom_pts2 = np.concatenate((pts2, ones), axis=-1).T<br>    Fp1 = F @ hom_pts1<br>    Fp2 = F.T @ hom_pts2<br>    p2Fp1 = (hom_pts2 * Fp1).sum(axis=0)<br><br>    error = (p2Fp1 **2) / (Fp1[0]**2 +Fp1[1]**2 + Fp2[0]**2 + Fp2[1]**2 + 1e-8)<br>    return error</pre><p>The results can be seen on the animation below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kC_eSWj-qCul7IJuy9R_YQ.gif" /><figcaption>The motion estimation with Sampson error: most of the pixels even for moving objects moves along the epipolar lines (see upper left) therefore can not be detected in motion mask (bottom right)</figcaption></figure><p>The reason why motion masking fails to separate moving from static objects is that <strong>our assumption is incorrect.</strong></p><p>In reality, many motion patterns occur within the epipolar plane. For instance, collinear motion, such as the red truck in the image, or opposite-line cars, as well as motion toward the FoE, like the white car</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*LD-H8VB9I1IL5BdMm7YRlg.png" /><figcaption>The example coplanar moving object: the point moves withing epipolar plane (along epipolar line) therefore can’t be detected by Sampson error</figcaption></figure><p>We need something else to improve our detection.</p><p><strong>The essential matrix estimation and camera motion modeling</strong></p><p>What else can we do?</p><p>We need more information to handle motion segmentation. What if we calibrate the camera? Can intrinsic calibration (matrix <em>K</em>) help us to improve detection?</p><p>Given calibration of the camera we actually can estimate camera relative motion.</p><p>We can rewrite coplanarity constraints as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*UGzrAvKtR0xt_au8hZtKUg.png" /></figure><p>The re-projection between corresponded point <em>p</em> and <em>p’ </em>depends on special 3x3 matrix <em>E</em> — essential matrix.</p><p>The essential matrix contains inside the rotation and translation of the camera and we can decompose it to get camera motion (up to scale).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qSRK9Wv6wVK-L8HZaLYAmw.png" /></figure><p>The code for estimation of the camera motion:</p><pre>import numpy as np<br>import cv2<br>def compute_camera_motion(flow, intrinsics):<br>    &quot;&quot;&quot;Compute camera motion from essential matrix.<br><br>    Note: intrinsic matrix<br>            | fx 0 cx |<br>        K = | 0 fy cy |<br>            | 0  0  1 |<br>    Args:<br>      flow: optical flow of the shape [height, width, 2(dx, dy)]<br>      intrinsics: the intrinsics matrix K of the shape 3x3<br>    <br>    Returns:<br>      R: 3x3 rotation matrix<br>      t: translation vector<br>    &quot;&quot;&quot;<br>    height, width, _ = flow.shape<br>    xx, yy = np.meshgrid(range(width), range(height), indexing=&#39;xy&#39;)<br>    pts1 = np.asarray(np.stack((xx, yy), axis=-1), dtype=np.float32)<br>    pts2 = pts1 + flow<br><br>    pts1 = np.reshape(pts1, (-1, 2))<br>    pts2 = np.reshape(pts2, (-1, 2))<br><br><br>    ones = np.ones((height*width, 1))<br><br>    hom_pts1 = np.concatenate((pts1, ones), axis=-1)<br>    hom_pts2 = np.concatenate((pts2, ones), axis=-1)<br><br>    norm_points1 = (np.linalg.inv(intrinsics) @ hom_pts1.T).T[:, :2]<br>    norm_points2 = (np.linalg.inv(intrinsics) @ hom_pts2.T).T[:, :2]<br><br><br>    # Compute essential matrix with 5 points algorithm, valid_mask -inliers<br>    E, valid_mask = cv2.findEssentialMat(norm_points1, norm_points2, focal=1.0, pp=(0., 0.), method=cv2.RANSAC, prob=0.999, threshold=3.0)<br>    # Here outputs<br>    # retval: A scalar value indicating the success of the operation. If it&#39;s true, the decomposition was successful.<br>    # R: The rotation matrix representing the relative rotation between the two camera views.<br>    # t: The translation vector representing the relative translation between the two camera views.<br>    # mask: A mask indicating the inliers. It&#39;s optional and may not be returned if not provided<br>    # Recover pose (rotation and translation)<br>    retval, R, t,  mask = cv2.recoverPose(E, norm_points1, norm_points2, focal=1.0, pp=(0.0, 0.0))<br>    return R, t</pre><p>Now, given rotation, we can use it to compute rotational homography. The idea is that all static environment points will have the same rotation, but dynamic points will have different rotations. Therefore, if we remove the rotation by applying a homography transformation, we will observe errors on moving objects due to their additional motion. The rotational homography can be formulated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*od7bycfj1Qw8kq0ZA5BvXg.png" /></figure><p>And the homography error can be formulated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F4_MzhmquR5cD6uYYEn6kQ.png" /></figure><p>here <em>d()</em> — Euclidean distance, in python can be implemented as following:</p><pre><br>import numpy as np<br>import cv2<br>def homography_error(pts1, pts2, R, intrinsics):<br>    &quot;&quot;&quot;Rotational homography error.<br>    <br>    Args:<br>        pts1: undistorted pixels coords [num_points, 2]<br>        pts2: undistorted pixels coords [num_points, 2]<br>        R: rotation between cameras<br>        intrinsics: 3x3 intrinsics<br><br>    Returns:<br>        error: homography error<br>    &quot;&quot;&quot;<br><br>    num_points, _ = pts1.shape<br>    ones = np.ones((num_points, 1))<br><br>    hom_pts1 = np.concatenate((pts1, ones), axis=-1).T<br>    hom_pts2 = np.concatenate((pts2, ones), axis=-1).T<br><br>    H =  intrinsics @ R @ np.linalg.inv(intrinsics)<br><br>    Hp2 = H @ hom_pts2<br><br>    Hp1 = np.linalg.inv(H) @ hom_pts1<br><br>    error = np.linalg.norm( (hom_pts1 -Hp2)[:2], axis=0) **2 + np.linalg.norm( (hom_pts2 -Hp1)[:2], axis=0) **2<br>    return error</pre><p>The results is not good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*blxmZoMiZ8A322D8MngXnA.gif" /><figcaption>Homography error in case of general motion seems not the best way to go</figcaption></figure><p>Another possible trick we can use is estimating the focus of expansion (FoE) by knowing the motion direction. We can then remove rotation with homography from the flow and attempt to check the directions of the flow vectors with respect to the FoE.</p><p>In the case of backward motion, static vectors point toward the FoE; otherwise, they point in the opposite direction.</p><p>However, even with these tricks, it is not possible to model all types of collinear motion.</p><p><strong>Analysis and Conclusions</strong></p><p>We tried several different techniques to derive motion segmentation from optical flow but were not able to succeed. It is possible to use optical flow for motion segmentation with a stationary camera (or in some particular cases like single-object segmentation with minimal camera motion). However, it is <strong>impossible to derive general motion segmentation solely from pure optical flow</strong>.</p><p>Why is that?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*Z6izeThn7eu2HbuZ2LeKPg.png" /><figcaption>Optical flow f(fx, fy) is the projection of the 3d motion vector (scene flow) to 2d plane (image a). During this projection the information is lost and it is not possible recover exact 3d motion vector for given 2d flow vector image source [15]</figcaption></figure><p>The optical flow is the projection of 3D motion onto a 2D plane, and many different motion patterns in 3D can correspond to the same flow. It turns out that optical flow alone is not sufficient to recover 3D motion. The problem is under-constrained, with a large degree of ambiguity.</p><p>Ideally, we need to model the <strong>full 3D scene flow</strong> to resolve general motion.</p><p><strong>Literature</strong></p><p>[1] Coursera: Robotics Perception course from Penn University</p><p>[2] Carla simulator: <a href="http://carla.org//">http://carla.org//</a></p><p>[3] FlowNet: Learning Optical Flow with Convolutional Networks: <a href="https://lmb.informatik.uni-freiburg.de/Publications/2015/DFIB15/flownet.pdf">https://lmb.informatik.uni-freiburg.de/Publications/2015/DFIB15/flownet.pdf</a></p><p>[4] FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks: <a href="https://openaccess.thecvf.com/content_cvpr_2017/papers/Ilg_FlowNet_2.0_Evolution_CVPR_2017_paper.pdf">https://openaccess.thecvf.com/content_cvpr_2017/papers/Ilg_FlowNet_2.0_Evolution_CVPR_2017_paper.pdf</a></p><p>[5] PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume, <a href="https://arxiv.org/abs/1709.02371">https://arxiv.org/abs/1709.02371</a></p><p>[6] RAFT: Recurrent All-Pairs Field Transforms for<br>Optical Flow <a href="https://arxiv.org/pdf/2003.12039.pdf">https://arxiv.org/pdf/2003.12039.pdf</a></p><p>[7] What Makes RAFT Better Than PWC-Net? <a href="https://www.researchgate.net/publication/359390010_What_Makes_RAFT_Better_Than_PWC-Net">https://www.researchgate.net/publication/359390010_What_Makes_RAFT_Better_Than_PWC-Net</a></p><p>[8] Unifying Flow, Stereo and Depth Estimation <a href="https://arxiv.org/abs/2211.05783">https://arxiv.org/abs/2211.05783</a></p><p>[9] Swin transformer: Hierarchical vision transformer using shifted<br>windows <a href="https://arxiv.org/abs/2103.14030">https://arxiv.org/abs/2103.14030</a></p><p>[10] What Matters in Unsupervised Optical Flow <a href="https://arxiv.org/abs/2006.04902">https://arxiv.org/abs/2006.04902</a></p><p>[11] SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping <a href="https://arxiv.org/abs/2105.07014">https://arxiv.org/abs/2105.07014</a></p><p>[12] Uncertainty Estimates and Multi-Hypotheses<br>Networks for Optical Flow <a href="https://openaccess.thecvf.com/content_ECCV_2018/papers/Eddy_Ilg_Uncertainty_Estimates_and_ECCV_2018_paper.pdf">https://openaccess.thecvf.com/content_ECCV_2018/papers/Eddy_Ilg_Uncertainty_Estimates_and_ECCV_2018_paper.pdf</a></p><p>[13] Epipolar geometry tutorial OpenCV: <a href="https://www.geeksforgeeks.org/python-opencv-epipolar-geometry/">https://www.geeksforgeeks.org/python-opencv-epipolar-geometry/</a></p><p>[14] Learning to Segment Rigid Motions from Two Frames: <a href="https://arxiv.org/pdf/2101.03694.pdf">https://arxiv.org/pdf/2101.03694.pdf</a></p><p>[15] Self-Supervised Monocular Scene Flow Estimation <a href="https://openaccess.thecvf.com/content_CVPR_2020/papers/Hur_Self-Supervised_Monocular_Scene_Flow_Estimation_CVPR_2020_paper.pdf">https://openaccess.thecvf.com/content_CVPR_2020/papers/Hur_Self-Supervised_Monocular_Scene_Flow_Estimation_CVPR_2020_paper.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68901cf01435" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing the power of fish eye cameras in Carla simulator]]></title>
            <link>https://medium.com/@d.d.tananaev/unleashing-the-power-of-fish-eye-cameras-in-carla-simulator-abb7ab5bc147?source=rss-a6caf4f5f26e------2</link>
            <guid isPermaLink="false">https://medium.com/p/abb7ab5bc147</guid>
            <category><![CDATA[fisheyelens]]></category>
            <category><![CDATA[carla]]></category>
            <category><![CDATA[fisheye]]></category>
            <dc:creator><![CDATA[Logos]]></dc:creator>
            <pubDate>Wed, 14 Feb 2024 08:29:32 GMT</pubDate>
            <atom:updated>2024-03-29T15:01:50.306Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r9hpIE0yxw1BgCHXcjV7og.jpeg" /><figcaption>Fish eye Carla Images: rgb image (left), semantic segmentation (middle), depth (right)</figcaption></figure><p>Carla [1] is an open-source simulator for autonomous driving. Over the years, it has witnessed significant enhancements, which makes it comparable to numerous professional autonomous driving simulators in terms of amount of supported features. Despite its growing capabilities, a notable absence of wide-view cameras limits its practicality for real-world data applications.</p><p>The wast majority of autonomous driving systems uses different combinations of narrow view cameras for far range perception (e.g. handling highway driving scenarios) and also wide-view or fish eye cameras for the near range scenarios (e.g. parking).</p><p>The Carla simulator supports only pinhole camera projection model which can be applicable for narrow-view camera modelling and wide-view with field of view (FoV) up to 120 degrees [2].</p><p>Theoretical maximal FoV for pinhole should be less than 180 degrees, but in practice when the FoV become larger than 120 degrees the fish eye camera model would be better choice for higher accuracy.</p><blockquote>The code with implemented fish eye camera: <a href="https://github.com/Dtananaev/CarlaFSD/tree/main">https://github.com/Dtananaev/CarlaFSD/tree/main</a></blockquote><h4><strong>Camera geometry</strong></h4><p>In order to understand the difference of pinhole cameras and fish eye we need to dig into camera geometry. Camera geometry consists of two parts: camera model and distortion model.</p><p><strong>Camera model</strong>: the mathematical model which explains projection of 3d point to 2d image.</p><p><strong>Distortion model</strong>: the mathematical model which explains mapping from 2d distorted image coordinates to 2d undistorted image coordinates. Usually it is used to model lens distortion which are not captured by the camera model.</p><p>Both camera and distortion models combined to get precise projection of 3d world to 2d image plane.</p><h4><strong>Pinhole camera model</strong></h4><p>The pinhole camera model is simplified model of the projection of the 3d point to 2d plane with assumption that there are no lenses (for more information see nice tutorial from professor Cyrill Stachniss <a href="https://youtu.be/uHApDqH-8UE?si=OEj9xkibrur0-Os1">here</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGH3QlXK3H6zwdlE9O1A0A.png" /><figcaption>The schematic representation of the pinhole projection: the example of projection of the 3d point X to the image plane ends up at pixels coordinate x (on the left). The example of relations between 3d point X the Y coordinate and focal length f (on the right). The equations of the projection of 3D point X(X, Y, Z) to pixels coordinate (x, y) in image (bottom of the image)</figcaption></figure><p>The projection of the 3D point <strong>X</strong> with coordinates <strong>(X, Y, Z)</strong> to the image plane with pixels coordinates <strong>(x, y)</strong> can be represented as equation in the picture above(where <strong>X , Y, Z</strong>— 3d coordinates, <strong>x, y </strong>— 2d coordinates in the image plane and <strong>f</strong>-focal length). It is important to note that after projection with this equation we will have the pixels coordinates origin <strong>(0, 0)</strong> in the principle point <strong>p(cx, cy)</strong> (around the center of the image). If we want to transfer it to the image coordinates with origin at top left corner we should shift all pixels coordinates by <strong>p(cx, cy)</strong> (see full equation below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRUY7UAbKE2P8F_GsyPpFw.png" /></figure><p>Finally for implementation in python we need vectorized representation as matrix multiplication:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pMw6s88qUYjhwPcKcNg1Jg.png" /></figure><p>You may note that pixels coordinates represented as 3d vector <strong>x = [x, y, 1]</strong>, where last element equal 1. This coordinates are called homogeneous (for more details see tutorial <a href="https://youtu.be/PvEl63t-opM?si=H6jU1tSqbsl2ptIq">here</a>). The main goal of homogeneous coordinates is to enable projection from 3d to 2d by using matrix multiplication, as soon as last element equal 1 the first two <strong>(x, y)</strong> corresponds to pixel position.</p><p>The calibration matrix <strong>K </strong>consists of focal length for x and y axis <strong>(fx, fy)</strong> and principle point <strong>p =( cx, cy)</strong>. This parameters (also called intrinsic parameters) can be received during camera calibration process (for example see <a href="https://youtu.be/26nV4oDLiqc?si=7zxNQ-ZZ9lC8IARG">this</a> tutorial.)</p><p>The projection function in python will look like:</p><pre>import numpy as np<br>def pinhole_projection(X: np.ndarray, K: np.ndarray)-&gt; np.ndarray:<br>    &quot;&quot;&quot;Projects 3d points to the image.<br><br>    Note: Here K - is camera intrinsic camera parameters<br>            This parameters can be defined by calibration process<br>                    | fx 0.0  cx |<br>                K = | 0.0 fy  cy |<br>                    | 0.0 0.0 1.0|<br>          here fx, fy - the focal length of the camera<br>          cx, cy - principle point (usually pixel position<br>          near the center of the image)<br>          (e.g. cx = width/ 2.0, cy = height/2.0)<br>    Args:<br>        X: the array of 3d points of the shape [num_points, 3]<br>        K: the intrinsic matrix of the shape 3x3<br>    <br>    Returns:<br>        x: homogeneos pixels coordinates of the shape [num_points, 3]<br>    &quot;&quot;&quot;<br>    # We need Z values for normalization of the shape [1, num_points]<br>    # this is depth values which are equal Z coordinate of 3d point<br>    Z = np.expand_dims(X[..., -1], axis=0)<br>    # Here we get tensor of the shape [3, num_points] with pixels coordinates<br>    x = (1.0 / Z) * (K @ X.T)<br>    # Transpose to get shape [num_points, 3 (x, y, 1)]<br>    x = x.T<br>    # Here x_pixel_coords, y_pixels_coords = x[:, 0], x[:, 1]<br>    return x</pre><p>Another important part of camera model is how to undo projection or how to get 3d point from 2d pixels position. Such model is called inverse camera projection. In python can be represented as follows:</p><pre><br>def inverse_pinhole_projection(pixels2d: np.ndarray, K: np.ndarray)-&gt; np.ndarray:<br>    &quot;&quot;&quot;Inverse projection from 2d pixels to 3d rays.<br><br>        Note: Here K - is camera intrinsic camera parameters<br>            This parameters can be defined by calibration process<br>                    | fx 0.0  cx |<br>                K = | 0.0 fy  cy |<br>                    | 0.0 0.0 1.0|<br>    Args:<br>      pixels2d: 2d pixels coordinates of the shape [num_points, 2 (x, y)]<br>      K: calibration matrix<br>    <br>    Returns:<br>        rays3d: 3d rays in camera coordinates of the shape [num_points, 3]<br>    &quot;&quot;&quot;<br>    num_points, _  = pixels2d.shape<br><br>    ones = np.ones((num_points, 1))<br>    # Make homogeneous pixels coordinates of the shape [num_points, 3 (x, y, 1)]<br>    # in order to enable matrix multiplication for inverse projection<br>    homogeneos_coords = np.concatenate((pixels2d, ones), axis=-1)<br><br>    # Here inverse operation <br>    rays3d = np.linalg.inv(K) @ homogeneos_coords.T<br><br>    # Here the 3d rays in camera coordinates of the shape [num_points, 3]<br>    rays3d = rays3d.T<br>    #  You may note that in order to get full 3d points<br>    # we need Z coordinates (depth)<br>    # X = rays3d * Z - 3d points recovered<br>    # The depth information  (Z) is lost during projection operation<br>    # And can&#39;t be recovered in inverse projection operation<br>    return rays3d<br><br></pre><p>One important drawback of camera projection is that it projects 3d point to the 2d plane and during this operation the depth (Z-coordinate in case of pinhole model) is lost. Therefore, if we apply inverse operation we can reconstruct the set of 3d rays and in order to completely restore 3d point we would need to get somewhere depth information (e.g. from depth from mono neural network or from structure from motion (SfM) techniques).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQ-DC-ogT3QIvoGugvXjaQ.gif" /><figcaption>The example of visualization of the 3d rays from pinhole camera. You may note all 3d rays located in plane due to missing depth</figcaption></figure><p>But even though we can’t recover full 3d points without depth we can use rays from pinhole camera to model fish eye camera (in next chapters.)</p><blockquote>The code for this section can be found <a href="https://github.com/Dtananaev/CarlaFSD/blob/main/carla_fsd/camera_fisheye/tutorial/pinhole_projection.py">here</a>.</blockquote><h4><strong>Distortion model</strong></h4><p>As we discussed above the pinhole camera model assumes no lens distortion, but in reality we always uses cameras with different lenses. If we want to make use of pinhole model for real cameras we need somehow model the lens effects and remove it before applying pinhole model projection. The process of removing lens effect from image is called undistortion. And mathematically it modeled as distortion model. Above we gave the definition of the distortion model as mapping from 2d distorted image coordinates to 2d undistorted image coordinates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*JAWq2gonOXpVF3LaLJoavA.png" /><figcaption>The example of the distorted rd and undistorted ru pixels position of the pinhole camera. Here Id represents image with distortion of the lenses and Iu image without distortion. Here (u, v) is a pixels coordinate along x and y axis correspondingly. (Image source [2])</figcaption></figure><p>In the image above you may see how pixels coordinate for the same 3d point <strong>W</strong> differs for distorted image <strong>I<em>d</em></strong> and undistorted image <strong>I<em>u</em></strong>.</p><p>We can represent the relation between this images by using popular polynomial distortion model which describes two types of distortion <strong>radial</strong> and <strong>decentering (tangential) </strong>[2] (see images below for more details).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cuyWbXRYFI10yM9NC9rVCg.gif" /><figcaption>The example of type of radial distortion: original image (on the left); barrel distortion (on the right)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TW7NEM0M7tER7W4sNEKugA.gif" /><figcaption>The example of type of radial distortion: original image (on the left); pincushion distortion(on the right)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Rc5Km2whOBoIt0SZsp8zw.gif" /><figcaption>The example of type the tangential distortion: original image (on the left); decentering distortion (on the right)</figcaption></figure><p>Such distortion mapping can be expressed as following equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4tqtz3Xr9h3RMZD1Feg8NA.png" /></figure><p>You may observe that the model explains mapping between distorted pixels position<strong> (x<em>d</em>, y<em>d</em>) </strong>and undistorted pixels position<strong> (x<em>u</em>, y<em>u</em>)</strong>. The pinhole camera model can work only with undistorted pixels position. Therefore before using pinhole model we first apply undistortion of the image by using equations above.</p><p>The distortion model also needs distortion coefficients (intrinsic parameters of the model) <strong>k1, k2, k3 </strong>— for radial distortion and <strong>p1, p2, p3</strong> — for the tangential. This coefficients also can be defined during camera calibration process.</p><blockquote>The python implementation of the distortion model can be found <a href="https://github.com/Dtananaev/CarlaFSD/blob/main/carla_fsd/camera_fisheye/camera_models/lense_distortion.py">here</a>.</blockquote><h4>Fisheye camera model</h4><p>The fisheye camera lenses are capable to grasp field of view (FoV) more than 180 degrees. In order to model so wide FoV we may try to use <strong>pinhole camera model + distortion model</strong> with strong non-linear radial distortion.</p><p>However, the results of such model usually not precise enough compare to explicit 3d rays based mathematical projection models, which provide explicit projection equations (like pinhole model), but for fish eye camera lenses. Therefore it is common practice to model fish eye lenses as <strong>fisheye camera model + distortion model</strong>.</p><p>You may ask why we still need distortion model? The reason is that fish eye camera model assumes perfect fish eye lenses, which is not true in reality. There could be production artifacts (e.g. imperfect lens curvature which leads to radial distortion) or mounting artifact (e.g. lenses mounted not exactly to match center of the camera (decentering distortion case)).</p><p>Lets look on example of the one popular rays-based fisheye model: equidistant projection model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*lon_kGSZ6sbW9_547WyvGg.png" /><figcaption>Equidistant fisheye projection model schema: The 3d point P can be projected in image plane by using pinhole camera model to the ru — position (in case if we have no fisheye lens distortion). But given fisheye distortion the distorted image position rd. (image source [3])</figcaption></figure><p>In the image above there is schematic representation of equidistant fish eye model projection. In case if we have pure pinhole model with no distortion we will reproject 3d point <strong>P</strong> to the image plane at <strong>r<em>u</em></strong> position.</p><p>But given fisheye lens distortion the distorted position will be <strong>r<em>d</em></strong>. We can explicitly model mathematically the mapping between 3d point <strong>P </strong>to <strong>r<em>d</em></strong>.</p><p>First we can model fish eye lens as sphere with radius equal focal length <strong>f </strong>(instead of plane like in pinhole case). In this case 3d points can be observed with field of view more than 180 degrees, since they will always intersect the sphere.</p><p>Now we need to find out how to map from sphere to the 2d image plane, since in the end we still need 2d image!</p><p>In equidistant camera model (in image above) we can assume that the pixels position <strong>r<em>d </em></strong>equivalent the length of the arc segment between axis z and the 3d points ray <strong>P </strong>(see bold segment of the sphere).</p><p>Mathematically such relation can be represented as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*oluKimzuY_jFCtWiVjiKUA.png" /></figure><p>The full derived mathematical model for the equidistant fisheye camera can be expressed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CyPrFUGT0GCjup0i5Agd8g.png" /><figcaption>Image source [4]</figcaption></figure><p>The projection equation explains how to project 3d point with coordinate <strong>(X,Y, Z)</strong> to normalized image coordinates<strong> (<em>x*, y*</em>)</strong>. In order to get pixels coordinates it is necessary to apply<strong> x_pixel = f<em>x*</em> + c<em>x</em></strong>;<strong> y_pixel = f<em>y*</em> + c<em>y</em></strong>.</p><p>Here <strong>f </strong>— focal length and <strong>(c<em>x</em>, c<em>y</em>) </strong>— coordinates of the principle point (calibration matrix <strong>K</strong> parameters).</p><p>The inverse projection represents inverse operation. First we need to get normalized coordinates from pixels coordinates<em> </em><strong><em>x*=</em>(x_pixel — c<em>x</em>) / f <em>; y*=</em>(y_pixel — c<em>y</em>) / f .</strong></p><p>And than apply projection to get unit sphere 3d rays (see image below.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H8ODgSXdn_I8fXG-60UFpg.gif" /><figcaption>The example of visualization of the 3d rays from fisheye camera. You may note all 3d rays located in unit sphere</figcaption></figure><p>In order to get 3d points this rays same as in case of pinhole camera should be multiplied by depth.</p><p>But there is a difference for given 3d point with coordinates X, Y, Z:</p><ol><li>In case of pinhole camera model depth is z coordinate: depth = Z</li><li>In case of fisheye camera model depth is L2 norm: depth = sqrt(X**2 + Y**2 + Z**2) (here sqrt — square root.)</li></ol><p>If you want to dive deeper in fish eye geometry check nice <a href="https://plaut.github.io/fisheye_tutorial/#cylindrical-images-to-the-rescue">tutorial</a> [5].</p><blockquote>The code for fisheye rays visualization can be found <a href="https://github.com/Dtananaev/CarlaFSD/blob/main/carla_fsd/camera_fisheye/tutorial/fisheye_projection.py">here</a>, the equidistant camera model <a href="https://github.com/Dtananaev/CarlaFSD/blob/main/carla_fsd/camera_fisheye/camera_models/equidistant_projection.py">here</a></blockquote><h4>Creating fish eye camera from five pinhole cameras</h4><p>Finally we can create fish eye camera in carla simulator. One of the possibility to do that would be to access ray casting and directly create fish eye spherical rays, find their intersections with 3d objects and save the rgb values for corresponding pixels.</p><p>More simple would be to use highly optimized pinhole camera implementation and reproject data from five pinhole cube to fish eye sphere (see the image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qUOOmWSAbJue-G14PPgGPw.gif" /><figcaption>The example of the five pinhole to fish eye rays projection</figcaption></figure><p>In terms of implementation we can stitch together five images from pinhole cameras and for every pixel of the fish eye image we need to find corresponding pixel of the stitched pinhole image (see the image below).</p><p>And than apply inverse warping (pick corresponding pixels values at a given pinhole images and write them into corresponding fish eye pixels.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QKpgsqPf7rF6cDBVzpzFQ.png" /><figcaption>Stitched five pinhole image (top), resulted reprojected fish eye image (bottom)</figcaption></figure><p>This mapping can be found by first inverse projection from fish eye image (we will get spherical rays) and projection of the spherical rays to every pinhole image.</p><blockquote>The code for visualizaiton can be found <a href="https://github.com/Dtananaev/CarlaFSD/blob/main/carla_fsd/camera_fisheye/tutorial/five_pinhole_to_fisheye.py">here</a>. The remapping from pinhole to fish eye cameras can be found <a href="https://github.com/Dtananaev/CarlaFSD/blob/main/carla_fsd/camera_fisheye/fisheye_camera.py">here</a>.</blockquote><h4>Carla package for fish eye cameras</h4><p>Finally you can spawn and use fish eye cameras inside carla by following simple example code <a href="https://github.com/Dtananaev/CarlaFSD/blob/main/carla_fsd/camera_fisheye/drive.py">here</a>.</p><p>You need to install carla and repository from here <a href="https://github.com/Dtananaev/CarlaFSD/tree/main">https://github.com/Dtananaev/CarlaFSD/tree/main</a></p><p>and than simple run:</p><pre>python -m carla_fsd.camera_fisheye.drive</pre><p>This will spawn fish eye camera and make it possible to drive manually with <em>w, a, s, d </em>keys.</p><p><strong>Literature</strong></p><p>[1] Carla simulator: <a href="http://carla.org//">http://carla.org//</a></p><p>[2] Fan et al. Wide-angle Image Rectification: A Survey, (2021): <a href="https://arxiv.org/abs/2011.12108">https://arxiv.org/abs/2011.12108</a></p><p>[3] Hughes et al. Accuracy of fish-eye lens models, (2010): <a href="https://www.researchgate.net/publication/44664797_Accuracy_of_fish-eye_lens_models">https://www.researchgate.net/publication/44664797_Accuracy_of_fish-eye_lens_models</a></p><p>[4] Steffen Abraham, Wolfgang Förstner. Fish-Eye-Stereo<br>Calibration and Epipolar Rectification, (2005): <a href="http://www.ipb.uni-bonn.de/pdfs/Steffen2005Fish.pdf">http://www.ipb.uni-bonn.de/pdfs/Steffen2005Fish.pdf</a></p><p>[5] Tutorial on fisheye geometry: <a href="https://plaut.github.io/fisheye_tutorial/#cylindrical-images-to-the-rescue">https://plaut.github.io/fisheye_tutorial/#cylindrical-images-to-the-rescue</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abb7ab5bc147" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>