The Geometry of a Ray & Finding Landmarks

So I’ve been working with cameras for a while now, e.g the ov2640 esp32 cam while playing with some ArUco fiducial markers (boring) and now while studying the mapillary app. What finally prompted me to studying cameras fully is SfM (structure from motion). SfM in simple terms allows you to convert a couple of 2D images to 3D. If you’ve read my other articles, you’d realize these things trigger a particular neuron in my head, makes me feel like, damn, what’s that, I bet I can do that…

Anyways, this is going to be a series of articles where I attempt to (actually will) complete a basic version of this SfM, and then make a 3d version of my Uni’s entrance, hence stick along yh? Cool, let’s get started. The full repo of all sections will be here: https://github.com/xpanvictor/sfm_pipeline/

The first article (this) is actually the very basics, it’s about even understanding how the pinhole camera works, how does it project a real world object on a camera plane (we call this the frontal plane, will still explain). We’ll also proceed to introduce the first layer of the SfM, feature extraction, which is how we pick out “important and unique” points in a picture.

Okay, let’s start with the pinhole projection, also I included image of the original note at the end. I’ll like to mention now, an image is really just a matrix of colors. Essentially something like at pixel 0,0 put black, 24, 9 put green… Combined together over a frame, you see an image. This is essentially how a camera projects a 3D object (e.g when you capture a box in front of you) into a 2D image. I particularly like this reference image of a pinhole camera view (source):

Let’s start breakdown, first thing, take an object located in the world as Xc, keep this location in reference to the position of the camera. In simpler terms, Xc is the position of the object if we take the camera’s position as the origin. Hence, the object is at Xc = [Xc, Yc, Zc]. There’s a standard used where x is the right axis of the camera, y is down and z is the invisible line that shoots right out of the camera lens perpendicularly. Given this z, the focal length is right on the z axis. Xc and Yc tells how far to the size and up/down the object is from the center line and the depth of Zc of the camera.

Quick deviation, the focal length should be behind the lens but instead we use a virtual image plane to avoid the fact that the sensor actually inverts the image upside down and left to right (Z = -f). Hence, we place this virtual screen at Z = +f, this screen is called the frontal plane.

Using the similar triangle image above where Pc is the object and the P’c is image projected on the frontal plane (you can derive this from the first image too, check here). We can use similar triangles to derive the following:

Quick note, the above is how the position of each pixel of your picture is computed relatively to the camera’s view point. For example if a black shade (0,0,0) is at (1, 2, 3) relatively to the camera of focal length 600, the “pixel” at x = (600 * (1/3)) = 200, y = (600 * (2/3)) = 400 is colored complete black… Note however, this causes a term known as perspective foreshooting where objects twice as far appear half as large, this also means depth information drops.

Now a quick correction, sensors don’t measure in millimeters from the center, they index pixels from top-left corner. Starting with, there is pixel scaling where the pixel densities (mx, my) are used to convert from “metres” to pixels then translated via the centres (cx, cy). This is just to correct that it’s not really just f but (fx, fy).

Now note, there’s a division in our conversion from camera view to pixels (division by Zc). This complicates processes because we’re already dealing with a matrix (numpy array), we should try to keep our entire process in matrix maths. Hence, we linearize this division using homogeneous coordinates. ‘w’ here helps us include the division since we have to convert back to euclidean pixel by division as expressed here, we essentially multiple the original u by w (Zc) to get the bar u:

‘K’ is seen as the intrinsic calibration matrix, those are essentially the “settings” your camera comes with.

Get xpan victor’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Now that we have the camera view and can easily compute it by just matrix multiplication, we can move on to the real world coordinates. What we have currently is an image based on the reference of the camera. However, what if we want to know the ‘stand’ of the object in reference to the “world” which could be another reference point, maybe the beginning of the road or sky, anywhere. This reference point essentially looks at the camera which is looking at the object, think, I’m looking at your eye to see what you’re looking at, where’s that object relatively to me… (over explaining).

This is where we have extrinsics. The points in the world are known in a global world frame Xw = [Xw, Yw, Zw] which isn’t relative to the camera center. We need to transform the world frame to the camera frame by first having a 3D rotation matrix (R) and then translating (moving the camera from origin of world reference) (t). Let ‭C be the camera center’s position measured in world coordinates, then to describe the real world object position Xw relative to the camera, we measure the vector from the camera center C, (Xw — C) then rotate it the offset vector into the coordinate orientation of the camera using the rotation matrix R hence Xc = R(Xw-C) as shown below:

Finally we can look at the full projection matrix. This is the essential flow we’ll run from the world view to the camera ray everytime.

World point (Xw) -> Extrinsics [R|t] -> 3D camera ray (Xc) -> Intrinsics [K] -> Pixel array (x).

Lol, this actually clicked immediately the second time reading through so take your time. Now, the final equation which the camera used to paint the appropriate pixels based on world view.

Here is a video demonstration of the full script I wrote:

https://vimeo.com/1224335179?share=copy&fl=sv&fe=ci

You can check and run the script here: https://github.com/xpanvictor/sfm_pipeline/blob/main/src/article_1_geometry/pinhole_cam.py

Now an introduction of the next chapter is feature extraction. Imagine you are standing on a street corner, holding two photos taken a few meters apart. Your task is to point to the exact same physical spot in both images — say, the top-right corner of a stop sign. This is easy for an human (we’re a dense mass of neural networks capable of …). However, how does a computer figure this?
A computer does not know what a “stop sign” is. To a computer, an image is just a massive 2D grid of numbers (pixel brightness from 0 to 255) as we mentioned earlier.
Feature extraction solves two fundamental questions:

1. Detection: Which pixels are actually worth tracking? (Finding the landmark)
2. Description: How do we describe that spot using math so we can recognize it in the other photo? (Writing the ID card)

Stay tuned as we dissect this in the next article.

My original note for the pinhole projection: