Depth Pro
In the last post, we covered how MiDaS and Depth Anything pushed monocular depth estimation forward. But there are other models who pushed it farther and I would like to introduce one of them — the Depth Pro.
To reconstruct 3D space from 2D image, depth map alone is not enough. You also need to know how the camera sees the world. This is where focal length comes in.
Focal length
Focal length is a physical property of a camera lens that determines how wide of narrow the field of view is. A telephoto lens has a long focal length — it captures a narrow view and makes distant objects appear larger. A wide-angle lens has a short focal length — it captures a broad view. Without knowing the focal length, AI cannot determine the actual distance of objects in an image. Even we have depth map, we can’t build 3D space without focal length.
Depth Pro
Depth Pro solves this problem by providing both a depth map and its corresponding focal length — all from a single image, without requiring any camera metadata.
Training Curriculum
The training process can be broken down into two tages: generalization and sharpening.
Stage 1: Generlization
Datasets
In this step, we use 3 types of datasets: real metric data(ARKitScenes, ScanNet, etc.), real non-metric data(HRWSI, BlendedMVG, etc.), and synthetic data(Hypersim, TartanAir, etc.). Metric datasets are those with reliable camera intrinsics and consistent scale — typically captured with LiDAR or RGB-D sensors. Non-Metric datasets lack this reliability, so scale normalization is applied before computing the loss.Synthetic data is generated by computers, so the exact depth value of every pixel is mathematically known.
Loss function
Not all depth data is created equal, so Depth Pro applies a different loss to each.
Metric data has real, sensor-measured depth values. But sensors are noisy — especially around glass and object boundaries. So Depth Pro uses MAE, and throws away the noisiest 20% of pixels before computing it. This is why it’s called “trimmed” MAE.
Non-metric data has depth labels. but the scale can’t be trusted. The same SSI appraoch MiDaS introduced applies here — normalize the scale first, then compute the loss. Structure can be learned even when absolute distance cannot.
Synthetic data has perfect ground truth, so gradient loss i added on top of MAE. This is the MSG Loss — it forces the model to pay attention to boundaries, not just overall depth values.
Stage2: Sharpening
Datasets
Stage 2 drops all real data. Only synthetic datasets remain — Hypersim, TartanAir, Synscapes, and others.
The reason is simple. Real-world sensors are unreliable around boundaries. Training on noisy boundary labels teaches the model to reproduce the noise, not the detail. Synthetic data doesn’t have this problem. Every boundary, every thin structure, every pixel is mathematically exact. So Stage 2 goes all-in on synthetic.
Loss function
Four losses run simultaneously.
MAE keeps the overall depth accurate — the foundation everything else builds on. MAGE targets edge sharpness through first-order gradients, pushing the model to trace boundaries precisely. MALE goes deeper with second-order Laplacian, capturing fine curvature and thin structures like hair and fur. MSGE uses squared gradients to stabilize high-frequency details without introducing noise.
Stage 1 builds the foundation. Stage 2 carves out the details. This is where Depth Pro’s signature sharpness comes from.
Model architecture
First : Downsampling and Spliting
Downsampling
Fisrt, Depth Pro downsamples the image into three resolutions: the original, half the original and a quarter of the original. By processing the image at multiple scales, the model captures both fine-grained details and the broader global structrue.
Splitting
Next, each resolution is split into patches of 384×384. The original 1536×1536 is divided into a 5×5 grid, yielding 25 patches. The 768×768 is split into a 3×3 grid, giving 9 patches. And the 384×384 is kept as a single patch.
Note that the patches are not cut by simply dividing the image — they overlap by 96 pixels between adjacent patches. This ensures every patch stays exactly 384×384 while preventing visible seams at the boundaries.
Merit of Splitting
Depth Pro uses a powerful transformer-based encoder. Transformers are great at capturing global context, but they come with a cost — attention complexity scales quadratically with the number of input pixels. Running a transformer directly on a 1536×1536 image would be computationally infeasible
Get BLED’s stories in your inbox
Join Medium for free to get updates from this writer.
Splitting solves this in two ways. First, by breaking image into 384×384 patches, the complexity drops dramatically. Second, since each patch is processed independently, they can all be run in parallel — making high-resolution depth estimation both accurate and fast.
Second : Encoder
Now we have 35 patches in total — 25 from the original, 9 from the half-resolution, and 1 from quarter-resolution — each encoded into a 24×24 feature map. Let’s see how the model processes them.
There are three Encoders, each serving a different purpose.
Patch Encoder
Before fedding the patches into the patch encoder, all 35 patches are concatenated and processed at once. As shown in Figure A, this yields 85 feature patches, each at a resolution of 24×24.
In side the encoder, each 384×384 patch is further divided into 16×16 tokens — giving a 24×24 grid per patch. Each 16×16 region is then embedded into a single vector, and the transformer runs over this 24×24 feature map. This prodcues 24×24 feature for each of the 35 patches.
For the finest scale — the 25 patches from the original resolution — Depth Pro also extracts intermediate features from the middle layers of the encoder, before the final output. This captures finer-grained boundary details, yielding an additional 50 features patches.
Feature
The encoder produces multi-scale features at different resolutions — from fine-grained boundary details to broad structural context.
Image Encoder
The 384×384 downsampled image is passed into a seperate image encoder. Unlike the patch encoder, there is no splitting here — the entire image goes through the transformer as a single input, capturing the global context of the scene.
Image Encoder for focal length
A seperate image encoder, trained independently for focal length estimation, works in parallel with the depth network to predict the camera’s field of view.
Third : Merge and Upsampling
The encoders have finished their job. Now we prepare the output for the decoder
merge
Merging is straightforward. Each feature patch is placed back into its original position, reassembling the fragemented patches into a single continuous feature map.
Upsampling
Just as image was downsampled at the beginning, each feature map is now upsamples — scaling it back up to meet the resolution the decoder expects
Final : Decoder, Focal Length Head
Now, we are ready for the final output.
Decoder
We use DPT-decoder. By all things we have, this decoder gives us estimated depth map.
Depth Pro uses the DPT decoder. Taking everything the encoders and merge steps ahve produced, the decoder outputs the estimated depth map at 1536×1536 resolution.
Focal Length Head
Before put the ingrident into Focal Length Head we are going to add Features 6 and output of image encoder for focal length head. The “Adding” itself, we learned adding make the result better by experience.
Then, put them into Focal Length Head. This FLH is differecnt from the DPT-decoder, it is additional seperated network.
Before feeding into the Focal Length Head, Features 6 and the output of the Image Encoder for focal length are added together. The choice of addition over concatenation came from experimentation — it simply worked better.
This combined feature is the Focal Length Head. Ulike the DPT decoder, the FLH is a small, separate network — a lightweight 3-layer convolutional network that outputs a single value: the focal length.
Conclusion
So, Depth Pro provides both a depth map and a focal length from a single image.
Then what does it matter? As we covered erlier, a depth map alone is not enough to reconstruct 3D space. You need to know how the camera sees the world.
In the next post, I will explore how AI reconstructs a 3D world from 2D images.
