<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lathika on Medium]]></title>
        <description><![CDATA[Stories by Lathika on Medium]]></description>
        <link>https://medium.com/@lathwath5?source=rss-f78c5dcde75e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nJPLdCXfRc4VkZTCHeuqpg.jpeg</url>
            <title>Stories by Lathika on Medium</title>
            <link>https://medium.com/@lathwath5?source=rss-f78c5dcde75e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:30:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lathwath5/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tri-Plane 3D Representation Explained.]]></title>
            <link>https://medium.com/@lathwath5/tri-plane-3d-representation-explained-a57f81e653cc?source=rss-f78c5dcde75e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a57f81e653cc</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[3d-representation]]></category>
            <category><![CDATA[3d-vision]]></category>
            <dc:creator><![CDATA[Lathika]]></dc:creator>
            <pubDate>Tue, 11 Mar 2025 15:34:25 GMT</pubDate>
            <atom:updated>2025-03-11T18:51:36.109Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mh7Gt_UIh3CGdSh7hixgJA.png" /></figure><p>The 3D world we live in, is mostly filled with free space. Using a 3D array to represent that 3D space might not be the best option. The quest for efficient and expressive 3D representations has driven computer vision, gaming, and VR innovations. This technique balances efficiency and high-quality rendering, making it valuable in computer vision, gaming, and virtual reality. We’ll delve into the background and the methodology of this method and its impact on current research.</p><h3>What is Tri-Plane Representation?</h3><p>Tri-plane representation is a hybrid implicit-explicit 3D modeling technique that encodes 3D scenes using three orthogonal 2D feature planes (XY, XZ, and YZ). For any 3D point, features are sampled from these planes by getting the corresponding projections of the 3D point into these planes, and aggregated to make the feature vector. Those features are then decoded into color and density values. This approach reduces memory utilization, compared to voxel grids while maintaining faster rendering speeds than purely implicit methods like Neural Radiance Fields (<a href="https://arxiv.org/abs/2003.08934"><em>NeRF</em></a>).</p><p>This method <em>retains the 3D geometry</em>, while using only three 2D arrays. Therefore, the <em>space requirement becomes considerably low</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3996IsVCzMLa0Ki2MU77Ag.png" /></figure><p><strong>Feature Storage</strong>: - Each plane stores a 2D grid of high-dimensional features, avoiding the cubic memory costs of voxels.</p><p><strong>Feature Aggregation</strong>: - For each 3D point, features are extracted from the three planes, combined by interpolation and addition. This final feature vector is used as a compressed descriptor for decoding.</p><p><strong>Decoding</strong>: - A lightweight neural network translates aggregated features into RGB color and density. This enables efficient volume rendering.</p><p>This efficient and lightweight method is ideal for 3D-GANs, real-time applications, and compression.</p><h3>Let&#39;s compare Tri-Plane representation with some traditional methods</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-B5gHpMMhwgGFm3iUt5EUQ.png" /><figcaption>Table_1: Camparison of different 3D representation methods</figcaption></figure><h3>Novel Studies Utilizing the Tri-Plane Method</h3><ol><li>EG3D: 3D GANs with High Accuracy</li></ol><p>Stanford and NVIDIA’s EG3D advanced 3D image generation by combining tri-plane representation with StyleGAN2. It produces fast, detailed 3D geometry and consistent 2D images, surpassing previous techniques. (<a href="https://arxiv.org/pdf/2112.07945"><em>Link</em></a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*kbH1gRCbEm4o8Kr_OuEmkA.png" /></figure><p>2. PanoHead and SphereHead: Full-Head Synthesising</p><p>PanoHead achieved 360° head synthesis but suffered from mirroring issues due to its Cartesian tri-plane approach. SphereHead solved this by using spherical coordinates, better fitting the head’s shape and eliminating those artifacts. (<a href="https://arxiv.org/pdf/2303.13071"><em>Link for PanoHead paper</em></a>, <a href="https://arxiv.org/html/2404.05680v1#:~:text=In%20response%2C%20we%20propose%20SphereHead,many%20of%20the%20generated%20artifacts."><em>Link for SphereHead paper</em></a>)</p><p>3. TriHuman: Real-Time Avatars</p><p>This research used tri-plane representation for dynamic human avatars, linking it to skeletal movement for real-time, pose-controlled results. It achieved top-quality 3D models and visuals at 30 frames per second. (<a href="https://arxiv.org/pdf/2312.05161"><em>Link</em></a>)</p><p>4. Triplane Meets Gaussian Splatting</p><p>This paper combined tri-plane and 3D Gaussians for fast single-view 3D reconstruction. Tri-plane provided quick, rough shapes, while Gaussians added fine detail, leading to a 10x speed improvement over NeRF methods. (<a href="https://arxiv.org/pdf/2312.09147"><em>Link</em></a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNep4iiNTfH1vEC9OGQVNg.png" /></figure><h3>Challenges in Tri-Plane Method</h3><p>As we can see from <em>Table_1 </em>above, even though the tri-plane methods achieve great efficiency with low memory usage, there are some challenges that need to be addressed. One main challenge is the symmetry artifacts in the cartesian coordinates. This happens because symmetrical points can have very similar representations, which leads to unnatural mirroring and feature duplication. For example, faces can appear perfectly symmetrical, lacking realistic variations. To fix this, researchers adapt the tri-plane method to specific object types. This involves using coordinate systems like spherical coordinates for heads or adjusting the planes based on pose, aligning the representation with the object’s natural geometry and improving realism. (For more details: <a href="https://arxiv.org/html/2404.05680v1#:~:text=In%20response%2C%20we%20propose%20SphereHead,many%20of%20the%20generated%20artifacts."><em>SphereHead Paper</em></a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*YjcjjiIa5wurXz5pM1KoFw.png" /><figcaption><em>Mirroring-face artifacts (</em><a href="https://arxiv.org/html/2404.05680v1#:~:text=In%20response%2C%20we%20propose%20SphereHead,many%20of%20the%20generated%20artifacts."><em>Link</em></a><em>)</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*EiduKQzETr3PsFQRe8wlVg.png" /><figcaption>(a) Tri-plane representation. (b) Spherical tri-plane representation. (<a href="https://arxiv.org/html/2404.05680v1#:~:text=In%20response%2C%20we%20propose%20SphereHead,many%20of%20the%20generated%20artifacts.">Link</a>)</figcaption></figure><p>As we can conclude, the tri-plane representation has revolutionized 3D neural rendering by balancing speed, memory, and realism. Its hybrid approach is used in different applications. Ongoing research to improve its capabilities suggests it will be a key technology in future 3D vision systems. Let’s sit back and observe!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a57f81e653cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Point Cloud Downsampling Methods and Python Implementations]]></title>
            <link>https://medium.com/@lathwath5/point-cloud-downsampling-methods-and-python-implementations-6f91a129ac48?source=rss-f78c5dcde75e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f91a129ac48</guid>
            <category><![CDATA[computer-graphics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[point-cloud]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[3d-reconstruction]]></category>
            <dc:creator><![CDATA[Lathika]]></dc:creator>
            <pubDate>Wed, 08 Jan 2025 06:23:47 GMT</pubDate>
            <atom:updated>2025-01-08T06:23:47.657Z</atom:updated>
            <content:encoded><![CDATA[<p>Point clouds are collections of data points that represent 3D shapes or objects and can contain millions or billions of points. While these details can be useful, it complicates processing and visualization. To overcome this, point cloud data is often downsampled, reducing the number of points while preserving the overall structure. There are various downsampling methods, each with its pros and cons, but all aim to maintain the integrity of the original data. Here I will discuss about 4 such methods.</p><ol><li>Random Downsampling</li><li>Voxel Grid Downsampling</li><li>Uniform Grid Subsampling</li><li>Curvature-Preserving Sampling</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/0*3rA1wCk12BxcpR7W" /><figcaption>Figure 1: Downsampling. Source:- <a href="https://www.researchgate.net/publication/349520013/figure/fig2/AS:994308547158020@1614073039857/Point-Cloud-Downsample-a-the-point-cloud-before-downsampling-b-the-point-cloud.png">Link</a></figcaption></figure><p>First, let’s assume we have a PLY file path, from which we can extract its basic properties using the following function. (Source:- <a href="https://github.com/graphdeco-inria/gaussian-splatting/tree/main"><em>3DGS_Git</em></a>)</p><pre>class BasicPointCloud(NamedTuple):<br>    points : np.array<br>    colors : np.array<br>    normals : np.array<br><br>def fetchPly(path):<br>    plydata = PlyData.read(path)<br>    vertices = plydata[&#39;vertex&#39;]<br>    positions = np.vstack([vertices[&#39;x&#39;], vertices[&#39;y&#39;], vertices[&#39;z&#39;]]).T<br>    colors = np.vstack([vertices[&#39;red&#39;], vertices[&#39;green&#39;], vertices[&#39;blue&#39;]]).T / 255.0<br>    normals = np.vstack([vertices[&#39;nx&#39;], vertices[&#39;ny&#39;], vertices[&#39;nz&#39;]]).T<br>    return BasicPointCloud(points=positions, colors=colors, normals=normals)</pre><h3>1. Random Downsampling</h3><p>This is a straightforward method for randomly sampling a subset of our points. However, if we reduce the number of points too much, we may risk losing the key properties of the structure, since the points are randomly selected. While this method is simple, it does carry some risks. Additionally, this method is computationally efficient.</p><pre>def downsample_random(point_cloud: BasicPointCloud, ratio: float) -&gt; BasicPointCloud:<br>    &quot;&quot;&quot;Downsamples the point cloud by a given ratio.&quot;&quot;&quot;<br>    num_points = point_cloud.points.shape[0]<br>    sample_size = int(num_points * ratio)<br>    indices = np.random.choice(num_points, sample_size, replace=False)<br>    return BasicPointCloud(points=point_cloud.points[indices],<br>                           colors=point_cloud.colors[indices],<br>                           normals=point_cloud.normals[indices])</pre><p>“ratio” is a float value between 0 and 1. As an example, if the ratio is 0.2, then it will preserve 20% of the points.</p><h3>2. Voxel Grid Downsampling</h3><p>This method divides 3D space into a grid of fixed-size cubes, known as voxels, and retains only one representative point for each voxel. This approach effectively captures the overall structure of the data while reducing the total number of points. The remaining points reflect the average position of all points within their corresponding voxel, calculated by taking the mean of the points contained in each voxel. Additionally, point selection can be done in two alternative ways: by choosing the first point encountered in the voxel or by selecting the centroid of the voxel (if there are points within it). The method of point selection can be adjusted based on specific requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*DmNo6x1G-PmH2-zD" /><figcaption>Figure 2: Original point cloud and downsampled point cloud, using voxel grid method. Source: <a href="https://www.researchgate.net/publication/377932709/figure/fig3/AS:11431281231980457@1711585420802/The-original-image-and-the-results-of-two-voxel-downsampling-methods-a-Original-image.tif">Link</a></figcaption></figure><pre>import open3d as o3d<br><br>def downsample_voxel_grid(point_cloud: BasicPointCloud, voxel_size: float) -&gt; BasicPointCloud:<br>    &quot;&quot;&quot;Downsamples the point cloud using voxel grid filtering.&quot;&quot;&quot;<br>    # Convert BasicPointCloud to Open3D format<br>    o3d_cloud = o3d.geometry.PointCloud()<br>    o3d_cloud.points = o3d.utility.Vector3dVector(point_cloud.points)<br>    o3d_cloud.colors = o3d.utility.Vector3dVector(point_cloud.colors)<br>    o3d_cloud.normals = o3d.utility.Vector3dVector(point_cloud.normals)<br>    <br>    # Apply voxel downsampling<br>    downsampled_cloud = o3d_cloud.voxel_down_sample(voxel_size=voxel_size)<br>    <br>    # Convert back to BasicPointCloud<br>    return BasicPointCloud(points=np.asarray(downsampled_cloud.points),<br>                           colors=np.asarray(downsampled_cloud.colors),<br>                           normals=np.asarray(downsampled_cloud.normals))</pre><p>“Voxel_size” is the voxel size used for downsampling. Smaller voxel sizes can be selected for higher resolutions. However, there are a few things to consider before selecting the “voxel_size”.</p><ul><li>If the point cloud represents a large object or environment (a room or landscape), use a <strong>larger voxel size</strong>.</li><li>If the point cloud represents a small object with fine details, use a <strong>smaller voxel size</strong>.</li></ul><p><strong>Typical Values</strong></p><ul><li><strong>Small, detailed object</strong>: Use a voxel size of <strong>0.001 to 0.01 units.</strong></li><li><strong>Medium-sized objects</strong> <em>(furniture or vehicles</em>): Use a voxel size of <strong>0.01 to 0.05 units</strong>.</li><li><strong>Large-scale environments </strong>(<em>buildings or outdoor scenes</em>): Use a voxel size of <strong>0.1 to 0.5 units</strong>.</li></ul><blockquote>As an example,</blockquote><blockquote><strong>0.005 to 0.01</strong>: Preserves fine details for small objects or localized structures.</blockquote><blockquote><strong>0.02 to 0.05</strong>: Useful for general-purpose downsampling of medium-size scenes.</blockquote><blockquote><strong>0.1 or greater</strong>: Reduces point count significantly for large-scale scenes while preserving global structure.</blockquote><h3>3. Uniform Grid Subsampling</h3><p>The space is divided into a 3D grid of fixed-size cells, and for each cell, a single point is selected by calculating the mean. However, this approach results in a more uniform spatial distribution across the entire point cloud. Grid-based pooling is often explicitly designed for spatial regularity and prioritizes uniformity, which may sacrifice some fine structural details compared to the voxel grid method.</p><pre>import numpy as np<br><br>def uniform_grid_subsample(point_cloud: BasicPointCloud, grid_size: float) -&gt; BasicPointCloud:<br>    &quot;&quot;&quot;Uniform grid subsampling of a point cloud.&quot;&quot;&quot;<br>    points = point_cloud.points<br>    colors = point_cloud.colors<br>    normals = point_cloud.normals<br><br>    # Compute grid indices for each point<br>    grid_indices = np.floor(points / grid_size).astype(np.int32)<br><br>    # Create a dictionary to store points by grid cell<br>    grid_dict = {}<br>    for i, idx in enumerate(grid_indices):<br>        key = tuple(idx)<br>        if key not in grid_dict:<br>            grid_dict[key] = (points[i], colors[i], normals[i], 1)<br>        else:<br>            pos, col, norm, count = grid_dict[key]<br>            # Accumulate values for mean computation<br>            grid_dict[key] = (pos + points[i], col + colors[i], norm + normals[i], count + 1)<br><br>    # Compute mean values for each grid cell<br>    downsampled_points = []<br>    downsampled_colors = []<br>    downsampled_normals = []<br>    for pos, col, norm, count in grid_dict.values():<br>        downsampled_points.append(pos / count)<br>        downsampled_colors.append(col / count)<br>        downsampled_normals.append(norm / count)<br><br>    return BasicPointCloud(points=np.array(downsampled_points), <br>                           colors=np.array(downsampled_colors), <br>                           normals=np.array(downsampled_normals))</pre><p>Here “grid_size” Determines the size of each grid cell. Larger values result in higher downsampling.</p><h3>4. Curvature-Preserving Sampling</h3><p>This method focuses on preserving points with high curvature, such as edges and corners, which mainly contribute to the overall shape. Curvature can be estimated by examining the eigenvalues of the covariance matrix within a point’s neighborhood. So when compared to previous methods, this is a bit different.</p><p>How does this work?</p><ul><li>First, select the nearest neighbors for a point. (say 5)</li><li>Then compute the mean of the neighbors (compute the mean point for 5 neighbors).</li><li>Then compute the covariance matrix using the normalized positions of the neighbors. (Here we expect a 5 by 5 covariance matrix calculated using the normalized positions (point — mean_point))</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*ZJh00rnAOdLnAwSYZbV8yw.png" /><figcaption>Convariance equation</figcaption></figure><ul><li>Then compute the Eigenvalues of the covariance matrix. The ratio of the minimum Eigan value to the sum of Eigenvalues is defined as the curvature.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/1*7PyKKbWkMXCufgT9dcvunw.png" /><figcaption>Curvature</figcaption></figure><ul><li>After ranking points w.r.t their curvature values, select a set of points with the highest curvature for the downsampled point cloud. This will preserve key structural features.</li></ul><pre>import open3d as o3d<br>import numpy as np<br><br>def curvature_preserving_sampling(point_cloud: BasicPointCloud, k_neighbors=20, fraction=0.2) -&gt; BasicPointCloud:<br>    &quot;&quot;&quot;Curvature-preserving downsampling using eigenvalue-based curvature estimation.&quot;&quot;&quot;<br>    o3d_cloud = o3d.geometry.PointCloud()<br>    o3d_cloud.points = o3d.utility.Vector3dVector(point_cloud.points)<br>    <br>    # Compute curvature using Open3D&#39;s KDTree search<br>    o3d_cloud.estimate_normals(search_param=o3d.geometry.KDTreeSearchParamKNN(knn=k_neighbors))<br>    <br>    # Calculate curvature as a function of eigenvalues<br>    curvatures = []<br>    kdtree = o3d.geometry.KDTreeFlann(o3d_cloud)<br>    for i in range(len(point_cloud.points)):<br>        _, idx, _ = kdtree.search_knn_vector_3d(o3d_cloud.points[i], k_neighbors)<br>        neighbors = np.asarray(o3d_cloud.points)[idx]<br>        cov = np.cov(neighbors, rowvar=False)<br>        eigenvalues, _ = np.linalg.eigh(cov)<br>        curvature = eigenvalues[0] / sum(eigenvalues)<br>        curvatures.append(curvature)<br><br>    # Sort points by curvature and keep top fraction<br>    sorted_indices = np.argsort(curvatures)[::-1]<br>    num_samples = int(len(point_cloud.points) * fraction)<br>    selected_indices = sorted_indices[:num_samples]<br><br>    return BasicPointCloud(points=point_cloud.points[selected_indices],<br>                           colors=point_cloud.colors[selected_indices],<br>                           normals=point_cloud.normals[selected_indices])</pre><p>“K_neighbors”: Number of neighbors to be used for estimating curvature.</p><p>“Fraction”: Fraction of points to keep.</p><h3><strong>Comparison</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*xOc6SJ4kkIwlnM-rQ_Q2qw.png" /><figcaption>Figure 3: Comparison</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f91a129ac48" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mip-Splatting: Alias-free 3D Gaussian Splatting Paper Explained]]></title>
            <link>https://medium.com/@lathwath5/mip-splating-alias-free-3d-gaussian-splatting-paper-explained-3cfb0a4748db?source=rss-f78c5dcde75e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cfb0a4748db</guid>
            <category><![CDATA[best-paper]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[3dgaussiansplatting]]></category>
            <category><![CDATA[vision]]></category>
            <dc:creator><![CDATA[Lathika]]></dc:creator>
            <pubDate>Tue, 03 Dec 2024 12:44:44 GMT</pubDate>
            <atom:updated>2024-12-03T13:24:52.516Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://arxiv.org/abs/2311.16493"><em>Paper Link</em></a></p><p>In this discussion, I will explain the Mip-Splatting paper, which was awarded the Best Student Paper at CVPR 2024 (Conference on Computer Vision and Pattern Recognition). Mip-splatting focuses on optimizing novel view synthesis outputs, building on the concepts introduced by the 3D Gaussian Splatting paper, which has made a significant impact in the field of 3D reconstruction.</p><p>Let’s start with some fundamental concepts.</p><p><strong><em>What is Novel View Synthesis?</em></strong></p><p>Novel view synthesis (NVS) refers to the process of generating new views of a scene from a set of pre-captured viewpoints (images). This challenge has been addressed by various methods, including interpolation techniques, multi-view stereo, and newer approaches like Neural Radiance Fields (NeRF) and 3D Gaussian splatting. NVS has numerous applications in fields such as virtual reality, robotics, gaming, medical imaging, and e-commerce, among others.</p><p><strong><em>What is 3D Gaussian Splatting?</em></strong></p><p>Imagine seeing shapes in the sky created by clouds. These clouds may not be situated close together, yet they form recognizable shapes. In this context, we treat these shapes as 3D Gaussian clouds acting as the basis for complex 3D structures. Each small cloud has several parameters: position, orientation, size, opacity, and color. These are trained through a multiview photometric loss. Of particular interest is the color representation, which utilizes the concept of spherical harmonics (angular solutions to Laplace’s equation in spherical coordinates). Here, color values are computed based on the viewing direction, allowing different colors to be generated for the Gaussians.</p><p>The 3D Gaussian model begins with a set of these Gaussian bases, adjusting their parameters while pruning and cloning to create a 3D representation of the scene. Even with a limited number of images of the scene, we can develop a fairly accurate 3D model. Novel views can be generated by simply rasterizing the 3D model based on the desired view.</p><p>The 3D Gaussian splatting technique has gained attention as an appealing alternative to multilayer perceptrons (MLPs) and feature grid-based representations. It stands out for its impressive results in novel view synthesis while achieving high-quality real-time rendering. For a deeper understanding, please refer to the “<strong>3D-Gaussian Splatting for Real-Time Radiance Field Rendering</strong>” paper or any related resources before moving on to the subsequent sections.</p><p>Refer 3D Gaussian Splatting Paper: —<em> </em><a href="https://arxiv.org/abs/2308.04079"><em>Link</em></a></p><h3><strong>Problems that have been identified in this paper</strong></h3><p>Although 3D Gaussian splatting is employed, several key issues have been identified in this work. When rendering the 3D model into screen space, a 2D dilation operation is applied for low-pass filtering. However, this process can produce artifacts when training with camera views that differ from the original screen, such as during zooming in and zooming out.</p><p>The paper discusses possible reasons for this, such as the lack of 3D frequency constraints and the use of a 2D dilation filter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/0*dxNPr25yishv6OY2" /><figcaption>Figure 1 :- Dilation and Erosion. <a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fpub.towardsai.net%2Fimage-processing-morphological-operations-with-python-7e0f8d1983eb&amp;psig=AOvVaw1QH8WgVdK9GXIFtSTZWzvi&amp;ust=1733225965657000&amp;source=images&amp;cd=vfe&amp;opi=89978449&amp;ved=0CBcQjhxqFwoTCMCcg4aAiY">Link</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*FGjpSN-q-wcbx8iX" /><figcaption>Figure 2: -Problem identification. Credit:- <a href="https://arxiv.org/abs/2311.16493">Mip-splatting paper</a></figcaption></figure><p><strong><em>Part a</em></strong>: - Imagine there are five pixels in the screen space, creating an ideal and accurate rendering situation. The 3D Gaussian (represented by a black Gaussian distribution line) is perfectly trained to match the 3D object (depicted as a blue circle). The 3D Gaussian aligns precisely with the original object. When projected into screen space, the Gaussian shadow (the corresponding 2D Gaussian of the 3D Gaussian) perfectly aligns with the size of a pixel. Therefore, the object is represented by that pixel. To achieve this representation, the Gaussian must cover the third pixel. Is this the only way to achieve this 2D representation?</p><p><strong><em>Part b</em></strong>: - If the Gaussian has a higher frequency, the method’s intrinsic shrinkage bias may exceed the sampling rate (which will result in smaller Gaussians). As a result, the Gaussian will be dilated and approximated to a larger Gaussian, represented by the highest frequency of the system,(meaning a pixel will be allocated for a smaller Gaussian). There will be no difference between the rendered outputs of Part A and Part B at this sampling rate. (The sampling rate can change depending on the camera distance or focal length.)</p><p><strong><em>Part c</em></strong>: - When the focal length is reduced or the camera distance is increased (zooming out), the sampling rate decreases. Here, the sampling rate refers to the number of pixels per unit of physical distance, which determines the level of detail in a digital image. With the correct 3D Gaussian as in “Part a”, if the sampling rate is reduced, the Gaussian will be significantly smaller than the size of the sampling pixel box. However, due to dilation, it will be approximated to the largest size of the sampling pixel box, resulting in exaggerated features, as described in Part C. This effect will lead to noticeable dilation results.</p><p><strong><em>Part d</em></strong>: - In a scenario where dilation effects initially caused degeneration of the Gaussian, increasing the sampling rate (by increasing focal length or reducing camera distance) may lead to artifacts and visible, underdeveloped Gaussians. Since the originally trained Gaussian was smaller, a higher sampling rate will capture its differences, resulting in erosion effects in the digital image, similar to the bike cable effect noted in “Part c”.</p><p>While the model produced accurate images from the training scale, noticeable artifacts, dilations, and erosions can occur at other scale views.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/0*wafslPE3HXdmX55T" /><figcaption>Figure 3: — Aliasing effects on different implementations. Credit:- <a href="https://arxiv.org/abs/2311.16493">Mip-splatting paper</a></figcaption></figure><h3><strong>How were these issues addressed?</strong></h3><p>According to Nyquist’s sampling theorem, to accurately capture a signal, it must be sampled at a rate that is at least twice its highest frequency. Sampling at a lower rate can lead to aliasing, which occurs when high-frequency components of the signal are misrepresented as lower-frequency components in the sampled signal, resulting in distortion. There are two options to address aliasing: one is to use a higher sampling rate, and the other is to limit the maximum frequency of the signal to half of the sampling frequency. In this case, the second option is the more viable solution.</p><p>The authors have introduced two primary methods to address the outlined issues:</p><p>1. A 3D smoothing filter for 3DGS that effectively regularizes the maximum frequency of 3D Gaussian primitives. This method resolves the artifacts observed during zooming in and out of distribution renderings found in previous approaches.</p><p>2. A replacement of the 2D dilation filter with a 2D Mip filter to tackle aliasing and dilation artifacts.</p><p>These implementations are straightforward and highly effective.</p><h3><strong>1. 3D Smoothing filter</strong></h3><p>It is a well-known issue that multiple 3D structures can produce identical 2D views. The highest frequency of the reconstructed 3D scene is limited by the sampling rate of the training views.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*91ugtMDC5PoREj_f" /><figcaption>Figure 4: — Same 2D shadow from different 3D structuers.</figcaption></figure><p>Following Nyquist’s principle, the authors aim to limit the maximum frequency of the 3D representation during optimization. The sampling interval can be calculated by considering the focal length and the depth of the Gaussian center.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/0*ZF9wHVqDmfdusGec" /></figure><p>In this context, we consider the sampling interval on the screen to be 1 (equivalent to 1 pixel length). According to the Angle-Angle-Angle (AAA) criterion, the corresponding sampling interval in 3D space, when dealing with a Gaussian distribution, can be expressed as <em>d/f, </em>where <em>d </em>represents depth (or distance) and <em>f </em>denotes the focal length in pixels. The term <em>v </em>refers to the sampling rate, and <em>T </em>is the sampling interval.</p><p>Reconstruction algorithms can accurately reconstruct components of a signal with frequencies up to <em>v</em>/2 when samples are taken at this frequency <em>v</em>. Consequently, a primitive smaller than <em>2T </em>can lead to aliasing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*Zi6XaBfs7EHIOasr" /><figcaption>Figure 5</figcaption></figure><p>They calculate the maximum sampling frequency for each Gaussian so that at least one camera (training view) can successfully reconstruct the corresponding primitive. This maximum frequency is then used to establish the boundary for the low-pass filter.</p><p>A low-pass filter is designed to allow lower frequencies to pass through while reducing higher frequencies. In the example mentioned, this low-pass smoothing filter ensures that the Gaussian values do not fall below <em>2T</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/0*DXAK89AMnZVRorq7" /></figure><p>Here<em> V</em>ₖ is the maximum sampling frequency for kᵗʰ Gaussian. <em>T</em>ₙ<em>(P</em>ₖ<em>)</em> is an indicator function that will be 1 for Gaussians that are visible and 0 otherwise. <em>N</em> is the number of images. Here for each training image, they check the sampling rate corresponding to the distance of the Gaussian and store the maximum sampling rate, such that at least one camera can reconstruct the respective primitive. Since the 3D Gaussian centers remain relatively stable during the training, the maximum sampling rate is computed only for every mᵗʰ iteration.</p><p>After acquiring the maximum possible sampling rate for a primitive, then a 3D Gaussian low pass filter is created accordingly. Let Gₖ be the kᵗʰ Gaussian, <em>Glow </em>be its low pass filter, and <em>Gₖ(X)reg</em> be the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/0*Ljl6vw5RIigodhKy" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*T7EQJ5N39YRjeLXq" /><figcaption>Figure 6: Convolution of Gaussians. Credits: -<a href="https://commons.wikimedia.org/wiki/File:Convolution_Animation_%28Gaussian%29.gif">Link</a></figcaption></figure><p>This Gaussian convolution results in a Gaussian with the covariance of <em>Σ_k+Σ_low</em> where <em>Σ_k</em> is the covariance of the primitive while <em>Σ_low</em> is the covariance of the low pass filter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/0*sRtpO7cpdebjvmct" /></figure><p>Here “s” is the scale hyperparameter that controls the filter size. The scale <em>s/vₖ</em> is unique to each Gaussian sinch the maximum frequency depends on the camera views and their respective visibility pattern. By employing the 3D smoothing filter, the authors make sure that each Gaussian will not exceed its maximum sampling rate boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/0*nVuNSB9O_5tVlw8D" /></figure><p>The 3D low-pass Gaussian filter remains constant after the training since it has become a basic requirement for 3D representation.</p><h3><strong>2. 2D Mip Filter</strong></h3><p>Although the 3D smoothing filter helps reduce aliasing that occurs due to high sampling rates, aliasing caused by low sampling rates (such as when zooming out or moving the camera further away) still needs to be addressed. The previous version 3DGS, employed a dilation filter, which constrained each 2D covariance to the size of the pixel without decreasing its opacity value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/0*6LuHQDJONwoHwCjl" /><figcaption>Dilation in 3DGS</figcaption></figure><p>In this context, 0.3f is an approximation for √1. Since this only affects the size of the Gaussian, a smaller Gaussian that is closer to the pixel has a higher likelihood of saturating the rasterization process while fully covering the entire pixel, even if there are smaller Gaussians behind it. In practice, these Gaussians in the background may not be completely covered by the Gaussian in front. Therefore, their effects should be considered to achieve more realistic renderings.</p><p>In practical scenarios involving a camera, when multiple rays of light hit the sensor that corresponds to a single pixel, the pixel value is typically determined by integrating all the relevant colors over the pixel area. If the sampling rates are low, the ideal approach would be to use a box filter to average all the rays. However, for optimization purposes, the authors have opted to implement a 2D Gaussian filter to replace dilation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/0*wNX19oy9W7eNwU6t" /></figure><p>The scale “s” is chosen to cover a single pixel in screen space. <em>Σ</em>ₖ²ᴰ is the 2D covariance matrix derived from the 3D covariance after applying a transformation (using Jacobian matrix) on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/0*gg9k0sT4-LI-I7t6" /><figcaption>Figure 7: Effects of dilation filter and Mip filter</figcaption></figure><p>A: There are two small Gaussians before rendering. In this case, the sampling rates are low.</p><p>B: This represents the ideal scenario we expect, assuming we could achieve higher sampling rates.</p><p>C: This illustrates the effect of dilation. The pixel is 95% yellow. Because the dilation filter does not reduce the opacity while rendering, there is no opportunity to integrate the effects of the green Gaussian. As a result, the pixel will appear mostly yellow, even though we expect it to average out like a camera pixel, leading to a more realistic image without artifacts.</p><p>D: By using a Mip filter, the Gaussians will have a minimum size of one pixel while simultaneously reducing their alpha values (which correspond to opacity). This is due to the square root of the ratio of the determinants of the 2D covariance. Consequently, Gaussians that are farther away will still effectively influence the pixel color (Green will also affect the pixel color).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/0*vk5OBUvILw2YlDJW" /></figure><p>In summary, the Mip-Splatting paper represents a significant leap forward in novel view synthesis, addressing the critical challenges faced by 3D Gaussian Splatting. By introducing a 3D smoothing filter to control frequency constraints and a 2D Mip filter to resolve aliasing, the authors provide an elegant and practical solution to common artifacts like dilation and erosion. These techniques ensure that the rendered outputs remain consistent and realistic across various camera scales, maintaining accuracy even in challenging scenarios like zooming in or out.</p><p>The methodology stands out for its simplicity and effectiveness, combining well-grounded principles like Nyquist’s sampling theorem with novel adaptations for 3D reconstruction. By aligning theory with practical application, the authors have set a foundation for more robust, scalable, and high-quality rendering in real-time systems. This work is proof of how thoughtful innovations can refine complex technologies, pushing the boundaries of what is possible in computer vision and 3D reconstruction.</p><blockquote>Additionally, the 3DGS (original paper) GitHub code was updated to address the aliasing issue, with the new release on August 21, 2024.</blockquote><blockquote>Check Aug 21 new release commits: <a href="https://github.com/graphdeco-inria/gaussian-splatting/activity?ref=main">Link</a></blockquote><blockquote>Changes done in the new release: <a href="https://github.com/graphdeco-inria/diff-gaussian-rasterization/compare/59f5f77e3ddbac3ed9db93ec2cfe99ed6c5d121d...eb015708bb0ae0468367b60ec8d809a5f1ec34fe#diff-21fc0605026bb360b391ae3bc00aea857add7ec5a3d77cf3f69ba82dca77f220">Link</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*6BnprUlDY5vzp5DZ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*nMgUqAaGlUFYsbWx" /></figure><blockquote>Mip Filter implementation in Mip-splatting paper: <a href="https://github.com/autonomousvision/mip-splatting">GitHub Link</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*4Ifk7vcV0HTGVbr0" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*deIRNYeh-tT2fs53" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cfb0a4748db" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>