Photogrammetry vs. NeRFs vs. gaussian splatting – pros and cons
A comparison of photogrammetry, NeRFs, and Gaussian splatting for 3D modeling, with Teleport by Varjo offering an intuitive iOS app for easy, high-quality 3D scanning and proce
Knut Nesheim
3D modeling of real-world objects can be done using many different techniques, and the three most commonly used ones are photogrammetry, neural radiance fields (NeRFs), and Gaussian splatting. Each of these techniques has unique strengths and weaknesses, making them best-suited for different tasks. In this article, you’ll learn more about each technique in detail.
Photogrammetry: Simple, detailed, and time-consuming
The oldest out of the three, photogrammetry is a technique based on large sets of photographs. It relies on multiple images of the same object taken from different angles. Photogrammetry software is then used to detect the points where these images should join together so a 3D model can be constructed.
As long as the source images themselves are high-quality, the result is a highly detailed and accurate 3D representation of the object, which can be anything from small items to very large objects such as terrain or large buildings. A 3D model created with photogrammetry can be measured and has a scale, and lighting conditions of the source images are also baked into the 3D image.
The primary advantages of photogrammetry are its simplicity and the high fidelity of the textures in the 3D models it produces – as long as the source images are of high quality as well. You can for example take high quality images using a high-end camera or use drone footage of large objects to create a model with photogrammetry and the detail will be preserved.
Pros of Photogrammetry
High fidelity and accuracy: Photogrammetry can produce highly accurate and detailed 3D models, capturing textures and fine details directly from photographs.
Cost-effective: Photogrammetry only requires a high-quality camera and some software, making it accessible and cost-effective for many applications.
Ideal for object scanning: The output of photogrammetry is a 3D mesh that is easy to interact with. It is easy to scale, rotate, translate, etc., making it well-suited for objects. However, because details are focused on the object you photograph, not on the external environment around it, photogrammetry is less ideal for environments.
Cons of Photogrammetry
Heavily dependent on photo quality: The quality of the photogrammetry model is only as good as the source image. Image resolution and lighting conditions during shooting matter significantly.
Poor Performance with Reflective or Transparent Surfaces: Photogrammetry often struggles with reflective, transparent, or featureless surfaces which might not capture well in photographs.
Highly visible artifacting: As photogrammetry is essentially a collection of images stitched together, it can very easily cause gaps in the capture that will produce highly apparent and jarring artifacts such as a hole in a mesh or a jagged edge that juts out from a scanned object.
Neural Radiance Fields (NeRFs): AI-driven models with dynamic lighting from smaller image sets
Neural radiance fields (NeRFs) is a more recent technology that creates 3D models from 2D images using deep learning. Unlike photogrammetry that builds 3D models from large, detailed sets of images, NeRF can create models using a much more limited set of photos.
NeRF models the light and color information as it moves through space, creating a continuous volumetric scene. This means it can also be used to easily generate new viewpoints that you did not originally capture in your images. As NeRF can render scenes with complex lighting and viewpoints with a more limited set of images, it’s particularly useful for creating virtual reality environments where the user can move around.
One advantage NeRFs have compared to photogrammetry is that they are better suited for capturing the environment around an object in addition to the object itself, so you can show skies, objects in the horizon, ceilings of indoor spaces, etc. more easily than with photogrammetry.
Pros of NeRFs
Relatively high quality: NeRFs do not consist of a mesh; they’re a volumetric representation of a scene. This means you can dynamically capture light, reflections, and opacity, leading to a vastly increased sense of realism that allows you to create fairly detailed models (and even lifelike visuals).
Requires less inputs: Even with limited image inputs, NeRF can construct complex models and scenes, making it ideal for visual effects and virtual reality scenarios.
Allows complex light interactions: NeRFs are excellent for modeling complex lighting scenarios, including reflections, refractions, and shadows, providing a more natural integration of virtual objects into real environments.
Dynamic Viewing Angles: Allows for dynamic viewing angles and perspectives that are not directly captured in the original images.
Cons of NeRFs
Computational intensity: NeRFs require a lot of computational power, so they can be slow to train and render. This also leads to low FPS while rendering, making NeRFs impractical for immersive use cases such as VR scenarios.
Limited by sparse data: While NeRFs require less input data than photogrammetry, very sparse or uneven image distributions can lead to poor-quality models or artifacting.
Not as detailed as photogrammetry and lacks measurement capabilities: Because NeRFs lack the ability for precise measurement, they are best used for visualization rather than precise 3D modeling of objects.
Gaussian splatting: Fast and ideal for real-time applications
Gaussian splatting is the most recent 3D visualization technique primarily used in the visualization of volumetric data. It models scenes as a collection of points that create the visual appearance of surfaces. This involves projecting data points into a visualization space and smoothing them using a Gaussian function, creating a continuous representation from discrete data points.
While gaussian splatting and NeRFs are both volumetric techniques, they have major differences. NeRFs model the entire scene as a continuous volumetric function, while gaussian splatting represents the scene as a collection of discrete “splats” – point clouds with gaussian profiles that help blend the data smoothly. The gaussian approach allows for faster, real-time visualization by reducing computational requirements, but lacks the ability to model intricate lighting interactions and tends to be less detailed compared to NeRF.
Gaussian splatting is particularly useful for rendering fuzzy objects or semi-transparent materials where you need to visualize density or intensity variations within a volume. Originally, it was less about creating accurate 3D models from images and more about effectively displaying complex 3D data.
A major benefit of gaussian splatting is that it requires lighter computation compared to the other methods, meaning it is well-suited for real-time applications and modeling large environments that would require processing vast amounts of data. It should also be noted that as gaussian splatting is a new technology, it is constantly being developed further and will be capable of more in the future.
Pros of Gaussian Splatting
Strong depth of field: Gaussian splatting is an excellent technique when you need to convey a sense of distances. It is very easy for the user to evaluate distances and tell what’s close and what’s far away – especially when compared to photogrammetry and 360 photos and videos.
Visualizes fuzzy or transparent materials: Gaussian splatting is excellent for rendering materials that have varying opacity or density, such as glass or biological tissues.
Lower computational load: Gaussian splatting requires lower computational resources compared to photogrammetry and NeRFs, making it ideal for real-time applications. It means hardware requirements are lower and a higher FPS can be achieved, which enables immersive viewing of these realistic scenes in VR for the first time.
Cons of Gaussian Splatting
Artifacts: While not as jarring as artifacting in photogrammetry, gaussian splatting can also introduce artifacts such as blurring or ghosting if the viewer moves far away from the path of the scan.
Lower interactivity: Unlike with photogrammetry or NeRFs, objects scanned with gaussian splatting are not an individual virtual object or 3D mesh, but rather a cloud of thousands of mini gaussians forming an object. This makes objects scanned with gaussian splatting very hard to move, scale or otherwise interact with. However, this limitation is being worked on and might not be as much of an issue in the future.
Teleport by Varjo makes capturing 3D objects and environments a breeze
The three methods outlined above are particularly well suited for specific tasks, but all of them require rendering on a dedicated computer, and often require you to take numerous photos and use a high-end camera for the best result.
Teleport removes these restrictions by allowing you to scan and upload environments using an intuitive iOS application, without needing specialized cameras. Scans are processed in the cloud using advanced machine learning, removing the need for manual processing or editing.
With Teleport, you can easily create photorealistic digital twins of any environment or object, from single rooms to town squares, with accurate lighting and shading, textured, reflections, and more. Once the scans are processed, you can explore your capture in your web browser, with a desktop app, or in real-world scale with a VR headset.
.jpeg)
.jpeg)
