“The 15 Most Frustrating AR/VR Development Bugs and How to Fix Every Single One”

Photo by Kevin Doyle on Unsplash

🔍 1. Tracking Loss on Featureless Surfaces

Why it happens: SLAM systems (ARKit, ARCore, OpenXR spatial tracking) rely on visual contrast and depth edges. White walls, glass, or low-texture floors give the algorithm nothing to latch onto. The fix:

• Enable hybrid tracking (IMU + vision + LiDAR if available)
• Add subtle digital markers or projection-mapped patterns in your app
• Fallback to raycast-based placement with confidence scoring Pro tip: In Unity, monitor ARCameraTrackingState.Tracking and trackingReason. If Reason == Limited, pause physics updates and show a subtle UI nudge instead of fighting the drift.

🌊 2. Jittery or Drifting Anchors

Why it happens: World anchors are continuously optimized by the backend. When relocalization or plane reclassification happens, anchor transforms get smoothed or snapped, causing visible pop/jitter. The fix:

• Enable anchor smoothing/interpolation on the CPU side
• Attach virtual objects to ARPlane center + normal, not raw anchor transforms
• Cache and blend between old/new poses using Mathf.Lerp or dual-quaternion interpolation Pro tip: Use a short history buffer (3–5 frames) of anchor poses and apply a Kalman or exponential moving average filter before feeding transforms to render loops.

🎭 3. Z-Fighting in 3D Overlays & UI

Why it happens: Two surfaces share nearly identical depth values, causing the depth buffer to flip-float between them. The fix:

• Increase polygon offset (glPolygonOffset, Unity’s Offset in materials, or Unreal’s PolygonOffset)
• Scale UI elements slightly along Z (0.98–1.02)
• Use a dedicated UI render queue/depth layer Pro tip: In stereo rendering, even 0.001m differences can cause per-eye flicker. Always test z-fighting in headset, not the editor viewport.

👁️ 4. Stereo Shader Flickering / Double Vision

Why it happens: Shaders using Camera.worldToCameraMatrix or gl_FragCoord don’t account for per-eye transforms or temporal reprojection. The fix:

• Use stereo-aware shader macros (UNITY_MATRIX_I_V, STEREO_PROJECTION_MATRIX, or OpenXR view indices)
• Avoid hard depth comparisons without per-eye offsets
• Disable motion vectors/temporal AA for diagnostic passes Pro tip: Flicker that disappears in monocular view is always a stereo-matrix issue. Log StereoRenderingPath and verify shader variants compile for multi-view.

🔄 5. Passthrough Misalignment & Depth Offset

Why it happens: MR headsets (Quest 3, Vision Pro) render passthrough with a slight latency + hardware calibration offset. Virtual objects placed at “0 distance” often sit behind/forward of the real world. The fix:

• Apply a calibrated Z-offset based on headset model
• Use environment depth APIs to align virtual geometry with real surfaces
• Run a quick “touch calibration” routine at app start Pro tip: Store per-device offset values in a config JSON. Vision Pro’s RealityKit and Quest’s PassthroughAPI both expose depth alignment tools—use them.

🎮 6. Controller Drift & Calibration Drift

Why it happens: IMU bias accumulates over time. Temperature changes, magnetometer interference, or firmware updates shift baseline calibration. The fix:

• Implement periodic recalibration on pose reset
• Use deadzones and velocity thresholds for input filtering
• Expose a manual “recalibrate” button tied to InputDevices.GetFeatureValuePro tip: Drift is rarely the controller’s fault—it’s usually the SDK’s fusion filter. Switch from Raw to Predicted or Smoothed tracking states in your input manager.

📏 7. Scale Mismatch Between Real & Virtual

Why it happens: AR platforms report in meters, but imported assets use centimeters/inches, or camera FOV assumptions don’t match real optics. The fix:

• Lock import scale to 0.01 (Blender → Unity) or enable metric units in-engine
• Validate scale with a real-world reference (e.g., 1m ruler plane at runtime)
• Use ARSessionOrigin.MakeContentAppearAt with explicit scale matrices Pro tip: Always bake scale into FBX/glTF exports. Runtime scaling breaks physics, lighting, and hand tracking colliders.

💡 8. Lighting Mismatch & “Floaty” Objects

Why it happens: IBL/cubemaps don’t match the real environment, or directional lights ignore estimated AR light estimation data. The fix:

• Feed ARLightEstimation (intensity, color temp, spherical harmonics) into your rendering pipeline
• Use probe placement near the anchor, not at the camera
• Add subtle contact shadows + ambient occlusion Pro tip: In Unity URP/HDRP, enable Realtime Reflection Probes with Blend Distance = 0.5–1.5m for seamless AR lighting transitions.

🔥 9. Thermal Throttling & Sudden Frame Drops

Why it happens: Mobile/XR SoCs drop clock speeds when thermals exceed thresholds. GPU-heavy shaders, unbatched draw calls, or background processes trigger it. The fix:

• Cap framerate to 72/90Hz consistently; avoid dynamic scaling
• Use LODs, baked lighting, and occlusion culling aggressively
• Monitor thermals via SystemInfo.thermalStatus and degrade effects before it hits Pro tip: Throttling feels like a bug, but it’s a hardware safeguard. Profile with RenderDoc or Android GPU Inspector before blaming the SDK.

🤲 10. Hand Tracking Occlusion & Joint Jitter

Why it happens: Depth cameras struggle with fingers crossing, low light, or fast motion. SDKs extrapolate, causing snap-back or floating joints. The fix:

• Apply joint velocity damping & positional clamping
• Fallback to controller/hybrid input when confidence drops below threshold
• Use skeletal retargeting with spring physics, not direct pose mapping Pro tip: Confidence < 0.4 means the data is noise. Mask it out. Never trust raw hand tracking without a smoothing layer.

📐 11. UI Elements Clipping or Floating at Wrong Depth

Why it happens: World-space canvases don’t respect depth testing, or camera near/far planes clip geometry in stereo. The fix:

• Set canvas render mode to World Space + Camera
• Match canvas Z to Camera.nearClipPlane + 0.1
• Use CanvasGroup fade instead of scaling for depth transitions Pro tip: In VR, UI that sits inside the focal plane causes vergence-accommodation conflict. Keep interactive UI between 1.5–3m.

🔊 12. Spatial Audio Latency & Misalignment

Why it happens: Audio engines use world transforms that lag behind head tracking, or Doppler/roll-off settings aren’t tuned for room scale. The fix:

• Use dedicated spatial audio SDKs (Meta Presence, Steam Audio, Resonance)
• Feed head pose at Update(), not FixedUpdate()
• Tune max distance & rolloff to match your play space Pro tip: If sound feels “behind” you, it’s usually a transform update order issue. Run audio pose sync before render.

💥 13. Build Crashes on Device (IL2CPP / AOT)

Why it happens: Reflection, dynamic linking, or unsupported generics fail in AOT compilation. Works in editor, dies on headset. The fix:

• Add link.xml to preserve required types
• Replace dynamic/Activator.CreateInstance with factory patterns
• Use #if ENABLE_IL2CPP guards for platform-specific code Pro tip: Always run Unity Editor → Build Settings → Development Build with Script Debugging on first to catch IL2CPP stripping issues early.

🕳️ 14. Mesh Reconstruction Artifacts & Holes

Why it happens: LiDAR/depth sensors miss reflective, transparent, or fast-moving surfaces. Reconstruction algorithms interpolate poorly. The fix:

• Enable mesh smoothing & hole filling in SDK
• Fallback to plane detection for large flat surfaces
• Add procedural filler meshes with transparent edge blending Pro tip: Don’t trust real-time meshes for collision. Use raycast-based surface proxies with confidence filtering.

🌐 15. Cross-Platform Input Mapping Conflicts

Why it happens: OpenXR profiles, Unity XR Interaction Toolkit, and Unreal Input Mapping Contexts handle button mapping differently. Trigger on one device = Grip on another. The fix:

• Abstract input behind a unified IInputProvider interface
• Use action-based input (XR Input System / OpenXR Actions) instead of device-specific polling
• Maintain a runtime config that remaps per headset Pro tip: Never hardcode Button 0 or Axis 3. Map to semantic actions (Select, Locomotion, Interact) and let the runtime resolve them.

🔧 The Real Secret to XR Debugging

Most of these bugs don’t come from bad code. They come from mismatched assumptions:

• Assuming tracking is perfect
• Assuming stereo renders like a flat screen
• Assuming the real world behaves like a static 3D scene

The best XR developers don’t just write code — they build fallbacks, smoothing layers, and graceful degradation into every system from day one.

Get ATNO For AR / VR / MR / XR Developer’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Start every feature with:

1. A tracking/pose validation layer
2. A smoothing/interpolation pass
3. A confidence threshold that triggers fallback UI
4. A runtime profiler hook for thermals & draw calls

Do that, and 90% of “bugs” become features waiting to be tuned.