In this article, we’ll explore the practical application of eye gaze tracking on a mobile screen. Leveraging the power of ARKit’s ARFaceAnchor and the dynamic lookAtPoint property, this project allows us to precisely determine where a user is looking on their device.

The code and resources you need are conveniently hosted in the GitHub repository. Basics about ARKit, ARFaceAnchor, BlendShapeLocations, and LookAtPoint covered in 1st part.

In a recent Apple event, the frontier of eye tracking technology was pushed further with the Vision Pro mixed-reality headset, showcasing a future where users can interact with applications using just their eyes

Local Coordinate Space Vs World Coordinate System

Local Coordinate Space:

Imagine the local coordinate space of the ARFaceAnchor as a 3D grid centered on the face. The origin of this grid is at the center of the face, and positions and rotations are defined relative to this origin. The lookAtPoint is a coordinate in this local space and indicates the direction the user is looking concerning their own face.

World Coordinate System:

Now, let’s expand this view to the world coordinate system. In the broader world space (the entire environment captured by the camera), every object, including the face anchor, has its own position and orientation. The world coordinate system provides a global reference frame for all these objects.

The lookAtPoint is a point expressed in the local coordinate space of the face anchor. To convert it into a meaningful global position, you need to consider the anchor's own position and orientation in the world coordinate system.

Simplified Explanation: Local Coordinate Space Vs World Coordinate System and their relation with lookAtPoint & ARFaceAnchor :

Think of the ARFaceAnchor as the center of a little world on your face. This is the Local Coordinate Space. In this space, everything is measured from the center of your face, like how you might give directions using your own body as a reference point (raise your right hand).

Now, imagine you’re in a larger world, like a room. This is the World Coordinate System. In this big world, there are not only things on your face (like the ARFaceAnchor or where you’re looking), but also everything else around you, including the device’s camera capturing your face.

So, when we talk about where you’re looking (lookAtPoint), we first figure it out in the little world on your face (Local Coordinate Space with the ARFaceAnchor as the center). Then, we take that information and translate it to make sense in the bigger world around you (World Coordinate System), where there’s more than just your face — there’s the whole environment captured by the camera. It’s like saying, “I’m pointing to my nose in my personal space, and now let’s find where that is in the entire room.”

lookAtPoint w.r.t. the world coordinate system

To convert the lookAtPoint from the local coordinate space of the ARFaceAnchor to the world coordinate system, we can use matrix multiplication. The process involves applying the transformation matrix of the ARFaceAnchor to the lookAtPoint. This will bring the point from the local coordinate space of the face anchor to the world coordinate system.

let lookAtPointInWorld = faceAnchor.transform * simd_float4(lookAtPoint, 1)

In this line, faceAnchor.transform represents the transformation matrix of the ARFaceAnchor, and lookAtPoint is extended to a simd_float4 to facilitate matrix multiplication. The multiplication operation combines the transformation matrix and the point, resulting in a new point that is now in the world coordinate system.

After this transformation, lookAtPointInWorld contains the coordinates of the lookAtPoint with respect to the world coordinate system.

Camera Transformations

The camera is a crucial component in AR, responsible for capturing the real-world environment and providing the necessary data for rendering virtual objects in the correct perspective.

In ARKit, the camera.transform represents the transformation matrix of the device's camera. The transformation matrix is a mathematical construct that encapsulates translation, rotation, and scaling operations in 3D space. Essentially, it contains information about how the camera is positioned and oriented relative to the world coordinate system.

cameraTransform = session.currentFrame?.camera.transform

Breakdown of what the camera.transform includes:

• Translation (Position): Specifies the location of the camera in 3D space (X, Y, Z coordinates).
• Rotation: Describes the orientation of the camera. This can include rotations around the X, Y, and Z axes.
• Scaling: Defines any scaling factors applied to the camera.

camera.transform is useful to transform the lookAtPoint (gaze direction) from the local coordinate space of the ARFaceAnchor to the world coordinate system.

In summary, the camera.transform provides a comprehensive representation of how the device's camera is positioned and oriented in 3D space, enabling accurate transformations between the real world and the virtual world in AR applications.

lookAtPoint onto the mobile screen

To find the reflection of a point in the XY plane (phone screen plane) of a given camera transform, you can follow these steps:

1. Transform the LookAtPoint to Camera Coordinates:

• We have the lookAtPointInWorld in global coordinates (world coordinate system).
• Use the inverse of the camera transform to convert it to camera coordinates. This new point is referred to as lookAtPointInCamera.

let lookAtPointInCamera = simd_mul(simd_inverse(cameraTransform), lookAtPointInWorld)

2. Reflection in the XY Plane (Phone screen plane):

• To reflect a point in the XY plane, we can neglect its z-coordinate since lookAtPointInCamera is relative to camera coordinates.
• Coordinates — (transformedLookAtPoint.x, transformedLookAtPoint.y)

In conclusion, by transforming the lookAtPoint from global coordinates to camera coordinates and reflecting it in the XY plane, we successfully obtained the user’s gaze point coordinates in the camera coordinates system.

FocusPoint on Mobile Screen

After obtaining the user’s gaze point coordinates in the camera coordinates system, the next step is to project this point onto the mobile screen. The process involves several key calculations:

let screenX = transformedLookAtPoint.y / (Float(Device.screenSize.width) / 2) * Float(Device.frameSize.width)
let screenY = transformedLookAtPoint.x / (Float(Device.screenSize.height) / 2) * Float(Device.frameSize.height)
        
let focusPoint = CGPoint(
    x: CGFloat(screenX).clamped(to: Ranges.widthRange),
    y: CGFloat(screenY).clamped(to: Ranges.heightRange)
)

Normalization:

• transformedLookAtPoint.y / (Float(Device.screenSize.width) / 2): This step normalizes the y-coordinate of the gaze point relative to half of the screen width, resulting in a value between -1 and 1.
• transformedLookAtPoint.x / (Float(Device.screenSize.height) / 2): Similarly, the x-coordinate is normalized relative to half of the screen height.

Scaling to Screen Size:

• * Float(Device.frameSize.width): The normalized x-coordinate is then scaled to the full width of the screen.
• * Float(Device.frameSize.height): Likewise, the normalized y-coordinate is scaled to the full height of the screen.

Creating a CGPoint:

• let focusPoint = CGPoint(...): The scaled coordinates are then used to create a CGPoint, representing the user's focus point on the mobile screen.

Clamping to Screen Boundaries:

• clamped(to: Ranges.widthRange) and clamped(to: Ranges.heightRange): To ensure the focus point stays within the screen boundaries, these clamping operations restrict the coordinates to valid ranges.

limitations of eye tracking with ARKit

• Head movement: Head movement can affect the accuracy of eye tracking, especially if the user is moving their head quickly.
• Blinking: Blinking can also affect the accuracy of eye tracking, as the camera may not be able to track the user’s eyes during a blink.
• Ambient lighting: Ambient lighting can also affect the accuracy of eye tracking, as the camera may not be able to see the user’s eyes properly if the lighting is too bright or too dark.
• Other Factors: Glasses and accessories, Eyelid occlusion, Eye conditions, Distance and angle, Limited field of view:, Environmental factors, Hardware limitations

Eye tracking technology can be a game-changer, unlocking new dimensions of user interaction and immersion. an AR experience that responds to the movement of your eyes, providing a level of engagement and personalization previously unattainable. Imagine scrolling Instagram or X with eyebrow movement.

In this article, we’ll work on eye tracking with a specific focus on implementing it using ARKit, Apple’s powerful augmented reality framework. The code and resources you need are conveniently hosted in the GitHub repository.

Eye tracking utilizes the front camera to monitor and track the gaze and movements of a user’s eyes. This technology enables AR applications to precisely discern the user’s point of focus, facilitating dynamic adjustments in the presentation of digital content based on their gaze direction.

ARKit & ARFaceAnchor

ARKit is Apple’s augmented reality (AR) framework that empowers developers to integrate AR experiences seamlessly into iOS applications. One crucial aspect of ARKit is its ability to track and analyze facial expressions using the ARFaceAnchor class.

ARFaceAnchor:

ARFaceAnchor is a class in ARKit that represents the face geometry tracked in an AR session. It provides information about the user’s facial features, such as the position and orientation of the face, as well as details about facial expressions through the use of blend shapes.

Blend Shape Locations:

Blend shapes, in the context of ARFaceAnchor, refer to specific facial muscle movements or expressions that ARKit can identify and track. BlendShapeLocation is an enumeration in ARKit that defines various regions of the face, each corresponding to a specific facial feature or expression.

Some common BlendShapeLocations include:

• .mouthSmileLeft and .mouthSmileRight: Indicates a smile on the left or right side of the mouth.
• .eyeBlinkLeft and .eyeBlinkRight: Represents a blink in the left or right eye.
• .browDownLeft and .browDownRight: Signifies a downward movement of the left or right eyebrow.
• .mouthPucker: Indicates a puckering or tightening of the mouth.

Use Case: Interactive Gaming

In gaming applications, facial expressions can be harnessed as input for character control or to trigger in-game events. Imagine building a game similar to Flappy Bird, but with a unique twist — this time, the bird takes flight with a simple blink of the eye. By utilizing blend shapes and ARFaceAnchor in ARKit, developers can introduce an entirely new level of interactivity, making gaming experiences more immersive and responsive to the player’s facial expressions.

Extending this concept to other applications, similar functionality can be implemented in social media apps, allowing users to scroll through content without physically touching the screen. This innovative use of facial expressions can redefine how users interact with digital environments, making the experience more intuitive and engaging.

ARFaceAnchor.LookAtPoint

The lookAtPoint property in ARFaceAnchor plays a crucial role in eye tracking by providing information about the point in 3D space where the user is looking. When a user's gaze is directed at a specific point, this property holds the coordinates of that point in the AR world.

How it works:

1. Gaze Tracking: ARKit meticulously tracks the position of a user’s eyes by utilizing information from leftEyeTransform and rightEyeTransform.
2. 3D Coordinate Calculation: Leveraging the detected eye transforms, ARKit performs calculations to pinpoint the user’s current gaze in the 3D space. This calculated point is then represented by the lookAtPoint.
3. Real-time Updates: The lookAtPoint property stays dynamically updated in real-time, continuously reflecting changes in the user’s gaze. This provides developers with up-to-the-moment information about the focal point in the AR environment.

The lookAtPoint property is a valuable tool for developers implementing eye tracking features. It provides a dynamic and accurate representation of a user's gaze point in 3D space, opening up possibilities for creating immersive and responsive AR applications.

Save AR Functionality

public func save(_ uiView: ARView) {
    var modelEntityDetailsDict: [String: [String]] = [:]
    
    // Iterate through AR anchors
    if let arAnchors = uiView.session.currentFrame?.anchors {
        for arAnchor in arAnchors {
            if let modelName = arAnchor.name,
               let identifier = arAnchor.identifier {
                
                // Retrieve model entity and transform details
                if let anchorEntity = uiView.scene.findEntity(named: identifier) as? AnchorEntity,
                   let modelEntity = anchorEntity.children.first as? ModelEntity {
                    let transformDetails = [
                        modelEntity.transform.translation.debugDescription,
                        modelEntity.transform.rotation.debugDescription,
                        modelEntity.transform.scale.debugDescription
                    ]
                    modelEntityDetailsDict["\(modelName)@\(identifier.uuidString)"] = transformDetails
                }
            }
        }
    }
    
    // Save AR Anchors as WorldMap
    let anchorDataURL = self.worldMapDir.appendingPathComponent("ARExperience.map")
    uiView.session.getCurrentWorldMap { worldMap, error in
        guard let worldMap = worldMap else { return }
        do {
            let anchorData = try NSKeyedArchiver.archivedData(withRootObject: worldMap, requiringSecureCoding: true)
            try anchorData.write(to: anchorDataURL, options: [.atomic])
        } catch {
            fatalError("Debug: Can't save map: \(error.localizedDescription)")
        }
    }
    
    // Save modelEntity transforms as JSON
    let modelEntityDataURL = self.worldMapDir.appendingPathComponent("ARExperience.json")
    do {
        let jsonData = try JSONSerialization.data(withJSONObject: modelEntityDetailsDict, options: [])
        try jsonData.write(to: modelEntityDataURL)
    } catch {
        print("Debug: \(error.localizedDescription)")
    }
}

Summary: The save method is responsible for saving the AR anchors and model entity transforms for AR persistence. It iterates through the AR anchors, retrieves the model entity and its transform details, and stores them in a dictionary. Then, it saves the AR anchors as a WorldMap using the getCurrentWorldMap method. Finally, it saves the model entity transforms as JSON. This method allows the user to save the positions and transformations of virtual objects placed in the scene for future retrieval and AR experiences.

Detailed Explanation:

Let’s go through the save function step by step:

1. Create a dictionary modelEntityDetailsDict to store the details of the model entities' transformations. The keys of the dictionary will be a combination of the model name and the anchor identifier.
2. Iterate through the AR anchors present in the current AR frame.
3. For each anchor, check if it has a name and an identifier. If both are available, proceed to retrieve the model entity and its transformation details.
4. Using the anchor’s identifier, find the corresponding AnchorEntity in the AR scene using the findEntity(named:) method. Then, access the first child of the AnchorEntity, assuming it is the model entity.
5. Retrieve the translation, rotation, and scale values of the model entity’s transformation and store them as an array of strings in the transformDetails variable.
6. Add an entry to the modelEntityDetailsDict dictionary using the combination of the model name and the anchor identifier as the key and the transformDetails as the value.
7. Define the URL for storing the AR anchors as a World Map. The file will be named “ARExperience.map” and stored in the worldMapDir directory.
8. Call getCurrentWorldMap on the AR session to obtain the current world map.
9. Inside the completion block, check if the worldMap is not nil. If it exists, archive the world map data using NSKeyedArchiver and write it to the anchorDataURL.
10. If an error occurs during the archiving and writing process, handle it by reporting a fatal error.
11. Define the URL for storing the model entity transforms as JSON. The file will be named “ARExperience.json” and stored in the worldMapDir directory.
12. Serialize the modelEntityDetailsDict dictionary into JSON data using JSONSerialization.data(withJSONObject:options:).
13. Write the JSON data to the modelEntityDataURL.
14. If an error occurs during the serialization and writing process, print the error description for debugging purposes.

In summary, the save function iterates through the AR anchors in the current frame, retrieves the transformation details of the associated model entities, and stores both the AR anchors as a World Map and the model entity transforms as a JSON file. This enables saving the state of the AR experience, allowing it to be reloaded and restored at a later time with the same spatial alignment and placed virtual objects.

Load AR Functionality

public func load(_ uiView: ARView) {
    // Load AR Anchors
    let anchorDataURL = self.worldMapDir.appendingPathComponent("ARExperience.map")
    
    do {
        let anchorData = try Data(contentsOf: anchorDataURL)
        if let worldMap = try NSKeyedUnarchiver.unarchivedObject(ofClass: ARWorldMap.self, from: anchorData) {
            let configuration = ARWorldTrackingConfiguration()
            configuration.initialWorldMap = worldMap
            uiView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
        }
    } catch {
        print("Debug: Can't load map: \(error.localizedDescription)")
    }
    
    // Load modelEntity transforms
    let modelEntityDataURL = self.worldMapDir.appendingPathComponent("ARExperience.json")
    
    do {
        let jsonData = try Data(contentsOf: modelEntityDataURL)
        if let modelEntityDetailsDict = try JSONSerialization.jsonObject(with: jsonData, options: []) as? [String: [String]] {
            for (key, transformDetails) in modelEntityDetailsDict {
                let components = key.components(separatedBy: "@")
                guard components.count == 2 else { continue }
                
                let modelName = components[0]
                let identifier = components[1]
                
                let anchorEntity = AnchorEntity()
                anchorEntity.name = identifier
                uiView.scene.addAnchor(anchorEntity)
                
                let modelEntity = try ModelEntity.loadModel(named: modelName + ".usdz")
                modelEntity.transform.translation = parseVector3(transformDetails[0])
                modelEntity.transform.rotation = parseQuaternion(transformDetails[1])
                modelEntity.transform.scale = parseVector3(transformDetails[2])
                anchorEntity.addChild(modelEntity)
                
                for animation in modelEntity.availableAnimations {
                    modelEntity.playAnimation(animation.repeat())
                }
            }
        } else {
            print("Debug: Can't load map")
        }
    } catch {
        print("Debug: \(error.localizedDescription)")
    }
}

Summary: The load method in the AR persistence code loads saved AR anchors and model entity transforms. It retrieves the ARWorldMap from the saved data and uses it to initialize the ARWorldTrackingConfiguration, which is then used to run the AR session. It also retrieves the model entity transform details from the JSON data and creates corresponding anchor entities with model entities in the AR scene. The method enables animations for the loaded model entities.

Detailed Explanation:

Let’s break down the load functionality in detail:

1. Define the URL for the stored AR anchors as a World Map. The file is named “ARExperience.map” and located in the worldMapDir directory.
2. Attempt to read the data from the anchorDataURL using Data(contentsOf:).
3. If the data is successfully loaded, try to unarchive it using NSKeyedUnarchiver.unarchivedObject(ofClass:from:) as an instance of ARWorldMap.
4. If the unarchiving is successful and the worldMap exists, create a new ARWorldTrackingConfiguration and set its initialWorldMap property to the loaded worldMap.
5. Run the AR session with the new configuration, specifying options to reset the tracking and remove any existing anchors.
6. If an error occurs during the loading process, print the error description for debugging purposes.
7. Define the URL for the stored model entity transforms as a JSON file. The file is named “ARExperience.json” and located in the worldMapDir directory.
8. Attempt to read the data from the modelEntityDataURL using Data(contentsOf:).
9. If the data is successfully loaded, try to deserialize it using JSONSerialization.jsonObject(with:options:) as a dictionary with string keys and array values.
10. If the deserialization is successful and the modelEntityDetailsDict exists, iterate through its key-value pairs.
11. Split each key into components using the “@” separator. Ensure that there are exactly two components; otherwise, skip the current iteration.
12. Extract the model name and identifier from the components.
13. Create a new AnchorEntity and set its name to the identifier.
14. Add the AnchorEntity to the AR scene using uiView.scene.addAnchor(_:).
15. Load the model entity using ModelEntity.loadModel(named:), appending the model name with the ".usdz" extension.
16. Set the translation, rotation, and scale of the model entity’s transformation using the parsed vector and quaternion values from the transformDetails array.
17. Add the model entity as a child of the AnchorEntity.
18. Play all available animations of the model entity in a repeating manner.
19. If the JSON deserialization fails or the modelEntityDetailsDict does not exist, print a debug message indicating the failure.
20. If an error occurs during the loading process, print the error description for debugging purposes.

In summary, the load function loads the previously saved AR anchors from a World Map file and restores the corresponding model entity transforms from a JSON file. It sets up the AR session with the loaded world map, creates anchor entities, loads model entities, applies their transformations, and plays animations if available.

Share AR Experience

To share the AR experience with others, the generated .map and .json files can be shared. These files contain the necessary information to recreate the saved AR anchors and model entity transforms. By sharing these files, other users can load the same AR scene with the exact positions and transformations of virtual objects. This allows for a consistent and synchronized AR experience across multiple devices, enabling users to collaborate, explore, and interact with the same augmented environment. Whether it's for educational purposes, gaming, or remote assistance, sharing these files empowers users to share their AR creations and immerse others in their augmented reality world.

AR Persistence

AR Persistence refers to the ability of an augmented reality (AR) application to save and retrieve virtual content in real-world locations across multiple sessions. It allows users to place virtual objects, annotations, or effects in the physical environment and have them remain in their designated positions when they return to the app later. This means that the virtual content becomes persistent and retains its spatial context, enabling users to interact with and share their augmented reality experiences over time. AR Persistence leverages technologies such as ARKit and RealityKit to accurately track and anchor virtual content in the real world, providing a seamless and continuous AR experience.

ARWorldMap

The ARWorldMap in ARKit represents a serialized representation of the AR session’s spatial mapping and anchor data. It captures the geometric and semantic information of the real-world environment, including detected surfaces, images, and the positions and orientations of ARAnchors.

The WorldMap is used for persistence, allowing you to save the state of an AR session and later load it to restore the previously mapped environment and placed ARAnchors. By saving and loading WorldMaps, you can preserve the spatial understanding of the scene and the positions of virtual objects, enabling a consistent AR experience across different sessions or devices.

While the WorldMap itself does not directly store the specifics of model entities or their properties, it provides the foundation for reconstructing the AR scene and can serve as a basis for loading and placing virtual content based on the stored anchor data.

In summary, the WorldMap is a crucial component in ARKit and RealityKit that allows for persistence by capturing the spatial mapping and anchor data of an AR session, facilitating the restoration of the environment and virtual content placement.

ARPersistence Package

import SwiftUI
import RealityKit
import ARKit
import ARPersistence


struct ARViewContainer: UIViewRepresentable {
    @Binding var modelName: String?
    @Binding var isResetOn: Bool
    @Binding var isSaveOn: Bool
    @Binding var loadSaved: Bool
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: UIScreen.main.bounds)
        arView.debugOptions.insert([.showWorldOrigin])
        return arView
    }
    
    
    func setPropertiesToModelEntity(_ uiView: ARView, _ modelEntity: ModelEntity) {
        modelEntity.transform.translation.x = 0
        
        modelEntity.generateCollisionShapes(recursive: false)
        uiView.installGestures([.rotation, .translation, .scale], for: modelEntity)
    }
    
    
    func place3DModel(_ uiView: ARView) {
        
        guard let modelName = modelName else {
            return
        }
        
        // AR Anchor
        let arAnchor = ARAnchor(name: modelName, transform: matrix_identity_float4x4)
        uiView.session.add(anchor: arAnchor)
        
        
        // Anchor Entitty
        let anchorEntity = AnchorEntity(anchor: arAnchor)
        anchorEntity.name = arAnchor.identifier.uuidString      // MUST
        uiView.scene.addAnchor(anchorEntity)
        
        
        // Model Entity
        let modelEntity: ModelEntity = try! ModelEntity.loadModel(named: modelName + ".usdz")
        setPropertiesToModelEntity(uiView, modelEntity)
        anchorEntity.addChild(modelEntity)
    }

    
    func updateUIView(_ uiView: ARView, context: Context) {

        if(loadSaved){
            uiView.scene.anchors.removeAll()
            ARPersistence.shared.load(uiView)
            loadSaved.toggle()
            return
        }
        
        if(isSaveOn){
            ARPersistence.shared.save(uiView)
            isSaveOn.toggle()
            return
        }
        
        if(isResetOn){
            uiView.scene.anchors.removeAll()
            isResetOn.toggle()
            return
        }
        
        place3DModel(uiView)
    }
}

In summary, the code allows users to store AR anchors and model entity transformations in .map and .json files using ARPersistence package, enabling them to retrieve and load the saved state in future sessions. This provides persistence and continuity to the AR experience, allowing users to resume their augmented reality environment with the previously placed 3D models.

Explanation

Code demonstrates an AR experience using SwiftUI, RealityKit, ARKit, and ARPersistence. It allows users to place 3D models in the AR scene, reset the scene, save the current state, and load a previously saved state.

The ARViewContainer struct represents the AR view and conforms to UIViewRepresentable. It sets up the ARView and handles the placement of 3D models, resetting the scene, saving the state, and loading a saved state using the ARPersistence library.

The saveState function saves the current AR anchors and model entity transformations by calling the save method from the ARPersistence library, which stores the data in .map and .json files.

The loadState function retrieves the previously saved AR anchors and model entity transformations by calling the load method from the ARPersistence library, which loads the data from the .map and .json files and applies it to the AR scene.

The updateUIView function is responsible for the main functionality. It checks the state of the isSaveOn, isResetOn, and loadSaved flags to determine whether to save the state, reset the scene, or load a saved state. If none of these flags are active, it calls the place3DModel function to place the selected 3D model in the AR scene.

More Details about package — ARPersistence

Occlusion

Occlusion refers to the ability of virtual objects to hide behind real-world objects in the scene, as if they were behind a physical barrier. Occlusion helps to create a sense of depth and realism, and it requires precise tracking and depth sensing, as well as accurate geometry and lighting of the virtual content.

// occlusion
self.environment.sceneUnderstanding.options.insert(.occlusion) 

import SwiftUI
import RealityKit
import ARKit
import FocusEntity
import Combine

struct ContentView: View {
    
    var body: some View {
        ZStack(alignment: .bottom) {
            CustomARViewContainer()
            
            Button(action: {
                ActionManager.shared.actionStream.send(.place3DModel)
            }, label: {
                Text("Place 3D Model")
                    .font(.headline)
                    .foregroundColor(.white)
                    .padding()
                    .background(Color.blue)
                    .cornerRadius(10)
            })
            .padding(.bottom, 50)
        }
    }
}

struct CustomARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> CustomARView {
        return CustomARView()
    }
    
    func updateUIView(_ uiView: CustomARView, context: Context) {}
}



class CustomARView: ARView {
    
    var focusEntity: FocusEntity?
    var cancellables: Set<AnyCancellable> = []
    
    init() {
        super.init(frame: .zero)
        
        // ActionStrean
        subscribeToActionStream()
        
        // FocusEntity
        self.focusEntity = FocusEntity(on: self, style: .classic(color: .yellow))
        
        // Configuration
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal]
        config.environmentTexturing = .automatic
        
        if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) {
            config.sceneReconstruction = .meshWithClassification
        }
        
        // occlusion
        self.environment.sceneUnderstanding.options.insert(.occlusion)
        
        self.session.run(config)
    }

    
    func place3DModel() {
        guard let focusEntity = self.focusEntity else { return }

        let modelEntity = try! ModelEntity.load(named: "toy_car.usdz")
        let anchorEntity = AnchorEntity(world: focusEntity.position)
        anchorEntity.addChild(modelEntity)
        self.scene.addAnchor(anchorEntity)
    }
    
    
    func subscribeToActionStream() {
        ActionManager.shared
            .actionStream
            .sink { [weak self] action in
                
                switch action {
                    
                case .place3DModel:
                    self?.place3DModel()
                    
                case .remove3DModel:
                    print("Removeing 3D model: has not been implemented")
                }
            }
            .store(in: &cancellables)
    }
    
    @MainActor required dynamic init?(coder decoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
    
    @MainActor required dynamic init(frame frameRect: CGRect) {
        fatalError("init(frame:) has not been implemented")
    }
}

enum Actions {
    case place3DModel
    case remove3DModel
}

class ActionManager {
    static let shared = ActionManager()
    
    private init() { }
    
    var actionStream = PassthroughSubject<Actions, Never>()
}

Functionality: This code sets up an AR view using SwiftUI and RealityKit that allows the user to place a 3D model in their real-world environment. The CustomARView class is a subclass of ARView that sets up the AR session and adds a yellow focus indicator for the user to interact with. When the user taps the "Place 3D Model" button, it triggers the place3DModel() function which loads a 3D model of a toy car and places it in the real-world environment at the position of the focus indicator.

The occlusion feature is implemented by adding the .occlusion option to the sceneUnderstanding.options of the CustomARView. This tells the ARKit to recognize and track the depth of the real-world objects and use that information to hide the virtual objects. The end result is a more realistic augmented reality experience where virtual objects interact seamlessly with the real world.

The ActionManager class is used to manage the user's actions and communicates them between different parts of the code. It uses a PassthroughSubject to send and receive actions. The subscribeToActionStream() function is called to listen for new actions being sent and performs the appropriate action, either placing or removing a 3D model.

Material

Material refers to the appearance and properties of the virtual objects, such as their color, texture, reflectivity, and transparency. Material is crucial for making the virtual content blend seamlessly with the real-world environment, and it can also convey meaning and emotion through visual cues.

    func place3DModel() {
        guard let focusEntity = self.focusEntity else { return }

        let modelEntity = try! ModelEntity.load(named: "toy_drummer_idle.usdz")
        
        // adding material
        var metal = SimpleMaterial(color: .clear, isMetallic: true)
        metal.metallic = 1.0
        modelEntity.model?.materials.append(metal)

        let anchorEntity = AnchorEntity(world: focusEntity.position)
        anchorEntity.addChild(modelEntity)
        self.scene.addAnchor(anchorEntity)
    }

Let’s get started with placing a 3D object on a horizontal or vertical plane using ARKit. With ARKit, we can detect the plane and add different user interactions to the 3D object. We can also explore the 3D model properties like animations or audio/video associated with them.

Plane Detection

When we use an AR app, we want to make sure that the virtual objects we place in the real world stay in place and move realistically as we move around them. The way we do this is by using something called “world tracking”.

World tracking configuration is like a tool that helps our app to understand and interact with the world around it. One of the things it can do is to detect flat surfaces in the real world, like a table or a floor. It does this by using the camera on our device to create a map of the area and then looking for places where the map is flat.

Once it detects a flat surface, it can add a virtual object on top of it, making it look like the object is sitting right there in the real world

ARWorldTrackingConfiguration is a class provided by ARKit framework that specifies the configuration and options for an AR session that uses the device’s motion and camera to track the device’s position and orientation relative to the world around it. It allows for high-quality AR experiences, as it uses 6DoF (six degrees of freedom) tracking to track the device’s position and orientation in 3D space.

import SwiftUI
import RealityKit
import ARKit

struct ContentView : View {
    
    var body: some View {
        CustomARViewContainer()
    }
}

struct CustomARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> CustomARView {
        return CustomARView()
    }
    
    func updateUIView(_ uiView: CustomARView, context: Context) { }
}


class CustomARView: ARView {
    
    init() {
        super.init(frame: .zero)

        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal]
        config.environmentTexturing = .automatic

        if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) {
            config.sceneReconstruction = .meshWithClassification
        }

        self.session.run(config)
    }
}

In this part of the code, we create a configuration object for ARKit’s world tracking system. The ARWorldTrackingConfiguration is a class that encapsulates various settings for ARKit's tracking capabilities, such as the types of features to detect and track, the environment texture mapping, and scene reconstruction.

Here, we set the planeDetection property to [.horizontal], which means that we want ARKit to detect horizontal planes such as tables, floors, and other flat surfaces in the environment. The environmentTexturing property is set to .automatic, which means that ARKit will automatically create an environment map to blend virtual objects with the real-world environment.

Next, we check if the device supports scene reconstruction with the depth information from a LiDAR sensor. The LiDAR sensor can measure the distance between the device and objects in the environment with high accuracy and speed, allowing ARKit to reconstruct a detailed 3D mesh of the environment. If the device supports this feature, we set the sceneReconstruction property of the configuration to .meshWithClassification, which means that ARKit will generate a mesh with classification information, allowing us to place virtual objects on surfaces with more accuracy.

It’s worth noting that not all iPhones have LiDAR sensors, which means that some devices may not support the scene reconstruction feature with depth information. In such cases, ARKit will still work but with lower accuracy and precision.

FocusEntity

Once we detect a plane in the AR scene, the FocusEntity helps to guide the user’s attention to the detected plane by displaying a visual cue, such as a crosshair or a circle, on the detected plane. This visual cue helps to inform the user that they can interact with the AR content on the detected plane.

Additionally, the FocusEntity provides functionality for adding AR content onto the detected plane. For example, if we want to place a 3D model onto the detected plane, we can use the FocusEntity to ensure that the 3D model is placed correctly on the plane and also provide visual feedback to the user as they move the 3D model around on the plane.

Here are the steps you can follow to ensure that the FocusEntity package is properly integrated in your Xcode project:

1. Add FocusEntity package to project. To do this, go to File -> Swift Packages -> Add Package Dependency and enter the package repository URL: https://github.com/maxxfrazer/FocusEntity.
2. Check if the package is listed under your project’s dependencies in the Project navigator. If it’s not there, try adding it again.

import SwiftUI
import RealityKit
import ARKit
import FocusEntity

struct ContentView : View {
    
    var body: some View {
        CustomARViewContainer()
    }
}

struct CustomARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> CustomARView {
        return CustomARView()
    }
    
    func updateUIView(_ uiView: CustomARView, context: Context) { }
}


class CustomARView: ARView {
    
    var focusEntity: FocusEntity?
    
    init() {
        super.init(frame: .zero)
        
        self.setUpFocusEntity()
        self.setUpARView()
    }
    
    func setUpFocusEntity() {
        self.focusEntity = FocusEntity(on: self, style: .classic(color: .yellow))
    }
    
    func setUpARView() {
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal]
        config.environmentTexturing = .automatic
        
        if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) {
            config.sceneReconstruction = .meshWithClassification
        }

        self.session.run(config)
    }
}

The FocusEntity package provides a lot of customization options that can be used to change the appearance and behavior of the focus square in your AR app. For example, you can change the color of the focus square, adjust its size and animation, and even add 3D objects to it. More Details - GitHub

Placing 3D model on plane

import SwiftUI
import RealityKit
import ARKit
import FocusEntity
import Combine

struct ContentView: View {
    
    var body: some View {
        ZStack(alignment: .bottom) {
            CustomARViewContainer()
            
            Button(action: {
                ActionManager.shared.actionStream.send(.place3DModel)
            }, label: {
                Text("Place 3D Model")
                    .font(.headline)
                    .foregroundColor(.white)
                    .padding()
                    .background(Color.blue)
                    .cornerRadius(10)
            })
            .padding(.bottom, 50)
        }
    }
}

struct CustomARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> CustomARView {
        return CustomARView()
    }
    
    func updateUIView(_ uiView: CustomARView, context: Context) {}
}



class CustomARView: ARView {
    
    var focusEntity: FocusEntity?
    var cancellables: Set<AnyCancellable> = []
    
    init() {
        super.init(frame: .zero)
        
        // ActionStrean
        subscribeToActionStream()
        
        // FocusEntity
        self.focusEntity = FocusEntity(on: self, style: .classic(color: .yellow))
        
        // Configuration
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal]
        config.environmentTexturing = .automatic
        
        if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) {
            config.sceneReconstruction = .meshWithClassification
        }
        
        self.session.run(config)
    }

    
    func place3DModel() {
        guard let focusEntity = self.focusEntity else { return }

        let modelEntity = try! ModelEntity.load(named: "toy_car.usdz")
        let anchorEntity = AnchorEntity(world: focusEntity.position)
        anchorEntity.addChild(modelEntity)
        self.scene.addAnchor(anchorEntity)
    }
    
    
    func subscribeToActionStream() {
        ActionManager.shared
            .actionStream
            .sink { [weak self] action in
                
                switch action {
                    
                case .place3DModel:
                    self?.place3DModel()
                    
                case .remove3DModel:
                    print("Removeing 3D model: has not been implemented")
                }
            }
            .store(in: &cancellables)
    }
    
    @MainActor required dynamic init?(coder decoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
    
    @MainActor required dynamic init(frame frameRect: CGRect) {
        fatalError("init(frame:) has not been implemented")
    }
}

enum Actions {
    case place3DModel
    case remove3DModel
}

class ActionManager {
    static let shared = ActionManager()
    
    private init() { }
    
    var actionStream = PassthroughSubject<Actions, Never>()
}

Functionality: When the user taps on the Button, the place3DModel function is called and a ModelEntity of a toy car is loaded and placed at the position of the FocusEntity.

You can ignore the functionality of ActionManager it simply let the button action call place3DModel directly.

Tap Gesture Code Explanation:

This code defines a SwiftUI ContentView that contains an AR view and a button labeled “Place 3D Model”. Tapping the button sends a message to the ActionManager’s shared PassthroughSubject, which is subscribed to by the CustomARView. When the action stream sends the “place3DModel” action, the CustomARView creates a ModelEntity from a 3D model file, creates an AnchorEntity at the position of the FocusEntity (a component for ARKit that helps with positioning objects in AR), adds the ModelEntity as a child of the AnchorEntity, and adds the AnchorEntity to the scene. The CustomARView also sets up an AR session with plane detection and environment texturing, and subscribes to the action stream to handle the “place3DModel” action.

AR gestures

AR gestures are interactions and movements made by users in an Augmented Reality (AR) environment to manipulate AR objects or interact with AR content.

In the context of AR with iOS, AR gestures can be used to enable users to interact with virtual objects overlaid on the real world through intuitive gestures.

Some common AR gestures

1. Tap Gesture: Users can tap on virtual objects to select or interact with them. For example, tapping on a virtual button to trigger an action, or tapping on a virtual object to rotate or scale it.
2. Pan Gesture: Users can use a pan gesture to move virtual objects in the AR environment. For example, dragging a virtual object to reposition it in the real world.
3. Pinch Gesture: Users can use a pinch gesture to scale virtual objects. For example, pinching on a virtual object to resize it or zoom in/out.
4. Rotation Gesture: Users can use a rotation gesture to rotate virtual objects. For example, rotating a virtual object with two fingers to change its orientation or alignment.
5. Long-Press Gesture: Users can use a long-press gesture to trigger actions or interactions with virtual objects. For example, long-pressing on a virtual button to reveal additional options or information.
6. Swipe Gesture: Users can use swipe gestures to trigger specific actions or interactions with virtual objects. For example, swiping left or right on a virtual object to get different colour variant of virtual object.

The implementation of AR gestures may require handling touch events, gesture recognizers, or other APIs provided by ARKit and RealityKit frameworks

AR gestures on ARViewContainer

import SwiftUI
import RealityKit

struct ContentView : View {
    
    var body: some View {
        ARViewContainer()
            .gesture(TapGesture().onEnded {
                // Handle tap gesture here
                print("Tap gesture detected")
            })
            .gesture(DragGesture().onChanged { value in
                // Handle drag gesture here
                print("Drag gesture detected: \(value.translation)")
            })
            .gesture(MagnificationGesture().onChanged { value in
                // Handle pinch gesture here
                print("Pinch gesture detected: \(value.magnitude)")
            })
            .gesture(RotationGesture().onChanged { value in
                // Handle rotation gesture here
                print("Rotation gesture detected: \(value.degrees)")
            })
    }
}

struct ARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        
        // Load the model from the app's asset catalog
        let modelEntity = try! ModelEntity.load(named: "toy_car.usdz")
        
        // Create an anchor entity and add the model to it
        let anchorEntity = AnchorEntity()
        anchorEntity.addChild(modelEntity)
        
        // Set the position of the anchor entity to 1 meter in front of the camera
        anchorEntity.position = [0, 0, -1]
        
        // Add the anchor entity to the scene
        arView.scene.addAnchor(anchorEntity)
        
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) { }
}

Code Explanation

The code provided demonstrates how to implement AR gestures (tap, drag, pinch, and rotation) on an AR view in SwiftUI using RealityKit.

1. ARViewContainer is a UIViewRepresentable that creates an ARView and loads a 3D model (in this case, "toy_car.usdz") from the app's asset catalog. It creates an anchor entity, adds the model as its child, sets the position of the anchor entity to 1 meter in front of the camera, and adds the anchor entity to the AR scene.
2. In ContentView, an instance of ARViewContainer is added as a subview. Then, gesture recognizers (TapGesture, DragGesture, MagnificationGesture, and RotationGesture) are attached to the AR view using SwiftUI's .gesture() modifier.
3. The .onEnded and .onChanged closures within each gesture recognizer handle the corresponding gestures. For example, when a tap gesture is detected, it prints a message "Tap gesture detected". When a drag gesture is detected, it prints the translation of the drag. When a pinch gesture is detected, it prints the magnitude of the pinch. When a rotation gesture is detected, it prints the degrees of rotation.

Issue with gestures on ARViewContainer

.gesture(DragGesture().onChanged { value in
    // Update the position of the modelEntity based on the translation value
    modelEntity.position += SIMD3<Float>(value.translation.width, value.translation.height, 0)
})

This code adds a DragGesture to an ARViewContainer screen in a SwiftUI app. The position of a modelEntity is updated based on the horizontal and vertical translation values of the drag gesture. The modelEntity.position is updated by adding the translation values to its current position using the += operator, resulting in interactive manipulation of the model's position in the scene based on the user's pan gesture.

The issue is that if you drag on any part of the ARViewContainer where the modelEntity is not even available, the modelEntity will be updated, and the position of modelEntity will change accordingly.

What will happen if we get multiple modelEntities placed in ARView? gestures on modelEntity

Gestures on modelEntity

This allows the user to interact with the model by performing gestures such as rotation, translation (pan), and scaling (pinch) using touch or other input methods.

import SwiftUI
import RealityKit


struct ContentView : View {
    
    var body: some View {
        ARViewContainer()
    }
}

struct ARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        
        // Load the model from the app's asset catalog
        let modelEntity = try! ModelEntity.load(named: "toy_car.usdz")
        
        // Create an anchor entity and add the model to it
        let anchorEntity = AnchorEntity()
        anchorEntity.addChild(modelEntity)
        
        // Set the position of the anchor entity to 1 meter in front of the camera
        anchorEntity.position = [0, 0, -1]
        
        // Add the anchor entity to the scene
        arView.scene.addAnchor(anchorEntity)
        
        // Install gestures on the modelEntity
        arView.installGestures([.rotation, .translation, .scale], for: modelEntity as! HasCollision)
        
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) { }
}

Overall, this code sets up an AR scene with a 3D model of a toy car and enables user interaction with the model through gestures in the ARView.

With Gestures on modelEntity, even if you place multiple 3D models, the user interaction with the models will be smooth.

Custom Gestures within CustomARView

[Note: same can be achieved with AR gestures on ARViewContainer]

import SwiftUI
import RealityKit

struct ContentView : View {
    
    var body: some View {
        CustomARViewContainer()
    }
}

struct CustomARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> CustomARView {
        return CustomARView()
    }
    
    func updateUIView(_ uiView: CustomARView, context: Context) { }
}


class CustomARView: ARView {
    
    init() {
        super.init(frame: .zero)

        let modelEntity = try! ModelEntity.loadModel(named: "toy_car.usdz")

        let anchorEntity = AnchorEntity()
        anchorEntity.addChild(modelEntity)
        
        anchorEntity.position = [0, 0, -1]

        self.scene.addAnchor(anchorEntity)
        
        self.installGestures([.rotation, .translation, .scale], for: modelEntity as! HasCollision)
        
        let longpress = UILongPressGestureRecognizer(target: self, action: #selector(self.handleLongPress(_:)))
        self.addGestureRecognizer(longpress)
    }
    
    
    @objc func handleLongPress(_ recognizer: UITapGestureRecognizer? = nil) {

        guard let touchInView = recognizer?.location(in: self) else {
            return
        }

        guard let modelEntity = self.entity(at: touchInView) as? ModelEntity else {
            print("modelEntity not found")
            return
        }
        
        print("Long press detected on - \(modelEntity.name)")
        
    }
}

Explanation : In this code, the long press gesture is set up to recognize when the user long presses on a 3D model loaded into the ARView. When a long press is detected, the code uses the entity(at: ) method to get the 3D entity (ModelEntity) at the location where the user long pressed on the screen. If the entity exists, the code prints out the name of the 3D model that was long pressed.

This code is useful when you want to perform an action when the user interacts with a 3D object in your AR app. For example, you could use this code to detect when the user selects a 3D object and display additional information or trigger an animation.

3D USDZ Models

what is usdz : USDZ is a file format developed by Pixar and Apple for sharing and displaying 3D content in AR and VR environments. It’s based on Pixar’s Universal Scene Description (USD), and is designed to be lightweight and efficient. USDZ files can contain 3D models, textures, animations, and other assets needed to render a complete 3D scene.

ARKit is a framework developed by Apple for building AR apps on iOS devices. While USDZ is the recommended format for creating and sharing 3D models in ARKit, the framework also supports other common 3D file formats such as OBJ, Collada (DAE), and Alembic (ABC).

Using USDZ format can have advantages over other formats, such as improved performance, better support for materials and lighting, and easier integration with Apple’s Quick Look and Reality Composer tools. Additionally, USDZ models can be easily previewed in AR using the Quick Look feature on iOS devices.

Some Resources for free USDZ models

1. Apple’s USDZ Sample Gallery: https://developer.apple.com/augmented-reality/quick-look/
2. Sketchfab: https://sketchfab.com/
3. TurboSquid: https://www.turbosquid.com/
4. CGTrader: https://www.cgtrader.com/
5. Free3D: https://free3d.com/
6. Or one can convert other 3D model formats to USDZ using Reality Converter

Adding 3D Model in project

1. First create Assets folder in project
2. Download add few USDZ models from Apple’s official site or any other source
3. Add these USDZ models in Assets with “File” > “Add Files to …” > select USDZ files > “Add”
4. Similarly, we can add images in Assests.xcassets as icons for these USDZ models

Few AR Concepts

• ARViewContainer: This is a UIViewRepresentable that wraps the ARView from the RealityKit framework. The ARView is the main view for displaying augmented reality content, and the ARViewContainer is responsible for managing the creation and updating of the view.
• ModelEntity: This is a type from the RealityKit framework that represents a 3D model that can be added to the augmented reality scene. It is loaded from a file using the load(named:) method.
• AnchorEntity: This is a type from the RealityKit framework that represents a position and orientation in the augmented reality scene. A ModelEntity can be added as a child of an AnchorEntity to place it in the scene. The AnchorEntity is added to the ARView's scene using the addAnchor(_:) method.

Code Explanation

overview

When the user selects a model, the corresponding 3D model is loaded using ModelEntity and added to an AnchorEntity. The AnchorEntity is then positioned 1 meter in front of the camera and added to the AR view scene.

import SwiftUI
import RealityKit

struct ContentView : View {
    
    let modelNames = ["toy_car", "robot_walk_idle", "toy_biplane_idle", "toy_drummer_idle"]
    
    // State variable for selected model index
    @State private var selectedModelIndex = 0
    
    var body: some View {
        VStack {
            // AR view container with selected model
            ARViewContainer(modelName: modelNames[selectedModelIndex])
                .frame(height: 400)
            
            // Picker for selecting the model
            Picker("Select Model", selection: $selectedModelIndex) {
                ForEach(0 ..< modelNames.count) { index in
                    Image(modelNames[index])
                        .resizable()
                        .frame(width: 50, height: 50)
                        .tag(index)
                }
            }
            .pickerStyle(SegmentedPickerStyle())
        }
    }
}

struct ARViewContainer: UIViewRepresentable {
    let modelName: String
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {
        // Remove any existing anchor entities from the scene
        uiView.scene.anchors.removeAll()
        
        // Load the model from the app's asset catalog
        let modelEntity = try! ModelEntity.load(named: modelName + ".usdz")
        
        // Create an anchor entity and add the model to it
        let anchorEntity = AnchorEntity()
        anchorEntity.addChild(modelEntity)
        
        // Set the position of the anchor entity to 1 meter in front of the camera
        anchorEntity.position = [0, 0, -1]
        
        // Add the anchor entity to the scene
        uiView.scene.addAnchor(anchorEntity)
    }
}

The ARView is initially empty when the app is launched, and the user needs to select a model from the Picker for it to be displayed in the ARView. The selected model is displayed at the position of the AnchorEntity which is 1 meter in front of the camera, and the user can move the camera to view the model from different angles and distances.

The ARViewContainer struct creates an instance of ARView, which is a UIView subclass provided by RealityKit. The makeUIView method returns the ARView instance when the view is created, and the updateUIView method is called to update the view. This method removes any existing anchor entities from the scene, loads the model from the app’s asset catalog, creates an anchor entity, adds the model to the anchor entity, sets the position of the anchor entity to 1 meter in front of the camera, and adds the anchor entity to the scene.

Augmented Reality (AR) is a technology that blends virtual objects and information with the real-world environment, enhancing the user experience. With the launch of ARKit and RealityKit, Apple has made it easier for iOS developers to create AR apps that can transform how we interact with the world. SwiftUI, Apple’s modern declarative UI framework, can be used to build the user interface for AR apps, making it easier to create engaging AR experiences. In this article, we will explore how to use SwiftUI, ARKit, and RealityKit to create AR apps for iOS devices.

Initial Setup

1. Open Xcode: Launch Xcode from the Applications folder or by searching for it in Spotlight.
2. Create a new project: Click on “Create a new Xcode project” on the welcome screen or go to “File” > “New” > “Project” from the menu bar.
3. Choose a template: Choose the template that best matches your project needs, such as “iOS App” > “Augmented Reality App” > Next
4. Configure the project: Provide a name for your project, choose a team, and select a location to save the project.

Default Code Explanation

import SwiftUI
import RealityKit

struct ContentView : View {
    var body: some View {
        ARViewContainer().edgesIgnoringSafeArea(.all)
    }
}

struct ARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> ARView {
        
        let arView = ARView(frame: .zero)
        
        // Load the "Box" scene from the "Experience" Reality File
        let boxAnchor = try! Experience.loadBox()
        
        // Add the box anchor to the scene
        arView.scene.anchors.append(boxAnchor)
        
        return arView
        
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {}
    
}

#if DEBUG
struct ContentView_Previews : PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}
#endif

ARViewContainer is a UIViewRepresentable struct that is used to create and manage an ARView instance. UIViewRepresentable is a protocol that allows SwiftUI to interoperate with UIKit views and controls. ARViewContainer acts as a bridge between the SwiftUI framework and the RealityKit framework, allowing you to create 3D AR experiences in your SwiftUI app.

makeUIView is a required method of the UIViewRepresentable protocol. It's called when the SwiftUI view is being created for the first time. In this method, you create and configure your ARView instance, load any 3D models or assets you need, and add them to the scene. Finally, you return the ARView instance you created.

updateUIView is another required method of the UIViewRepresentable protocol. It's called whenever the SwiftUI view needs to be updated based on changes to its state or environment. In this method, you update the ARView instance to reflect any changes in the state of your SwiftUI view.

More about updateUIView

struct ARViewContainer: UIViewRepresentable {
    @Binding var isModelDisplayed: Bool
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {
        if isModelDisplayed {
            // Load and add the 3D model to the AR scene
            let boxAnchor = try! Experience.loadBox()
        
            // Add the box anchor to the scene
            uiView.scene.anchors.append(boxAnchor)
        } else {
            // Remove the 3D model from the AR scene
            uiView.scene.anchors.removeAll()
        }
    }
}

In this example, updateUIView is used to add or remove a 3D model from the AR scene based on the value of the isModelDisplayed Boolean state variable. If the value of isModelDisplayed is true, the method loads and adds a 3D box model to the AR scene. Otherwise, it removes all anchors (including the 3D model) from the AR scene.

This method is called whenever the value of the isModelDisplayed state variable is changed, and it allows the ARView instance to be updated accordingly. However, if you don’t do anything in the updateUIView method to respond to changes in that state variable, the ARViewContainer view won't be affected.

Augmented Reality (AR)

AR (Augmented Reality) overlays virtual content on the real world, allowing users to interact with virtual objects in their real environment.

Examples

1. Snapchat filters
2. Ikea Place
3. Pokémon GO
4. Lamborghini

Virtual Reality (VR)

VR (Virtual Reality) creates a fully immersive digital environment, where users are transported to a virtual world and can interact with objects and environments that are entirely computer-generated.

Examples

1. Oculus Rift
2. PlayStation VR

Mixed Reality (MR)

MR (Mixed Reality) combines elements of both AR and VR, allowing virtual objects to interact with and appear as if they are part of the real world. Users can interact with virtual objects while still being aware of and able to see their real environment.

Examples

1. Microsoft HoloLens
2. Magic Leap

Difference between AR & MR

The main difference between AR (Augmented Reality) and MR (Mixed Reality) lies in the level of virtual object integration with the real world.

In AR, virtual content is overlaid onto the real world, allowing users to see and interact with virtual objects in their real environment. However, the virtual objects do not typically interact with the real world in a physically realistic way. They may appear as if they are floating or hovering above the real world, but they do not have a tangible connection to the physical environment.

On the other hand, in MR, virtual objects are integrated into the real world in a more realistic and seamless manner. They can interact with the real world objects and environments in a way that appears natural, creating a more immersive experience. MR often involves real-time tracking and mapping of the physical environment, allowing virtual objects to respond to and interact with the real world in a dynamic and interactive way.

In summary, while both AR and MR involve overlaying virtual content onto the real world, MR takes it a step further by integrating virtual objects into the real world in a more realistic and interactive manner.

Example, Lamborghini VS Microsoft HoloLens