Building Drone Detection Application For Quest 3

In this post, I will guide you how to create a real-time drone detection system using YOLO11. I will share code examples and walk you through the process step-by-step: training the model in Python, saving it as an ONNX file, and bringing it into Unity 6 and Sentis for the Meta Quest 3.

Workflow

First Test of Quest 3 Drone Detection

The workflow for this project follows a seamless path from data to edge deployment. First, I trained a YOLO11 model on a custom drone dataset for 100 epochs using Python, which generated the optimal best.pt weights file and converted best.pt to ONNX file for Unity Inference Engine use. Finally, I compiled the Unity project into an Android APK and deployed it directly onto the Meta Quest 3.

Python: Training the Model and Exporting to ONNX

The Meta Quest 3 has limited GPU and processor power. A heavy model would lag or crash the headset. I chose YOLO11 nano because it is lightweight and provides the best balance of speed and accuracy for mobile hardware. I used a single-class dataset focused only on drones. It contains 7260 images labeled to train the model specifically for this task.

Dataset Link: Military Drone Dataset on Roboflow

We will create two code snippets: one for the training process and another for converting the best.pt file to ONNX.

Get Özgür SANLI’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In this step, I trained the model for 100 epochs using my local GPU to ensure high performance. One of the biggest advantages of using YOLO is the .yaml configuration format; it automatically handles the organization of training, validation, and testing image sets, making the data management process seamless.

from ultralytics import YOLO

# Load the lightweight YOLO11 nano model
model = YOLO("yolo11n.pt")

# Define the path to my data configuration file
yaml_path = "data.yaml"

# Start the training process
results = model.train(
    data=yaml_path,
    epochs=100,       # Optimized for detection without overfitting
    patience=15,     # Early stopping if no improvement
    batch=16,        # Adjust based on my GPU memory
    imgsz=640,       # Standard resolution for YOLO
    device=0,        # Use GPU (CUDA)
    workers=2        # Number of parallel threads
)

The training process took approximately 73 minutes, reaching a high mAP50 accuracy of 96.4%

Result of Training

Using the code below, we converted the best.pt file into the ONNX format required by Unity. This step is essential because it allows the Unity Inference Engine to process the model locally.

from ultralytics import YOLO

# Load your trained model weights
model = YOLO("best.pt")

# Export to ONNX format with optimization
model.export(
    format="onnx",
    dynamic=False,
        imgsz=640,
        half=False,
        simplify=True
    )

    print("Model successfully converted to ONNX format!")

Unity Integration & Meta Quest 3 Setup

Now that our ONNX model is ready, it’s time to bring it into the VR world. Setting up the environment correctly is crucial for performance and compatibility.

1. Environment Setup

I started by creating a new project using the 3D Sample template. To target the headset, I switched the Build Profile to Meta Quest. This ensures that the engine optimizes the rendering for mobile VR hardware.

Switch Profile to Meta Quest

2. Installing the SDKs

I integrated the Meta XR SDK All-In-One package. For the model to run, I also added the Meta XR Inference and Object Detection tools from the Meta XR Tools menu. These packages provide the necessary libraries to handle neural networks and passthrough-based detection.

Meta XR All-in-One SDK

3. Resolving Plugin Conflicts

A common issue during setup is the conflict between the legacy Oculus plugin and the modern XR Plugin Management. To ensure a stable build, I removed the Oculus plugin and focused entirely on the XR SDK workflow. This step is vital to prevent crashes and deployment errors on the Quest 3.

Select Meta XR Feature Group

4. Meta Toolkit and Building Blocks

We installed the Meta XR SDK, which automatically added the Meta Toolkit menu to Unity. This toolkit provides a specialized interface called Building Blocks, designed to make VR development modular and straightforward.

Meta XR Tools

Select Object Detection

Installing the Object Detection block also sets up the Camera Rig and Passthrough building blocks. Here is the final hierarchy:

Final Hierarchy

5. Deploying the ONNX File

Load the ONNX file as shown below. You must click “Optimize and Save Model” to calibrate it. Once the “__converted” file is generated, select it for the Model Asset option in the inspector. This is a critical step!

Click On UnityInferenceEngine in Object Detection

Connect the Quest 3 to your PC and select “Build and Run”. This generates the APK file and automatically installs the project onto the headset for testing.

The final APK size is 115 MB, and the ONNX model is only 12 MB. This shows how lightweight and efficient the system is for mobile VR hardware.

Note: Don’t forget to enable “Unknown Sources” in your Meta Quest 3 settings to run the APK successfully.

Conclusion

In this project, we successfully bridged the gap between a theoretical Python model and the latest edge technology using Unity. It was an incredibly fun and rewarding experience to see a concept come to life in a real-world application. Deploying a custom-trained model onto a powerful device like the Meta Quest 3 truly demonstrates the potential of combining AI with immersive VR technology.