Data annotation interview prep

nuPlan data is a massive real-world driving dataset and closed-loop planning benchmark

3D Gaussian Splatting is a modern digital graphics technique that rebuilds real-world scenes into photorealistic 3D environments from a set of regular photos or videos.

Closed-loop simulation is a testing environment where the autonomous vehicle’s (the “ego” vehicle) actions actively change the simulation’s state. Every move the vehicle makes generates a feedback loop that alters the next step of the simulation.

When testing an autonomous vehicle planner using nuPlan’s simulation framework, the surrounding agents (non-ego vehicles) can be set to different behaviors:

• Non-Reactive Agents: The autonomous car can deviate from its original path, but the background traffic strictly repeats the pre-recorded logs.
• Reactive Agents (IDM): Surrounding cars respond directly to the autonomous car using rule-based formulas like the Intelligent Driver Model (IDM) to slow down, brake, or change lanes to avoid it.
• Learned Reactive Agents: Advanced variations utilize machine learning models to simulate highly unpredictable, human-like reactions and adversarial maneuvers from surrounding drivers.
• Supervised Fine-Tuning (SFT): The baseline method used to train driving models (like Vision-Language-Action or VLA models) via imitation learning. The model clones human expert behavior by trying to replicate the exact coordinates of a recorded driving trajectory.
• Open-Loop PDMS (PDM Score): The specialized, safety-aware metric formula popularized by frameworks like NAVSIM and nuPlan. Instead of measuring simple coordinate deviation, PDMS runs a rule-based check on a predicted trajectory to evaluate Safety, Comfort, Progress, and Drivable Area Compliance.

closed-loop SR: It measures the exact percentage of long-tail, high-risk, or safety-critical driving scenarios that an autonomous vehicle model successfully completes without crashing or breaking major traffic rules in a fully interactive simulator.

Moving from static datasets to generative loops is the defining shift in modern autonomous vehicle (AV) training. It represents the evolution from passive learning (watching recorded videos) to active learning (practicing in an interactive, AI-generated sandbox)

In autonomous vehicle research, SR (Success Rate) and PDMS (PDM Score / Planning Domain Metric Score) are the two most important metrics used to grade how safely and effectively an AI driver navigates a simulation.

Each tool addresses a different engineering challenge when training or testing an AI driver:

1. Hydra: The Configuration Manager

Hydra (created by Meta AI) is an open-source Python framework that elegantly handles complex configurations.

• The Problem: An autonomous vehicle project has thousands of variables (e.g., choice of machine learning model, simulation type, city map, weather conditions, metric weights like safety vs. progress). Manually editing files or writing giant command-line arguments becomes a nightmare.
• What it does: Hydra lets developers dynamically compose configuration files using a hierarchical structure. With a single command-line line, a researcher can switch their code from training on Boston data to testing on Singapore data without changing a line of code.

2. Ray: The Distributed Computing Engine

Ray (created by Anyscale) is a powerful library for scaling Python applications and machine learning across multiple computers (clusters).

• The Problem: Training an end-to-end driving model or running millions of closed-loop simulation scenarios requires massive computing power. A single computer would take months to process the data.
• What it does: Ray takes the simulation workload and automatically splits it across dozens or hundreds of cloud servers. In nuPlan or NAVSIM, Ray distributes individual driving scenarios to different GPUs, running hundreds of virtual simulations in parallel and compiling the final PDM Scores instantly.

3. Rendering: The Visual Simulator

Get Ronel Solomon’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Rendering is the process of translating raw 3D data (bounding boxes, lidar points, map lanes, and vehicle trajectories) into a visual format that humans or AI models can see.

• The Problem: Driving logs are just lists of numbers, coordinates, and vectors. To evaluate a model visually, or to train a vision-based AI (which takes camera pixel data as input), those numbers must be turned into an image.
• What it does: In benchmarks like nuPlan, the rendering engine creates bird’s-eye-view (BEV) animations showing the ego vehicle navigating around other actors. In advanced “generative loop” setups, rendering uses neural graphics techniques such as NeRFs to generate photorealistic synthetic camera views from any perspective in real time.

1. MMDetection3D: The Foundational Infrastructure

MMDetection3D (part of the OpenMMLab ecosystem) is an open-source, highly versatile 3D object detection framework.

• What it does: It provides pre-built machine learning building blocks (backbones, necks, heads) to process raw data like camera images and LiDAR point clouds. It converts these inputs into localized 3D bounding boxes around obstacles like cars, pedestrians, and cyclists.
• How it’s used: It acts as the “eyes” or base perception engine. Advanced end-to-end frameworks use MMDetection3D to extract Bird’s-Eye-View (BEV) features before they can start planning trajectories.

2. UniAD (Unified Autonomous Driving): The Milestone End-to-End Pioneer

UniAD (CVPR 2023 Best Paper) was the first major framework to tightly bind full-stack driving sub-tasks into a single, unified Transformer network.

• The Breakthrough: Before UniAD, separate systems handled tracking, lane mapping, and predicting where other cars would go. If the tracking system made a tiny mistake, that error snowballed and ruined the final driving plan.
• How it works: UniAD organizes tasks sequentially inside one model. The tracking module passes raw tokens to the mapping module, which feeds the trajectory predictor, which immediately tells the planner how to steer. By sharing mathematical “queries,” the whole system prioritizes safe planning above all else.

3. VADv2 (Vectorized Autonomous Driving v2): The Fast, Probabilistic Planner

VADv2 (ICLR 2026) evolved autonomous driving by throwing away heavy, pixel-dense mapping imagery in favor of pure vectorized geometry and probabilistic planning.

• The Breakthrough: Real driving is inherently uncertain. Traditional deterministic models try to guess exactly one perfect path, which can cause the AI to panic or freeze when a human behaves unpredictably. Furthermore, processing massive image grids makes them slow.
• How it works: VADv2 models the driving policy as a broad probability distribution. It converts maps and traffic agents into explicit geometric vectors (lines, coordinates, and boundaries) and maps out a large “planning vocabulary”. It calculates the probability of various driving maneuvers, enabling it to swiftly choose the safest, most human-like path forward while running up to 9 times faster than older systems.

4. HydraMDP: The Champion of Hybrid Expert Distillation

Hydra-MDP (NVIDIA, CVPR 2024 Challenge Winner) solves a major imitation learning loophole: the tendency of AI models to copy human mistakes or freeze out of fear.

• The Breakthrough: It introduces a Teacher-Student architecture (Hydra-Distillation). Instead of having the AI model learn only from recordings of human drivers, it forces the model to learn simultaneously from both the human logs and strict, rule-based computational experts. A algorithmic safety rules being used.
• How it works: HydraMDP utilizes a multi-head decoder network. Each decoder head evaluates trajectories against different target metrics (e.g., one head focuses strictly on comfort, another on keeping distance from vehicles, and another on hitting traffic light targets). By synthesizing rules and human intuition, it produces highly adaptable, accident-free driving plans

“How do you approach fact-checking information that you aren’t familiar with?”

• Good Answer: “I break the text down claim by claim. I cross-reference specific names, dates, and historical events using multiple authoritative databases or primary sources, rather than relying on a single search result. I always look for consensus across neutral platforms.

“If two AI models both give highly creative answers, but one completely misses a formatting instruction, which one do you rank higher?”

• Good Answer: “I always rank the one that followed the instructions higher. In data annotation, truthfulness and strict adherence to user constraints always beat creative writing quality.

“Data annotation can be repetitive and requires intense focus for hours. How do you maintain high accuracy and prevent fatigue?

• Good Answer: “I treat every single prompt as an independent puzzle. I use a structured checklist for every evaluation — checking intent first, formatting second, and factual accuracy third — which keeps my workflow consistent and prevents me from skimming.