Large language models are moving XR beyond floating chat windows toward world‑aware assistants that understand context, act through tools, and adapt to the physical environment. Instead of asking a headset to open a menu, a technician can say “pin the torque spec next to this valve”, and an assistant translates deictic language into a precise action anchored in space. This shift, from text boxes to embodied agents, changes how we design interfaces, orchestrate data, and measure success.

The fit is natural. Headsets privilege hands, eyes, and voice. LLMs excel at turning messy, multimodal input into intent. They compress torrents of sensor data, planes, meshes, anchors, and hand rays into concise, actionable steps a person can follow while staying heads‑up and hands‑free. Crucially, they also act as glue, calling scene APIs, querying knowledge bases, and invoking detectors so the system can do things, not just say things.

Practical use cases are already compelling. Field service teams use hands‑free procedures where manuals become stepwise guidance attached to real surfaces. Designers sketch with words, asking for a workstation layout “along this wall”, then refine it in place. Live translation and scene descriptions make spaces more accessible. Spatial search and retrieval let people ask where they left the 10‑mm socket yesterday and get an arrow to the last known anchor. Each scenario relies on the same capabilities: grounding language to the scene, retrieving the right knowledge, and executing with reliable placement.

Under the hood, an effective architecture keeps the render loop fast while the assistant thinks in parallel. The perception layer streams pose, planes, meshes, and anchors. A context layer builds a spatial knowledge graph that links documents, notes, and prior actions to specific places. The LLM layer plans and decides, preferably with function calling, so it can ask for scene queries, create anchors, and place UI panels. An action layer executes those tool calls in the engine. A feedback loop closes the gap with confirmations by gaze or pinch, visual gizmos that show intent, and conservative policy checks.

Spatial retrieval is the quiet superpower. Instead of tossing every paragraph of every manual into a single vector store, attach the right chunks to the right anchors and regions. At runtime, retrieved by both semantic similarity and spatial proximity, so a query near Panel B prioritizes Panel B’s history and parts list. The result is a smaller, more relevant context window, faster responses, and fewer hallucinations.

Latency disciplines make or break presence. Motion‑to‑photon budgets for comfortable VR live around tens of milliseconds and must stay sacred. Keep the LLM off the render thread, stream tokens and audio so the user hears “placing…” before the full sentence finishes, and cache retrievals for time‑to‑first‑token that feels conversational. Favor on‑device perception, use efficient prompts, and degrade gracefully when the network hiccups. If the assistant stalls, the interface should never freeze. Neutral placeholders and subtle progress afford a sense of control.

Prompt design in 3D benefits from a few reliable patterns. Map deixis, “this”, “that”, “over there”, to gaze and hand‑ray hit tests plus small spatial neighborhoods. Inject affordance constraints that reflect physics and ergonomics, such as preferred distances and surface normals for readable UI. Maintain stateful memory bound to anchors, so returning to a machine recalls what happened there last week. Encourage tool‑first reasoning when the answer depends on the scene, the model should query or scan before composing prose.

Safety and privacy deserve explicit boundaries. Do not assert physical safety or structural claims without verified sensors and policy gates, offer conservative guidance, and encourage confirmation with certified tools. Minimize video and audio egress by default, blur sensitive content when capture is needed, and keep logs scoped to anchors and sessions with sensible retention. Treat the assistant as an operator in a shared, physical workplace, not just a chatbot in a window.

Evaluation should look beyond language metrics. Track time‑to‑first‑token and full turn latency for voice interactions. Measure placement precision in centimeters against intended targets and the persistence of anchors across sessions. Instrument task success and time on task for procedures, and capture user comfort and cognitive load. Monitor confabulation rates for scene‑related claims and count safety intercepts to validate policy coverage.

Common pitfalls are easy to recognize and fix. A “talking head” that hovers in the user’s face breaks immersion, attaches guidance to relevant surfaces at readable distances, and allows occlusion by real geometry. Jittery overlays usually mean weak anchoring or poor surface selection, prefer stable planes, and re‑snap after head motion. Ungrounded answers come from speculation. They require tool use before claims about the scene. Platform lock‑in creeps in unnoticed, abstract core pose/input, and keep content pipelines portable.

Getting started is simpler when the job‑to‑be‑done is narrow. Pick one workflow, filter replacement on a specific model, warehouse pick‑assist in one aisle, and instrument it end‑to‑end. Attach knowledge to the world rather than floating it in space. Stream everything that can stream, log events spatially, and test in situ under real lighting, noise, and occlusion. Small wins in these focused scenarios build the muscle to scale responsibly.

The destination is clear: assistants that see what we see, understand where content belongs, and help us get real work done without breaking presence. LLMs are not merely another input method for AR/VR, they are the orchestration layer that turns spatial computing into useful computing.

LLMs in AR/VR: Turning Spatial Computing into Useful Computing was originally published in Data Has Better Idea on Medium, where people are continuing the conversation by highlighting and responding to this story.