SimWorld is a simulation platform for developing and evaluating LLM/VLM AI agents in complex physical and social environments.

• 2025.9 SimWorld has been accepted to NeurIPS 2025 main track as spotlight paper! 🎉
• 2025.6 The first formal release of SimWorld has been published! 🚀
• 2025.3 Our demo of SimWolrd been accepted by CVPR 2025 Demostration Tack! 🎉

SimWorld is built on Unreal Engine 5 and offers core capabilities to meet the needs of modern agent development. It provides:

• Realistic, open-ended world simulation with accurate physics and language-based procedural generation.
• Rich interface for LLM/VLM agents, supporting multi-modal perception and natural language actions.
• Diverse and customizable physical and social reasoning scenarios, enabling systematic training and evaluation of complex agent behaviors like navigation, planning, and strategic cooperation.

SimWorld consists of three layers:

• the Unreal Engine Backend, providing diverse and open-ended environments, rich assets and realistic physics simulation;
• the Environment layer, supporting procedural city generation, language-driven scene editing, gym-like APIs for LLM/VLM agents and traffic simulation;
• the Agent layer, enabling LLM/VLM agents to reason over multimodal observations and history while executing actions via an action planner;

SimWorld's architecture is designed to be modular and flexible, supporting an array of functionalities such as dynamic world generation, agent control, and performance benchmarking. The components are seamlessly integrated to provide a robust platform for Embodied AI and Agents research and applications.

simworld/               # Python package
    local_planner/      # Local planner component
    agent/              # Agent system
    assets_rp/          # Live editor component for retrieval and re-placing
    citygen/            # City layout procedural generator
    communicator/       # Core component to connect Unreal Engine
    config/             # Configuration loader and default config file
    llm/                # Basic llm class
    map/                # Basic map class and waypoint system
    traffic/            # Traffic system
    utils/              # Utility functions
data/                   # Necessary input data
config/                 # Example configuration file and user configuration file
scripts/                # Examples of usage, such as layout generation and traffic simulation
docs/                   # Documentation source files
README.md

• Python Client Make sure to use Python 3.10 or later.

git clone https://github.com/renjw02/SimWorld.git
cd SimWorld
conda create -n simworld python=3.10
conda activate simworld
pip install -e .

• UE server Download the SimWorld server executable from S3:

  • Windows. Download the SimWorld Windows 64-bit Server (v0.1.0) and unzip it.
  • Linux. Download the SimWorld Linux 64-bit Server (v0.1.0) and unzip it.

We provide several examples of code in scripts/, showcasing how to use the basic functionalities of SimWorld, including city layout generation, traffic simulation, asset retrieval, and activity-to-actions. Please follow the examples to see how SimWorld works.

SimWorld uses YAML-formatted configuration files for system settings. The default configuration files are located in the ./simworld/config directory while user configurations are placed in the ./config directory.

• ./simworld/config/default.yaml serves as the default configuration file.
• ./config/example.yaml is provided as a template for custom configurations.

Users can switch between different configurations by specifying a custom configuration file path through the Config class:

To set up your own configuration:

1. Create your custom configuration by copying the example template:

   cp ./config/example.yaml ./config/your_config.yaml

2. Modify the configuration values in your_config.yaml according to your needs

3. Load your custom configuration in your code:

   from simworld.config import Config
   config = Config('path/to/your_config')    # use absolute path here

Once the SimWorld UE5 environment is running, you can connect from Python and control an in-world humanoid agent in just a few lines:

from simworld.communicator.unrealcv import UnrealCV
from simworld.communicator.communicator import Communicator
from simworld.agent.humanoid import Humanoid
from simworld.utils.vector import Vector
from simworld.llm.base_llm import BaseLLM


# Connect to the running Unreal Engine instance via UnrealCV
ucv = UnrealCV()
comm = Communicator(ucv)


class Env:
    def __init__(self, comm: Communicator):
        self.comm = comm
        self.agent: Humanoid | None = None
        self.agent_name: str | None = None
        self.target: Vector | None = None

    def reset(self):
        """Clear the UE scene and (re)spawn the humanoid and target."""
        # Clear spawned objects
        self.comm.clear_env()

        # Blueprint path for the humanoid agent to spawn in the UE level
        agent_bp = "/Game/TrafficSystem/Pedestrian/Base_User_Agent.Base_User_Agent_C"

        # Initial spawn position and facing direction for the humanoid (2D)
        spawn_location = Vector(0, 0)
        spawn_forward = Vector(0, 1)
        self.agent = Humanoid(spawn_location, spawn_forward)

        # Spawn the humanoid agent in the Unreal world
        # NOTE: name is ignored for humanoid type, but required by the API.
        self.comm.spawn_agent(self.agent, name=None, model_path=agent_bp, type="humanoid")

        # Cache the generated UE actor name
        self.agent_name = self.comm.get_humanoid_name(self.agent.id)

        # Define a target position the agent is encouraged to move toward (example value)
        self.target = Vector(1000, 0)

        # Return initial observation (optional, but RL-style)
        observation = self.comm.get_camera_observation(self.agent.camera_id, "lit")
        return observation

    def step(self, action):
        """Move the humanoid forward a bit and compute reward."""
        # Parse the action text and map it to the action space
        if "step_forward" in action:
            self.comm.humanoid_step_forward(self.agent.id, 2.0)

        # Get current location from UE (x, y, z) and convert to 2D Vector
        loc_3d = self.comm.unrealcv.get_location(self.agent_name)
        # loc_3d is a numpy array; explicitly use x, y to build our 2D Vector
        location = Vector(loc_3d[0], loc_3d[1])

        # Camera observation for RL
        observation = self.comm.get_camera_observation(self.agent.camera_id, "lit")

        # Reward: negative Euclidean distance in 2D plane
        reward = -location.distance(self.target)

        return observation, reward


if __name__ == "__main__":
    # Create the environment wrapper
    env = Env(comm)
    obs = env.reset()
    llm = BaseLLM("GPT-4o")  # Make sure OPENAI_API_KEY is set as an environment variable

    system_prompt = "Your system prompt here"

    # Roll out a short trajectory
    for _ in range(100):
        user_prompt = f"Current observation: {obs}\nPlease output an action."
        action = llm.generate_text(system_prompt, user_prompt)
        obs, reward = env.step(action)
        # Plug this into your RL loop / logging as needed

We use Google docstring format for our docstrings and the pre-commit library to check our code. To install pre-commit, run the following command:

conda install pre-commit  # or pip install pre-commit
pre-commit install

The pre-commit hooks will run automatically when you try to commit changes to the repository.

All commit messages should be clear, concise, and follow this format:

<type>: <short summary>

[optional body explaining the change]

Recommended types:

• feat: A new feature
• fix: A bug fix
• docs: Documentation changes
• refactor: Code restructuring without behavior changes
• style: Code style changes (formatting, linting)
• test: Adding or updating tests
• chore: Non-code changes (e.g., updating dependencies)

Example:

feat: add user login API

• Use clear titles starting with [Bug] or [Feature].
• Describe the problem or request clearly.
• Include steps to reproduce (for bugs), expected behavior, and screenshots if possible.
• Mention your environment (OS, browser/runtime, version, etc.).

• Fork the repo and create a new branch (e.g., feature/your-feature, fix/bug-name).
• Keep PRs focused: one feature or fix per PR.
• Follow the project’s coding style and naming conventions.
• Test your changes before submitting.
• Link related issues using Fixes #issue-number if applicable.
• Add comments or documentation if needed.

We appreciate clean, well-described contributions! 🚀