Splat-Sim A Closed-Loop Driving Simulator with Gaussian Splatting

Junhao Ge^1*, Zuhong Liu^1*, Longteng Fan^1*, Yifan Jiang^1*, Jiaqi Su¹,
Yiming Li², Zhejun Zhang³, Siheng Chen^1,4

¹Shanghai Jiao Tong University ²New York University
³ETH Zurich ⁴Shanghai Artificial Intelligence Laboratory
^*Indicates Equal Contribution

Arxiv Paper Code

Interactive, High-fidelity, Closed-loop Simulation Platform with 3D Gaussian Splatting

Splat-Sim simulates traffic flow with HD map, and then reconstruct the scenario with pre-trained foreground gaussian models and background models.

Map

Foreground Model

Background Model

Abstract

We introduce Splat-Sim, a simulation framework with Gaussian Splatting for testing autonomous driving (AD) systems, tailored for sensor-realistic closed-loop evaluation in scenarios involving dynamic agent interactions. Splat-Sim generates agent vehicles that interact with ego vehicle in configurable scenarios. We reconstruct scene backgrounds, integrating agent vehicles from a pre-trained 3D Gaussian Splatting (3DGS) vehicle library. Our framework renders images from the perspective of ego vehicle, dynamically adjusting the agent vehicles to produce novel viewpoint images in real time. In our experiments, we generate additional data via our platform to train end-to-end models (E2E models), validating the quality of the synthetic data and the effectiveness of our closed-loop simulation. Our findings indicate that open-loop evaluations of E2E models do not fully reflect performance limitations and reveal the need for improved generalization across real-world datasets. Our platform demonstrates the potential to fuse and extend real-world datasets, offering a robust simulator for AD model development and testing.

Splat-Sim Framework

Pipeline Image

Splat-Sim consists of three components. Scene Controller is initialized with a world configuration and an HD map, responsible for generating and managing interactive traffic flow, including both ego and agent vehicles. Based on this traffic flow data, Scene Renderer renders new images according to a camera configuration, which is then used as sensor input by Driving Planner. Finally, Driving Planner makes updated trajectory plans for Scene Controller.
Scene Controller. Scene Controller utilizes HD maps derived from real-world datasets. By building detailed topological representations, it supports precise vehicle navigation and localization. Scene Controller generates global trajectories, manages agent vehicle behaviors, and uses a physics constraint handler for realistic vehicle interactions, ensuring smooth and consistent motion.
Scene Renderer. Scene Renderer leverages 3D Gaussian Splatting to render high-fidelity driving images. It integrating background models and foreground models seamlessly within a unified coordinate system, ensuring spatial-temporal consistency in rendering and producing images that reflect realistic driving scenes.
Driving Planner. Splat-Sim supports both expert planners and end-to-end (E2E) models for driving behavior. The expert planner uses map-based navigation to generate vehicle trajectories. E2E model takes images and ego status as inputs and then update ego trajectory. In different scenarios, Driving Planner can be switched to adapt to different requirements.

Reconstruction Library

Before simulation, we pre-build a 3D Gaussian model library for dynamic foreground objects and static backgrounds, which serves as the source for rendering scenes and dynamic objects within them. Using Blender and BlenderNeRF, we capture digital assets and record camera positions, initializing the foreground model with mesh data containing RGB and positional information. For the Background Gaussian models, we remove dynamic objects from the scene, obtaining backgrounds entirely composed of static objects.

Foreground Gaussian Model Data Generated with Blender and BlenderNeRF

Images Rendered with Pre-trained Background Gaussian Models

Open-Loop Benchmark

In open-loop mode, an expert planner generates control signals for the ego and agent vehicles based on predefined trajectories. Scene Controller sets up the scenario, and Scene Renderer creates images from updated vehicle positions. Following data is generated with Waymo Open Dataset and FPS is set to 2Hz.

Scene: 019, Agent Num: 20, 2Hz

Scene: 155, Agent Num: 20, 2Hz

Scene: 276, Agent Num: 20, 2Hz

Closed-Loop Simulation

In closed-loop mode, E2E driving models interacts with the environment and adjusts the ego vehicle's trajectory in real-time. Key metrics like Route Completion (RC), Vehicle Collision Rate (VCR), and Layout Collision Rate (LCR) are used to evaluate the model's performance, focusing on adaptability and robustness in dynamic scenarios.

Scene: 019, Success in Static Driving Scene

Scene: 019, Fail Reason: Collide with Other Vehicles

Scene: 276, Success in Static Driving Scene

Scene: 276, Fail Reason: Collide with Other Vehicles

BibTeX

BibTex Code