training, eval, fleet
A flight simulator is valuable. It is still not the weather at the real airport.
Pilots use simulators because crashing a model is cheaper than crashing a plane. Driving teams use simulators for the same reason: you can spawn a mattress, a fog bank, or a thousand left turns before breakfast.
Simulation shines when you already know the question. How does the planner behave if a cone field is two feet left? How often does the controller overshoot a tight curve? You can repeat the test. Real video does not let you rewind the physical world.
Driving teams use simulators for the same reason: you can spawn a mattress, a fog bank, or a thousand left turns before breakfast.
The gap is fidelity. Simulated rain is a recipe for rain. Real rain is dirty lenses, inconsistent droplet size, and a wiper smear across one camera only. A pedestrian model does not have every awkward human gait. If a team trains only in the simulator, the network can ace the game and stumble on the street.
The practical pattern is mixed. Use simulation to stress a known weakness and to run cheap regression tests. Use fleet video to discover weaknesses you did not know how to write down. Treating either source as the whole diet is how models get oddly confident.