prediction, planning
Knowing where everything is is not enough. The system has to guess where it is about to be.
A quarterback does not only read where defenders are standing. He reads where they are about to be. The snap-to-throw window is short enough that acting on current positions means acting on stale information.
Driving has the same structure. By the time a vehicle finishes a hard brake, the scene has already moved. So the software runs a continuous forecast: this pedestrian is facing the curb and slowing, so they may step out. That vehicle is drifting inside its lane, so it may merge. That cyclist glanced over a shoulder, which often precedes a lane change.
He reads where they are about to be.
Prediction is multi-agent and interactive. Every actor is forecasting everyone else, and those forecasts change behavior. If the car creeps at a four-way stop, another driver may yield — which means the forecast has to include how the car's own motion changes what happens next.
There is also no single right answer. A person at a curb might step out or might not. A careful system carries both possibilities instead of collapsing them into one confident story. A lot of what riders call hesitation is a prediction layer refusing to treat an uncertain scene as settled.
It is the least visible part of the stack and, in dense traffic, often the hardest.