What You Lose With End-to-End Driving

Anyone who has used a game's debug console knows the value of being able to inspect state, change one variable, and watch the result. Remove the console and you are testing from the outside, inferring the inside.

Learned end-to-end driving removes a lot of that console. With written rules, a bad behavior could often be traced to a specific condition and patched without touching the rest. With one large learned system, there is no such line. Behavior comes from a huge set of internal values nobody set one-by-one.

Remove the console and you are testing from the outside, inferring the inside.

Three practical costs follow.

First, fixes are indirect. Improving one scene usually means curating data and retraining, then checking whether the change actually landed.

Second, improvements can regress unrelated scenes. Because many behaviors share the same internals, teaching one case can quietly degrade another. Teams spend a lot of time watching for that.

Third, verification gets slower. You generally cannot prove a property by reading the policy as if it were a checklist. You have to show it across many examples, which costs more time than an inspection.

Teams accepted those costs because the hand-written recipe had a ceiling. The costs are still real, and they help explain why progress often arrives in lumps instead of a straight line.