Honestly I'm not too worried about it, any of the plans I've looked at always seem to mention overrides and the like for the speed limiter. Limiting a car to a 'low' speed potentially brings up all sort of legal questions in the event of a crash that could be attributed to the car being unable to utilize its own performance potential, as said in the overtaking a tractor scenario and the like.

Plus you have the same problem as autonomous cars face; how do these new limited cars play nice with older unlimited cars on the same roads?

Ultimately I see full autonomy being the death of fun driving far more so than these kind of measures. I always thought the driving scenes in iRobot (the movie) were a distinct possibility for the distant future of transport. Using technology to make transport far more capable and efficient than it currently is seems far more like the kind of progress the world will push for rather than endlessly trying to make the weakest link in the current system, us, work better.