I havent used a clay mitt before, but I have a mental block in my head that would stop me using it - yes, it makes sense for the reasons you say, but you are unable to knead the clay to a fresh surface like you would with a clay bar....so contamination is always on the surface of the clay mitt, which in turn you are dragging on the surface.

But as I said, Ive never used one, so perhaps it isnt an issue - definitely not if you plan on machine polishing the surface afterwards anyways!