The drivers side one has the usual wear already so won't make it any worse,

It should be simple enough, I'm just worried about the glue and will it ruin the trim or will it come off if wanted in the future ?