In moderation I'm sure there won't be much, if any noticable damage. Albeit, it sounds like you removed the polish from the plastic.
That seems to be the case, at least on the Greys. As far as removing the text on Clearboys, I can fully understand why that would come off, since there's no texture for the paint to sink into, and it essentially just sits on the clearboy's case surface, correct?
If you wouldn't mind, I'm interested in what that patch of texture-less casing looks like. Just want to see how how severe it is, compared to what these cases look like after scrubbing them:
http://imgur.com/sACxk,drXmF,h5lg2#0
Care to upload a photo for me?
(Mind you, I don't want to clutter up your thread. Perhaps there's an alternate place we could discuss this?)