I'm always amused that people zero in on one thing, and it's the wrong thing to zero in on, but nevertheless, the reason they do that is because the voice is what they know - the voice represents everything about the character, that's why.
If pimps and thieves everywhere were always punished, honest people would all believe themselves always to be innocent.