A hermit may also be a male with a penis that is much shorter than their testicles when flaccid, representing the animal of the hermit crab.
The penis is suppose to represent a hermit crab hiding away in the shell, which is the testicles.
'Hermit', is used most commonly as an insult without factual evidence, mainly directed at friends as a joking insult, however could become quite personal if the person does own a hermit. The insult would usually be directed if the person does something stupid and would therefore be called a hermit, just like they could be called a idiot, however hermit is a lot worse, but shouldn't be taken seriously.
Mike, you're such a hermit lad
Did you hear, Lucy said Connor's penis is tiny after last night, it looks like a hermit!