unattractively unkept, dirty, or deranged. it may but doesn't necessarily have to encompass the scientifically-recognized communicable skin disease by the same name.
Jenna: check out the guy over by the bar
Alex: you mean the one with the crooked teeth, unwashed hair, and putrid body odor?
Jenna: yeah, he's got that scruffy look to him
Alex: un-uh. he's definitely got the mange. i wouldn't let him ruffie me if his last name was Darcy AND he had a british accent.