Why do we call bad family members "black sheep"?

Shepherds and sheepowners of old disliked black sheep because they were less valuable on the market than white ones. A “black sheep” in the family has come to be known as a disgraceful member who shames the rest