bane

Something that causes great distress, annoyance, or ruin to a person or thing.

"The constant traffic noise was the bane of his quiet life."