sweater thread

Something better left alone where one thing leads to another resulting in disaster, like the thread of a sweater that when pulled unravels the whole garment.

That whole relationship was a sweater thread.

Don't go there. That is one sweater thread you do not want to pick at.