If I stop buying imported T-shirts, what are the repercussions for workers?

I don't want to support the practice of exploited labor in the clothing industry, but if I – alongside everyone else – stop buying shirts from, say, China (where long hours and fair pay are not always regulated and shipping to the US produces emissions), will those workers be out of a job and worse off when demand disappears?