I just feel it's really divisive and causes a lot of bad feeling and divisions in society. There are ways of discussing and recognising everyone's challenges, that don't need to put down one person or group in order to elevate another.
I did see you use the term in a way that described people who didn't believe that your hardships had actually happened. You said their attitude showed how privileged they were to not even be able to imagine what had happened to you!
The way you used it there, I completely understood and felt the use was appropriate. But so often, it isn't.
By the way, I believe you. :)