Oh dear. I remember meeting my sister's foster kids years and years ago. They were straight out of Africa and had been sent to the UK for their education because their dad lived here. He was abusive, so they ended up in foster care. My sister looked after them. I expected them to speak another language so was amazed how good their English was. I complimented them and said, "Your English is very good!" They were very polite and said 'thank you'.
I felt like a total jerk when my sister said, "English is their first language". I said, "Oh."
Felt like a right prat.
But honestly, it's not supposed to be aggression. People who take it as an insult are just being oversensitive and making an issue out of nothing. I was genuinely impressed and obviously didn't realise that English was their first language! Now I know.