Some get kicked out by Medium for bad behaviour or because they're bots. Some only follow to get a follow and then unfollow, whether you've followed them or not. There are a lot of not very interested followers out there. Of my 2400ish, only about 20-30 engage regularly. Don't worry about the numbers going up and down. Just follow who you're interested in reading, and keep going.