Even though most professionals suggest that a fling can only have negative effects on your personality, there are not always reasons to be worried about.
What you need to think about first is not if you have flings or not, but rather why you have them. If your motivation is right, then you need sex, and not something else that you think it would provide. If you want to have fun and enjoy yourself, or if you want to experiment in bed then it can only give you pleasure.
However, as soon as you start having casual sex because you feel bad about yourself or your relationship, and you need something to boost your ego, then having sex is nothing better than lighting a cigarette or having a shot of vodka at the nearest bar.
On the other end, if you have sex because your partner wants it – but you do not – or if you hope sex can turn a friendship into a long-term relationship, then you give yourself an illusion that will never come true. If you do not want to have sex, then don't do it. This is what lowers your self-esteem not the fact that you do it.
As long as you feel that a hook-up is all that you want, and not something it will never give you, have fun an enjoy life. But if you made a wager, or want to take revenge or just something in return, your mental well-being is at stake, so keep your pants on.