Protecting ourselves from toxic energy
Have you heard of "energy vampires"? These are people (and places for that matter) that suck our energy away from us. Think of times when you spoke with someone then walked away feeling depleted, or more depressed than when you started. It's like that. So first, you're right - they ARE taking away your energy. To be fair though, they don't intend to and very likely have no idea that they are doing it.
There IS a solution - we need to protect ourselves. Since the issue is around energy, the solution is too. We need to protect our energy. Psychologically, what you are doing is failing to get "attached" to their words, and/or overly empathize with them (similar to "not caring", but instead it's more like you are refusing to let this effect you even though you feel badly for them). Energetically though, we want to (literally) protect our energy from being taken/effected/manipulated in the first place. We all have an energy field that is ours, and we have the right and the power to decide what happens with that energy. With that in mind, the first thing we want to do is hold an intention (in other words, be VERY clear) that our energy is ours, and no one is allowed to effect it unless they are offering love, light and/or energy that is nourishing to us. To support this, imagine that you and your energy exist in a bubble that has mirrors on the outside. Anything people try to do with your energy that fails to serve you is reflected back to them. Anything that is about light, love and all things wonderful are allowed in. Boom. While you're at it, attach an umbilical cord to that bubble that goes to the center of the Earth and exhale any heavy/toxic energy out, while breathing in all the great light that the center of the Earth offers. This process is called GROUNDING. You can actually do it any time you feel off, whether from someone else or not. In theory, we should always be grounded.
Last thing - when you're with someone who has any toxicity, send them "blessings"/"prayers"/"light"/"love". There is no limit to the amount of this to send anyone, so why not send lots?! We all can use it. In fact, I officially send you a bunch right now. Breathe it in deeply. Feels good right? ;-)