It's due to a very distinct pattern formed in your sub-conscious when you were a child. It's not that you "seek" abusers, it's that you "recognize" them. Their mannerisms, behaviors and attitudes are familiar to you on a sub-conscious level and once a potential abusers says or does something that you recognize, a light bulb goes off in your head, like throwing a light switch and "bing!" you've found what your sub-conscious was looking for...familiarity.
That is why it is difficult to break this and all habits (good or bad) because you're "wired" to seek "familiarity" or that which we have been conditioned to know. What is key is re-conditioning or re-wiring your mind. This is possible. But the same way you are conditioned to fear dogs, you will have to teach your mind to recognize "good" and "bad" people/behavior.
You will know what I mean once people and their behavior deviates from your current programming. For example, you might be leery of someone who touches your shoulder or knee while talking/ laughing. You may (cuz I don't know) have been conditioned in the past that when people touch you, regardless of how casual, they want "something" from you (ie, sex). A part of you will react reflexively to your current programming (ie, anger, distrust, "sex object mode", uneasiness...) but when you try to download the "new program" (ie, it was an innocent touch...no harm, no foul...) your mind will go into defending the old program and second-guessing You. It will say how you never know what's truly best and how last time you trusted someone you got yourself into trouble. It will say that everybody wants something and that nobody can be trusted...yadda, yadda
Your own mind will berate you into submission and you will be left "trusting" whatever defense mechanisms you've created over the years. You will ask yourself "if this(old programming) is working why are you so miserable" and your mind will say..."sure, we're miserable but we're safe..."
This is merely my opinion and personal experience, because I don't know you...However, someone once said that "...only the brave, love..." Nobody is 100% trustworthy because people are generally driven by their own self-interests, except those that have been abused. Abuse victims either become abusers, doormats or .....
.....whatever they WANT to be.....
You can have it your way, Mike! Just be honest when you see ANYTHING that reminds you of your abusers in others. Sometimes we see the abusers but we go into "caretaker" mode and try to fix them. This is because we are sub-consciously trying to re-play/reconstruct what happened w/ our abusers. THIS IS FUTILE. Don't try to save anybody! Fuk 'em! You have spent too much of your life trying to appease and help others, now it's time to be selfish and self-serving like everybody else. I am sure the river of guilt runs deep when it comes to taking care of yourself, first. But who cares! None of the critics came to your aid when you most needed them. You survived the physical abuse on your own, now it's time to deal w/ the psychological aftermath.
Don't worry, I've got your back!
Take care. Sorry this ends abruptly, I have to catch a bus...