When I read this I can somewhat relate to your brothers reluctance. I was the big brother and though I did not commit sexual or physical abuse I came to realise as an adult that my behaviour might have been really harmful to my younger brothers. It still pains me to think about my failures as a big brother, going back and forth between being kind and being angry, protecting them but then bullying them. I was well into my twenties before I realised that my behaviour was harmful bullying and emotionally abusive, and that the "weakness" in them was valid feelings and needs in children. They did not need to "toughen up".
We have no contact for other reasons and I think it is for the best, there is not really any realtionship that can be saved. I miss having brothers, I miss that bond but I don't miss them as persons. But for you and your brother I sincerely hope you can find that connection again and that he can process his own guilt and responsibility in this.
There is something healing for me in reading your words about your brother. Now as an adult you look back at both of you as the boys you were, and you have this empathy and forgiveness. It touches on the big brother wound in me, my failures as a big brother, the wounds of not having had the strength and empathy back then in that situation to care for and protect my brothers the way i wish I would have. I was a kid too. But I was a bigger kid. It is a difficult feeling.