Dennis,
First of all, you did nothing except what a kid should do - you told your mother. That was already incredibly brave; many boys would not have been able to do that. Your disclosure didn't produce anything positive, but the blame for that goes not to you, but to the dysfunctional nature of the home you were in.
Whatever I say here next isn't meant to excuse your mother's neglect of you. Once you told her what was happening she should have put you first; nothing should have been more important.
That said, however, it isn't uncommon for a woman who is being brutalized at home by an abusive husband to be reduced pretty quickly to a state where she feels entirely powerless. This is what happened years ago to my aunt (my Dad's sister). Her husband became violent and abusive, but my aunt kept hiding what was going on and she made excuses for her husband for years. Finally he beat her bad enough to cause serious injuries and she told my Dad; he went up there, threw her husband out of the house and helped my aunt get a divorce. Lots of differences from your case, I know, but the point is how powerless the woman feels.
I think it's all the worse since these things are happening in the woman's own home, where she ought to feel safest. She has kids to think about, she's living from day to day, and she probably feels guilty - the bastard hitting her will of course stress what a failure and unsexy miserable wife she is. And she probably fears the future as well, even if there is a women's shelter in her area: will she really be safe there?, what will happen to her after the shelter?, what about the children?
The problem now is, as you suggest, the elephant in the room every time you meet. This is a big problem and it affects both you and Brian.
The "what next?" question just raises several others. Okay, you are angry with your mother, but what do you want from her? What do you need? For example, do you want/need a clearing of the air, answers to some questions, and then progress towards a reconciliation? How would that help you? Or do you want a confrontation and a parting of the ways? How would that help you? Don't feel you are being selfish if you think this way, putting your needs first. You and Brian should not be in this position in the first place, but here you are. Now it is up to YOU to do what should have been done years ago - put first things first. The bad luck of the situation leaves a heavy burden of responsibility on you personally, Dennis, and if you don't or cannot act to secure what you and your brother need, then both of you will just be victimized all over again.
If you do want a reconciliation there are various ways to work towards this that don't involve a shouting match. But the point right now is, as I said, deciding what you want. You have every right to be angry, but addressing that anger isn't the issue here; what would assuage your anger (if anything) might not be identical to what really serves your long-term interests and needs.
Much love,
Larry