I've never had to deal with an abuser so close at hand. You would probably need to consider the cost/benefit of a head on confrontation with him. Considering it would devastate your mum, unless the situation is preventing your recovery it may be better to continue to ignore it. But if it is interfering with your emotional well being, or any progress in healing from the CSA I think you should definitely confront him. Sometime you just have to have a ME consideration instead of what's best for the family.
One of my abusers was a cousin who I rarely see except for funerals for family members. For our grandmother's funeral we were both pall bearers, and I had a terrible feeling this was going to be traumatic for me. But he seemed really oblivious to the fact that our past was anything but friendly, so it was easy to pretend every thing was fine between us. If I had contact with him on a regular basis, I don't know if I could handle it so well.