Dennis,
If this were my 17-year-old brother I would EXPECT him to say nothing happened - that's his code for "I can't talk about it". My version of that when I was a teenager was to run up to my room, stay there and refuse to talk to anyone. But at the same time I desperately WANTED to talk! Go figure.
What's different between Brian's case and mine is (apart from three decades) that you know what happened to him - my parents didn't and I was too afraid to tell them.
Brian will in general be awash with negative feelings about himself - often feelings buffeting him all at once. At times he may need to talk, but he may be so confused he doesn't know where to start. I get that a lot when I talk to young survivors here. When someone tells me he wants to talk but he's scared/can't/too ashamed/doesn't know where to start, I suggest "start anywhere"; take a deep breath and just talk, the conversation will find it's own direction soon enough.
Brian may also REALLY feel that what he has to talk about is so shameful and embarrassing that you would not understand, or perhaps would even disbelieve him. That's the fear talking, of course. He needs to know that you are always there for him and that he can trust you 100%.
The problem here is that talk is cheap, as teens love to point out. He needs to see this in deeds as well. Try to pick up on any opportunity that arises to support him, validate him, and show him you care and can be trusted. Include him in things you do when you can; take him to the races, a cool film, a restaurant for his favorite food, things that will show him that he matters and is special. Remember that an abused kid has learned that trust is a stupid idea; recovering from that devastating lesson requires more than words, he needs to see the proof.
You may remember yourself some of the things that could come up in school. I remember, for example, going to school and thinking I had "fucked a million times" written on my back with Christmas lights; I thought that at any moment someone might figure out what had happened to me. The changing rooms and showers in gym class were especially bad, because I wondered did anything show and I hated being naked. I also wondered if someone even looked at me: I thought do they hate me, do they know, did I do something stupid, are they talking about me, blah, blah, blah.
And a final thing: have you ever told Brian about your own abuse and how you felt - what YOU went through and how you felt? The information might help him, and again, the fact that you were open with him is a gesture of trust that would encourage him to reciprocate.
All this said, Dennis, I do have to end on the note that sometimes things DO get to be too much for a traumatized teen. That is, it is indeed possible that he can't continue in his present state of mind. If that's the case, and he continues under pressure, the result could be a nervous breakdown.
If Brian's trauma over school seems to be assuming the character of an ongoing reaction to school, I would try to get a professional counselor into the picture. That will be rough - yet another strange adult for him to cope with. But it might be a good idea to get a professional opinion of how bad things really are.
I really feel for your position Dennis; it's so great how you are supporting your little brother. Hang in there, and know you can speak out and rely on us for support here.
Much love,
Larry