Help! Cat Shy And Afraid

hi there. I have two tabby/mane coon kitties about 5 years old. I adopted them when they were both kittens (brother and sister from the sameness litter). A week ago my girl cat woke me up with a blood curdling cry in the middle of the night and had somehow managed to rip one of her claws out entirely. I took her to the vet and got her treated but ever since then her brother has been acting afraid (pupils dilated, sniffing everything, not wanting to be affectionate which is VERY uncharacteristic of him). I’ve checked and he hasn’t no injuries but just isn’t acting right. I’ve got the felliway diffuser going but it doesn’t seem to help. I don’t know what to do

He probably has non-recognition aggression, except without the aggression part. It's quite normal for one cat to be fearful of another after they come back from the vet. They smell different and they no longer recognize their buddy.
If you can, rub a towel on your male kitty then rub it on your female, and back again. It will help to get their scents mixed up again on each other. Sometimes this can last a few hours and sometimes it can last a week. Just try to encourage their normal routine, play with them together, feed them together (as long as he's not attacking her)...you want him to know that he doesn't have to be afraid of her. As their scents mix again and the vet smell goes away, he should get back to normal.
One thing to consider though, if you are giving her any medication or using any ointments on her that he does not like or changes her scent this could continue until treatment is done.
He could also be afraid of her due to the scream she let out when she got hurt. In that case, doing the same things should work. But if he is fearful of something (or even you) that he associates her getting hurt with, then you may have to try and figure out what it is that he has associated that incident with. Have you noticed that he avoids a certain thing, room or person in the house? Does a certain noise or object "set him off"?

That’s just it, he’s not afraid of his sister at all. The opposite actually. He’s still very affectionate and protective of her but very weary and cautious of me and our two dogs. He always has to be able to see everyone and his sister. It’s been a week now and I just don’t know how to help him feel safe and not scared. His sister isn’t in any medications and this behavior started immediately after the incident, not invented I brought his sister home from the vet.

Her scream really scared him. Let him take it at his own pace. Don't try to show him affection unless he asks. Do play with and cuddle her where he can see. Hopefully he will see she's not afraid and will relax.

Then I would assume that his sister's scream is what terrified him, and unfortunately, he has no idea why/what "hurt her", so now he's just scared. And now that he's just scared, any big moving unpredictable "thing" (you, dogs, etc...) will make him jumpy.
I know it's been a week, but hang tight, it will get better, he just needs time.

My husband accidentally hit my cat with a pillow in the middle of the night a few months ago when he was throwing it on the floor. After that Chester (cat) didn't even want to come into our bedroom. After a week or so I was finally able to get him back in our room, but never at night and if the blankets or pillow was moved, he immediately ran away. Thankfully Chester is food motivated so I continued to call him to our room and onto the bed for treats. Then started doing it at night and before bed (as we were getting into bed). It took almost a month before he felt comfortable coming in again and didn't flinch if the blankets or pillow moved.

So I would try and create routine and fun for him. Play with him a couple of times a day followed by a meal or treat. Letting him "act out" his feline instincts of stalking, catching and eating "prey", helps build up confidence in a scared kitty.
Stick to a routine as much as possible so he has confidence in his surroundings.
Make you, the dogs and the room where "the incident" happened a happy and rewarding place/thing. If the dogs are being rambunctious and scaring him, maybe that would be a good time to try and engage him in play and a treat. He's already "stimulated" due to the dogs, but you may be able the change his mindset to a good thing rather than to be fearful.