You have given way too little info. Does the sitter like cats? If she likes cats does she possibly have one of her own (or even another animal at all) that your cat smells on her? Does your cat act like this around other people or just her? You say babysitters, so does the cat attack all of them? Also just to be clear is this a babysitter for human kids? or someone taking care of the cat? If it's for the cat, I highly suggest finding a new sitter, you want one your cat likes.
If it's for kids then my suggestion would be to just put him in another room or crate, with food, water, toys, litter box, etc. while she is there. Have people that come to babysit give him high value treats (something he really likes but doesn't get normally) but no touching, basically come in give treat, then leave him. Over time he should learn that the sitter means he gets a treat. Or depending on if you cat is more toy motivated, get a toy he really likes, and he only gets that toy from the sitter when they come. You basically want him to associate someone new coming into his house with him getting something good.
Another thing to keep in mind, although I hate to have to bring it up and is unlikely if the cat is doing this to multiple people, is that pets can be protective of their owners. If the sitter is (or appears to the cat to be) hurting or being mean to the kids the cat may try to defend them.