This is your vet that recommended it and you are looking for advice on an online forum because you are second guessing her? I have heard of cats and dogs being given benadryl by vets. The thing you need to worry about is dosage, and your vet gave you that. Yes, you are being paranoid.
