It's tough to answer your question without more information. How long has your cat been with you? What's his history - did you get him as a kitten (and from whom), or was he in another home before yours? Have there been big changes in his life recently (ex.- have you moved, had a baby, have a new roomate, etc.) What's his personality like usually - is this behavior new for him? Is he otherwise eating, drinking, using the litterbox, etc. normally?
I seriously doubt your cat "hates" you, but he does sound fearful. Please give us some more information and we'll be able to offer suggestions.
If you meet a cat with a direct stare it can be intimidating to them and they can consider it a threat. Ragdolls are generally laid-back wonderful cats is there a possibility he might be ill? Or did he come from a breeder with overcrowded conditions?