When was he neutered? It takes several weeks after the surgery for the swell to go down and fur to grow back. After that it's flattened and soft to the touch. Flattened as in not 100% flat but with a little tiny wee bulge.
Generally, in order to tell if a cat has been neutered, you would check if he has visible or palpable testicles. If you feel the sack on a whole male, it's generally easy to feel the testicles inside; they'll feel like rubbery marbles. On a neutered male you'll just feel a soft sack.
Here's an example of a one year old sexually mature male, so you can see what an adult male looks like if he isn't neutered. Of course the testicles can vary in size, so it's not always as obvious as this.
When they are neutered they get to keep their "harbl" - Internet slang... So we always called it that. The testicles are removed from their bean bag and it's stitched back up.
Stewart was my first non-neutered male, and it took me by surprise. My boyfriend had put him in the crate since he'd just come in as a stray. I went to lift him out of the crate "let's see who we have here" and I ended up groping him accidentally just right that I let out a very loud "Whoa...oops" because we didn't know each other well yet, and I clearly ended up with a palm full of testicles.
"Um. He's definitely male." I told the boyfriend. "We've got to get him fixed."