Given that I work only with strays and ferals and have done a lot of TNR, I have neutered and spayed cats who were definitely adult. If the cat was in good health, and often it was only that they appeared to be healthy as no medical assessment is done most of the time with TNR, I went ahead and fixed them. I don't consider 4 to be old at all.
I agree with all the above members, 4 yrs. old is definitely not too old to neuter a male cat. Many cats who are strays and brought to a shelter at that age or older are safely neutered before being adopted to new homes.