I don't think my two males were quite as old as your cat, but I think both were at least three when they were neutered (they were shelter cats). No problems at all.
Given that I work only with strays and ferals and have done a lot of TNR, I have neutered and spayed cats who were definitely adult. If the cat was in good health, and often it was only that they appeared to be healthy as no medical assessment is done most of the time with TNR, I went ahead and fixed them. I don't consider 4 to be old at all.
I agree with all the above members, 4 yrs. old is definitely not too old to neuter a male cat. Many cats who are strays and brought to a shelter at that age or older are safely neutered before being adopted to new homes.