I know its not mites. I did clean it with a qtip and it was a little dirty. What other reason would it be bothering him? He scratches at it, keeps trying to clean it and has it down sometimes.
It could be something deeper in the ear canal that you can't reach with a Qtip, or even see. Could be an ear infection. You might want to take him in to be checked out by a Vet, since it's clearly bothering him.