I think if you dont keep it clean it will grow bacteria. Bacteria can get in the air you breathe, when the cat uses the litter box. Or the cat can track it around the house and contaminate what he touches. So i would say yes. For your health, and the health of your cat you should keep them a clean litter box.
It certainly can. Not only can it become a breeding ground for bacteria, but the ammonia can cause breathing issues. And it can make your cat sick, as well. Their lungs are also affected by ammonia.