Most breads have yeast and cats like yeast. Nutritional yeast can be used as a cheese substitute for lactose intolerant or vegan people. My brothers ex used to sprinkle it on popcorn too. I'd imagine the yeast in bread might be a lighter version of that and cats can smell it in the bread. I'd also guess that cats who have had bread like the sugar too but that's probably a learned thing.
A little bit of plain bread isn't bad for cats. Avoid savory breads because onion and garlic (is not good for cats). While bread isn't going to hurt your cat in small amounts it should be avoided and limited. It's a high carb food and cats need very little carbs in their diet. Plus the added sugars doesn't do cats any good.