I disagree. I think men find breasts attractive because they are something (most) men lack. It's a secondary sex characteristic. In our society, long hair is also largely (but not exclusively) a secondary sexual characteristic, and a lot of men find that attractive. (Not that I can't find short-haired women attractive.... but I usually like the long hair better.) This is probably related to why women's hair was traditionally covered up in all the Abrahamic religions.... Christians and Jews used to do it, too (and still do, in some parts of the world). Women who walked around with long hair in public was considered indecent, I suspect because of this. And even though most women don't do this anymore, men still find it attractive.
But I do think the example with hair shows that men would probably able to control themselves if it did happen. It would still be kind of sexual though.