I don't think you're correct, though, that the average guy doesn't have any privilege. I know that it isn't something done on purpose, and I believe most men (and women) are good-hearted and wouldn't participate consciously. But that's why these things are so insidious. I don't believe my husband WANTS to have privilege, but he acknowledges that it exists (against his will- against most people's will, actually) and works to correct it when he can. Example: right now, today, my husband can go to his job full time and nobody on the planet will question his worth as a father. Yet when a mother of young children makes the same decision, she is subject to a chorus of "WHAT ABOUT THE CHILDREN?" Why her? Why not her husband? (Note- I'm not angry or huffy as I post this- I'm calmly, rationally, pointing out an example of privilege that goes unnoticed much of the time.)
Calling the acknowledgment of this kind of thing whining or getting in a huff or being a victim is a great way to keep propagating it.