If it's any consolation, I often wear one white and one black sock, just to see if anyone will notice. They almost never do.
I don't match my socks unless I'm wearing shoes that show them, and in that case, I just wear a pair of blacks because those are really easy to match or pseudomatch.
I have socks with all sorts of different patterns and graphics on them, and I can't be bothered to find their respective mates, so I just throw them all into a drawer and grab a pair when I'm ready to shoe up. My socks like the diversity; there's no sense in making friends with only the ones who are similar to us. Where's the fun in that?