Am I being too simplistic when I think that the verse in question means "peace on earth to men (or all people) of good will"? That the explicit wish to bring peace to "people of good will" was not a mistake and only bestowed upon some people, those of good will.
Also, there are many other verses confirming that peace is a gift to the few and under certain conditions and that some people just don't get peace.