I think YOUR line of thinking shows that you are sheltered and aren't putting yourself in a woman's shoes.

As a man, if you walk by two men at night, you don't immediately need to feel nervous. For one thing, you don't have anything they want, unless they indicate otherwise. And for another thing, you can feel reasonably confident in your own ability to either run away, or fight back enough to make attacking you not worth their time. The situation is vastly different for a woman. If the woman is even moderately attractive, she inherently has something that every man might want - her body. And at the same time, she likely has far less confidence in her ability to manage the situation if it goes badly. Women can't run as fast as men (and aren't typically wearing shoes which let them sprint anyways). They are also physically weaker. A woman walking by two men is both more likely to be attacked, and less likely to escape unharmed if they do attack, than you would be in the same situation.

Humans operate by considering relative danger. Travelling in a car is statistically much more dangerous than flying in a plane. Yet far more people are afraid of flying. Why is that? Because if something DOES go wrong while you're in the air, you aren't in any position to control the situation, and you're much less likely to survive than in an average traffic accident. Flying is scarier than driving, despite being technically safer.

Now apply the flying vs. driving comparison to men vs. women walking down a street at night. A man and a woman might be comparably safe, but things can go MUCH worse for a lone woman than for a lone man. That's why a woman would (and should) be nervous walking by two men at night. In a perfect world, she wouldn't have to feel nervous. But this world is not perfect, and she can't account for the intentions of the two men. As sad as it is that a woman can't just walk down a street and trust that she won't be assaulted, better safe than sorry. Better to be naturally suspicious of men (especially at night when you're alone) than to pay the price for your naiveté.