I can only assume that you mean the negative view of feminism, notably the weird what is often called "tumblr feminism".
And it's the same reason why most Americans think so poorly of muslims. When your interactions with a group are limited, you base them off of what few interactions you DO have, and notably you tend to remember negative interactions far worse.
So when your average person has relatively few interactions with people openly identifying as "Feminists" and one of those interactions is someone who perhaps jumps down their throat for something that is say, unintentionally construed as sexist ...Then given that limited interaction, some people's limited view of feminism can be poor.
An example might be a girl who is in a white water rafting group getting upset when the instructor says "Are you guys ready to go white water rafting?" and she is the only girl.
While the instructor may have simply meant "Guys" in a gender neutral sense, the girl might easily see it as a slight against her. She may construe it as the instructor assuming she's not going because she's a girl.
This might have just been the final straw, perhaps her trip to Colorado has been nothing but people assuming she's a bystander, and not a participant on the mountain biking, climbing, or other activities because she's a girl. Before, she's simply corrected them and enjoyed herself.
But this time, it sets her off, and she blows up on the guy.
Now the guy, who has probably encountered maybe 5 self identified feminists in his life, has just had a terrible interaction, and probably even been called a "Misogynist" or something.
Now thanks to a failing of communication, he probably thinks shittily of feminists.
That's really assuming a best case scenario, and ignoring the people we all know exist for the sole purpose of seeking out people to be outraged by and flame.
That being said, if there were a checklist for "Being a Feminist", a vast majority of people that still claim "They are not feminists" or are even "Anti-Feminism" would likely check most if not all of the boxes on that list.
I hope that helps answer your question.