This is partially correct, but it ignores that government action can exist in like-instances. To set the stage, one must recognize that the First Amendment carries costs for the State. Schneider v. New Jersey (State cannot ban leafleting on grounds that it leads to litter which the state must clean up itself. Such a cost must be borne by the state.) The state has a duty to reasonably protect a speaker's right to speak, even against a hostile audience. If the speaker is unable to exercise her right to speak in a public place due to an unruly crowd, we have what is called a "heckler's veto." The government action occurs if the government steps in to stop the speech for fear that the speaker will be harmed for speaking. Such action is an impermissible infringement on the speaker's First Amendment unless the government took reasonable action to protect the speech but was outmuscled by the crowd. Terminiello v. Chicago; more recently, Bible Believers v. Wayne County.

There was no State action here because Milo was not removed from the venue by the police, and thus there was no First Amendment violation, but that does not support your blanket statement that crowd reaction is irrelevant in determining whether a First Amendment violation occurs. That stance is wholly foreign to First Amendment jurisprudence.

For those passionate about the First Amendment, specifically in this context, I urge you to read the majority opinion in Bible Believers. Here is a link to Eugene Volokh's coverage of the decision, and inside the article you will find a link to the opinion itself. It has been praised by First Amendment scholars for providing an accurate and concise history of the issue.