I appreciate that you clearly put a lot of time and thought into this. Thanks for caring about reddit enough to bother! I stand by this data, and genuinely don't feel that we're spinning it.

By the way, you mention that we're "trying to validate something that is clearly unpopular." I suspect your definition of "clear unpopularity" is based on ... public commentary. This is a great example of why surveys like the one we ran are helpful. People can express opinions and concerns that they feel might be unpopular. When there are patterns in that data, we take notice.

There are a few things to consider when addressing product issues - severity and size. One might prioritize a less prevalent issue which causes horrible things to happen, over more prevalent and less severe issue (say, visual appeal.) Hence, while there might be a lower number of people who answered the question about why they wouldn't recommend reddit, or are extremely dissatisfied, its pretty important to us to know what about reddit would make them feel that way.

For that reason too, we wanted to get the opinions of more than those who follow the blog; we want to hear from the lurkers, and those who hadn't created accounts. What was holding them back?

Keep in mind that we asked all respondents what they dislike about reddit. Out of ~16k total responses, we got ~10k responses to that question. Even relatively satisfied users (those who put down 6 or 7 for overall satisfaction) can have things to dislike about the site. And the top issue was community, at 25%.

On recommendations Your interpretation that 93.5% of people would recommend reddit is simply incorrect. We did not ask whether people would or would not recommend reddit - we asked if they had in the past (asking about actual behavior is much better than predicted behavior), and provided two options for "no." It's an important distinction.

• The overall number for people who had recommended reddit is 75%.
• 17% answered that question with "No, but I might"
• 6% answered with "No, and I probably won't."
  

This is all in the spreadsheet. I suspect you may have only looked at the "No, and I probably won't" number alone, but not at the question itself (first row.)

On the lack of the words "hate" and "offensive" Had we asked about hate and offensive content specifically, that would likely add in another sort of bias, a la "Now that you mention it, I suppose I have been harassed." Those words appeared, unprompted by us, in open ended responses. Again, those responses were questions generically asking what they didn't like about reddit, and follow-ups to why people were extremely dissatisfied, and wouldn't recommend it. That so many felt so intensely about it (severity) and also that it was the top issue across those questions, speaks pretty strongly.

On selection bias (the fact that people who opt-in to surveys are different from people who take other surveys) It is certainly true that selection bias affected this survey, as it does all surveys. Some people just don't take surveys. There has been much discussion as to whether the opinions of these people are vastly different from the populace. We'll just never know. Were we to post the survey on the reddit blog as suggested here, I agree that it would get a certain set of reddit users. I disagree that they would necessarily be representative of active community members. It would simply represent those who read the blog. If you look at the data on how people use the site, a number of them just browse (and have been doing so for 3+ years), or just look at one or a few subreddits. We care about their experience, even if they don't care about the official reddit blog.

On incentivizing users to participate in surveys Providing incentives (usually money) will increase response rate, but won't really affect quality. It's also less effective over time, and we intend to continue doing surveys like this over time. Here's a good pdf.

On response rate This was a pretty long survey (thanks again to those who made it through), promoted through an ad. Online ads typically have a pretty low conversion rate. The response rate was actually a little higher than what we'd expected, and we're happy with it. Also, "Choosing not to participate," as you put it, is different from "had better things to do," wanted to read a post instead, or good old ad blindness.