The number of Asians (both East and South) among American readers is pretty surprisingly low - 43/855 ~= 5%. This despite Asians being, e.g., ~15% of the Ivy League student body (it'd be much higher without affirmative action), and close to 50% of Silicon Valley workers.

Calibration Score

Using a log scoring rule, I calculated a total accuracy+calibration score for the ten questions together. There's an issue that this assumes the questions are binary when they're not- someone who is 0% sure that Thor is the right answer to the mythology question gets the same score (0) as the person who is 100% sure that Odin is the right answer to the mythology question. I ignored infinitely low scores for the correlation part.

I replicated the MWI correlation, but I noticed something weird- all of the really low scorers gave really low probabilities to MWI. The worst scorer had a score of -18, which corresponds to giving about 1.6% probability to the right answer. What appears to have happened is they misunderstood the survey, and answered in fractions instead of percents- they got 9 out of 10 questions right, but lost 2 points every time they assigned 1% or slightly less than 1% to the right answer (i.e. they mean to express almost certainty by saying 1 or 0.99) and only lost 0.0013 points when they assigned 0.3% probability to a wrong answer.

When I drop the 30 lowest scorers, the direction of the relationship flips- now, people with better log scores (i.e. closer to 0) give lower probabilities for MWI (with a text answer counting as a probability of 0, as most were complaints that asking for a number didn't make sense).

What about Tragic Mistakes? These are people that assign 0% probability to a correct answer, or 100% probability to a wrong one, and under a log scoring rule lose infinite points. Checking those showed both errors, as well as highlighting that several of the 'wrong' answers were spelling mistakes- I probably would have accepted "Oden" and "Mitocondria."

(Amusingly, the person with the most tragic mistakes- 9 of them- supplied a probability for their answers instead of an answer, so they were 100% sure that the battle of Trafalgar was fought off the coast of 100, which was the state where Obama was born.)

There's a tiny decline in tragic mistakes as P(MWI) increases, but I don't think I'd be confident in drawing conclusions from this data.

MIRI Mission/MIRI Effectiveness .395 (1331)

This result sets off my halo effect alarm.

Once again pandemic is the leading cat risk. It was the leading cat risk last year. http://lesswrong.com/lw/jj0/2013_survey_results/aekk It was the leading cat risk the year before that. http://lesswrong.com/lw/fp5/2012_survey_results/7xz0

Pandemics are the risk LWers are most afraid of and to my knowledge we as a community have expended almost no effort on preventing them.

So this year I resolve that my effort towards pandemic prevention will be greater than simply posting a remark about how it's the leading risk.

WHY ISN’T THERE AN OPTION FOR NONE SO I CAN SIGNAL MY OBVIOUS OBJECTIVITY WITH MINIMAL EFFORT

This is why I didn't vote on the politics question.

This is a sign that most Less Wrongers continue to neglect the very basics of rationality and are incapable of judging how much evidence they have on a given issue. Veterans of the site do no better than newbies on this measure.

Theory: People use this site as a geek / intellectual social outlet and/or insight porn and/or self-help site more than they seriously try to get progressively better at rationality. At least, I know that applies to me :).

I decided to take a look at overconfidence (rather than calibration) on the 10 calibration questions.

For each person, I added up the probabilities that they assigned to getting each of the 10 questions correct, and then subtracted the number of correct answers. Positive numbers indicate overconfidence (fewer correct answers than they predicted they'd get), negative numbers indicate underconfidence (more correct answers than they predicted). Note that this is somewhat different from calibration: you could get a good score on this if you put 40% on each question and get 40% of them right (showing no ability to distinguish between what you know and what you don't), or if you put 99% on the ones you get wrong and 1% on the ones you get right. But this overconfidence score is easy to calculate, has a nice distribution, and is informative about the general tendency to be overconfident.

After cleaning up the data set in a few ways (which I'll describe in a reply to this comment), the average overconfidence score was 0.39. On average, people expected to get 4.79 of the 10 questions correct, but only got 4.40 correct. My impression is that this gap (4 percentage points) is smallish compared to what overconfidence research tends to find, but I don't have any numbers at hand to make direct comparisons with the numbers in the published literature.

People were most overconfident on question 6 (densest planet: 18% correct, 35% average estimate) and question 10 (bestselling video game: 7% correct, 22% average estimate) and most underconfident on questions 4 (Norse God: 87% correct, 75% average estimate) and 2 (Obama's state: 82% correct, 71% average estimate).

Overconfidence was correlated with a few other variables at p<.01:

SATscoresoutof2400 -.185 (242)
SATscoresoutof1600 -.160 (329)
IQ -.157 (368)
PCryonics .116 (1112)
MinimumWage .086 (1055)

That is, people who were more overconfident had lower test scores, assigned a higher probability to cryonics working, and were more in favor of raising the minimum wage. On PCryonics, I think my comments about the cryonics questions on the 2011 Survey are related to what's going on.

Overconfidence had no significant relationships with any of the other numerical variables, including the various other probability estimates and political views, age, finger ratio, or charitable donations. It was also uncorrelated with scales measuring growth mindset and self-efficacy.

When I turned them into numbers, various measures of ties to the LW community were correlated with overconfidence in the expected direction (closer ties to LW --> less overconfident), but not at p < .01 (perhaps in part because they weren't really intended to be continuous variables). So I combined several questions about ties to LW into a simple composite variable where a person gets one point each for: having read the sequences, having joined the community before 2010, having at least 1000 karma, having read all of HPMOR, having attended a full CFAR workshop, having posted in main, regularly attending meetups, regularly interacting with LWers in person, and having a romantic partner that they met through LW. This composite variable (which ranged from 0 to 8) correlated with overconfidence at r = -.085, p < .01. In other words: people with closer ties to LW were less overconfident.

But it's probably more informative to compare means on these variables, instead of turning them into an ad hoc continuous variable. Here is the average overconfidence score among various subgroups (where the full sample was overconfident by 0.39 questions out of ten, etc.):

Everyone 0.39 (4.79 pred - 4.40 actual) (n=1141)
Read HPMOR 0.35 (4.70 pred - 4.35 actual) (n=753)
Active in-person 0.26 (4.55 pred - 4.29 actual) (n=171)
Read the sequences 0.23 (4.69 pred - 4.46 actual) (n=357)
Attended CFAR 0.15 (4.42 pred - 4.27 actual) (n=91)
High test scores 0.15 (5.14 pred - 4.99 actual) (n=260)
1000 karma 0.14 (4.98 pred - 4.83 actual) (n = 127)

(The active in-person group includes everyone who answered yes/regularly/all the time to any of the 3 questions: in-person interaction, attending meetups, or LW romantic partner. The high test scores group includes anyone who was in the top 25% of reported scores on any one of the 4 test score questions: IQ (146+), SAT out of 1600 (1560+), SAT out of 2400 (2330+), or ACT (35+).)

Compared to the full sample (which was overconfident by 0.39 questions), there was less than half as much overconfidence among people who attended CFAR, have 1000 karma, or have high test scores. Other indicators of LW involvement were also associated with less overconfidence, though with smaller effect sizes.

Note that being overconfident by 0.14 questions is a small enough gap to be accounted for entirely by a single one of the 10 questions. If we remove the video game question, for example, then the people with 1000+ karma are within 0.01 question of being neither overconfident nor underconfident. So these results are consistent with the 1000+ karma group being perfectly calibrated (although they still count as some evidence in favor of that group being a bit overconfident).

In summary: LWers show some overconfidence, probably less overconfidence than in the published literature, and there's less overconfidence among those with close ties to LW (e.g., high karma or CFAR alumni) or with high test scores. Pretty similar to what I found for other biases on the 2012 LW Survey.

Thanks for showing us that there are autistic cryonics patients in the world. I am more likely to sign up when I am old enough to legally do so without parental permission, because now I know I wouldn't be the only autistic person in the future, no matter what happens when people develop a prenatal autism test.

Myth: Americans think they know a lot about other countries but really are clueless.

Verdict: Self-cancelling prophesy.

Method: Semi-humorous generalization from a single data series, hopefully inspiring replication instead of harsh judgment :)

I decided to do some analysis about what makes people overconfident about certain subjects, and decided to start with an old stereotype. I compared how people did on the population calibration question (#9) based on their country.

Full disclosure: I'm Israeli (currently living in the US) and would've guessed Japan with 50% confidence, but I joined LW (unlurked) two days after the end of the survey.

I normalized every probability by rounding extreme confidence values to 1% and 99% and scored each answer that seemed close enough to a misspelling of Indonesia according to the log rule.

Results: Americans didn't have a strong showing with an average score of -0.0071, but the rest of the world really sucked with an average of -0.0296. The reason? While the correct answer rate was almost identical (28.3% v 28.8%) Americans were much less confident in their answers: 42.4% confidence v 46.3% (p<0.01).

Dear Americans, you don't know (significantly) less about the world than everyone else, but at least you internalized the fact that you don't know much*!

Next up: how people who grew up in a religious household do on the Biblical calibration question.

*Unlike cocky Israelis like me.

Thanks for doing this!

[This question was poorly worded and should have acknowledged that people can both be asexual and have a specific orientation; as a result it probably vastly undercounted our asexual readers]

I find the "vastly" part dubious, given that 3% asexual already seems disproportionately large (general population seems to be about 1%). I would expect for asexuals to be overrepresented, and I do think the question wording means the survey's estimate underestimates the true proportion, but I don't think that it's, say, actually 10% instead of actually 4%.

May is missing from Birth Month.

Do you have some links to calibration training? I'm curious how they handle model error (the error when your model is totally wrong).

For question 10 for example, I'm guessing that many more people would have gotten the correct answer if the question was something like "Name the best selling PC game, where best selling solely counts units not gross, number of box purchases and not subscriptions, and also does not count games packaged with other software?" instead of "What is the best-selling computer game of all time?". I'm guessing most people answered WOW or Solitaire/Minesweeper or Tetris, each of which would be the correct answer if you remove on of those restraiints.

But it seems hard to guess beforehand that the question you thought you were answering wasn't the question that you were being asked! So you'd end up distributing that model error relatively evenly over all the questions, and so you'd end up underconfident on the questions where the model was straightfoward and correct and overconfident when the question wasn't as simple as it appeared.

It looks to me like everyone was horrendously underconfident on all the easy questions, and horrendously overconfident on all the hard questions.

I think that this is what correct calibration overall looks like, since you don't know in advance which questions are easy and which ones are tricky. I would be quite impressed if a group of super-calibrators had correct calibration curves on every question, rather than on average over a set of questions.

I think that there are better analyses of calibration which could be done than the ones that are posted here.

For example, I think it's better to combine all 10 questions into a single graph rather than looking at each one separately.

The pattern of overconfidence on hard questions and underconfidence on easy questions is actually what you'd expect to find, even if people are well-calibrated. One thing that makes a question easy is if the obvious guess is the correct answer (like a question about Confederate Civil War generals where the correct answer is Robert E. Lee). On those sorts of easy questions, a bunch of people who made the obvious guess with moderate confidence will turn out to be correct, and they'll look underconfident. Whereas on questions where the correct answer is more obscure or counterintuitive, those guesses will be wrong and they'll look overconfident.

I'll see what other analyses I can do with the data. First I'll need to make some not-entirely-straightforward decisions about what to do with misspellings and other interesting responses. This may be fairly important, since those account for a decent chunk of all responses. For example, here are the most common responses to question 8 (in descending order of frequency):

Mitochondria
mitochondria
Mitochondrion
mitochondrion
Nucleus
Mitocondria
mitochondrium
nucleus
mitocondria
Mitochondrium
Mithocondria
Mitochondira
Ribosome

P Supernatural: 6.68 + 20.271 (0, 0, 1) [1386]

P God: 8.26 + 21.088 (0, 0.01, 3) [1376]

The question for P(Supernatural) explicitly said "including God." So either LW assigns a median probability of at least one in 10,000 that God created the universe and then did nothing, or there's a bad case of conjunction fallacy.

This is not a human universal - people who put even a small amount of training into calibration can become very well calibrated very quickly. This is a sign that most Less Wrongers continue to neglect the very basics of rationality and are incapable of judging how much evidence they have on a given issue. Veterans of the site do no better than newbies on this measure.

Can someone who's done calibration training comment on whether it really seems to represent the ability to "judge how much evidence you have on a given issue", as opposed to accurately translate brain-based probability estimates in to numerical probability estimates?

This was the first time that I failed to take the survey, because I kept going "I don't have a scanner at home for the finger ratio thing, so I'll do it when I'm at the university" and then each time when I was at the university, I forgot. (I wasn't there often.)

Yayy! I was having a shitty day, and seeing these results posted lifted my spirits. Thank you for that! Below are my assorted thoughts:

I'm a little disappointed that the correlation between height and P(supernatural)-and-similar didn't hold up this year, because it was really fun trying to come up with explanations for that that weren't prima facie moronic. Maybe that should have been a sign it wasn't a real thing.

The digit ratio thing is indeed delicious. I love that stuff. I'm surprised there wasn't a correlation to sexual orientation, though, since I seem to recall reading that that was relatively well-supported. Oh well.

WTF was going on with the computer games question? Could there have been some kind of widespread misunderstanding of the question? In any case, it's pretty clearly poorly-calibrated Georg, but the results from the other questions are horrendous enough on their own.

On that subject, I have to say that even more amusing than the people who gave 100% and got it wrong are the people who put down 0% and then got it right -- aka, really lucky guessers :P

Congrats to the Snicket fan!

This was a good survey and a good year. Cheers!

Any correlation between digit ratios and gender default?

For the warming question, much of the difference between responses is likely due to differing judgments on what is "significant" as opposed to on what will/has been actually happening. I think it would be better to try to split these components. A first question could ask for example, how much do you think human influence (e.g. CO2 emitted due to human activity) has raised the temperature at the current moment (give a value that you think has a 50% chance of being above or below the correct value), also how much it will have raised the temperature in 2050 and 2100, this could be preceded by a graph showing temperature change due to all causes for the last 80 years or so, to provide a common anchor point. Then one or more additional questions could be aimed at how big a problem people think it is and how much they think should be done about it.

I would be really interested in hearing from one of the fourteen schizophrenic rationalists. Given that one of the most prominent symptoms of schizophrenia is delusional thinking, a.k.a. irrationality... I wonder how this plays out in someone who has read the Sequences. Do these people have less severe symptoms as a result? When your brain decides to turn against you, is there a way to win?

I also find it fascinating that bisexuality is vastly overrepresented here (14.4% in LW vs. 1-2% in US), while homosexuality is not. My natural immediate interpretation of this is that bisexuality is a choice. I think Eliezer said once that he would rather be bisexual than straight, because it would allow for more opportunities to have fun. This seems like an attitude many LW members might share, given that polyamory a.k.a. pursuing a weird dating strategy because it's more fun is very popular in this community. (I personally also share Eliezer's attitude, but unfortunately I'm pretty sure I'm straight.) So to me it seems logical that the large number of bisexuals may come from a large number of people-who-want-to-be-bisexual actually becoming so. This seems more likely to me than some aspect or correlate of bisexuality (and not homosexuality) causing people to find LW.

Alternatively, and now that I think about it probably more realistically, perhaps the vast majority of people in America who are attracted to two genders decide to keep their same-sex attraction to themselves, concluding (arguably rationally) that the added sexual opportunities aren't worth the stigmatization. However, LW members are more likely to be unashamed of being weird, and also more likely to socialize e.g. with a bunch of nerds in the Bay Area, meaning that the risk of stigmatization is much lower.

Or perhaps the true answer is some sort of combination of the two I just postulated.

[Poor calibration] is not a human universal - people who put even a small amount of training into calibration can become very well calibrated very quickly.

Is there a source on this?

I'm losing a lot of confidence in the digit ratio/masculinity femininity stuff. I'm not seeing a number of things I'd expect to see.

First, my numbers for correlations don't match up with yours. With filters on for female gendered, and answering all of BemSexRoleF, BemSexRoleM, RightHand, and LeftHand, I get a correlation of only -0.34 for RightHand and BemSexRoleM, not -0.433 as you say. I get various other differences as well, all weaker correlations than you describe. Perhaps differences in filtering explain this? -.34 vs -.433 seems to be high for this to be true though.

Second, Bem masculinity and femininity actually seem to have a positive correlation, albeit tiny. So more masculine people are... more feminine? This makes no sense and makes me more likely to throw out the entire data set.

Thirdly, I don't see any huge differences between Cisgender Men, Transgender Men, Cisgender Women, or Transgender Women on digit ratios. I would expect to see this as well. I get 95% confidence intervals (mean +/- 3*sigma/sqrt(n), formatted [Lower Right - Upper Right / Lower Left - Upper Left]) for the categories as follows:

• F (Cis): [0.949 - 0.996 / 0.956 - 1.004]
• M (Cis): [0.962 - 0.978 / 0.963 - 0.979]
• M (Trans): [0.907 - 0.988 / 0.818 - 1.070]
• F (Trans): [0.935 - 1.002 / 0.935 - 1.019]

There's pretty significant overlap between all 4 categories. I made a dotplot that I can't upload and it doesn't look to me like there's any difference in the distributions, but I don't think we have enough of a sample size to have a meaningful distribution on anything except cis males and maybe cis females.

Lastly, I guess I just skipped over the favorite LessWrong post? Not sure what I would have answered, but it would have been either When None Dare Urge Restraint or Intellectual Hipsters and Metacontrarians. Surprised to see neither of those on the list.

Edit: As always, thanks for doing this!

So I noticed a few people from Portugal in the public raw data. I am one of those people and from what I saw in the data, the remaining are all lurkers.

Viva pessoal! Want to join the community, even if just to say hi? =)

Thanks for all the hard work!

SAT Scores out of 1600/SAT Scores out of 2400 .844 (59)

I'm surprised that this correlation wasn't higher. They're both pretty much the same test, right?

Three explanations I thought of:

1. I'm missing something/I have an inaccurate model of the difference between the two tests.

2. There's a lot of random difference between SAT scores from different testings. If this is true, I would expect there to be a correlation of around .844 between one test score and a later test score under the same grading system.

3. SAT scores are correlated with age (no idea whether this is true or not) and people take the two tests some time apart, and thus have better scores on the second.

Any ideas?

It looks like the median age is 27.67, but I'm curious to see the age-range breakdown as I've frequently assumed I'm "old" for the group (over 40). Oh....never mind.....just saw the link to the Excel spreadsheet and will sort it myself.

I'd like to see the correlations between Bem Sex Roles and actual Gender answers.

Among the correlations the first I found surprising are

• Minimum Wage/Feminism .378 (1286)

• Immigration/Feminism .365 (1296)#

I wouldn't have guessed these from the corresponding memeplexes and can't see a plausible relationship. Anybody volunteer?

Good job on running the survey and analyzing the data! I do wish that one of the extra credit questions had asked whether or not readers were fans of My Little Pony: Friendship is Magic.

This post had more statements of the type “p < 0.01” than I would expect at LW. I recently read “Frequentist Statistics are Frequently Subjective” here.

I only identify with my birth gender by default: 681, 45.3%

I'm surprised at this. Is there a special term for "only identifying with one's gender by default" or keywords I can use to look for statistics for among the general population? (a brief googling didn't uncover anything). I would've guessed this number to be much lower, and now I'm wondering whether this is signaling or whether my model of other people in this particular instance is completely wrong.

Thanks Alex and Ozy!

To compare, are there any public stats on LessWrong readership, such as how many new and returning visitors the site gets?

It looks to me like everyone was horrendously underconfident on all the easy questions, and horrendously overconfident on all the hard questions.

Have you ranked questions by their easiness before you looked at the results?

I want to be friends with the write-in worshiper of CelestAI mentioned :) PM if you like!

Under the profession listings, it says 35 people and 4% for Business. 35 is 2.7% of 1500.

Do I understand correctly that a more masculine finger ratio correlated strongly to support for feminism in both men and women?

I am also amused to note that, despite our extreme sex ratio, our BEM gender masculinity and femininity are almost exactly equal -- way below error.

It's interesting to compare these results to those of the 2014 Survey of Effective Altruists. These will be released soon, but here are some initial ways in which effective altruists who took this survey compare to LessWrong census takers:

• Somewhat less male (75% vs 87%)
• More in the UK
• Equally atheist/agnostic
• More consequentialist (74% vs 60%)
• Much more vegan/vegetarian (34% vs 10%)
• Witty, attractive, and great company at parties

What's up with a whole 10% being 'Atheist and spiritual'? It doesn't seem to be a family thing, as you get only 4.9% with that belief in the family section, and the numbers don't match up with the P(Supernatural) question.

I was worried about this last year when it was 8.1%, and the number seems to be increasing. Is this Will Newsome's post-rationality faction or what?

As always, thanks for doing this.

people can both be asexual and have a specific orientation

Huh? This is worded as a question about orientation rather than practice, so people who have an orientation have an orientation, no? Or is the issue something else?

Why are some results ordered and some listed alphabetically? That's a bit confusing.

What's this easy becoming calibrated through training? Bonus for a mainstream-ish source rather than LWsphere.

The questions about sexual orientation and gender identity seem to be quite well thought-out, as I would expect, but I have some suggestions in that regard for next year's survey:

1) Include the option to report as a man who has sex with men or a woman who has sex with women.

This could either be incorporated into the question of sexual orientation, or, perhaps more appropriately, because it is not technically a sexual orientation, it could be included as a standalone question.

2) Include the option to report one's sexual orientation using the Klein Sexual Orientation Grid.

If the Grid is too complex, the Kinsey scale is a simpler alternative. However, this survey is already so complex that I fail to see how that case could be made.

Rationale for 1):

Warning: Long anecdote! TL;DR is below. Inexorably, it comes with the territory of men who self-report as MWHSWM. Even though I am male, I have a lot of fun feeling feminine. This was the reason that I began to have sex with other men, rather than any attraction to the men themselves (although I have actually grown to appreciate the male form over the years, and now I do become aroused when I see a penis or a fit, well-dressed man.) I have always had very nice hair, and am usually too lazy to cut it on a regular basis; I am spindly; and when I was a young teenager and my bones were still growing and my shoulder-to-hip ratio was closer to 1:1, in the right clothes I could pass for a moderately attractive girl. I certainly cannot be mistaken for a woman at this age, but sometimes people mention in passing that in subtle ways I am quite feminine in appearance for a male, and not in a disparaging way; I enjoy that! Yet, I have never wanted to have hormone treatments or sex reassignment surgery. There are days when I enjoy feeling masculine, and I have never felt trapped by my male form. Oral and anal sex are enough for me anyway; even if receiving vaginal sex is fun to imagine, I consider it hardly worth losing the ability to penetrate with an innervated sex organ! That's what lucid dreams are for. My gender identity has always been very fluid. Should certain future events come to pass, that fluidity will likely become (virtually) physical. Reporting as a MWHSWM comes into play for me when I consider how to relate this to someone else. LessWrong is one of few places where I could explain all of this. There are many places where people would become repulsed or bored (maybe some of you are) before I finish my paragraph-long explanation of my sexuality. There are other places where people would think I was weird. (And there are still more where people would plot to slaughter me.) For this reason, if I can get away with it safely, I truncate this to: "I'm bisexual." This is inarguably a lossy compression, and one for which there is no need here.

TL;DR: Men who identify as heterosexual may self-report as bisexual and skew the results if they can't self-report as a heterosexual man who has sex with men.

Rationale for 2):

The Grid considers how sexual preferences have changed over time, and even what sexual preferences one prefers oneself to have, among many other things. I think that we could get some really interesting results from that. Someone said in this comment that Eliezer has said that although he identifies as heterosexual, he actually has a preference to be bisexual because there are more opportunities for fun! That's just one person and one example.

I think one logical correlation following from the Simulation Argument is underappreciated in the correlations.

I spotted this in the uncorrelated data already:

• P Supernatural: 6.68 + 20.271 (0, 0, 1) [1386]

• P God: 8.26 + 21.088 (0, 0.01, 3) [1376]

• P Simulation 24.31 + 28.2 (1, 10, 50) [1320]

Shouldn't evidence for simulations - and apparently the median belief is 10% for simulation - be evidence for Supernatural influences, for which there is 0% median belief (not even 0.01). After all a simulation implies a simulator and thus a more complex 'outer world' doing the simulation and thus disabling occams razor style arguments against gods.

Admittedly there is a small correlation:

• P God/P Simulation .110 (1296)

Interestingly this is on the same order as

• P Aliens/P Simulation .098 (1308)

but there is no correlation listed between P Aliens/P God. Thus my initial hypothesis that aliens running the simulation of gods being the argument behind the 0.11 correlation is invalid.

Note that I mentioned simulation as weak argument for theism earlier.