Comments on “Pop Bayesianism: cruder than I thought?”

David -

This conversation has moved in interesting ways since I was last here!

Your comments suggest you are looking at "pop-Bayesianism" through the lens of your own work on meaningness, which is probably unfair, at least as far as groups like Less Wrong are concerned.

Meaningness is about meaning(lessness), value, purpose, significance. Bayes (actually now I think they are all about Solomonoff Induction as The Answer, which I think has pretty obvious problems that are being overlooked) is about drawing conclusions and making decisions under conditions of uncertainty about FACT, not value.

Now, these groups are doing work on ethics elsewhere (in this area I think many are hindered by the fact that they consider the philosophical tradition wrongheaded and beneath them and so ignore the many contributions other thinkers have made on the same topics they are exploring), but at least at their more basic/fundamental levels, these Rationality groups are trying to formulate methods to come to more correct beliefs and to more effectively achieve one's ends (they do not necessarily specify what these ends should be).

You stated,

"What Bayesianism claims to offer is an optimal response to nebulosity."

I would say this is untrue, or at least only true in the area of factual uncertainty (vs. value or purpose or meaning -type ambiguities).

"That makes it appealing if you can mis-use it to blind yourself to nebulosity in general. "Situation is ill-defined, unstable, complex, mostly unknown? No problem! We'll just apply Bayesianism, and that guarantees we'll take the best possible action!""

Do you think it is a priori impossible to develop best-practices for decision making under conditions of uncertainty?

Less Wrong-types think that it is possible, and they are trying to build it. Interpreting them charitably (and I think accurately), I would say they accept that there are problems with human cognition (heuristics & biases), they want to do better, they need a standard for what better means, and they use mathematical probabilities (with Bayes as the poster child) as that ideal standard. Their question is how to get real humans in real-world situations closer to that ideal. Currently there's a muddle in the middle.

Another way to say it would be that they are trying to apply the theoretical insights of (aspects of) probability theory and cognitive psychology in real life and/or daily life. A very different project than yours.

So is there meaning to life or not?

no not really

@ David

You said:

I hope to provide a box of tools for atheists to identify and free themselves from such ideologies, by going to their root—the stances, and emotional needs they address—rather than arguing with belief systems.

It is an excellent project. On my site, I flip between 'arguing with belief systems" so as to point them to their root emotional uses (or pragmatic uses), and then 'going to theirour shared roots' hoping to point how various belief systems can address them. (see my last diagram).

So I agree with your project, but do feel arguing belief systems can be useful. And that being specific/concrete can be useful.

Seque:

I love your reply to Jason Clark on the question "Is there meaning to life or not?" And especially your approach that understanding a person's point-of-view can help us make more meaningful replies -- speaking in the abstract can often be a waste of time.

But some additional thoughts:

(1) The mistaken notion that there is some THING called "Life" has been clear to me for a long time. We are so easily fooled by our own intention -- language. But to say "life is too complicated" to have meaning seems odd -- because complicated things have meaning all the time. Perhaps a persons life is not a "purpose intended thing" and thus can't be evaluated as a whole to have meaning, much like a play or a novel. Only a mythologizer can accomplish that.

(2) I have no understanding of what you mean by "meanings are not purely subjective" unless by "meaning" you mean "patterned relationships." Which is a different sense than the normal nuance implied in "The Meaning of Life", I think. Point: "Meaning" has a lot of senses that gets a conversation into knots quickly.

I don't think it contributes much to the discussion, but I found interesting that it wasn't mentioned that Thomas Bayes was a Presbyterian minister.

From Wikipedia:

He is known to have published two works in his lifetime, one theological and one mathematical:

1- Divine Benevolence, or an Attempt to Prove That the Principal End of the Divine Providence and Government is the Happiness of His Creatures (1731)

2- An Introduction to the Doctrine of Fluxions, and a Defence of the Mathematicians Against the Objections of the Author of the Analyst (published anonymously in 1736), in which he defended the logical foundation of Isaac Newton's calculus ("fluxions") against the criticism of George Berkeley, author of The Analyst

By the way, I studied probability (and Bayes' theorem) in high school, and I liked it, but it was pretty obvious that it was pretty useless in everyday life (as is almost everything teached in high school, but that's another issue)

Also, the formula is almost never useful in everyday life.

On the other hand, once you understand the basic principles of probability,
Bayes’ Rule is obvious, and not particularly important.

I disagree. Although it's true that the formula is almost never useful in everyday life in the sense that you'd need to do any calculations with it, it's very useful to understand the extent to which it's crucial for all your reasoning processes.

For example:

• Understanding Bayes' Rule helps clarify just what kinds of things should count as "strong evidence" - it tells you that Y is strong evidence about X being true to the extent Y is the kind of thing that you would only see if X were true (i.e. only X and nothing else would make you see Y). That's a very generally applicable rule that's useful for evaluating the correctness of a lot of different things. Yes, it is certainly possible to understand the above even without knowing the exact formula, but at least I personally found that knowing the exact reason (read: the exact math) for that rule made it feel like I understood it better.
• It also helps understand why people tend to hear what they expect to hear instead of what the other speaker is saying: see e.g. http://lesswrong.com/lw/hv9/rationality_quotes_july_2013/9alt . Not only that, it makes me personally more aware of the way in which I personally might misunderstand the claims or experiences of others, and reminds me to consciously consider alternative explanations to somebody's words/motives besides just the first explanation that pops to mind, since I know that my
  priors may or may not be correct.
• http://lesswrong.com/lw/2el/applied_bayes_theorem_reading_people/ (not very happy with this post, but it should get the rough ideas across)
• And generally it helps me remember that in order for my beliefs to be accurate, I have to update them in a way that actually makes them correspond with reality, and Bayes' theorem shows many of the necessary preconditions for that.

That said, I do agree that just saying "Bayes' theorem!" is often unhelpful, and that one would instead need lots of worked examples of its implications to make it theorem really useful. (And maybe you could just give the examples and skip the formula entirely.) I tried to do something like that with the "What is Bayesianism" article as well as the "Applied Bayes' Theorem" one, but I don't think I did very well on either.

Hi David,

your offered formulation isn't quite what I was after, though I do appreciate the attempt to find a sympathetic interpretation. Let me try to rephrase...

If I understood you correctly, you said that "pop Bayesianism" doesn't seem to make sense, because it claims to use Bayes' Rule - which is a rule for calculating probabilities with numbers - despite almost never actually invoking any specific numbers. Would any of these examples make more sense to you?

• A physicist, after having learned Newton's laws of motion, knows that his infant child should be tightly secured while in a car, because the child will continue its motion even during a sudden stop, and won't be easily held in place.
• A computer scientist, after having learned the formal definitions for computational complexity classes and had some experience with applying them, finds that knowledge useful in doing informal guesses of how hard some problem might be, despite never running any numbers while doing so.
• A philosopher, after having learned formal logic, intuitively recognizes patterns of logic in people's statements, even without doing a formal analysis on them.

What I'm trying to convey here is the notion that once you have learned a formal rule describing some phenomenon, then you will start recognizing the structure described by that rule even in situations where you don't have the exact numbers... that even if you can't do an exact calculation, knowing the formalism behind the exact calculations will make it possible to get a rough hunch of how the thing in question could be expected to behave, given the formalism and your intuitive guesses of the general magnitude of the various numbers involved. The physicist might not bother to calculate the exact speed at which an unsecured child would be moving during a sudden stop, but he knows that it is far too fast.

The examples in my previous post, then, were intended as examples of ways in which knowing Bayes' Rule and having some experience of some of its applications lets you recognize the "Bayes structure" of things and get a rough handle of how they behave, even when you can't do the exact numbers.

All of that said, I do admit that I am still not personally sure of exactly how much of the "pop Bayesian" influence on my thought has actually come from an understanding of the actual Bayes' rule itself, since I've only spent a rather limited time actually plugging in numbers to it. It could be that what most benefited me were the various qualitative examples of how to reason in a probabilistic manner, and general discussions about things like priors, the origins of our knowledge, and what counts as valid evidence. I would guess the same to be the case with many other "pop Bayesians", so you could be right that it's more of a symbol than anything.

Still, I do think that there is value in also teaching the rule itself, since it can help make the various informal arguments more clear and concrete...

The claim that "knowing Bayes' Rule is heuristically valuable even in the absence of numbers" is empirical. Is there any evidence for it?

All of my evidence is purely anecdotal, with all the confidence that that implies. :-)

However, I think that teaching the "balls in boxes" (frequentist) formulation for probability would probably be much better. Once you understand that formulation, it's easy to see how to apply it to many different sorts of probabilistic circumstances.

Oh, I've been implicitly presuming all along that the target audience already knows the very rudiments of probability, and the basic frequentist formulation of it. Though now that you point it out, that's probably an artifact of me spending excessive amounts of time in geek circles, and not very representative of the population at large...

But yeah, if I personally were to write an introduction to these kinds of things now, I might not even mention Bayes' rule until some later "advanced course".

Well, hmm, it seems to me that LW has built an enormous edifice, or castle in the air, on anecdotal evidence for the efficacy of Bayesianism. Oughtn't that to bother those in the movement? Especially since evidence is the sacred principle of the movement?

When you're saying "Bayesianism", here, are you referring specifically to using Bayes' rule as an instruction tool, or the whole broader emphasis on how to use probabilistic thinking? What I meant to say was that I admit that the specific claim of "knowing Bayes' Rule is heuristically valuable even in the absence of numbers" doesn't necessarily have strong support. But "Bayesianism" in the LW sense covers much more than that, and is more broadly about probabilistic thinking and e.g. the nature of evidence in general...

And from what little contact I've had with them, CFAR's agenda is even more broad - one of their instructors was in town and trialed a small workshop that was basically about staying calm in an argument and being able to analyze each person in the argument actually wanted, so that you could deal with the ultimate issues instead of degenerating into a shouting match. Of course, they probably wouldn't claim that that falls under "Bayesianism", but I don't think they'd claim that Bayes' Rule is all you need to know about rationality, either. (Though I don't know CFAR that well.)

Does the average LW reader actually understand classical decision theory and how to apply it in real-world situations? I now think: probably not.

I would guess so, too. Though of course, the average LW reader is also likely to be a lurker, and less likely to be loudly praising Bayesianism; but the statement may quite likely still hold even if you change it to refer to the average LW commenter. Then again, I'm not sure of the extent to which the average LW commenter will be loudly praising Bayesianism, either... actually, now that I think of it, Yudkowsky is the only one whose LW writings I've seen explicitly saying that Bayesianism is something great and fantastic and incredible.

The closest thing that LW has to an introduction to classical probability is probably Eliezer's Intuitive Explanation of Bayes' Theorem, though I think that it already assumed some understanding of what probability is; I'm not entirely sure.

One thing that's worth keeping in mind is that LW's definition of "Bayesianism" is somewhat idiosyncratic. While Eliezer does often make disparaging comments about frequentist statistics, the main ideas aren't so much about how to apply formal Bayesian statistics - indeed, formal statistics of either kind haven't been very much discussed on LW. What LW calls "Bayesian" is more of a general mindset which says that there exist mathematical laws prescribing the optimal way of learning from the information that you encounter, and that although those laws are intractable in practice, you can still try to judge various reasoning processes and arguments by estimating the extent to which they approximate the laws.

"Frequentists" or "non-Bayesians" in LW lingo doesn't refer so much to people who use frequentist statistical methods, but to people who don't think of reasoning and probability in terms of laws that you have to obey if you wish to be correct. (Yes, this is a confusing way of using the terms, which I think is harmful.) For example, from Eliezer's Beautiful Probability:

And yet... should rationality be math? It is by no means a foregone conclusion that probability should be pretty. The real world is messy - so shouldn't you need messy reasoning to handle it? Maybe the non-Bayesian statisticians, with their vast collection of ad-hoc methods and ad-hoc justifications, are strictly more competent because they have a strictly larger toolbox. It's nice when problems are clean, but they usually aren't, and you have to live with that.

After all, it's a well-known fact that you can't use Bayesian methods on many problems because the Bayesian calculation is computationally intractable. So why not let many flowers bloom? Why not have more than one tool in your toolbox?

That's the fundamental difference in mindset. Old School statisticians thought in terms of tools, tricks to throw at particular problems. Bayesians - at least this Bayesian, though I don't think I'm speaking only for myself - we think in terms of laws.

Looking for laws isn't the same as looking for especially neat and pretty tools. The second law of thermodynamics isn't an especially neat and pretty refrigerator.

The Carnot cycle is an ideal engine - in fact, the ideal engine. No engine powered by two heat reservoirs can be more efficient than a Carnot engine. As a corollary, all thermodynamically reversible engines operating between the same heat reservoirs are equally efficient.

But, of course, you can't use a Carnot engine to power a real car. A real car's engine bears the same resemblance to a Carnot engine that the car's tires bear to perfect rolling cylinders.

Clearly, then, a Carnot engine is a useless tool for building a real-world car. The second law of thermodynamics, obviously, is not applicable here. It's too hard to make an engine that obeys it, in the real world. Just ignore thermodynamics - use whatever works.

This is the sort of confusion that I think reigns over they who still cling to the Old Ways.

No, you can't always do the exact Bayesian calculation for a problem. Sometimes you must seek an approximation; often, indeed. This doesn't mean that probability theory has ceased to apply, any more than your inability to calculate the aerodynamics of a 747 on an atom-by-atom basis implies that the 747 is not made out of atoms. Whatever approximation you use, it works to the extent that it approximates the ideal Bayesian calculation - and fails to the extent that it departs.

Bayesianism's coherence and uniqueness proofs cut both ways. Just as any calculation that obeys Cox's coherency axioms (or any of the many reformulations and generalizations) must map onto probabilities, so too, anything that is not Bayesian must fail one of the coherency tests. This, in turn, opens you to punishments like Dutch-booking (accepting combinations of bets that are sure losses, or rejecting combinations of bets that are sure gains).

You may not be able to compute the optimal answer. But whatever approximation you use, both its failures and successes will be explainable in terms of Bayesian probability theory. You may not know the explanation; that does not mean no explanation exists.

So you want to use a linear regression, instead of doing Bayesian updates? But look to the underlying structure of the linear regression, and you see that it corresponds to picking the best point estimate given a Gaussian likelihood function and a uniform prior over the parameters.

You want to use a regularized linear regression, because that works better in practice? Well, that corresponds (says the Bayesian) to having a Gaussian prior over the weights.

Sometimes you can't use Bayesian methods literally; often, indeed. But when you can use the exact Bayesian calculation that uses every scrap of available knowledge, you are done. You will never find a statistical method that yields a better answer. You may find a cheap approximation that works excellently nearly all the time, and it will be cheaper, but it will not be more accurate. Not unless the other method uses knowledge, perhaps in the form of disguised prior information, that you are not allowing into the Bayesian calculation; and then when you feed the prior information into the Bayesian calculation, the Bayesian calculation will again be equal or superior.

When you use an Old Style ad-hoc statistical tool with an ad-hoc (but often quite interesting) justification, you never know if someone else will come up with an even more clever tool tomorrow. But when you can directly use a calculation that mirrors the Bayesian law, you're done - like managing to put a Carnot heat engine into your car. It is, as the saying goes, "Bayes-optimal".

The power in that mindset, I would say, is that you can no longer just believe in something or think something and just automatically assume that it's correct. Instead, you are forced to constantly question and evaluate your thought processes: is this the kind of an inference that would actually cause me to have true beliefs?

As for your question of how you actually use it... in my original comment I gave some examples of ways to check your reasoning processes by checking whether they follow Bayes' rule. There are a bunch of other LW articles that apply either Bayes' rule or the more general mindset of lawful reasoning. It feels a little rude to throw a dozen links at someone in a conversation like this, but since you asked for examples, some semi-randomly picked ones: Absence of Evidence is Evidence of Absence, Conservation of Expected Evidence, Update Yourself Incrementally, One Argument Against An Army, What is Evidence?, What Evidence Filtered Evidence?, and Privileging the Hypothesis (I'll single out "What is Evidence?" as one that I particularly like, and "Privileging the Hypothesis" points out a fallacy that I often started to realize I was guilty of, after I read the article).

I wouldn't agree with your characterization of Bayesianism - at least in its LW version - offering certainty, however. Yes, it talks about laws that might lead us to the truth if we follow them... but it also makes a strong attack on various ways to rationalize pleasant beliefs to yourself, and undermines the implicit notion of "I can just believe whatever I want" that many people subconsciously have, even if they wouldn't admit it out loud.

This undermining happened to me - for example, I would use to have beliefs that I didn't want to talk about in public because I knew that I couldn't defend them, but reading LW for sufficiently long made me actually internalize the notion that if I can't defend it, I don't have any reason for believing in it myself. That might sound obvious, but it is a lot easier said than done.

The message I get out of LW-style Bayesianism, and what a lot of people seem to get out of it, is rather one of massive uncertainty - that we cannot know anything for sure, and that even if we make our best effort, nothing requires the universe to play fair and not screw us over anyway... certainly to me, it feels like reading LW made me become far, far less certain in anything.

As for your question of "where did the non-probability parts of rationality go" - well, I've only been discussing the probability parts of rationality because those were the topic of your original post. Certainly there's a lot of other rationality discussion on LW (and CFAR), too. Though LW has traditionally been most focused on epistemic rather than instrumental rationality, and probability is a core part of epistemic rationality. I gather that CFAR is more focused on instrumental rationality than LW is. I would assume that this "Checklist of Rationality Habits" on CFAR's website would be more representative of the general kind of stuff they do.

David,

I thought you might appreciate some additional comments; for background, I'm the guy that wrote LW's introduction to Ron Howard-style decision analysis. LWers do seem to appreciate discussion of the technical bits, though I don't think everyone follows them.

As someone who understands all the technical detail, I agree with you that the quasi-religious aspects of LW are troubling. But I think a lot of that is that it's more fun for EY to talk that way, and so he talks that way, and I don't think that it's a significant part of the LW culture.

I think the actual quantitative use of Bayes is not that important for most people. I think the qualitative use of Bayes is very important, but is hard to discuss, and I don't think anyone on LW has found a really good way to do that yet. (CFAR is trying, but hasn't come up with anything spectacular so far to the best of my knowledge.)

I think that tools are best when they make the hard parts hard and the easy parts easy, which I think Bayes is good at doing and non-Bayesian tools are bad at doing. With Bayes, coming up with a prior is hard (as it should be) and coming up with a decision rule for the posterior is hard (as it should be). With, say, a null hypothesis test with p=.05, the prior and decision rule are both selected at default. Are they appropriate? Maybe, maybe not- but such discussions are entirely sidestepped without Bayes, and someone can go through the motions without realizing that what they're doing doesn't make sense. (This xkcd strip comes to mind. Yes, they used the wrong test- but is it obvious that they did so?)

For instance, I don't believe there is any physicist who deduced that he should use a child's car seat (even without numbers). You do that because everyone else does, because it's legally required, and because it is an obviously good idea, based on your own felt experience with seatbelts and cars stopping suddenly.

You might be interested in this anecdote, in which a physicist explains to a mother what deceleration is like. The relevant bit:

“But I’m holding on to him!”, the woman started saying, but by the time she made it to the end of her sentence, she was full-on screaming, completely losing her cool. “Nothing bad is going to happen to him!!”

“Right, please hear me out,” Syd got them back on track. “Say that you go from 30 miles per hour to zero in the space of about a foot and a half, right? That is an acceleration of about 20 times the earth’s gravity. That means that your 1-stone baby would go from 1 stone of weight in your arms, to about 20 stone moving away from you.”

The woman just stared at Syd, but her eyes showed that she was trying to envision holding on to a 20-stone baby.

“I guess what I am asking: Would you be able to hold a 20-stone item the size of a large watermelon in your arms in the middle of a car crash?”

“I…”, she said.

“Be honest.” he said. “No, you wouldn’t. You wouldn’t have a chance. Which means that if you guys had been in a crash, your baby would have gone flying straight into the windshield. There, he wouldn’t have had the benefit of being slowed down gradually: Instead of being slowed down by the seat belt over a distance of about a foot and a half, he would be brought to a stop in an inch or less. Then…”

Syd was on a roll, and I could tell that he was about to launch into a further explanation. He was so caught up in his own discussion, that he hadn’t seen how pale the woman had gotten. I shook my head at him. He spotted me, and stopped his train of thought.

“You get the picture; I don’t need to explain. Suffice to say that it’s unlikely that your baby would survive an impact like that.”

The woman went even more pale, and was hugging her child closely. She looked, for a moment, as if she might keel over, but her husband stepped in and put an arm around her.

Hey David, you have quite a lot of thought-provoking stuff here on this website. I found a link to this article while browsing Lesswrong, and, after seeing you display high level of sanity in comments, have read through most of your "book contents".

Unfortunately, I can't quite seem to make sense of your core ideas and I'd appreciate you clarifying a few points.

• What is your definition of the word "meaning" that you use throughout the book? I have tried substituting both my own intuitive definition and the dictionary one - both substitutions seem to result in nonsense.
• Are stances (within one specific dimension) discrete categories of thinking that each person falls squarely in, or are the two confused stances bounds of the worldview distribution, with most people being in-between?
  If it's the continuous case, is the complete stance just a roughly 50% A 50% B mix? You describe the completes like they are transcendent with respect to confuseds, not merely a partway point.

  It seems to me that the discrete case is what you believe in, but the continuous one is what fits my real-world experiences the best.

• You go in detail on each of the complete stances, but are they the only/the only true compromises between confuseds? In most dimensions, I partially agree with/compromise between/partially accept both confuseds, and yet my worldviews of those dimensions look nothing like your descriptions of the complete stances
• Are your views falsifiable? Can you think up and tell us a few specific examples of evidence that, if observed, would cause you to abandon your whole worldview of "meaningness"?
• The whole worldview kind of looks like yet another attempt at the Answer to Life, the Universe, and Everything, but you said you don't like those. What am I missing here that makes it special?

As someone who lurks on LW without a background in classical statistics, I thought you might find my experiences helpful.

The "small, practical insights" are great! Unfortunately they are often found within walls of not-small text, inter-mixed with personal narrative and quasi-religious language. It all seems a bit cult-ish, which Eliezer obviously realizes and pokes fun at. However the feeling remains that he hasn't engaged the wider scientific community and defended his beliefs.

The lack of discussion of non-bayesian methods is especially troubling. I rarely use much statistics in my work (software engineering); when I do I look up specific tools which apply to the task at hand, use them, then largely forget about them. I just don't use statistics often enough to stay skilled in it. So when I encountered LW I thought "oh neat, they've found a simpler, more general theory of probability I can more easily remember".

So I started fiddling with and thinking about Bayes' Rule, and it seemed like you could infer some important insights directly from it. Things like "absence of evidence is evidence of absence" (which we already felt, but its nice to know), and how priors influence our beliefs as new evidence is acquired.

Useful stuff, I thought. But without any contrast to classical methods, I don't know how unique or event valid these insights are.

When I need to Get Something Done that requires I learn a concept in statistics, decision theory or rationality (this has happened all of twice), I will always consult wikipedia and its peer-reviewed sources before LW. LW feels good for provoking thought and discussion, but not actually getting stuff done. In the Hansonian sense it feels "far", while sources such as wikipedia are "near".

I just watched Galef's brief video, and I must say the point of here talk seemed to me to be something that you have apparently totally missed. What she is talking about is a formal but flawed mode of reasoning used frequently by scientists, and something that happens all the time when people use informal reasoning. It's called the base-rate fallacy, it's crappy reasoning, and when you know what Bayes' theorem means, it's obviously wrong: P(H | D) is not the same as P(D | H). This is what her example about being or not being a good driver was about - its not enough that the hypothesis fits the data, you must look at how other hypotheses fit the data also.

For your convenience, regarding the base-rate fallacy, see my brief article
http://maximum-entropy-blog.blogspot.com/p/glossary.html#base-rate-fallacy
and linked material.

(By the way, the commenter above me, Marc, complains that Bayes' theorem doesn't specify a hypothesis space. This is correct, it's called theory ladeness, and its just something we have live with - no other procedure can provide this, or any alternative. To complain about not having the hypothesis space laid out in detail is to complain about not being omniscient. If there was some unique, true hypothesis space, attainable by some procedure, then what would it look like? Why would it contain any hypotheses beyond the one true proposition? Where would this miraculous information come from?)

Hi,

I agree with the pedagogical issues with teaching Bayes, and the issue of the worship of the all mighty Bayes rule.

However, you mention that pop Bayesianism might do more harm than good (beyond the religiosity) and I am not convinced. The only evidence you have given is that you have observed some Bayesians being very confident in beliefs that you think that they shouldn't be confident in, though you don't tell us any of those. I would like to hear some of those. And are these beliefs associated with the meme cluster that Bayesianism tends to be attached to (LW things like FAI, cryonics, etc.)?

By the way, here is a brief summary of the Bayes unit at CFAR. The basic tool is to used how surprised you are of certain odds to approximate numerical probabilities based on your beliefs. You use this to generate an approximate numerical prior in the form of odds, and a likelihood ratio, then multiply to get an order of magnitude approximation of the posterior. I find the most useful part to be the calculation of the likelihood ratio.

Would I be surprised if he got in a life threatening car accident 1 time for every 1,000,000 times he goes out with friends? Yes? How about 5:1M? Yes? 20:1M? Not really. Lets go with 10:1M (you could probably use actual stats for this). Now, would I be surprised if he got in an accident 1 times for every 10,000 times? Yes. [etc.] Ok, lets go with 30:1M. So a good prior would be about 20:1M.

Now this probably isn't the greatest example since the numbers are so large, but I think that the likelihood ratios are the interesting part:

Now lets say he got in a car accident. How often would he fail to call by 11PM? Essentially 1/1. Now lets say he didn't get in an accident. How often would he fail to call by 11PM? [...uses above method for finding a probability...] about 3/10 , he is a little disorganized and forgetful. So our likelihood ratio is 10:3 or about 3:1. So he is only 3 times more likely to get in a fatal accident if he fails to call, or 60:1M chance, or 0.006%. I should stop my unreasonable worrying.

Anyways, as some other people have mentioned, I find that the most useful part of Bayes is the qualitative stuff. It also tells you why you should update in the ways that seem intuitively reasonable.

The rest of the video goes on to say that Bayesianism boils down to “don’t be so sure of your beliefs; be less sure when you see contradictory evidence.”

Now that is just common sense. Why does anyone need to be told this? And how does the formula help?

What do you mean by common sense? And I think you are being a little bit optimistic. These things (and other such things mentioned your article) seem like common sense upon reflection, but you probably won't notice your brain violating those rules anyways.

I'm actually more confident about FAI and the Singularity than LW folks. I think their probability in the next 40 years is < 0.1, whereas the average LW person might say 0.5.

The 2011 Less Wrong Survey had a question for what people thought was the most likely date of the Singularity. "The mean for the Singularity question is useless because of the very high numbers some people put in, but the median was 2080 (quartiles 2050, 2080, 2150)." The results page of the 2012 survey doesn't mention the Singularity dates, but downloading the raw data, it looks like that year's quartiles were 2045, 2060, 2080.

It might also be relevant to mention the results for the "Which disaster do you think is most likely to wipe out greater than 90% of humanity before the year 2100?" question. In 2011, "Unfriendly AI" got 16.5% of the votes and came out second (behind bioengineered pandemics at 17.8%). In 2012, it got 13.5% of the votes and came out third (behind bioengineered pandemics at 23% and environmental collapse at 14.5%).

What I find particulary weird about this LW obsession with singularity is that I haven't seen in LW any evidence whatsoever that people are getting results that get them closer and closer to singularity.

This is markedly different from the LessWrong survey results you mention. I find that striking; but I've no idea what it implies about LessWrong (or AI). Any thoughts about that?

No, not really. I could try to come up with a hypothesis, but I'd just be making stuff up. :-)

Incidentally, the reason why I personally find AI in 50 years to be quite plausible is because of the progress we're making in neuroscience. We've already successfully created brain prostheses that replicate hippocampal and cerebrellar function in rats, and there are claims that we wouldn't need any big conceptual breakthroughs to implement whole brain emulation, just gradual improvement of already existing techniques. One distinguished computational neuroscientist was also willing to go on record about us not needing any huge conceptual breakthroughs for creating neuroprostheses that would mimic human cortical function, and co-authored a paper about that and its consequences with me.

If we're already this far now, it wouldn't seem at all implausible that we'd have reverse-engineered the building blocks of intelligence within the next 50 years. Though obviously that's also what the AI gurus of 50 years ago thought, so I wouldn't say that "AI within 50 years" is certain, just plausible.

Above, you claimed that a major problem with Bayesianism is that the universe does not come prepackaged with hypotheses. However, human brains are born with intuitions about causality and inference, which I think suffices as a starting point for Bayesianism or something closely akin to it, at least once badly conflicting intuitions are recognized and repaired through reflective equilibrium. I do not think any epistemic framework can avoid relying on human inferences, so I do not see why you think it is a problem with Bayesianism that the universe does not hand us hypotheses or reference classes or objective priors.

Sorry to dig up this old post, again there's a topic here that's been in my head a lot recently on and off. I've tried not to make this too ranty, I really like the LW people but have a similar fascination/frustration dynamic to you.

In the comments here you write:

Part of LessWrong 2.0 could be a new presentation of the small practical insights that didn't involve the big, wrong claims about Bayesianism.

I think LW has already made some nice steps towards this under the heading of 'instrumental rationality'. The bit that interests me most is that that side of LW is often very insightful on the kinds of preverbal/emotional content that ideas have associated with them: there's a pretty sophisticated vocabulary of 'ugh fields' and 'affective death spirals', and Alicorn wrote a whole sequence of posts on introspection and trying to pick apart the background affective tone you associate with beliefs, with the intention of maybe changing them (or in Silicon Valley terms 'hacking yourself to liking different stuff').

This is very close to the territory I'm going over with my mathematical intuition obsession, so it's helpful for me to read. And they write in a language I can understand! Normally I'm worried that I'm going to have to go and read reams of obscure text by continental phenomenologists or something, so I definitely appreciate this.

CfAR seems to do even more of this, but I think they're also increasingly tightly coupled to MIRI/x-risk... not really sure what's going on there.

I really don't understand how this side of LW fits with the side that's obsessed with formalising everything in some improbably narrow mathematical framework. They seem completely at odds. And maybe I'm being unfair but it always seems to have the feeling on LW of being a kind of sideshow, like it's being treated as a set of interesting soft questions for the less technically minded, while the grown-ups get on with the serious business of trying to make ethical statements take values in the real numbers or whatever. I'm convinced it's the good bit though!

Thanks for the reply! It's true that I'm missing a lot of context. I've hung round the edges of 'rationalist-adjacent tumblr' quite a bit, but that's really its own separate thing, and geographically the interesting stuff is mostly going on thousands of miles from me. I'm fascinated by what I can glean online, though - it's one of those things where some parts of the community really appeal to me and some really don't, so I just keep coming back and poking around some more.

That's interesting that CfAR stopped the Bayesian part of their curriculum - I was still under the impression that that was important to them. Some details of their new content have been published on the LW discussion pages, though, and 'mostly psychological techniques for getting past personal limiting assumptions' is probably accurate. It looks good to me!

I guess what I was trying to get at is that the new CfAR-type-stuff seems to point to quite a complex, messy account of how we learn new things and change our minds, whereas old-school Less Wrong seemed to be obsessed with manipulating clean formal 'representations in the head'. I'm pretty sure that in at least some cases these are the same people (CfAR and MIRI are still closely linked, and the MIRI forum still looks like 'old school Less Wrong' to me), so I'm just interested in how they resolve the tension!

drossbucket:

I really don't understand how this side of LW fits with the side that's obsessed with formalising everything in some improbably narrow mathematical framework. They seem completely at odds.

Not sure exactly which parts of LW you are referring to when you're talking about "formalizing everything in math", but for some parts (e.g. anything to do with decision theory) at least, the answer is that it's the LW/MIRI flavor of AI research. It's meant to be something that you use for thinking about how to build an AI; it's not meant to be a practical guide to life any more than a theoretical physicist's work on string theory is meant to help him with his more mundane concerns.

David:

I'm sure you realize that if you're curious about CFAR, they do run workshops you could attend, right? ;) If the cost is the deciding factor, it's negotiable. (It would probably be gauche to elaborate on the exact deal that I got, but let's just say that I was pretty penniless when I contacted them about attending but we came to a mutually satisfactory arrangement anyway.)