In discussions around existential risk, I sometimes hear the phrase “coordination is hard”.  Most of the time, the person saying this is using this as a reason to give up on strategies involving decentralized coordination and instead pursue more-centralized strategies, such as “a small number of people being extremely rational individuals and using other people as advisers and actuators”.  In this context, I think the phrase “coordination is hard” is very misleading.

Here are some examples of successful coordination:

• Labor movements manage to hold strikes despite individual incentives to break the strike.
• Open source software is usually created by volunteers and is often of fairly high quantity and quality.  Some projects, such as Linux, have hundreds or thousands of contributors.
• CFCs have been largely phased out globally due to their impact on the ozone layer.
• Communities, in-person and online, develop discourse norms, which can implement significant functionality for the community (e.g. fact-checking, preventing people from unjustly harming others)
• Some groups of people distributed among different countries (e.g. workers, moderates, various racial/ethnic groups, intellectuals) will feel affinity with each other and help each other out (e.g. intellectuals in the US might support immigration as a way of assisting intellectuals in other countries)
• Militaries usually have significant decentralized coordination.  Soldiers usually fight bravely; defectors are relatively rare.
• Many societies have a well-functioning justice system.  This system requires people to perform many different functions (judge, juror, witness, expert, …).

None of these examples are in perfect analogy with current existential risk issues, but given them, the generic statement “coordination is hard” looks silly.  Upon dismissing this generic statement, more interesting questions emerge:

• When is coordination easy or hard?
• How does ability to coordinate vary across different societies historically?
• What different coordination strategies are there?  How do they work?  What are their strengths and weaknesses?

In many cases, humans seem to be better at coordinating than Homo economicus would predict; anyone concerned with the future of humanity ought to find this fact extremely interesting.

 

 

 

 

[epistemic status: confident this distinction exists and is important, unsure about details/framing]

A contrast:

• Naïve epistemology: simple models; mathematics and logic; “prima facie”; original seeing; “natural” descriptions of facts on the ground; “from the mouth of a babe”; not trusting others’ models/narratives (but often trusting their reports of basic observable facts); blunt
• Savvy epistemology: responding to the dominant opinions in a social group such as expert consensus (agreeing with parts of it and/or disagreeing with parts of it); “this is a complex topic”; not wanting to omit considerations that others know about; making sure to be “informed” and “up to date”; educated; responsible; legitimate; sophisticated; subtle; trust or cynicism regarding others’ models/narratives

Naïve epistemology mostly models the world; savvy epistemology mostly models other people’s models of the world.  Naïve epistemology avoids the social field; savvy epistemology swims in it.

The virtues of naiveté

In the past year, I’ve strongly updated towards using more naïve epistemology than savvy epistemology.  Savvy epistemology is more informed in some ways, but is also more corrupt.  It will tend to be slanted by whoever has local political power, rather than being purely optimized for accuracy.  Some examples:

• Tetlock’s book Superforecasting shows that some non-experts using relatively naïve methods can predict world events significantly better than savvier experts.  This indicates that these experts’ opinions are not very well optimized for accuracy (and are most likely optimized for something other than accuracy).
• I am convinced by Robin Hanson’s argument that medical care has little association with health outcomes in the US.  The “savvy” responses to his “naïve” article (linked at the bottom of the page) seem to basically be misdirections intended to avoid taking seriously his main point (that cutting medical spending by 50% across the board would make little difference to health outcomes).
• AI risk was discussed on LessWrong many years before the topic received serious mainstream attention, mostly from a naïve epistemology perspective.  The savvy expert view started out saying it wasn’t a serious problem (for various reasons), and has since drifted somewhat towards the LessWrong view.  From thinking extensively about AI alignment and talking with people over the past few years, I feel that I have gotten significant personal evidence in favor of the LessWrong view that the alignment problem is very difficult and that no one has a good idea of how to solve it.

The corruption of savvy epistemology does not make it useless: it usually contains a great deal of information not captured by naïve epistemology, and use of savvy epistemology allows for easier coordination with others using it.  But it does mean that it’s highly valuable to develop a naïve perspective in addition to a savvy one.

Why be afraid of naïve epistemology?

There are legitimate reasons to be afraid that someone taking naïve epistemology seriously will cause harm:

• Uncooperative actions: Taking one’s own models seriously can lead one to take actions that are uncooperative with others who have different models, such as dishonesty or violence.  See The Unilateralist’s Curse.  The “social field” that savviness swims in is a coordination mechanism that, among other things, prevents people from taking some uncooperative actions.
• Safety: Common recommendations for what to believe, though often false, at least offer safety in the medium term for most people.  Having and acting on unusual beliefs can be dangerous to one’s self and others, since at that point you’re not using a set of beliefs that’s already been optimized for medium-term safety.  (In the long term, safety from some risks such as large-scale wars and some medical issues might require nonconformist epistemics)
• Disharmony: To the extent that words ostensibly about the world are used to coordinate actions, people using different words (e.g. those generated by naïve epistemology) can generate confusion about what actions the group is supposed to be taking.
• Politics: Saying naïve things can disrupt politics, by e.g. offending allies or potential allies.  This can trigger a defensive reaction.
• Legibility bias: Naïve models tend to be legible.  It can be tempting to make oversimplified models that don’t actually summarize the fiddly details well, leading to failures such as utopian totalitarianism.

Social technology for being naïve

I think that people who want good epistemics should have the affordance to use and act on naïve epistemology.  This requires addressing the problems in the previous section.  Here are some tools I use to give myself the affordance to do naïve epistemology without paying large costs:

• Decision theory: Preferentially use naïve epistemology when making bets using one’s own resources, and use shared epistemology when making bets using others’ resources.  Detect “prisoner’s dilemma” or “unilateralist’s curse” situations (including epistemic ones) and make a good-faith effort to achieve mutual cooperation.  Develop and signal the intent to honestly inform others and to learn from others.  Take the true outside view and act as you would want others in your position to act.  (See also: Guided by the beauty of our weapons)
• Testing things: If a naïve model implies some surprising prediction, it can be good to test it out (if it isn’t very costly).  If the prediction is confirmed, great; if not, then perhaps create a new naïve model that takes this new piece of evidence into account.
• Separate magisteria: Calling forms of model-building “naïve” puts these models in a separate magisterium where it isn’t socially expected that they will be taken seriously, triggering less defensive reaction.  (I bet many people would be less receptive to this post if I had called it “world-modeling epistemology” rather than “naïve epistemology”).  Other ways of putting model-building in a separate magisterium: calling yourself “a contrarian acting on a weird inside view”; saying you’re “discussing weird ideas”; etc.
• Translation: Some naïve models can eventually be translated into savvy arguments.  (In practice, not all of them can be translated without losing fidelity).
• Arguing against savviness:  In general, internalizing arguments against “mainstream” opinions makes it psychologically and socially easier to do “non-mainstream” things.  This post itself contains arguments against savvy epistemology.  Getting arguments against specific savvy beliefs/worldviews can also be helpful.
• Playing with identity: In general, some identities (“independent”, “truth-seeking”, “weird”) will create a local incentive to lean towards naïve epistemology,  while others (“responsible”, “official”, “educated”) will create a local incentive to lean towards savvy epistemology.  Someone who leans savvy might benefit by creating a context where they can embody a more naïve identity.

I would like it if more people had the affordance to develop a naïve epistemology, so that they can make important inferences that savvy epistemology obscures.  I don’t think that naïve epistemology is an “endpoint” of epistemology (it seems desirable to eventually develop epistemology that is fundamentally truth-tracking the way naïve epistemology is in addition to being highly informed and coordinated across people), but I expect it to be an important stepping stone for people currently embedded in savvy epistemology.

[epistemic status: confident this is a thing, not sure if I’m framing it right]

I’ve noticed that my writing is usually better when I write it quickly and don’t edit it as much.  Continuously adjusting it results in something blander.  I think this is about defensibility.  Some examples:

• Academic papers are more defensible than blog posts.
• Public relations speech is more defensible that private speech.
• Proofs are more defensible than intuitive arguments.
• Slowly-written things are more defensible than quickly-written things.
• “Literal” speech is more defensible than “metaphorical” speech.

Defensibility is about resisting all possible attacks from some class of adversaries.  The more you expect your expression to be picked apart and used against you, the more defensible your expression will be.  Slowly-written things look more polished and the flaws stand out more, so they get nitpicked.

Defensibility can be good.  In mathematics, a defensible argument (i.e. a proof) is more likely to be correct.  In science, a defensible statistical result is more likely to replicate.  Defensible results become blocks of knowledge that others can (and sometimes must) build upon.

Defensibility can be bad.  Academic papers are usually worse at explaining things than blog posts.  PR is usually highly misleading.  Fully formal proofs are usually harder to understand than intuitive arguments.  Defensible art is bland.

Defensibility requires conformity.  If anything you say can and will be used against you, it is better to say the same things others are saying.  Defensible expressions happen in a shared ontology, such as formal logic, or the “ordinary official speech” ontology that Wikipedia uses.

Some expression should be defensible (against different classes of adversaries).  Some shouldn’t be.  It is virtuous to be flexible about defensibility.

[epistemic status: optimistic vision that currently seems highly useful]

In Christopher Alexander’s work, a pattern consists of:

1. A way of perceiving an existing tension in a living system.
2. A model for why the tension exists.
3. A way of adapting the system in a way that resolves the tension.

Christopher Alexander describes hundreds of patterns for towns and buildings in A Pattern Language.  Consider pattern #126, SOMETHING ROUGHLY IN THE MIDDLE:

A public space without a middle is quite likely to stay empty.

…

if there is a reasonable area in the middle, intended for public use, it will be wasted unless there are trees, monuments, seats, fountains—a place where people can protect their backs, as easily as they can around the edge.

…

Therefore:

Between the natural paths which cross a public square or courtyard or piece of common land choose something to stand roughly in the middle: a fountain, a tree, a statue, a clock-tower with seats, a windmill, a bandstand.  Make it something which gives a strong and steady pulse to the square, drawing people in twoards the center.  Leave it exactly where it falls between the paths; resist the impulse to put it exactly in the middle.

This pattern points out a tension: a public space without a middle will feel weird and people will avoid it.  Next, it gives a model for the tension: it’s because people want their back to be against something.  Finally, it gives a way to resolve this tension: put something roughly in the middle.

Patterns that work well with each other form a pattern language.  SOMETHING ROUGHLY IN THE MIDDLE works with SMALL PUBLIC SQUARES; along with the other patterns, they form a pattern language.

The initial tension will sometimes be diffuse and hard to pin down.  People might feel “off” when they’re in a large empty space but not have the words to say why.  They might not even consciously realize there is a problem and instead just have slightly elevated stress levels.  The tension becomes more pointed if it is explicitly pointed out and especially if there is a model for it.

Many of the rationality techniques discussed in the LessWrong-descended rationality community are patterns in this sense.  The technique of leaving a line of retreat is about noticing a tension (some possibility X seems hard to seriously consider) and resolving it by imagining a world in which X is true in detail (in order to allow yourself to compare the world where X is true and the world where X is false even-handedly).  The technique of finding a true rejection is about noticing a tension (people have arguments where they give false reasons for their beliefs and don’t actually change their minds when these reasons are refuted) and resolving it by urging people to find the actual historical reasons for their beliefs.  CFAR‘s debugging techniques are also usually patterns of this form (they resolve some sort of tension within a person or between a person and their environment, usually reducing stress levels).

Resolving tensions fully brings to mind the image of a placid lake.  But resolving tensions doesn’t necessarily lead to placidity, because systems involving humans are alive.  Solving all of someone’s immediate problems might make them bored, and might make them want to help others.

Christopher Alexander praises the way that well-built towns and buildings bring out life in The Timeless Way of Building (I highly recommend reading the first 2 chapters):

There is one timeless way of building.

It is thousands of years old, and the same today as it has always been.

The great traditional buildings of the past, the villages and tents and temples in which man feels at home, have always been made by people who were very close to the center of this way.  It is not possible to make great buildings, or great towns, beautiful places, places where you feel yourself, places where you feel alive, except by following this way.  And, as you will see, this way will lead anyone who looks for it to buildings which are themselves as ancient in their form, as the trees and hills, and as our faces are.

It is a process through which the order of a building or a town grows out directly from the inner nature of the people, and the animals, and plants, and matter which are in it.

It is a process which allows the life inside a person, or a family, or a town, to flourish, openly, in freedom, so vividly that it gives birth, of its own accord, to the natural order which is needed to sustain this life.

…

But as things are, we have so far beset ourselves with rules, and concepts, and ideas of what must be done to make a building or a town alive, that we have become afraid of what will happen naturally, and convinced that we must work within a “system” and with “methods” since without them our surroundings will come tumbling down in chaos.

…

The thoughts and fears which feed these methods are illusions.

I find Christopher Alexander’s vision highly appealing.  Something happened in Western architecture that interfered with the timeless way of building; this probably had to do with the increase in degrees of freedom in the transition from pre-modern to modern societies.  Similar phenomena have happened in other domains.  Luckily, the way can be regained through the discipline of pattern languages, and eventually through dissolving disciplines and doing what is natural.  There are obvious parallels with Taoism.

I want pattern languages for improving people’s psychological functioning, for improving groups’ ability to work together, for improving models of the world, for making good institutions, for all sorts of things.  Groups of people are developing pattern languages in many different places, mostly without explicitly knowing what they are doing.  I think a more explicit understanding of pattern languages and knowledge of how past pattern languages were created will help to design better ones in the future.  Maybe with the right pattern languages (and with the right going-beyond-them), we can become much more alive, as individuals and as groups, and much more able to coordinate to accomplish very hard things.

[epistemic status: strong opinions weakly held, not very original]

The false outside view says that if you have a contrarian inside view, you’re probably wrong, so you should not act like you’re correct. The false outside view is driven by intuitions about social status and/or fear of being wrong.

The true outside view says that a great deal of good comes from contrarians pushing on their contrarian inside views (acting on them, talking about them with others, etc), so if you have one of those you should maybe do that, while additionally taking others’ perspectives seriously, testing your ideas often, being honest, avoiding some potentially-catastrophic unilateral actions, etc.

The true outside view follows from any halfway-decent decision theory (something like “updateless decision theory for humans”, as gestured at before. This decision theory might look “epistemically modest” sometimes and “epistemically arrogant” other times, but these descriptions impose ego on something that by construction doesn’t have an ego.