A simple, inexpensive, relatively easy step that EA organizations could take to improve their research quality is to submit their paper drafts to a paid peer review service at an academic publisher like Wiley. Wiley charges $200-300 for a 10-day turnaround. That’s assuming the draft is no more than 5,000 words — longer drafts cost more to review.
Wiley’s service is not the only one like this, or necessarily the best, it was just the first I came across. I also can’t personally attest to the quality of Wiley’s service (or any of the other similar services). I hope that at least one of these companies offers a good service, but I don’t know for sure that they do.
Another thought: it could potentially be disheartening to get this kind of feedback at the end of a research project, when the paper is almost ready to post online. So, maybe it would be even better to get input from experts in relevant fields at the earliest stage. Experts could review your research proposal and offer input, potentially saving you tons of time and heartache if you were about to make an avoidable error.
For example, if METR had gotten its research proposal for its AI time horizons work reviewed by some external experts, there are some avoidable errors in that work that potentially could have been averted. More discouragingly, but still important, if METR had submitted the draft of its paper on the time horizons work to a peer review service prior to posting it on its website, the paper could have better disclosed some of the errors and limitations.
Peer review of both research proposals and paper drafts would be useful for two major reasons. First, it would be intrinsically useful because it would lead to better research. If the point of research is to tell us the truth, and we want to know the truth, well, then, better research will tell us the truth better.
Second, it would be instrumentally useful. An important goal for many EA organizations is to persuade a broader community of people about something — experts, policymakers, regulators, the general public, potential recruits to the EA movement. Higher-quality research is more persuasive. It’s also a good way to earn credibility and trust. Low-quality research is unpersuasive, and can even persuade people in the opposite direction. (“If that’s the best you could come up with, surely your conclusions must be wrong!”) Publishing low-quality or fatally erroneous research also damages credibility and trust.
Assuming Wiley’s service is as useful as I hope, $200-300 and 10 days of waiting is a tiny cost compared to the intrinsic and instrumental value of doing better research.
Two potential subcultural stumbling blocks:
- There is a strong undercurrent in the EA community of opposition to mainstream institutions — mainstream journalism, mainstream academia, and mainstream, institutional science. Even to mainstream society and culture.
- Not unrelatedly, there is a strong desire in the EA community to treat the community as an enclave (or conclave!), rather than a part of the wider world. For EA to rely only on itself for ideas, for input, for intellectual evaluation.
I probably can’t convince anyone that these attitudes are wrong for intrinsic epistemic reasons. But maybe I can convince them that, in order to have a strong and durable influence on the wider world, it will be necessary for EA organizations to “play ball” and engage with the rest of the world on its terms.
The EA community has certain beliefs, particularly about how close the world is to creating AGI, that most experts, forecasters, policymakers, and members of the general public disagree with. Some EA organizations just want to do technical research and don’t need to worry about what anyone else thinks. But other organizations want to persuade the world of the danger. Maybe some people feel cynical and don’t dare hope that the world could actually be persuaded on the basis of high-quality scientific evidence.
Although scientific thinking and Enlightenment values are embattled, and there is a lot of misinformation out there, I still think scientific evidence matters a lot to a lot of people, including experts, policymakers, and the general public. The world is open to being persuaded. But you have to “play ball”.
(Related post here.)
