A simple, inexpensive, relatively easy step that EA organizations could take to improve their research quality is to pay external experts to do peer review.
Another thought: it could potentially be disheartening to get this kind of feedback at the end of a research project, when the paper is almost ready to post online. So, maybe it would be even better to get input from experts in relevant fields at the earliest stage. Experts could review your research proposal and offer input, potentially saving you tons of time and heartache if you were about to make an avoidable error.
For example, if METR had gotten a research proposal for its AI time horizons work reviewed by some external experts, there are some avoidable errors in that work that potentially could have been averted. More discouragingly, but still important, if METR had submitted the draft of its paper on the time horizons work for peer review by external experts prior to posting it on its website, the paper could have better disclosed some of its errors and limitations.
Peer review of both research proposals and paper drafts would be useful for two major reasons. First, it would be intrinsically useful because it would lead to better research. If the point of research is to tell us the truth, and we want to know the truth, well, then, better research will tell us the truth better.
Second, it would be instrumentally useful. An important goal for many EA organizations is to persuade a broader community of people about something — experts, policymakers, regulators, the general public, potential recruits to the EA movement. Higher-quality research is more persuasive. It’s also a good way to earn credibility and trust. Low-quality research is unpersuasive, and can even persuade people in the opposite direction. (“If that’s the best you could come up with, surely your conclusions must be wrong!”) Publishing low-quality or fatally erroneous research also damages credibility and trust.
Two potential subcultural stumbling blocks:
- There is a strong undercurrent in the EA community of opposition to mainstream institutions — mainstream journalism, mainstream academia, and mainstream, institutional science. Even to mainstream society and culture.
- Not unrelatedly, there is a strong desire in the EA community to treat the community as an enclave (or conclave!), rather than a part of the wider world. For EA to rely only on itself for ideas, for input, for intellectual evaluation.
I probably can’t convince anyone that these attitudes are wrong for intrinsic epistemic reasons. But maybe I can convince them that, in order to have a strong and durable influence on the wider world, it will be necessary for EA organizations to “play ball” and engage with the rest of the world on its terms.
The EA community has certain beliefs, particularly about how close the world is to creating AGI, that most experts, forecasters, policymakers, and members of the general public disagree with. Some EA organizations just want to do technical research and don’t need to worry about what anyone else thinks. But other organizations want to persuade the world of the danger.
Maybe some people feel cynical and don’t dare hope that the world could actually be persuaded on the basis of high-quality scientific evidence. Although scientific thinking and Enlightenment values are embattled, and there is a lot of misinformation out there, I still think scientific evidence matters a lot to a lot of people, including experts, policymakers, and the general public. The world is open to being persuaded. But you have to “play ball”.
(Related post here.)
Note: I got tricked by deceptive SEO into thinking that a paid peer review service that used an academic publisher's name a lot was run by that academic publisher. Google Gemini Pro also lied to me during my search for such services and told me the academic publisher and the paid peer review service were one and the same. But it's totally my fault for not catching this. Clara Torres Latorre caught this mistake in this comment and I updated this post with this correction at 22:04 UTC on Wednesday, May 27, 2026.
I agree with the general point of the post.
But I disagree with specifically using Wiley's services to obtain peer review [1].
I would be excited for people in EA sending their "research" pieces to mainstream academic journals more often.
[1] I'm a math researcher in academia. When I peer review an article, I'm usually granted 60 to 90 (sometimes more) days, depending on length. Adding admin, sometimes multiple times of revision, the time that editors need to find a suitable reviewer, missed deadlines... the whole process usually amounts to at least 6 months, and sometimes it takes years.
I don't think 10 days is a reasonable timeline for peer review, and I find it even short to find a relevant and willing reviewer.
Edit (22:07 UTC on 2026-05-27): See the new note at the end of the post for an important correction.
So, your skepticism comes from the 10-day turnaround time? If it were 60 days or 90 days instead, you wouldn’t feel skeptical?
I wonder how/why they are able to offer such fast turnarounds and whether it’s by sacrificing quality. Do you think if you got paid, say, $150-250 per review you’d make time to do them faster? Or would it just be impossible regardless?
There are a number of other services similar to
Wiley’s. I don’t know if any of them are any good.Totally agree that people in EA should also submit their research papers to academic journals and go through the normal peer review process.
Take this with a huge grain of salt, because thing vary enormously field to field.
10 days turnaround sounds too good to be true to me. If it was 6 months I would maybe give it a try for value of information.
But there's more:
Price seems cheap: Last paper I reviewed took me 10-15 hours, so 300-500 + tax would be a reasonable price range and 150-250 sounds meagre. And you need to factor in admin, infra and costs that are not just paying the reviewer.
In my field, peer review is "pro bono" but done during the "working hours" of people with public salaries mostly. And there's the understanding that since we publish and someone else reviews, we also should review some. However, that means that the availability to review varies a lot depending on teaching, research, admin, etc, and we usually fit it in with low priority.
That means if you want someone to review a paper, even if they only need 2 days, the people that have the expertise usually have work to do and might not be willing to give up on next weekend.
About "what if I got paid", it's complicated, bc it depends on if I really have the time (meaning, no plans on the only weekend in the 10 day window, and willing to work), but probably not.
And there's also an ugh factor about getting paid for something that we usually do for free / as part of a salaried job. I'm not decided on it.
I think you're selling yourself short at 300-500 USD. Gemini estimates 1600-4200 USD (for 3 reviewers total), Opus 400-1000 USD (for a single reviewer spending only 4-6 hours). I endorse those estimates.
Prompt for those curious: If academic peer reviews were compensated at market rate (ie, relative to industry pay for someone with the relevant expertise), how much would it cost to have a typical academic paper reviewed?
I computed the time that it takes me * my salary, approx.
Ofc if I did this as a freelancer I would charge more.
Hm, interesting! Thanks for weighing in!
My wild guess about the turnaround time is that they just have so many reviewers “on call” that even if most are unavailable within the 10-day window, at least some people will be available.
The price does seem kind of low. I wonder if the actual average price ends up being more than the list price? E.g., if drafts are above 5,000 words?
I do wonder if the price and turnaround time is too good to be true.
One thing that is worth thinking about is what the response would be if the peer reviews are negative. Open Philanthropy used to commission quite a number of academic reviews for some of their better-known reports (perhaps they still do this?). Not all of the reviews were enthusiastic, and there wasn't always action taken on those reviews.
I didn't know that about Open Philanthropy!
If EA organizations commission academic reviews and ignore them, then, yeah, it's pointless. I guess there has to be some underlying belief that academic feedback is epistemically valuable. Or at least an underlying commitment to move ideas out of the EA echo chamber into wider acceptance by doing research that is persuasive to people outside of EA.
I see two discouraging signs. One, an anti-academic prejudice in EA. (Often along with a belief that EA is intellectually or epistemically superior to academia, and possibly the rest of the world, too.) Two, low patience for attempts to persuade people outside of EA about ideas that are popular within EA but unpopular outside it (e.g., a 50%+ chance of AGI within the next decade).
If people in EA want to switch gears from operating EA as an elite enclave (or conclave) to a movement that can influence the world at a large scale, including the policies of large liberal democracies like the United States, this change will be painful. People will have to learn how to go from having the majority opinion (in EA) to the minority opinion (in the world). From having the power to decide which opinions can and can't be expressed (in EA) to fighting to be heard in contexts where others have that power (in the world). This is as much about emotional regulation as it is about intellectual discipline.
Thanks for the comment, David.
By the way, I clicked the link (finally), and:
I wouldn't even give it a chance, this aren't a couple red flags, we talking November 1917 situation here.
Which makes me sad, because I really like the broader point of engaging with mainstream academia and playing ball, and I've been nerdsniped by a discussion about peer review and feel that I'm derailing the comment section.
Hold on, you're right! They say "Wiley" a lot, but they aren't actually affiliated with Wiley! I think the "Wiley" thing was just an SEO trick! Okay, well now this company definitely seems sketchy, and I wouldn't trust them!
I was looking at this at the same time I was looking at Springer Nature's scientific editing service — which is affiliated with Springer Nature, but it's just editing, not peer review — and ended up thinking it was a similar service. (Google Gemini Pro lied to me/fell for Meritpeer's SEO and told me Metritpeer was Wiley, but it's totally my fault for not fact checking this better when I clicked through to Meritpeer's website.) I'm going to edit my post.
By the way, you're not derailing at all, this is an extremely important and helpful contribution!
The general idea of paying for external expert review or peer review still makes sense, but it would require more doing on the part of EA organizations to make it happen if it's not an off-the-shelf service. Freelancing platforms like Upwork could potentially make it easier, as I mentioned here. I say potentially because I don't know if you could reliably find good peer reviewers on Upwork.
I think it would be an interesting exercise to send a handful of articles to these services and see the quality of the feedback. I would also doubt it would be very good quality but would reserve judgement before trying!
Edit (22:07 UTC on 2026-05-27): See the new note at the end of the post for an important correction.
Where does your doubt come from? Do you doubt that peer review in general is good quality? Or does this service seem too cheap or too fast to be any good?
There’s also the EA organization called The Unjournal, which commissions reviews of EA research from external experts. But I don’t know if this is a better option than
Wiley’s service.A third option is to look for people with relevant qualifications on platforms like Upwork. Here’s a recent freelance job posted on Upwork:
Years ago, I paid someone on Upwork with a PhD in a relevant field to review a paper published by Waymo. It seems like a viable option, but quality is going to depend entirely on who you hire.
And of course option #4 is to submit papers to peer-reviewed journals.
I would have found this much more persuasive if you'd tried these services yourself and found them valuable. Without that, my median expectation is that they will do a worse job than Claude Opus 4.7.
Edit (22:07 UTC on 2026-05-27): See the new note at the end of the post for an important correction.
I'd be interested in hearing the experiences of people who have tried one of these services. I hope they're good, but I don't know that they are. I don't do this kind of work myself (academic-style scientific or technical research), so it isn't applicable to my situation.
A digression on whether you should rely on Claude to do peer review. I found some funny and striking examples to demonstrate the perils of relying on LLM chatbots for this sort of thing:
These were cases where I suspected it would probably give ridiculously high probabilities, and I chose questions unflattering to EA because people in the EA community would be less likely to accept the chatbot's answers. I also asked it a flattering question though:
I tried the same prompt three times and ChatGPT gave probabilities of 3%, 0.1%, and 5%. Again, just ridiculously high probabilities.[1]
In the course of organically using ChatGPT and Google Gemini, I've also encountered tons of weird behaviours. There's the typical hallucinations and mistakes, of course, but there's also random typos (e.g. "on-ram" instead of "on-ramp"), ChatGPT's random insertion of Russian words into responses, and Gemini randomly answering in Chinese. GPT-5.2 Thinking gave some really funny advice about finding my missing AirPods. One of the craziest was when I asked GPT-5.4 Thinking (with "Extended thinking") to do a simple time zone conversion. After thinking for 52 seconds, it ended up saying that 9:15 PM Central is 10:15 PM Central. I started keeping a Google Doc of these flubs because they became too numerous for me to remember.
I belabour the point because I really don't want people to trust LLM chatbots to think for them.
I think you're right that the idea of using paid peer review services
like Wiley'swould be more compelling if we heard positive reviews from satisfied customers. This is worth looking into further.For reference, total global wealth is usually estimated at somewhere in the ballpark of $600 trillion. Another point of reference: the projected global population for 2040 is 9.2 billion people. Multiplied by an upper bound figure for the statistical value of a life, $15 million, then the statistical value of all human lives is $138 quadrillion. Still not even close to $1 quintillion.
Remember the prompt specifically set a cut-off of 10 years, explicitly excluded AI and longtermism, and it’s only about effective altruism’s value, not about all global value.