Getting across (scientifically) profitable notions of statistics to non-statisticians (as well as fellow statisticians) ain’t easy.
Statistics is what it is, but explaining it as what it ain’t just so it is easy to understand (and thereby likely to make you more popular) should no longer be tolerated. What folks take away from easy to understand incorrect explanations can be dangerous to them and others. Worse they can become more gruesome than even vampirical ideas – false notions that can’t be killed by reason.
I recently came across the quoted explanation in the title of this post in a youtube a colleague tweeted How Not to Fall for Bad Statistics – with Jennifer Rogers.
The offending explanation of statistics as the alchemy of converting uncertainty into certainty occurs at around 7 minutes. Again, “Statistical significance just tells us whether or not something definitely does or definitely doesn’t cause cancer.” So if you were uncertain if something caused cancer, just use statistical significance to determine if it definitely does or definitely doesn’t. Easy peasy. If p > .05 nothing to worry about. On the other hand, if p < .05 do whatever you can to avoid it. Nooooooo!
Now, if only a statistician was doing such a talk or maybe a highly credentialed statistician – but at the time Jennifer Rogers was the Director of Statistical Consultancy Services at the University of Oxford, an associate professor at Oxford and still is vice president for external affairs of the Royal Statistical Society. And has a TEDx talk list on her personal page. How could they have gotten statistical significance so wrong?
OK, at another point in the talk she did give a correct definition of p_values and at another point she explained a confidence interval as an interval of plausible values. But then she claimed for a particular confidence interval at around 37 minutes “I would expect 95% of them between 38 and 66” where she seems to be referring to future estimates or maybe even the “truth”. Again getting across (scientifically) profitable notions of statistics to non-statisticians (as well as fellow statisticians) ain’t easy. We all are at risk of accidentally giving incorrect definitions and explanations. Unfortunately those are the ones folks are most likely to take away as they are much easier to make sense of and seemingly more profitable for what they want to do.
So we all need to speak up about them and retract ones we make. This video has had almost 50,000 views!!!
Unfortunately, there is more to complain about in the talk. Most of the discussion about confidence intervals seemed to be just a demonstration of how to determine statistical significance with them. The example made this especially perplexing to me being that it addressed a survey to determine how many agreed with an advertisement claim – of 52 surveyed 52% agreed. Now when I first went to university, I wanted to go into advertising (there was even a club for that at the University of Toronto). Things may have changed since then, but then getting even 10% of people to accept an adverting claim would have to considered a success.
But here the uncertainty in the survey results is assessed primarily using a null hypothesis of 50% agreement. What? As if we are really worried that 52 people flipped a random coin to answer the survey. Really? However, with that convenient assumption it is all about whether the confidence interval includes 50% or not. At around 36 minutes if the confidence interval does not cross 50% “I say it’s a statistically significant result” QED.
Perhaps the bottom line here is that as with journalists who would benefit from statisticians giving advice as to how to avoid being mislead by statistics, all statisticians need other statisticians to help them avoiding explanations of statistics that may instil misleading notions of what statistics are, can do and especially what one should make of them. So we all need to speak up about them and retract ones we make.
It is bad enough when Dr. Gelman engages in this drivel. No idea who Kevin O’Rourke is or why Dr. Gelman allows this cretin to pollute a wonderful blog.
PS How come my comments re: cancer and cell phones and leukemia and cell phone towers is being censored?
https://www.cnn.com/2018/05/02/health/brain-tumors-cell-phones-study/index.html
https://scienceblog.cancerresearchuk.org/2016/10/31/sellafield-radiation-and-childhood-cancer-shedding-light-on-cancer-clusters-near-nuclear-sites/
https://www.cbsnews.com/news/cell-tower-shut-down-some-california-parents-link-to-several-cases-of-childhood-cancer/
Excellent parody comment, well played.
Keith [sic] O’Rourke is a long-time commenter and contributor to the blog: a quick Google search takes you to their Google Scholar page, https://scholar.google.ca/citations?user=R064zwoAAAAJ&hl=en
I don’t think your comments were censored; I saw at least one comment of yours containing these links on the previous (“horns and cell-phone towers”) post.
You seem to be mixing together effects of radioactivity (ionizing radiation, e.g. from nuclear power plants) and several different kinds of non-ionizing radiation (e.g. cell-phone towers vs cell-phone use). You’re also citing a mixture of news stories about *concern* (e.g. “a cancer cluster occurred, some parents are concerned that it’s caused by cell-tower proximity”) and scientific studies.
The conversation on this site is generally more polite than the average internet comment section (e.g. “drivel” and “cretin” seem unnecessary).
Could you suggest or provide a link to correct content (usable/presentable/understandable) for explaining p_value and confidence to non-statisticians?
There’s a lot that’s been written on these topics so it’s hard to recommend a short list of resources but I think the most accessible stuff has been written in the past few years because it’s recent enough to address many controversies around the topics. I list them in order of difficulty (least difficult to most difficult)
Denworth, L. (2019), “The Significant Problem of P Values,” Scientific American, Available athttp://www.scientificamerican.com/article/the-significant-problem-of-p-values/. https://doi.org/10.1038/scientificamerican1019-62.
Amrhein, V., Greenland, S., and McShane, B. (2019), “Scientists rise up against statistical significance,” Nature, 567, 305. https://doi.org/10.1038/d41586-019-00857-9.
(Discussion of the paper above on this blog: https://statmodeling.stat.columbia.edu/2019/03/20/retire-statistical-significance-the-discussion/)
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016), “Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations,” European Journal of Epidemiology, 31, 337–350. https://doi.org/10.1007/s10654-016-0149-3.
Gelman, A., and Greenland, S. (2019), “Are confidence intervals better termed ‘uncertainty intervals’?,” BMJ, 366. https://doi.org/10.1136/bmj.l5381.
Chow, Z. R., and Greenland, S. (2019), “Semantic and Cognitive Tools to Aid Statistical Inference: Replace Confidence and Significance by Compatibility and Surprise,” arXiv:1909.08579 [stat.ME]. https://arxiv.org/abs/1909.08579.
Greenland, S. (2019), “Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution with S-values,” The American Statistician, 73, 106–114. https://doi.org/10.1080/00031305.2018.1529625.
Thanks Zad.
Actually think Chow, Z. R., and Greenland, S. (2019) would be the best bet for Chris K and other similar inquiries.
You might also try the notes linked from https://web.ma.utexas.edu/users/mks/CommonMistakes2016/commonmistakeshome2016.html .
They start with fundamental (but common) misunderstandings involving uncertainty (such as “Expecting too much certainty”, “Terminology-inspired confusions”, and “Mistakes involving causality”), since these basic misunderstandings contribute a lot to misunderstanding statistical inference.
I feel much better. I thought this was like a blog-specific rejection of the basis of all scientific research. but clearly i was wrong. thank you to Zad and apologies to Keith. I cannot really believe that the editor of psychological review has stopped caring about p-values. or the editor of science or nature or cognitive neuroscience. seems like a weird little faux-controversy. will read the scientific american article as soon as i score some ketamine.
Beatrice:
No need to apologize, no enemies here just allies who are having difficulty understanding what to make of what other’s are writing.
(With apologies to Oscar Wilde.)
You mean cat Valium?
Thanks for posting this Keith. I’m really astonished that anyone in statistics could claim that a single number or decision-making framework could definitively tell them something about a phenomenon. I guess Aschwanden nailed it with her article on P-values, even those who have thought and written about them for a long time have difficulty explaining them
https://fivethirtyeight.com/features/not-even-scientists-can-easily-explain-p-values/
From the article:
“The p-value won’t tell you whether the coin is fair, but it will tell you the probability that you’d get at least as many heads as you did if the coin was fair. That’s it — nothing more. “
This is a great example. The simple experiment – a coin flip – can’t possibly go wrong.
Now translate that “coin flip” into the context of the power pose: “the probability that you – a student posing as a job candidate – would get a fake job, offered by another student posing has a person offering a job – after doing a power pose in the mirror”. Then remember that the coin flip is an actual event, but your power pose experiment is a simulation of an actual event, roughly comparable to a five year simulating cooking dinner with cardboard kitchen appliances.
P is perfectly legitimate for simple experiments, and as a cut-off value in assessing some minimum level of efficacy. If people would just use it for those experiments, it would be fine.
I’d like to morph your first line slightly:
“Science is what it is, but explaining it as what it ain’t just so it is easy to understand…should no longer be tolerated. “