What a difference four years makes. I will still describe Thinking, Fast and Slow as an excellent book – possibly the best behavioural science book available. But during that time a combination of my learning path and additional research in the behavioural sciences has led me to see Thinking, Fast and Slow as a book with many flaws.
First, there is the list of studies that simply haven’t held up through the “replication crisis” of the last few years. The first substantive chapter of Thinking, Fast and Slow is on priming, so many of these studies are right up the front. These include the Florida effect, money priming, the idea that making a test harder to read can increase test results, and ego depletion (I touch on each of these in my recent talk at the Sydney Behavioural Economics and Behavioural Science Meetup).
It’s understandable that Kahneman was somewhat caught out by the replication crisis that has enveloped this literature. But what does not sit so well was the confidence with which Kahneman made his claims. For example, he wrote:
When I describe priming studies to audiences, the reaction is often disbelief . . . The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true.
I am surprised at the blind spot I had when first reading it – Kahneman’s overconfidence didn’t register with me.
As I was also, Kahneman is a fan of the hot hand studies. Someone who believes in the hot hand believes that a sportsperson such as a basketball player is more likely to make a shot if they made their previous one. Kahneman wrote:
The hot hand is entirely in the eye of the beholders, who are consistently too quick to perceive order and causality in randomness. The hot hand is a massive and widespread cognitive illusion. [Could the same be said about much of the priming literature?]
The public reaction to this research is part of the story. The finding was picked up by the press because of its surprising conclusion, and the general response was disbelief. When the celebrated coach of the Boston Celtics, Red Auerbach, heard of Gilovich and his study, he responded, “Who is this guy? So he makes a study. I couldn’t care less.” The tendency to see patterns in randomness is overwhelming – certainly more impressive than a guy making a study.
And now it seems there is a hot hand. The finding that there was no hot hand the consequence of a statistical error (also covered in my recent talk). The disbelief was appropriate, and Auerbach did himself a favour by ignoring the study.
As I’ve picked on Dan Ariely for the way he talks about organ donation rates, here’s Kahneman on that same point:
A directive about organ donation in case of accidental death is noted on an individual’s driver licence in many countries. The formulation of that directive is another case in which one frame is clearly superior to the other. Few people would argue that the decision of whether or not to donate one’s organs is unimportant, but there is strong evidence that most people make their choice thoughtlessly. The evidence comes from a comparison of organ donation rates in European countries, which reveals startling differences between neighbouring and culturally similar countries. An article published in 2003 noted that the organ donation rate was closer to 100% in Austria but only 12% in Germany, 86% in Sweden but only 4% in Denmark.
These enormous differences are a framing effect, which is caused by the format of the critical question. The high-donation countries have an opt-out form, where individuals who wish not to donate must check an appropriate box. Unless they take this simple action, they are considered willing donors. The low-contribution countries have an opt-in form: you must check a box to become a donor. That is all. The best single predictor of whether or not people will donate their organs is the designation of the default option that will be adopted without having to check a box. …
When the role of formulation is acknowledged, a policy question arises: Which formulation should be adopted. In this case, the answer is straightforward. If you believe that a large supply of donated organs is good for society, you will not be neutral between a formulation that yields almost 100% donations and another formulation that elicits donations from 4% of drivers.
As Ariely does, Kahneman describes the difference between European countries as being due to differences in form design, when in fact those European countries with high “donor rates” never ask their citizens whether they wish to be donors. The form described does not exist in the high-donation countries. They are simply presumed to consent to donation. (The paper that these numbers come from, Do Defaults Save Lives?, might have been better titled “Does not asking if you can take people’s organs save lives?”. That could have saved some confusion.)
Further, Kahneman talks about the gap between 100% and 4% as donation rates, when these numbers refer to those who are presumed to consent in the high-donation countries. Actual donation rates and the gap between the different types of countries are much lower.
All the above points are minor in themselves. But together the shaky science, overconfidence and lazy storytelling add up to something substantial.
What I also find less satisfying now is the attempt to construct a framework around the disparate findings in behavioural science. I once saw prospect theory as a great framework for thinking about many of the findings, but it is as unrealistic a decision making model as that for the perfectly rational man – the maths involved is even more complicated. It’s might be a useful descriptive or predictive model (if you could work out what the reference point actually is) but no one makes decisions in that way. (One day I will write a post on this.)
It will be interesting to see how Thinking, Fast and Slow stands up after another five years.
Another great post Jason… Agree that the replication crisis has put priming and framing effects on the back foot – but as a business book – which TFS is first and foremost (see the chapter endings), Kahneman does an excellent and valuable job at demolishing the unrealistic over-intellectualised and over-intellectualising model of thought we use to explain and predict behaviour, and to inform business decisions (so-called System 2). We think less than we think we think.
For me the untapped value in TFS lies in the second part of the book that explores the distinction between the ‘experiencing self’ and the ‘remembering self’ – which has a bunch of real-world implications (e.g. peak/end rule suggests that customer experience could adopt a ‘finish-strong’ mantra)…
“as a business book”
And just imagine what this level of scrutiny would do to most other business books.
Here is a decent first step for any and all arguments, models and theories about behavior – and the brain – let’s see evidence of similar mechanisms and processes in other animals who evolved far before humans. A rule of biology is “Descent with modification.” so any behavioral theory or mechanism has to have “descended” from earlier animals. Thus, the human exceptionalism that is the foundation of all economics, and BE models, is false – by definition…duh…
I couldn’t agree more. The hot hands criticism also misses the more valuable message about probabilistic decision making which is far more subject to the probabilities and the laws of large numbers, than any intuitive investing skill.
Until the last, say, three years it seems that almost nobody understood the true extent of rot in (social) science. I certainly didn’t. There were a few guys like Jacob Cohen and Paul Meehl who appear to have realized it a long time ago, but even when their arguments were read and accepted at some abstract level, most people just couldn’t absorb the idea that entire research areas, especially their own ones, could be just smoke and mirrors. So Kahneman’s mistakes are understandable.
These days, however, it seems that lots of people have adopted an attitude of pervasive skepticism towards research that does not fulfill the highest standards — and I think this attitude is very welcome. It’s like we have a new cognitive tool that greatly improves our thinking about science. Previously, this tool was available only to the rare Cohen or Meehl but now it’s accessible to everybody.
TFS is one of the beat books of the past 10 years. Sure some of the studies he based it on have been debunked, but its still a magnificent book, and will always be recognized as the book that introduced “behavioral science” to so many people. It is not a scoentific paper, thats why DK writes with so much “confidence”, he is trying to persuade the audience. Im sure many more studies will be debunked, but the value is in the way of thinking about things.
And im specially amazed about your comment about prospect theory, it seems like you don’t even understand the basic point of PT. It sounds like you interpret it as a model people ought to use to evaluate alternatives, its simply an alternative model that explains much better the decisions we make, compared to the “rational model”. It is not meant to be a physical law.
There is no doubt there are a lot of advancements to be made, you should focus on those , instead of giving presentations looking for errors in the works of the people who showed the way.
The value in their work is that they recognized things were looked at the wrong way, and pointed us in the right direction
Considerable agreement here. Reading this article felt like I was watching someone having a “whoosh” moment at best. At worst it sounds like “seriously I’m smart: look at me taking shots at the big dogs.”
If you spend any time with the literature, you should realize that prospect theory does not present itself an end-all model with which decision making can be perfectly understood. Rather it offers a more robust framing of the topic which corrects for glaring errors in applying expected utility to the real world. One also wonders what Collins means by “more mathematical.” Apparently mathematicalness is terrible, and can be measured by counting variables…
The real value in the book was not its strong detailing of specific effects, but rather the totality of the message: our minds have notable limitations with respect to rationality. It’s written for a lay-audience. If you don’t understand that that may entail more assertive language, perhaps you need to spend more time with people outside of academia.
I’m now reading Thinking Fast and Slow, I find the blog informative and understand how even Daniel Kahneman was subjected to irrational behavior. Thanks for writing.
Pop culture is to make money. Science is to produce facts and truth which are expressed as statements that predict measurable events in the future. The professions solve problems.
The way to make money is to sell people what they want to buy and tell them what they want to hear. Mostly our brains want to hear magical thinking, wishful thinking, to be told what we want to hear which is usually lying. The basis of all pop culture is to pretend, really simple magical ideas work – they don’t. Blame our brains.
Behavioral economists, and economists overall, are great money makers and self promoters. They are not selling truth but to further their careers. DK is one of the best careerists. His ideas are rubbish but rubbish always sells best – for example: “free”, life after death, that people “like” you on facebook, “new and improved.” Nothing in economics is evidence based, economics is just more verbose magical thinking, like philosophy and theology and self-help.
Brain science is really, really hard – and getting harder and more complex as genetics starts to take over. There can no more be pop brain science then pop nuclear engineering, heart surgery or accounting.
There is evidence that Dk and the career beh economists engaged in systematic bad faith decades ago. Predictably;y, it produced smarmy pop culture nonsense. Hey, they are taking ti too the bank, while other academics are on a starvation diet.
Eventually, professionals of all kinds will have to get in line with biological facts and brain science, which is all just physics anyway. Economists, beh econmists and pop culture will be a quaint throw back of really bad and wrong ideas – they made their proponents some money. Ho hum… With anything new the con men always rush to the front of the parade….then leave for another one.
I study and write on this with a book planned, but for professionals – never the general public. Trust me, you do not want to kow the truth about the brainand behavior and won’t take the time to try ot understand it. Approx 80% of the general public, in America believe the character…like, ahh – is a “living” thing… http:brain4biz.wordpress.com
The trouble with behaviourists: 😜
– They think no one can think.
– They don’t believe in beliefs.
– In their opinion, the are no such things a opinions.
The problem with true believers selling “cognition”, free will, “thinking” and human exceptionalism is they lie – and are proudly ignorant of the latest and best medical and brain science. This includes Kahneman. But lying is the basis for most pop success – including pop academic success.
The trouble with people selling “beliefs” – they lie. Try out opinions and beliefs next time a loved one goes to the hospital – RIP.
Regarding organ donations. I just wish to add that Sweden, among those countries with high donation rate, does have an opt-in system for organ donations. Though you have no say over what part of the body might be donated nor can you influence who the organ recipient should be.
However, it seems a bit unwise to speak in too certain terms as Kahneman did, especially in this kind of research.
Regarding transplanations, I’m not sure I see the difference between Kahneman’s “The high-donation countries have an opt-out form” and your description that “those European countries with high ‘donor rates’ never ask their citizens whether they wish to be donors.”
As I read it, he’s not claiming that it’s due to a difference in wording on a specific form, but that the default action is different – either that you are considered a donor if you don’t send in a form (high donor rates), or that you are only considered a donor if you send in a form (low donor rates).
Along the same lines, you write “these numbers refer to those who are presumed to consent in the high-donation countries” but to me that’s exactly what Kahneman says with “Unless they take this simple action, they are considered willing donors”.
So I’m not sure if you’re actually arguing against what Kahneman has written, or if you’re reading him as saying something different than he does.
The “Do defaults…” paper also does include a short section on effects on actual donations: “there is a strong effect of the default: When donation is the default, there is a 16.3% (P < 0.02) increase in donation … looking only at 1999 for a broader set of European countries … Gimbel et al. (5) report an increase in the rate from 10.8 to 16.9, a 56.5% increase”.
He is claiming there is a difference in wording on a specific form – “These enormous differences are a framing effect, which is caused by the format of the critical question.” Similarly, it is not that “you must check a box to become a donor” – there is no box to check. The simple action Kahneman implies they need to take – checking a box – is instead knowing that you are a presumed donor, contacting the relevant Government department to get a form to register as not wanting your organs taken, completing and lodging the form.
The paper does, of course, have a section on actual donations. Yet Kahneman instead chooses the more striking numbers.
I took a look at actual donation rates in Nordic countries.
http://emilkirkegaard.dk/en/?p=6093
So, dual process brain-behavior processes: 1) Are unique to humans?, 2) Evolved at what point in proto-human-primate-mammal evolution? 3) Brain processes, across all species have 2 modes!?
These are truly silly ideas, but easy to sell. DK is either deeply ignorant about animal-human brain neurology or deeply dishonest – or both. But the, so-called, “Nobel”prize in econ seems to reward both.
Look, if anything as simple minded as “nudges” can remediate human behavior – great. It will not be that simple…..
Excellent article pointing out some of the dangers in being overly assertive in writing a general text for public consumption. Ever since Kahneman and his followers hit the airwaves with this fast and slow thesis, I have been wondering if research has been done on the actual dynamics behind a sudden very rational idea as it’s formed, such as the one that we should not be taken in by a sudden intuition or hunch that seems to be accurate – even if it seems to be strongly urging us on to take action or to accept that it’s accurate. For example, If we tease apart what occurs when a supposedly ‘slow’ process is used to come to some kind of a final well reasoned and evidence-based decision, for instance, we might actually find that there are several points of sudden ‘fast’ insights and decisions that are perhaps subconsciously taken as that unfolding slow process takes place….. One needs to be careful in not jumping to conclusions too quickly, however, if such a fast action is to be robustly investigated and tested, of course!……..
Kahneman’s ideas are ignorant and wholly uninformed about the current and best new science. They are easily debunked by biology, evolutionary principles, medicine, physiology and anatomy, animal ethology, molecular and cellular biology – and common sense.
He is a smarmy con man, but so are most economists-humanities folks, social scientists and philosophers.
Behavior is a medical and physiological science – period. Other ideas are a waste of time.
Currently reading the book, will take the skepticism home.Taking a grain of salt is always useful in the face of any information from any sources in the pursuit of truth.
Sadly, but predictably, the ideas in behavioral and neuro econ are so misinformed and wrong – according to the medical facts – they deserve to just be ignored. But economists tell ppl what they want to hear so, like religion, they are popular.
He also claims that those who “predicted” the financial crisis in 2008 only did so out of luck which just shows that he does not understand how economy works.
He should watch “The big short” and read up. Because it was obvious that things were just like they were just before 1930 and that it could only lead to a crash.
It was just a matter of months or year before it would come crashing down.
I doubt a movie has much intelligent to contribute to – anything. Nothing in economics, finance or the other humanities is evidence-based so can never predict. Animal behavior is immensely complex, human behavior maybe even a bit more so. It will take decades, generations and perhaps centuries to understand – and predict.
Currently, brain science and animal ethnology is the best knowledge. Econ, the other humanities and social sciences are just trading in smarmy, pop cultural beliefs and cant – aka – lying. But lies always sell best.
Most of the comments above confirm why I felt underwhelmed and uncomfortable reading TFS. Too slick, too anecdotal, weak on applying good science.
We do have to give credit to the behavioral econ folks for being excellent marketers. self promoters and careerists. If you look at the history they were driven by major foundations. Sadly but predictably, their personal success has blocked real brain science based problem solving. What economists and other social scientists and philosophers who have zero training in brain science, biology and medical physiology have to contribute to behavioral matters is mainly to boost their careers with silly pop culture ideas.
Kahneman and Tversky do not offer a binary choice between man as a completely rational decision-maker and man as a decision-maker who ALWAYS makes systematic errors. They say that very often the decisions made are correct and rational, but that they can be subject to systematic errors and that we must be aware of this possibility.
Have just been reading the book and must agree that it struck me (in the era when so much social science is being called into question) in the same way as JC, even allowing that it was written for the lay public.
As for all the blather of BMM about brain science — it only shows BMM has no idea where brains science really is today.
One could say that unless everything is proven true, nothing should be dared. The comments herein have so far been nothing short of rubbish. Mr Kahneman’s contribution (book) is to be applauded, praised and thought after. It never seizes to amaze me how fast some people are willing to criticise things they don’t understand. What a surprise there no less.