What Medical Science Can and Can’t Do
I’m an academic researcher studying the functions and dysfunctions of the microcirculation. Recently I attended the Experimental Biology conference in Chicago. Nearly all the big players in physiology research were there, and I got to see presentations on topics ranging from the body’s responses to lung injury in sepsis to the effects of behavioral interventions aimed at reducing childhood obesity. I even got to brainstorm with some of my favorite scientists on the possible theoretical implications of data they’ve published. There’s a lot of exciting advances happening, and it’s a phenomenally interesting time to be alive as a medical researcher. All around, it was a very positive, enlightening experience.
Unfortunately, one thing I noticed is that a lot of medical research happening right now doesn’t appear to be groundbreaking and probably won’t withstand the test of time. I had previously flirted with these same suspicions due to my observations from reading the literature and attending smaller conferences, but this year’s Experimental Biology conference pushed me over the edge to accepting the uselessness of a plurality of academic research as part of my worldview. This is not an indictment of the average medical researcher (who is, to the contrary, a highly skilled and hard working individual with a strong desire to help others), but instead a meta-level critique of the frontiers that the scientific community at large has chosen to pursue.
For example, I saw scores of posters making claims to the effect of “molecule X controls process Y.” Sometimes that kind of research is useful. Insulin fits neatly into this paradigm (insulin controls blood glucose), and I personally would not have lived to see six years old without that discovery. Much more often, however, “molecule X controls process Y” findings turn out to have missed something. Maybe molecule X needs molecules P, Q, and R in just the right amounts to work, and those other molecules have dozens of upstream regulators of their own. Maybe molecule X is important in mice/rats but doesn’t exist in humans, or does exist but does something completely different. Maybe process Y turns out to have been a red herring to begin with. In any case, I predict that most of those “molecule X controls process Y” studies I saw last week will be totally forgotten ten years from now.
Of course, just because a study doesn’t give us the next breakthrough drug doesn’t mean that it’s a waste of time. We do learn quite a bit from studies that yield promising results in one context but fail to replicate in another. But there’s a fundamental problem with this learn-by-failure approach, and the “molecule X controls process Y” paradigm makes a good example.
To illustrate, let’s derive an estimate of the number of potential molecule X’s we could investigate. For simplicity, we’ll only consider proteins. Proteins are made up of sequences of amino acids, and any one of 28 possible amino acids can fill each position in the sequence. The largest known protein is titin (gives muscles their elastic recoil), which contains about 30,000 amino acids. The smallest known protein is trp-cage (Gila monster venom), which contains 20 amino acids. With 28 possible amino acids, the number of permutations in amino acid sequence contains every combination ranging from 20 to 30,000 in length, or 28²⁰+28²¹+28²²…+28³⁰⁰⁰⁰. Crunch the numbers, and that yields an upper bound of 3.6*10⁴³⁴¹⁵ unique proteins that could theoretically exist in the body, as compared to about 10⁸⁰ atoms in the known universe.
This analysis doesn’t even take into account the possibility that molecule X might not be a protein, and it also doesn’t take into account permutations in protein folding patterns, glycosylation, and protein-protein interactions. The number of possible molecule X’s you could investigate as potential regulators of process Y is literally innumerable, and that’s before you even consider the possibility that process Y might actually be a red herring in your real goal, which is usually to cure, prevent, or treat some disease or another. The process of elimination is not a valid approach to molecular biology.
The crux of the issue is this: Unless the hypotheses tested are constrained by sufficient theoretical consideration and empirical observation, hypothesis-driven biological research is bound to turn up far more false positives than true positives, no matter how rigorous the methods employed may be.
What Medical Science Can Actually Do
Observational medical science is, for the most part, really good at finding and documenting relationships that show up in the real world. We know that people who lift heavy weights tend to have stronger spines, and that people who run tend to have stronger leg bones. We know that people who drink sugary drinks tend to be fatter, and that people who exercise regularly tend to live longer. We know that people who are fat tend to be afflicted with chronic diseases, and that people who live in neighborhoods with nowhere to go for a walk are generally fatter. In cases where the findings can be replicated and have a likely causal link (as is true for all the examples I just gave), this sort of information can be very useful.
When we fully understand why a causal link would be present, or when the effect size is unusually large, we’re also really good at experimentation to figure out what we can do with the phenomena we’ve observed. Low-calorie diets really do cause weight loss and limit inflammation. Exercise really does improve a variety of health outcomes. Vaccines really do prevent infectious diseases. As long as the findings are robust and can be replicated (as is true for all of the examples I just gave), intervention studies work well to reveal the tools that we can use to improve human health.
With a detailed enough view of what happens in the real world and sufficient knowledge of which parameters influence which, we can create detailed theories to model how it all works. This is especially true when we can invoke the properties of mathematically well defined, archetypal natural phenomena (such as the laws of physics) to explain the associations and interactions we’ve observed. A clearly defined theory adequately informed by prior observations enables us to predict phenomena we would not otherwise have known to look for. At this stage, the formal Scientific Method is an excellent way to find out whether our theory is broadly valid or not. All models are wrong; some models are useful; a theory becomes useful when its predictive value is demonstrated.
Insulin therapy for type 1 diabetes is illustrative of each of these arenas in which medical science can and does succeed. For centuries, doctors had observed that people with diabetes were excessively thirsty, urinated excessively, and that their urine was very sweet. Once these symptoms appeared, the patient would waste away and die a gruesome death within a few weeks or months. After sufficient documentation of these observations over the years, the natural course of untreated type 1 diabetes became well known in the scientific community.
Eventually, physicians drew upon the observation that diabetics’ urine was unusually sweet to try feeding them low-carbohydrate diets. This was effective in slightly extending their patients’ lifespans, and thus it became a part of the medical dogma that diabetes is a disease of sugar mismanagement. In the late 19th century, scientists Oscar Minkowski and Joseph von Mering at the University of Strasbourg discovered by chance while studying the role of the pancreas in digestion that removing a dog’s pancreas would cause the dog to develop diabetes. Around the same time, German physiologist Paul Langerhans discovered “islets” of cells in the pancreas that didn’t appear to be connected to the ducts leading to the intestines.
In the early twentieth century, Canadian researchers Frederick Banting and Charles Best discovered that an extract made from these pancreatic islets could treat diabetes in dogs. This discovery won them the Nobel prize, and opened up avenues for private sector development of advanced insulin formulations and delivery systems which, although overpriced, are a technology to which diabetics like literally owe our lives.
The story doesn’t end there, though, because insulin injections are a poor substitute for physiological insulin secretion. The algorithms I use to decide my insulin dose are much less sophisticated than those that the pancreas uses. Over the course of the twentieth century, further observational research illuminated the major sites and mechanisms of insulin transport and action. Enter Richard Bergman, a former engineer, who derived a minimal but comprehensive model of insulin and glucose transport. This theory proved to be quite robust, and in the twenty years since, other engineers have incorporated it’s predictive value into the Minimed 670g, the first insulin pump cleared by the FDA to automatically determine basal insulin infusion rates. I’m incredibly excited by the prospect of having my blood sugar management on autopilot overnight, and that possibility is only available to me due to the successes of medical research.
Now let’s recap the story of insulin therapy using the paradigm I laid out at the beginning of this section- 1) Observation: physicians observe that diabetes is a disease of sugar dysregulation, 2) More observation: researchers discover by chance that removing the pancreas causes diabetes, 3) Even more observation: anatomists discover that there are clusters of cells in the pancreas that don’t appear to be involved in digestion, 4) Experimentation: injecting an extract from these pancreatic islet cells to see if it treats diabetes in dogs, 5) Observation again: observing when and where insulin acts in vivo, 6) Theory: Bergman’s minimal model of insulin and glucose transport, 7) Empirical validation of theory: twenty or so years of experiments defining the scope of where the minimal model does/doesn’t work, and let’s not forget 8) Commercial development: advanced formulation of insulin, application of the minimal model, development of the necessary biosensors, etc.
The development of insulin therapy is arguably one of the greatest achievements in the history of medical science. Rather than dying in early childhood, people like me become cyborgs and productive members of society. And you know what? There was surprisingly little application of the canonical “Scientific Method” (come up with a hypothesis and then experimentally test it) in the process that led us to the futuristic, self-dosing insulin pumps that are now reaching the market. Much more of what wound up being useful in the long run was simple observational reports. When the formal Scientific Method was used, it was mainly within the context of testing an explicit, mathematical theory that had already been published on its own merits (more on this later) and couldn’t be revised post-hoc to simply make a good story out of the data.
What Medical Science Can’t Do
Medical science can’t save people from themselves. The medical research community is general pretty good at coming up with technological solutions to health problems. But there are many cases where, from a cost-benefit standpoint, even the very best technological interventions are objectively inferior to lifestyle interventions. To use a relatively non-controversial example, the CDC spent around $54 million on anti-smoking advertising in 2012. Also in 2012, the NIH spent around $233 million on lung cancer research. Smoking makes you around 100 times more likely to get lung cancer. With modern technology, there’s still a greater than 80% chance of dying within five years once you’re diagnosed with lung cancer. I can guarantee you without any reservations that dollar for dollar, the CDC’s cancer prevention spending bought more years of human life and alleviated more human suffering than did the NIH’s lung cancer research spending.
Science is a great tool for obtaining knowledge of how to solve or avoid problems, but that knowledge isn’t actually useful unless we put it to work. To continue with the smoking example, once we found out that smoking causes lung cancer (which was many decades ago, for what it’s worth), we got much more bang for our collective buck by creating a stigma towards smoking than by developing fancy chemo drugs to treat lung cancer. Fancy chemo drugs are still a good thing, of course, but relying on hypothetical future technologies to save us from our own decisions would be poor risk management. We, as a society, did the right thing by making smoking taboo, and that taboo is not something that scientific research did for us. Science is not an engine of moral responsibility.
Now let’s take this concept a step further. Right now, about half of american adults have diabetes or prediabetes. Diabetes costs the healthcare system around $13,700 per patient per year, and is a leading cause of kidney failure, blindness, and amputations. We’re talking about a disease that can literally make your flesh rot off of your bones. The NIH spends around a billion dollars per year on diabetes research (and pays my salary in the process), and when I was at Experimental Biology last week, it showed: dozens of posters about how to prevent diabetes or it’s complications. But you know what? We already pretty much know the answer to that question. Diet and exercise can prevent type 2 diabetes.
Of course, learning more about the human body and the diseases that can afflict it isn’t a bad thing at all. But no amount of NIH funding is going to fix the diabetes epidemic (which is, in reality, just one small facet of the obesity epidemic). Genuinely fixing these problems will require a cultural change, similar to what we saw with smoking. We need social taboos for sedentary leisure behavior and junk food just the same as we needed them for tobacco use. We now know enough about why type 2 diabetes happens to solve the problem. Scientific knowledge isn’t the limiting factor here. We just need to find the will to actually act upon the solutions that we already know will work, and the same goes for most other lifestyle-related diseases, which comprise a big part of the USA’s healthcare system issues.
Medical science also cannot generate meaningful hypotheses without adequate background knowledge. This goes back to the heart of my criticism of the “molecule X controls process Y” approach to biotech research. If your hypothesis leaves some wiggle room (e.g. which way does the effect on process Y go?), you’re going to spit out false positives. If your hypothesis lacks theoretical understanding (e.g. why would molecule X do that?), you’re going to spit out false positives. And if your hypothesis would be equally tenable with some substitutions (e.g. many other molecule X’s would make equally plausible candidates), you’re going to spit out false positives. And when the scientific community publishes a lot of false positives, we lose legitimacy as objective arbiters of knowledge.
The difference between a legitimate hypothesis and a meaningless one is the degree to which the hypothesis is constrained by prior knowledge. The hypothesis that insulin regulates blood glucose falls neatly into the “molecule X controls process Y” paradigm that I’ve roundly mocked, but there’s a critical distinction- the scientific community already knew that the molecule X they were looking for was in the pancreas, they knew that it functioned independently of the pancreas’ digestive functions, they knew that there were clusters of cells in the pancreas that weren’t anatomically associated with the digestive portions of the organ, and before testing the hypothesis that insulin was the molecule they were looking for, they knew that an extract prepared from the pancreatic islets lowered blood glucose.
With those constraints, there are only so many candidates for molecule X, and trial and error actually is a valid approach. Where I think a lot of the projects I saw at Experimental Biology went wrong is that the necessary background knowledge just didn’t exist. At best, the background section of most of those posters boiled down to “this molecule is up/down in disease, and this process is up/down in disease.” Lots of things change in disease. “X and Y are both different than they were before” is not a good reason to suspect causality. In order for any formulaic scientific hypothesis (of which “molecule X controls process Y” is just one of many templates) to be taken seriously, there ought to be 1) sufficient prior observations to make us question the likelihood of a spurious relationship, and 2) a good theoretical reason that the hypothesis would be true.
And with that, let me speak briefly to the concept of theory in medical science. Good theories, such as the minimal model of insulin transport and action I described in the previous section, tend to share three key attributes. The first is that they are tightly constrained by background knowledge, just like a good hypothesis. Otherwise it’s a fantasy, not a theory. The second is that they’re defined in explicit, usually mathematical terms. Otherwise they’re subject to interpretation and confirmation bias. The third is that they have few enough variables that you can actually prove them wrong. Otherwise it’s at best a statistical learning technique, and not a theory. The theory of general relativity and Bergman’s minimal model of glucose transport both meet all three of these criteria.
Many contemporary theory papers in bioscience, however, do not. I saw a handful of computational projects at the conference that relied on speculation in place of observation to constrain the mathematical relationships that they explored. Some of them had dozens of variables, many of which can’t practically be measured for purposes of experimental validation. Computational projects are pretty much always defined in explicit mathematical terms, but outside of computational bioscience, I saw some well-respected researchers present “theory” graphics that could be interpreted to nearly the same degree as an ancient religious prophecy could be. These trends are especially dangerous when some researchers present the implications of their theories as conclusions rather than as hypotheses that will need to be tested experimentally.
What We Can Do Better
It wouldn’t be very constructive of me to write up a long diatribe about problems within the scientific community (and in society’s relationship with science) if I didn’t also propose some practical solutions. In this section, I’ll give my two cents about how we can do better. By “we” I mean both the scientific community and the general public. Since I’m primarily writing for a lay audience in this essay, I’ll avoid going into excessive detail, and instead just stick with the big picture. If you’re a scientist, I’d encourage you to think about these next few paragraphs when you write your next grant. If you’re not a scientist, I’d encourage you to consider these ideas in your own lifestyle management and in your interpretation of scientific findings reported in the news media.
- Drop the pretense of “translational academic research.”
The institutional structure of academic research is much better suited to figuring out the rules of the game than it is to developing technologies to exploit them. While Richard Bergman may have been an academician when he derived his mathematical models of glucose metabolism, it wasn’t him that translated those theories into clinical practice. For the most part, it was companies like Medtronic that did that. Likewise, it wasn’t Banting and Best that made insulin therapy available to the masses; that was companies like Eli Lilly. It’s all the rage right now in academic medicine to look for “therapeutic targets” and to belabor the “translational potential” of new findings. That’s all a waste of energy in most cases. The free market will translate your discoveries for you as soon as there’s a viable path to a cost-effective therapy. What we really need from academic bioscience right now isn’t another two-dozen failed attempts to cure cancer, but instead comprehensive documentation of the phenomena that show up in the real world and plausible, testable theories to explain them. - Publish theory papers as a superior form of hypothesis pre-registration.
An alarming number of scientific papers fail to replicate. Some people assume foul play, but there’s also a much less cynical explanation: lots of false positives. A lot of those false positives happen when people don’t find what they’re looking for and then go digging through the data set to see what was statistically significant. The obvious solution to this problem, which has received plenty of attention, is to have scientists publish their hypotheses in advance rather than back-fitting their narrative to whatever shows up in the data. That’s a great idea, but it’s not much use if all we learn is that 99% of our hypotheses are wrong. As I’ve said several times before in this article, hypotheses need to be adequately constrained by prior knowledge. One mechanism that already exists in the scientific community for preregistration of exactly these well-constrained hypotheses is publication of theoretical papers. With a clearly defined theory, every hypothesis that logically follows from the theory is effectively preregistered upon the theory’s publication. Subsequent experimental testing then tells us if our interpretation of prior observations is grounded in reality. Publishing this sort of theory paper requires a fair bit of moral courage (if you’re wrong you could wind up losing your job), but it’s the most robust mechanism we have right now to differentiate confirmation bias from genuine understanding. - Use science for what it is, not as a universal authority.
In this section, I’ll return to the idea that social stigma and/or cultural values may be more useful than science in addressing the obesity (and diabetes and heart disease) epidemic. In the post-enlightenment Western world, there’s a strong temptation to treat scientific expertise as a sort of religious authority of the new church. This tendency is called scientism, and philosophers such as Nietzsche and more recently, Nassim Nicholas Taleb, have written prolifically on the topic. The relevance of scientism to the message of this essay is as follows: science can tell you the rules of the game, but it can’t decide your strategy for you, much less execute upon that strategy. When it comes to the lifestyle related diseases that are currently swamping the healthcare system, we already know more than enough about the rules of the game. We just need to eat our veggies and play outside. If instead we eat french fries and watch TV all day, our flesh will degenerate into a festering mess. Science can figure those rules out for you, but it can’t change them. It’s up to you to follow through on living a healthy lifestyle or suffer the consequences of an unhealthy one. - Wait for replication before believing anything.
Or, if you don’t read enough scientific literature to follow attempts at replication of experimental findings, wait and see if we’re all still talking about it in 5–10 years before you believe it. There’s more than enough questionable science out there for exciting findings to turn out to be completely wrong. There’s also more than enough questionable science journalism out there for mundane findings to be over-hyped. Even without this sort of foul play (which I believe is a small minority of the problem), false positives happen. Some findings withstand the test of time, while others turn out to have been flukes or frauds. Time is the easiest way to tell the difference between real findings and flukes. And it this stage, I should also explicitly state the converse: if a finding has been replicated and we really have been talking about it for ten-plus years, it’s probably time to incorporate it into your worldview. It’s good to be skeptical, but it’s no good at all to be a know-nothing or live in denial.
In summary, we need to reign in the scope of the hypotheses we consider worthy of testing, we need to admit that while science can help us find solutions, it can’t act upon them, and we need to be less credulous about novel findings. Since I want to finish on a positive note, I should point out that there was also a lot of really good medical science presented at Experimental Biology this year. I’d like to think that my presentation was one of the good ones. And we don’t need all of the science, or even most of the science, to be done right in order to advance. We just need enough of the science to be done right, and I’m quite confident that this is a tenable goal.