(Note: this blog post was originally submitted on 09/14/2010)
"When an hypothesis is submitted to the experimental method, it becomes a theory, while if it is submitted to logic alone, it becomes a system. A system, then, is an hypothesis with which we have connected the facts logically with the help of reason, but without experimental, critical verification. A theory is a verified hypothesis, after it has been submitted to the control of reason and experimental criticism. The soundest theory is one that has been verified by the greatest number of facts. But to remain valid, a theory must be continually altered to keep pace with the progress of science and must be constantly resubmitted to verification and criticism as new facts appear.
"If we consider a theory perfect and stop verifying by daily scientific experience, it becomes a doctrine. A doctrine, then, is a theory which we regard as immutable, which we take as a starting point for later deduction, and which we believe we are no longer obliged to submit to experimental verification." -Claude Bernard, An Introduction To The Study of Experimental Medicine, 1957.
I’ve been following what is considered to be the induction phase of the Atkins diet for more than 10 years, so when the latest headlines were smattered across the pages of the major news outlets (just after I got published in the LA Times for following such a diet) telling me my Atkins diet is going to send me to an early grave, as CBS News reported, I couldn’t help but listen.
Newspaper headlines generally try to give the reader an idea of the story that follows. See if you can determine what the following articles are about given their headlines:
Atkins Diet Mortality: Is Too Much Meat Killing Low-Carb Eaters? (CBS News)
Low-Carb and Healthy: New Study Finds Atkins Got It Half-Right (TIME)
Meat Protein Risky in Low-Carb Diets (WebMD)
Atkins Diet Increases All-Cause Mortality (Huffington Post, Dean Ornish)
Animal-based protein diets increase mortality rate (CNN)
Cutting carbs? Replace them with proteins from plants (USA Today)
Low-Carb Diets Heavy on Meat May Raise Health Risks (BusinessWeek)
On a low-carb diet? You may live longer if you make it vegetable-based (Los Angeles Times)
I have replaced refined carbohydrates, such as sugar, flour, and rice with animals to become more healthy, but that’s not going to help me, according to Dr. Frank Hu, one of the authors of the study that has attracted the headlines (1). Hu also happens to be a professor of nutrition and epidemiology at the Harvard School of Public Health.
"You can have the initial Atkins-type of low-carb diet, which is loaded with sausages, bacon, steaks, and you can have healthy versions of the low-carb diet with more vegetable- or plant-based protein and fat,” said Dr. Hu, in an interview with Bloomberg Businessweek (2).
Sounds like Hu might have a little bias towards vegetable-based, i.e., "healthy versions" of the low-carb diet, no?
"We looked at these two versions of low-carb diets and found that the impact of the two are drastically different. Those who follow the animal-based low-carb diet have an increased risk of total mortality and cancer mortality in particular."
The “initial Atkins-type of low-carb diet,” according to the Atkins diet, is called the induction phase. Carbohydrates are limited to 20 net grams per day (fiber grams are subtracted from the carbohydrate total), and all the meat, fish, fowl, and eggs you can eat are allowed during this phase. Up to 3-4 ounces of cheese and up to three cups of vegetables are also allowed per day.
Technically, the Atkins diet could look like what Dr. Hu is suggesting, with copious amounts of bacon and sausage (just make sure it’s not processed, since these should be limited, according to Atkins), and it could also consist of wild-caught sockeye salmon, grilled chicken, and an arugula salad, with sides of mushrooms, spinach, artichokes, and onions, for example (but that doesn't sound as homicidal and thus it usually isn't invoked in these sound bytes).
But the induction phase of the Atkins diet cannot look like the version that Hu and colleagues observed in their epidemiological study.
The authors used a “low-carbohydrate diet score (Halton et al., 2006 (3))” based on food-frequency questionnaires, to determine just how low-carb their low-carb dieters were. The study collected the information from their subjects and then placed them into ten different categories, or deciles, with “1” representing the lowest adherence to a low-carbohydrate diet, and “10” representing the highest level of adherence and most carbohydrate restriction. Both the “animal-based” and “vegetable-based” diets had their own decile for comparison. There was a 30-point ceiling, in which as many as 10 points could be given for eating low-carbohydrate, 10 for eating high fat, and 10 for eating high protein.
Because the researchers were allegedly looking at the initial phase of the Atkins diet, they focused on the tenth decile of the animal-based low-carbohydrate diet. This group was getting 37% of their daily caloric intake of approximately 1700 calories from carbohydrates. The dieters were eating over 157 grams of carbohydrates per day, a 686% increase in the amount of carbohydrates they were supposed to be eating if they were adhering to the induction phase of the Atkins diet. In other words, Hu and colleagues did not look at the diet they alleged to be looking at.
Chris Masterjohn (4) wrote that the authors had a “pretty funny way of determining what constitutes ‘low-carbohydrate’.” The highest protein decile consumed more than 26 percent of their calories as protein, the highest dietary fat decile consumed more than 47 percent of their calories as fat, and those with the lowest carbohydrate intake consumed less than 29 percent of their calories as carbohydrates.
Masterjohn notes, “this means that if you consumed 10 to 20 percent of your calories as carbs, it wouldn’t bump your low-carb score up any more than someone who ate 25 percent of their calories as carbs. If you ate 50 percent of your calories as fat and 15 percent of your calories as protein, you couldn’t gain any points from eating 10 more percent of you[r] calories as fat, but you could gain a whopping NINE POINTS out of a thirty point score by eating 10 more percent of your calories as protein.”
The scoring was based off another paper from the Nurses’ Health Study (Halton et al, 2006) that stated “those with the lowest carbohydrate intake received 10 points and those with the highest carbohydrate intake received 0 points.”
Based on a 2000 calorie a day diet, the typical induction phase of Atkins calls for 20 grams of carbohydrate consumption, which amounts to 4% of calories as carbohydrates. Consuming 37% of calories as carbohydrates (as the people observed) is an eight-and-a-half-fold increase in the percentage of carbohydrates consumed.
Dean Ornish recommends (or at least recommended before he started claiming an “Eco-Atkins” diet as his own, or “essentially the same diet that I have been recommending and studying for more than 30 years" (5)) less than 10% of calories from fat. If we keep calories the same (2000 kcal/d), and the authors said they were looking at an ‘Ornish-type diet,’ and the subjects were reportedly eating 85% of calories from fat (an eight-and-a-half-fold percentage increase), would Dean Ornish accept this as a reasonable representation of his diet plan? It would actually be a more accurate representation of an “Atkins-type” induction diet.
For Ornish to publish an article with the title “Atkins Diet Increases All-Cause Mortality” is a mighty ironic, or dare I say, hypocritical, sword to wield given his objections to interpretations of a “low-fat” diet using similar methodology.
In a recent article in the Huffington Post, David Katz noted (6), “The other principal complaint of the Atkins' camp is that this isn't the Atkins' diet. Perhaps not, but ... soy what?”
Katz is implying that the Atkins diet is a moving target and that the new version of the Atkins book emphasizes more plant sources from protein, however, one component of the Atkins diet that has never changed is restricting carbohydrates, and restricting them to no more than 20 grams per day during induction.
Should proponents, followers, or employees of the Atkins approach not care after reading news headlines insinuating that an Atkins diet is going to send them to an early grave (“it just might,” according to CBS News (7)), and then find out that the diet the authors were referring to doesn’t even come close to the Atkins diet? "Soy what?"
What if I published a study that concluded “drinking a couple glasses of wine per day decreases your life span,” but as it turned out, my “low-wine” drinker-decile were actually reportedly consuming more than three BOTTLES of wine per night. They were drinking 16 glasses of wine, but 'so what?' Despite what you would like to believe, Dr. Katz (“Do I think eating a high-meat, low-plant diet increases risk of death and disease? Hell ya!”), nutrition is still a science, and it should be treated as such.
Most people agree that limiting fat intake to 30% or less calories from fat is considered a low-fat diet. I’m sure, as mentioned before, Dean Ornish would disagree with this classification, and if a study published saying that a low-carbohydrate (40%+ calories as carbohydrates, for example) diet was more effective at weight loss than a low-fat diet (30%+ calories from fat), it wasn’t a true test of the Ornish diet or the Atkins diet. And researchers shouldn’t be able to conclude that an Atkins diet was more effective than an Ornish diet for weight loss in this case, despite many author’s inclinations to do just that. These types of observations illuminate the need for a more rigorous clinical standard in diet studies. Terms such as “low-carbohydrate” and “low-fat” should not be blanket terms for various levels of carbohydrate and fat intake, respectively, or a clinical definition needs to be established so there is less confusion and error when interpreting the results of diet studies.
Because carbohydrate-restricted diets, are not really diets per se, that is, they don’t restrict calories, it’s difficult to define them by percentage of calories since this will vary from individual to individual. Instead, a focus should be placed on the absolute intake of carbohydrates, such as 20 grams or less for the classification of a “severely-restricted-carbohydrate diet,” for example. With low-fat diets, they tend to be calorie-restricted, which makes definitions based on percentage of calories more feasible, but less effective in comparing a low-fat, calorie-restricted diet to a low-carbohydrate, unrestricted-calorie diet.
An intake of 37.2% of carbohydrates is a “moderately high carbohydrate intake,” as William Yancy pointed out in an editorial (8) that ran in the same issue of the study in question. Also, groups of individuals in the study still included in the “low-carbohydrate” stratum, were averaging more than 60% calories from carbohydrate, which stretches the imagination of the definition of “low-carbohydrate” beyond belief. However, this is perfectly acceptable methodology at Harvard University and the peer-reviewers don't see fit to question it, either.
Yancy also pointed out several noteworthy issues, among them that several dietary assessment and classification issues cloud interpretation of the study and that dietary intake patterns can change over time in an individual, especially when it’s over the course of three decades. Also, in observational studies, subjects are not assigned or randomized to a specific diet, so diet labels such as “low-carb” and “low-fat” are retrospective attributions. Not only that, there are no clinical standards for what constitutes a “low-fat” or a “low-carb” diet.
To give you an idea of the different intakes of carbohydrates in different studies and approaches, take a look at the following table:
"You can have the initial Atkins-type of low-carb diet, which is loaded with sausages, bacon, steaks, and you can have healthy versions of the low-carb diet with more vegetable- or plant-based protein and fat,” said Dr. Hu, in an interview with Bloomberg Businessweek (2).
Sounds like Hu might have a little bias towards vegetable-based, i.e., "healthy versions" of the low-carb diet, no?
"We looked at these two versions of low-carb diets and found that the impact of the two are drastically different. Those who follow the animal-based low-carb diet have an increased risk of total mortality and cancer mortality in particular."
The “initial Atkins-type of low-carb diet,” according to the Atkins diet, is called the induction phase. Carbohydrates are limited to 20 net grams per day (fiber grams are subtracted from the carbohydrate total), and all the meat, fish, fowl, and eggs you can eat are allowed during this phase. Up to 3-4 ounces of cheese and up to three cups of vegetables are also allowed per day.
Technically, the Atkins diet could look like what Dr. Hu is suggesting, with copious amounts of bacon and sausage (just make sure it’s not processed, since these should be limited, according to Atkins), and it could also consist of wild-caught sockeye salmon, grilled chicken, and an arugula salad, with sides of mushrooms, spinach, artichokes, and onions, for example (but that doesn't sound as homicidal and thus it usually isn't invoked in these sound bytes).
But the induction phase of the Atkins diet cannot look like the version that Hu and colleagues observed in their epidemiological study.
The authors used a “low-carbohydrate diet score (Halton et al., 2006 (3))” based on food-frequency questionnaires, to determine just how low-carb their low-carb dieters were. The study collected the information from their subjects and then placed them into ten different categories, or deciles, with “1” representing the lowest adherence to a low-carbohydrate diet, and “10” representing the highest level of adherence and most carbohydrate restriction. Both the “animal-based” and “vegetable-based” diets had their own decile for comparison. There was a 30-point ceiling, in which as many as 10 points could be given for eating low-carbohydrate, 10 for eating high fat, and 10 for eating high protein.
Because the researchers were allegedly looking at the initial phase of the Atkins diet, they focused on the tenth decile of the animal-based low-carbohydrate diet. This group was getting 37% of their daily caloric intake of approximately 1700 calories from carbohydrates. The dieters were eating over 157 grams of carbohydrates per day, a 686% increase in the amount of carbohydrates they were supposed to be eating if they were adhering to the induction phase of the Atkins diet. In other words, Hu and colleagues did not look at the diet they alleged to be looking at.
Chris Masterjohn (4) wrote that the authors had a “pretty funny way of determining what constitutes ‘low-carbohydrate’.” The highest protein decile consumed more than 26 percent of their calories as protein, the highest dietary fat decile consumed more than 47 percent of their calories as fat, and those with the lowest carbohydrate intake consumed less than 29 percent of their calories as carbohydrates.
Masterjohn notes, “this means that if you consumed 10 to 20 percent of your calories as carbs, it wouldn’t bump your low-carb score up any more than someone who ate 25 percent of their calories as carbs. If you ate 50 percent of your calories as fat and 15 percent of your calories as protein, you couldn’t gain any points from eating 10 more percent of you[r] calories as fat, but you could gain a whopping NINE POINTS out of a thirty point score by eating 10 more percent of your calories as protein.”
The scoring was based off another paper from the Nurses’ Health Study (Halton et al, 2006) that stated “those with the lowest carbohydrate intake received 10 points and those with the highest carbohydrate intake received 0 points.”
Based on a 2000 calorie a day diet, the typical induction phase of Atkins calls for 20 grams of carbohydrate consumption, which amounts to 4% of calories as carbohydrates. Consuming 37% of calories as carbohydrates (as the people observed) is an eight-and-a-half-fold increase in the percentage of carbohydrates consumed.
Dean Ornish recommends (or at least recommended before he started claiming an “Eco-Atkins” diet as his own, or “essentially the same diet that I have been recommending and studying for more than 30 years" (5)) less than 10% of calories from fat. If we keep calories the same (2000 kcal/d), and the authors said they were looking at an ‘Ornish-type diet,’ and the subjects were reportedly eating 85% of calories from fat (an eight-and-a-half-fold percentage increase), would Dean Ornish accept this as a reasonable representation of his diet plan? It would actually be a more accurate representation of an “Atkins-type” induction diet.
For Ornish to publish an article with the title “Atkins Diet Increases All-Cause Mortality” is a mighty ironic, or dare I say, hypocritical, sword to wield given his objections to interpretations of a “low-fat” diet using similar methodology.
In a recent article in the Huffington Post, David Katz noted (6), “The other principal complaint of the Atkins' camp is that this isn't the Atkins' diet. Perhaps not, but ... soy what?”
Katz is implying that the Atkins diet is a moving target and that the new version of the Atkins book emphasizes more plant sources from protein, however, one component of the Atkins diet that has never changed is restricting carbohydrates, and restricting them to no more than 20 grams per day during induction.
Should proponents, followers, or employees of the Atkins approach not care after reading news headlines insinuating that an Atkins diet is going to send them to an early grave (“it just might,” according to CBS News (7)), and then find out that the diet the authors were referring to doesn’t even come close to the Atkins diet? "Soy what?"
What if I published a study that concluded “drinking a couple glasses of wine per day decreases your life span,” but as it turned out, my “low-wine” drinker-decile were actually reportedly consuming more than three BOTTLES of wine per night. They were drinking 16 glasses of wine, but 'so what?' Despite what you would like to believe, Dr. Katz (“Do I think eating a high-meat, low-plant diet increases risk of death and disease? Hell ya!”), nutrition is still a science, and it should be treated as such.
Most people agree that limiting fat intake to 30% or less calories from fat is considered a low-fat diet. I’m sure, as mentioned before, Dean Ornish would disagree with this classification, and if a study published saying that a low-carbohydrate (40%+ calories as carbohydrates, for example) diet was more effective at weight loss than a low-fat diet (30%+ calories from fat), it wasn’t a true test of the Ornish diet or the Atkins diet. And researchers shouldn’t be able to conclude that an Atkins diet was more effective than an Ornish diet for weight loss in this case, despite many author’s inclinations to do just that. These types of observations illuminate the need for a more rigorous clinical standard in diet studies. Terms such as “low-carbohydrate” and “low-fat” should not be blanket terms for various levels of carbohydrate and fat intake, respectively, or a clinical definition needs to be established so there is less confusion and error when interpreting the results of diet studies.
Because carbohydrate-restricted diets, are not really diets per se, that is, they don’t restrict calories, it’s difficult to define them by percentage of calories since this will vary from individual to individual. Instead, a focus should be placed on the absolute intake of carbohydrates, such as 20 grams or less for the classification of a “severely-restricted-carbohydrate diet,” for example. With low-fat diets, they tend to be calorie-restricted, which makes definitions based on percentage of calories more feasible, but less effective in comparing a low-fat, calorie-restricted diet to a low-carbohydrate, unrestricted-calorie diet.
An intake of 37.2% of carbohydrates is a “moderately high carbohydrate intake,” as William Yancy pointed out in an editorial (8) that ran in the same issue of the study in question. Also, groups of individuals in the study still included in the “low-carbohydrate” stratum, were averaging more than 60% calories from carbohydrate, which stretches the imagination of the definition of “low-carbohydrate” beyond belief. However, this is perfectly acceptable methodology at Harvard University and the peer-reviewers don't see fit to question it, either.
Yancy also pointed out several noteworthy issues, among them that several dietary assessment and classification issues cloud interpretation of the study and that dietary intake patterns can change over time in an individual, especially when it’s over the course of three decades. Also, in observational studies, subjects are not assigned or randomized to a specific diet, so diet labels such as “low-carb” and “low-fat” are retrospective attributions. Not only that, there are no clinical standards for what constitutes a “low-fat” or a “low-carb” diet.
To give you an idea of the different intakes of carbohydrates in different studies and approaches, take a look at the following table:
Note the clinical issues raised when there are different interpretations of what constitutes a “low-carb” diet - or the fact that the Atkins diet is a moving target in which carbohydrate consumption can be progressively increased by the subject while not violating the prescription of the diet.
Note in the table above that the Gardner et al (2007)(10) subjects were trying to follow an Atkins diet, but their intake of carbohydrates were actually greater than other studies in which the design was intended to provide an opposing diet to low-carb, such as Yang and Van Itallie’s (1976)(11) “mixed” diet, or Golay et al.’s (1996)(12) 45% carbohydrate diet. Note that the health professionals in the Nurses’ Health Study (NHS) were reportedly eating more carbohydrates than these groups (see this post for more details on macronutrient composition).
Also, adherence, or lack thereof, to the diets are known limitations to diet studies (as witnessed in Gardner et al. above), and the observations of Fung et al (2010) are probably no exception. In fact, Dr. Jeff Volek, co-author of the Atkins diet, noted, “What strikes me about this study is the assumption by the authors that that a massive number of subjects overcomes the well known limitations associated with using food frequency questionnaires (FFQ). Even if we assume some level of accuracy in assessing carbohydrate and overall nutrient intake by FFQ, the authors only examined food intake once: at the start of the 20-plus-year study. If a person changed his diet at any point during the two-decade study it would not be reflected in the analysis. (13)”
I think it's reasonable to assume that the composition of a diet can change over the course of 20-plus years.
Should we assume “some level of accuracy in assessing carbohydrate and overall nutrient intake by FFQ?” Not so much, says Christopher Masterjohn, especially when the data is being collected from a specialized population, notably health professionals: “About two-thirds of the population were from the Nurses' Health Study, and about one-third were from the Health Professional's Follow-up Study. They are employees of the medical establishment. The fact that they really believe what the medical establishment teaches can be seen by the fact that all the groups listed in the paper had PUFA intakes between fifty percent higher and almost double the national average of 6-7% of calories, something the American Heart Association recommends, albeit something that randomized controlled trials have shown is harmful and fatal. It is in any case a clear seal of belief in the mainstream position.”
Masterjohn also provided a graphic representation of a validation study (Salvini et al., 1989)(14) demonstrating the accuracy of FFQs used in the NHS:
Masterjohn explains that “In the chart, each food has two bars. The one on the left represents the accuracy of the FFQ at the beginning of the year and the one on the right represents the accuracy of the FFQ from the end of the year.”
Masterjohn also does a nice job of explaining the results: “The FFQ's ability to accurately predict egg intake is the best of all the animal foods, coming in at almost 50%. The other animal foods don't look so hot. I included tea and beer, ranking in at 75-80%, just to prove the point that an FFQ isn't inherently pathetic. But look at the accuracy for hamburgers — only 1.4%! Even after meticulously weighing out and recording their foods for 28 days, the participants still only reported their hamburger intake on the FFQ with 6.8% accuracy.”
The authors in Salvini et al (1989), concluded that, “Focusing on the second questionnaire, we found that butter, whole milk, eggs, processed meat, and cold breakfast cereal were underestimated by 10 to 30% on the questionnaire. In contrast, a number of fruits and vegetables, yoghurt and fish were overestimated by at least 50%. These findings for specific foods suggest that participants over-reported consumption of foods often considered desirable or healthy, such as fruit and vegetables, and underestimated foods considered less desirable. . . . This general tendency to over-report socially desirable foods, whether conscious or unconscious, will probably be difficult to eliminate by an alteration of questionnaire design.”
In layman’s (and Masterjohn’s) terms: “a study showing a statistical relationship between hamburger consumption and some health-related variable is telling us less about hamburger consumption and more about health professionals' overwhelming propensity to lie about how much hamburger they eat.”
In a critique of the Fung et al. (2010) paper, Denise Minger concluded from the data that "differences in mortality appear to be unrelated to animal product consumption. Changes in cancer and cardiovascular risk ratios occur out of sync with with changes in animal food intake." In other words, she came to the opposite conclusion of the authors.
So, what did she attribute to the "vegetable" group's lower mortality hazard ratios?
"Here’s a clue. Every time the researchers made multivariate adjustments to the data to account for the risk factors they did document (including physical activity, BMI, alcohol consumption, hypertension, and smoking, among other things), the hazard ratio went down for the Animal Group (meaning it got better) and it went up for the Vegetable Group adherents (meaning it got worse). That indicates pretty clearly that the Animal Group adherents had more proclivity to disease right from the get go, regardless of meat consumption, and the Vegetable Group adherents may have been more health-aware than most folks. (To see what I’m talking about, look at the mortality tables under the “10″ column, and compare the “Age- and energy-adjusted HR” with the “Multivariate-adjusted HR” for each group.) "In other words, it looks like what this study really measured was a Standard American Diet group (aka highest Animal Group decile) and a slightly-less Standard American Diet group (aka highest Vegetable Group decile). Both ate sucky diets, but the latter had slightly less suckage. You can bet the farm that neither was anything close to “low carb.” And if you have two farms, you can bet the other one that neither diet group was anything near plant-based, so I’m not sure the vegan crowd has much to gloat about here."
In other words, we have what Gary Taubes referred to as the Healthy User Bias. "At its simplest, the problem is that people who faithfully engage in activities that are good for them — taking a drug as prescribed, for instance, or eating what they believe is a healthy diet — are fundamentally different from those who don’t," notes Taubes in the New York Times. "One thing epidemiologists have established with certainty, for example, is that women who take H.R.T. differ from those who don’t in many ways, virtually all of which associate with lower heart-disease risk: they’re thinner; they have fewer risk factors for heart disease to begin with; they tend to be more educated and wealthier; to exercise more; and to be generally more health conscious."
"Considering all these factors, is it possible to isolate one factor — hormone-replacement therapy [or perhaps meat consumption] — as the legitimate cause of the small association observed or even part of it?" asks Taubes.
Also, "adherers" to the animal-based diet were more likely to smoke and had higher BMIs than the vegetable-based group. Minger astutely noted that, "Along with influencing mortality outcomes, this suggests the Animal Food group, in the highest decile, may have been somewhat less health-conscious than the dieters lumped into the highest decile for the vegetable category. And that’s the type of thing that has repercussions for other diet and lifestyle choices that weren’t measured in the study."
Minger also noted that the vegetable-based group was "nowhere near plant-based." "They derived almost 30% of their daily calories from animal sources (animal fat and animal protein), versus about 45% for the Animal Group. If we compare the middle (fifth) decile, the Vegetable Group was eating a greater percent of total calories from animal foods than the Animal Group was."
A few other key points that Minger addresses:
- "The Vegetable Group adherents ate more fruits, vegetables, and whole grains than the Animal Group adherents—which begs the question: What kinds of carbohydrates filled this macronutrient void for the animal-food eaters? Could it’ve been refined grains and processed carbs, which the study conveniently forgot to document?"
- "For the Vegetable Group, cancer and cardiovascular mortality was lower in the tenth decile than the first decile, even though both deciles ate exactly the same amount of red meat and nearly the same amount of total animal foods. This suggests animal products aren’t the driving force behind differences in mortality rates."
- "Similarly, at the fifth decile, the Vegetable Group had a lower cardiovascular mortality hazard ratio than the Animal Group (0.99 versus 1.21), even though the Vegetable Group was eating a slightly greater proportion of animal foods (33.3% versus 29.9% of total energy for women; 32.9% versus 31% for men)."
What might be the most egregious component of the study is the assumption that epidemiological, observational studies can pass for experimental science. In other words, observational studies can produce observations, nothing more. From these observations we can develop hypotheses about how the world works, why there don’t seem to be any leaves on the trees in December, why lower income households have a greater prevalence of obesity, etc.
Masterjohn covers this topic and points out, just as any scientific method 101 course would, that first, we make observations.
As Claude Bernard put it: “To have our first idea of things, we must see those things; to have an idea about a natural phenomenon, we must, first of all, observe it. The mind of man cannot conceive an effect without a cause, so that the sight of a phenomenon always awakens an idea of causation. All human knowledge is limited to working back from observed effects to their cause (15)."
The ‘awakened idea of causation’ is the second step in the process of the scientific method. That is, we form a hypothesis based on the observations.
After we form our hypothesis, we deduce ways in which we can test it, and refute it. If our hypothesis is confirmed, it doesn’t necessarily mean that it’s true, however the hypothesis can become more robust with each confirmation, but doesn’t preclude the possibility that the hypothesis can be later refuted.
As Masterjohn notes and graphically depicts, “if we pretend [the steps to the scientific method] is a map and pay close attention to the arrows, we can see why the app[r]oach of this study is a bit like trying to travel from California to Virginia by going west. You're going to get pretty wet."
“These authors decided to test their hypothesis by making more observations:
Because the leading causes of death in the United States — cardiovascular disease (CVD) and cancer — develop over many years, long-term studies of low-carbohydrate diets are needed to evaluate effects on mortality. However, randomized trials of low-carbohydrate diets on mortality are not feasible because of the difficulty in maintaining adherence and follow-up over many years. . . . Therefore, we prospectively examined the relationship between different types of low-carbohydrate diets and all-cause and cause-specific mortality in 2 large cohorts in the United States (Fung et al., 2010).
“Splash! I hope they enjoy swimming in the Pacific, but the arrows simply don't flow backwards from hypothesis to observation, and amassing more and more observations to "test" your hypothesis leaves you with nothing but the same hypothesis and a huge stack of paper.”
As Masterjohn concludes, researchers can’t use observation to confirm a hypothesis, rather it’s the opposite: hypotheses are ideas or questions that try and explain the observations. We then move forward and test the hypothesis. Or at least that’s what we’re supposed to do if we’re adhering to the scientific method. If we’re not, then we should remove these types of studies from the medical and scientific journals, publish them in tabloid and other interest magazines and call it a day.
Just because it might be difficult, or "not feasible," it does not absolve the field of nutrition from conducting good science. If the leading researchers don't think it's feasible to adhere to the scientific method, which is ostensibly what they're implying in the paper, then it's time to step aside, hand over the keys, and allow people who are interested in conducting randomized trials and metabolic ward studies to test hypotheses, such as William Yancy, who rightfully noted, "the current state of the evidence is such that no one can legitimately claim that a low-carbohydrate diet is either harmful or safe with any degree of certainty until a large-scale, randomized study with meaningful clinical end points is done."
Yancy continues: "Such a study would require input from experts in administering each of the diets to minimize bias, and it would require an innovative design to enable completion of the study in a reasonable time frame for an achievable study budget. The authors claim that such trials are not feasible because of difficulties with follow-up and with dietary adherence. However, such a design is more feasible today than ever before given the possibility of a ubiquitous health information infrastructure emerging in the United States. Furthermore, a likely major detractor to dietary adherence is the plethora of mixed messages society receives about what and what not to eat. Isn’t adherence likely to be higher if we had greater certainty about which dietary patterns are healthy and which are not?"
Until then, I feel that since the literature isn’t being properly peer-reviewed, we should be grateful for the people who remain skeptical and ask the kinds of questions that the researchers themselves should be asking.
A recent article in the New York Times (16) reported a movement in an alternative method of peer-review, which is called “crowd sourcing,” where experts are invited to post, in a wikipedia-esque fashion. It was reported that mathematicians used blogs and wikis to evaluate a supposed mathematical proof (17) “in the space of a week – the scholarly equivalent to warp speed.”
I initially read the articles in the mainstream media (for example, I read a related article in the USA Today (18) on September 6th), about the Fung et al. (2010) study, which led me to read the study itself. I noticed many flaws in the article which made me want to write, or at least vent to someone, about the flaws of the study.
On the same day I read the story and study, I read an accompanying editorial (Yancy, 2010) and a release by Atkins Nutritionals (PR Newswire). Since then, there have been many compelling blog posts (a number of which aforementioned) and coherent arguments being made that critique the Fung et al. (2010) paper.
I propose we do something similar in the blogosphere. A wikipedia of sorts. If the studies and papers in nutrition are not going to be subjected to the scientific method in peer-review, then it’s incumbent upon us to make sure that they are. It's already happening, really, but I'm wondering if there was a collaborative source, it would provide a more effective message.
Here is a timeline of the Fung et al. (2010) paper and some sound critiques. I encourage you to visit the sites and read the articles in their entirety. Enjoy:
Article: “Low-Carbohydrate Diets and All-Cause and Cause-Specific Mortality (Fung et al., 2010)” published in the September 7, 2010 issue of the Annals of Internal Medicine.
September 6:
PR Newswire: Conclusions Flawed in Comparision of Animal-and Plant-Based ‘Low-Carbohydrate Diets:
“Although the headlines may say the study suggests that long-term adherence to a low-carb diet based heavily on animal protein may reduce lifespan, Dr. Funga, who was the lead author of the study, says that her research "is not representative of popular low-carb eating plans." She goes on to note that her study "is observational and based on a limited pool of health professionals rather than a large-scale, clinical trial based on a varied population. Other issues in the design of the study, such as depending on food frequency questionnaires, impact its conclusions." So why all the fuss? Headlines make news but headlines can be misleading; and that's why Atkins wants to clarify study conclusions and correct any misperception.”
September 7:
Jimmy Moore: Study Claims An Animal-Based Low-Carb Diet Will Kill You - - Not So Fast! (19):
“In fact, it was Dr. Hu himself who released this NIH-funded observational study in November 2006 published in the The New England Journal of Medicine that found the long-term heart health concerns about low-carb diets were unfounded. What data pool did he draw from for his research? The same one he did for this new one–The Nurses’ Health Study. Of course, he was already leaning towards looking at the veggie-based low-carb diet in that study claiming it produced better outcomes. But this latest release pretty much damns an animal-based low-carb diet from being healthy. Dr. Yancy points out the confusing nature of how the data was interpreted now compared with that 2006 study in his editorial.”
Diabetes Update: Why The Latest Low Carb Study Scare is Flawed (20):
“[F]our immediate red flags. . . .
1. Based on Inaccurate Questionnaire Data.
2. Misleading Definition of "Low Carb Diet”
3. “Meat-based diet" May Mean "Fast Food Meat and Potatoes Diet"
4. “Plant based Low Carb diet" Highly Suspect”
September 8:
Wall Street Journal: Call for a Randomized Trial to Settle the Low-Carb Diet Questions (21):
“But because of the inherent limits to observational studies, particularly those involving diet, this doesn’t really settle the question we all want to know, namely, what’s the best way to eat (if one actually exists) to live as long and well as possible?”
Denise Minger: Brand-Spankin’ New Study: Are Low-Carb Meat Eaters in Trouble? (22):
“I’ll sum it up. Some of these “low carbers” were eating up to 60% of their diet as carbohydrates (first decile), which—last time I checked—is kind of not low-carb. Even the lowest low-carb eaters were still eating over 37% of their calories from carbohydrates. Whoever decided to call this study “low carbohydrate” is nuttier than a squirrel turd. That doesn’t mean it can’t offer anything useful, though, so let’s look at what else is going on in the highest decile for each group (which is the only decile the researchers really looked at):
- Folks adhering the most to the animal-based diet were more likely to smoke and had higher BMIs than the best adherents of the Vegetable Group. Along with influencing mortality outcomes, this suggests the Animal Food group, in the highest decile, may have been somewhat less health-conscious than the dieters lumped into the highest decile for the vegetable category. And that’s the type of thing that has repercussions for other diet and lifestyle choices that weren’t measured in the study.
- The Vegetable Group was nowhere near plant-based: They derived almost 30% of their daily calories from animal sources (animal fat and animal protein), versus about 45% for the Animal Group. If we compare the middle (fifth) decile, the Vegetable Group was eating a greater percent of total calories from animal foods than the Animal Group was. D’oh!
- The Vegetable Group adherents ate more fruits, vegetables, and whole grains than the Animal Group adherents—which begs the question: What kinds of carbohydrates filled this macronutrient void for the animal-food eaters? Could it’ve been refined grains and processed carbs, which the study conveniently forgot to document?
- For the Vegetable Group, cancer and cardiovascular mortality was lower in the tenth decile than the first decile, even though both deciles ate exactly the same amount of red meat and nearly the same amount of total animal foods. This suggests animal products aren’t the driving force behind differences in mortality rates.
- Similarly, at the fifth decile, the Vegetable Group had a lower cardiovascular mortality hazard ratio than the Animal Group (0.99 versus 1.21), even though the Vegetable Group was eating a slightly greater proportion of animal foods (33.3% versus 29.9% of total energy for women; 32.9% versus 31% for men).”
Chris Masterjohn: New Study Shows that Lying About Your Hamburger Intake Prevents Disease and Death When You Eat a Low-Carb Diet High in Carbohydrates:
"The end result is that this study constitutes an observation, and cannot be used to support a hypothesis of any kind. Hypotheses are ideas developed to try to explain observations. You cannot test a hypothesis by making more observations. It is not impossible to test a hypothesis about diet over the long-term, and indeed trials have done this in the past, usually quietly swept under the rug because the establishment didn't like their results. Compliance with low-carb diets will never be perfect, but it will probably be better than the 1.4% accuracy with which food frequency questionnaires predict hamburger intake. Logical fallacies cannot substitute for the scientific method just because the scientific method seems difficult or even infeasible.
"Nevertheless, logical fallacies will not be disappearing from the scene any time soon. So there's lots of work in the blogger world ahead."
Tom Naughton: The ‘Atkins’ Study (ahem, ahem) According To Ornish (23):"As I noticed immediately and Ms. Minger pointed out as well, even if we grant that the researchers could accurately determine dietary intake from a dozen questionnaires mailed out over 20 years, these people weren’t on anything close to the Atkins diet. At the high end, their diets were 60% carbohydrates. At the lowest end, the diets were 37% carbohydrates. According to the study tables, the average calorie intake was right around 2000 calories per day (which sounds low to me, but we’ll roll with it). So let’s do the math:
"(2000 * .37) / 4 = 185 carbohydrates per day … for the lowest carb group.
"Could you please point out a page in any of the Atkins books where he recommends consuming 185 carbohydrates per day? The Atkins diet starts at 20 grams per day and gradually increases the carbs until weight-loss slows down — which for most of us is well under 100 grams per day. I rarely consume more than 50. That makes my diet less than 10% carbohydrates.
"Dr. Ornish, whenever yet another clinical study demonstrates that people on low-carb diets experience greater improvements in cardiovascular markers than people on low-fat diets, you immediately say the results are illegitimate because the low-fat dieters didn’t restrict their fats as much as you recommend. Okay, fair enough: if they get 20% of their calories from fat, it’s not really the Ornish diet.
"But when people in an observational (not clinical) study consume 185 carbs per day — at least three times what most Atkins dieters consume — that somehow becomes the Atkins Diet. Very consistent of you."
"Then there’s old missing-data problem. You can pretty much guess how they handled that:
A multiple imputation procedure was used with 20 rounds of imputation and included all covariates to account for missing dietary and covariate data. The analysis was repeated by using noncumulative updating of dietary information, in which we used the most recent diet data to predict mortality rate.
Same as in the recent low-fat vs. low-carb study: they performed mathematical magic to fill in the missing data. As Dr. Mike Eades like to say, if you torture the data long enough, it will tell you whatever you want to hear. This data appears to have been water-boarded until it screamed “Yes! Yes! Animal foods are deadly!”
September 11:
Tom Naughton: Food Questionnaires (24):
The latest “Low-Carb Diets Are Deadly!” study, the topic of my previous post, was based on data extracted from about a dozen food questionnaires mailed out over a 20-year span. Because so many studies rely on these questionnaires to draw headline-grabbing conclusions about associations between diets and disease, it’s important to understand just how lousy the accuracy rate is."
September 12:
Fred Hahn: Dean Ornish – Pioneer of Health or Profiteer for Wealth? (25):
Moreover, Dr. Ornish cleverly calls the paper a “study.” In truth, it is a type of study called an epidemiological survey – a method of looking at data that AT BEST can suggest an association. As any good scientist knows, associations can tell us something — they can be jumping off points for further study. However, associations can NEVER prove cause. We are left to assume that either a) Dr. Ornish knows that most people won’t catch this or call him out on it or b) he is simply not a good scientist.
We, as lay people, rely on physicians, scientists and experts that can accurately read and assess scientific papers for the betterment of our health and well being. At the very least we assume that they can and will without bias. When such experts fail us and fail us in such a careless and (as it appears to me) biased manner, it not only ruins our faith in that individual but in others of his esteemed profession.
We deserve better. We deserve the truth – not one man’s quest for profit at the expense of our health and scientific integrity. Or is it that he is sincere and merely a poor scientist. Either way, he should not be who we rely on or look to for sound nutritional advice.
Addendum (added 09-15-2010, 8:17 AM) links:
Preston: Why Dean Ornish is Wrong (September 11, 2010):
"Once we see how flawed the conclusions and design of this study are, it becomes obvious not only how wrong Ornish is in his article but how scary it is that someone so highly regarded and respected by the mainstream can get away with writing such rubbish and pulling the wool over so many unsuspecting readers’ eyes."
"The vegetable eaters in this study were still eating lots of animal products- about 30% of daily calories from animal fat and protein. Certainly not vegetarians! So that definitely muddies the waters. Minger even points out that in the 5th decile they ate more animal food than the meat eaters! And by the way, this same 5th decile group of meat-eating vegetable eaters (huh?) had a lower CV mortality hazard than the less-meat eating meat eaters. I’m confused."
Jay Wortman: Atkins Diet Increases All-Cause Mortality - NOT! (September 12, 2010):
"In this study, the food recall questionnaires considered “plausible” reported caloric intakes ranging from 500 to 3500 kcal per day. How many people get by on 500 kcal per day, do you think? Not too many. So, in the analysis, median or average numbers were chosen which still look, in all groups, to be significantly lower than what any American presently eats. The men reportedly ate about 2000 kcal per day while the women ate between 1600 and 1800. That should make for a pretty slim bunch of typical Americans! Interestingly, these absurd numbers for caloric intake are averaged while the analysis is done to determine the amounts of carbs, proteins, fats, etc in the various deciles. One wonders what would the data have looked like if they averaged the macronutrients and broke the caloric intake into deciles. The first decile would be 500 kcal per day, the next 800 kcal per day, the next 1100 kcal per day, etc etc. Do you see the problem? Of course it would be absurd to think that the first groups in the series could eat so few calories and actually live. That study would never get published. However, when these data are flipped around and the analysis is done the other way to determine macronutrient proportions by decile the study gets published. In what universe does that make any sense?"
1. Fung TT, van Dam RM, Hankinson SE, Stampfer M, Willett WC, Hu FB. Low-Carbohydrate Diets and All-Cause and Cause-Specific Mortality: Two Cohort Studies. Annals of Internal Medicine. 2010;153:289-298.
2. Bloomberg Businessweek. Low-Carb Diets on Meat May Raise Health Risks. Available at: http://www.businessweek.com/lifestyle/content/healthday/642824.html. Accessed September 11, 2010.
3. T. L Halton et al., “Low-carbohydrate-diet score and the risk of coronary heart disease in women,” New England Journal of Medicine 355, no. 19 (2006): 1991.
4. Masterjohn C. New Study Shows that Lying About Your Hamburger Intake Prevents Disease and Death When You Eat a Low-Carb Diet High in Carbohydrates. The Daily Lipid. Available at: http://blog.cholesterol-and-health.com/2010/09/new-study-shows-that-lying-about-your.html. Accessed September 11, 2010.
5. Ornish D. Atkins Diet Increases All-Cause Mortality. Huffington Post. Available at: http://www.huffingtonpost.com/dr-dean-ornish/an-atkins-diet-increases-_b_707005.html. Accessed September 8, 2010.
6. Katz D. The Beef With Atkins. Huffington Post. Available at: http://www.huffingtonpost.com/david-katz-md/as-ye-soy-the-beef-with-a_b_708865.html. Accessed September 11, 2010.
7. Freeman DW. Atkins Diet Mrtality: Is Too Much Meat Killing Low-Carb Eaters? CBS News. Available at: http://www.cbsnews.com/8301-504763_162-20015867-10391704.html. Accessed September 9, 2010.
8. William S. Yancy, Matthew L. Maciejewski, and Kevin A. Schulman, “Animal, Vegetable, or … Clinical Trial?,” Annals of Internal Medicine 153, no. 5 (2010): 337 -339.
8. William S. Yancy, Matthew L. Maciejewski, and Kevin A. Schulman, “Animal, Vegetable, or … Clinical Trial?,” Annals of Internal Medicine 153, no. 5 (2010): 337 -339.
9. Kaplan B. Low-Carb vs. Low-Fat: Part IV: Macronutrient Composition. Nerd Safari.
10. Gardner CD, Kiazand A, Alhassan S, Kim S, Stafford RS, Balise RR, et al. Comparison of the Atkins, Zone, Ornish, and LEARN diets for change in weight and related risk factors among overweight premenopausal women: the A TO Z Weight Loss Study: a randomized trial. JAMA. 2007;297:969-77. [PMID: 17341711]
11. Yang MU, Van Itallie TB. Composition of weight lost during short-term weight reduction. Metabolic responses of obese subjects to starvation and lowcalorie ketogenic and nonketogenic diets. J Clin Invest. 1976;58:722-30. [PMID: 956398]
12.Golay A, Allaz AF, Morel Y, de Tonnac N, Tankova S, Reaven G. Similar weight loss with low- or high-carbohydrate diets. Am J Clin Nutr. 1996;63: 174-8. [PMID: 8561057]
13. PR Newswire. Conclusions Flawed in Comparison of Animal-and-Plant Based ‘Low-Carbohydrate’ Diets. Available at: http://www.prnewswire.com/news-releases/conclusions-flawed-in-comparison-of-animal-and-plant-based-low-carbohydrate-diets-102303154.html. Accessed September 8, 2010.
14. S Salvini et al., “Food-based validation of a dietary questionnaire: the effects of week-to-week variation in food consumption,” International Journal of Epidemiology 18, no. 4 (December 1989): 858-867.
15. Bernard C. An Introduction To The Study of Experimental Medicine. 1957.
15. Bernard C. An Introduction To The Study of Experimental Medicine. 1957.
16. Cohen P. For Scholars, Web Changes Sacred Rite of Peer Review. New York Times. August 23, 2010. Avaiable at: http://www.nytimes.com/2010/08/24/arts/24peer.html. Accessed September 13, 2010.
17. Markoff J. Debate Over P vs. NP Proof Highlights Web Collaboration. New York Times. Available at: http://www.nytimes.com/2010/08/17/science/17proof.html?_r=2&scp=1&sq=markoff%20theorists%20&st=cse. Accessed September 13, 2010.
17. Markoff J. Debate Over P vs. NP Proof Highlights Web Collaboration. New York Times. Available at: http://www.nytimes.com/2010/08/17/science/17proof.html?_r=2&scp=1&sq=markoff%20theorists%20&st=cse. Accessed September 13, 2010.
18. Hellmich N. Cutting carbs? Replace them with proteins from plants. USA Today. Available at: http://www.usatoday.com/yourlife/food/diet-nutrition/2010-09-07-lowcarb07_ST_N.htm. Accessed September 6, 2010.
19. Moore J. Available at: http://livinlavidalowcarb.com/blog/study-claims-an-animal-based-low-carb-diet-will-kill-you-not-so-fast/8946. Accessed September 13, 2010.
20. Diabetes Update. Available at: http://diabetesupdate.blogspot.com/2010/09/why-latest-low-carb-scare-study-is.html.
21. Wall Street Journal. Available at: http://blogs.wsj.com/health/2010/09/08/call-for-a-randomized-trial-to-settle-the-low-carb-diet-questions/.
22. Minger D. Availabe at: http://rawfoodsos.com/2010/09/08/brand-spankin-new-study-are-low-carb-meat-eaters-in-trouble/
22. Minger D. Availabe at: http://rawfoodsos.com/2010/09/08/brand-spankin-new-study-are-low-carb-meat-eaters-in-trouble/
23. Naughton T. Available at: http://www.fathead-movie.com/index.php/2010/09/09/the-atkins-study-ahem-ahem-according-to-ornish/.
24. Naughton T. Available at: http://www.fathead-movie.com/index.php/2010/09/11/weekend-bonus-food-questionnaires/.
25. Hahn F. Available at: http://slowburnfitness.com/dean-ornish-pioneer-of-health-or-profiteer-for-wealth/.
No comments:
Post a Comment