Credibility of Evidence and Affiliation in the Quest for Proper Science

by Bart Penders and Kim Hendrickx

Lorry drivers who regularly visited South and Eastern Europe were struck by the beauty and shapeliness of local women. This observation give rise to the “Edric Original” breast growth pill, filled with hop, the apparent key dietary difference between well-endowed Southern and Eastern European and everyone else (Scholtens, 1997). The full treatment will set you back €540—a bargain.

What is it about this offer that makes us doubt its credibility? Why do we evaluate some claims as more credible than others? Kim Hendrickx is currently finalizing a PhD thesis on the credibility of claims on food products. Bart Penders and Kim Hendrickx give a commented preview of the argument.

The European Union has set up an infrastructure to rid itself of ill-supported claims associated with foods. One type of health claim that fell victim to this is that associated with pre- and probiotic foods.

But what is the proper foundation for a health claim? Who is going to decide upon this? And what effects does such an infrastructure have on nutrition science? This debate has become prominent since the European Food Safety Authority (EFSA) was charged with the evaluation of health claims, resulting in a list of permitted claims for the EU in 2012.

In December 2012, a list of permitted claims was approved based upon a scientific opinion from EFSA. It contained 222 general function health claims, 30 nutrition claims, and 21 health claims referring to the reduction of disease risk factors or child development. A whopping 1631 health claims were not approved (included the breast growth claim at the top of this post) and needed to be gradually removed from the public domain; 2303 health claims remain under consideration, awaiting more data (Mathioudakis, 2013).

That list was disappointing to both industry and academics. Most allowed claims dealt with vitamins and minerals, while pre- and probiotic claims were rejected. They are considered one of the most promising segments in the functional foods market and the discussions between industry, academics and EFSA have led to the creation of interest groups on pre- and probiotics. They have written letters to scientific journals and to the Commission, and a new scientific community emerged under the name gut health scientists, with a website to defend their cause and their right to contribute to the definition of proper nutrition science.

There are currently two “fault lines” running through the nutrition research community with respect to the health claims debate and the underlying struggle to define proper nutrition science as the rightful referee. First, there is an institutional fault line that discursively positions scientists with industry involvement in a seeming opposition to those who are not. Second, there is an epistemological one, distinguishing between the different logics of evaluating, valuing and working with evidence.

Today, most academic researchers in nutrition come to collaborate with food companies at some point in their career, either through collaboration in concrete research projects or through increased involvement in public-private research consortia. Lesser et al. (2007) point at a relationship between funding source and study result, while Katan (2007) notes that this need not reflect the quality of science but may rather reflect the subset of science that is actually done. Nevertheless, industrial relationships and perceived breaches of integrity influence public trust in researchers and institutions (Marks, 2013), and professional evaluations of credibility (Kesselheim et al., 2012).

Industrial science suffers from credibility problems—about which industrial scientists utter their discontent: “[I]t is unscientific, as well as unfair, to discard or discount a study based solely on which investigator or institution conducted or funded it” (Barrow and Conrad, 2006). However, during their assessment, EFSA panellists do not reserve a separate treatment for health claim dossiers provided by companies, simply because such dossiers are almost by definition provided by such companies. Industry being the default applicant, all that can be taken into account as a variable in the judgment of dossiers is the scientific content. The idea is that scientists recognize sound science when they see it, and that it is thus possible to make consensus-based judgments.

Deciding which type and which amount of evidence is sufficient to support any scientific claim is a question generally settled through consensus. Evidence based medicine (EBM) is such a consensus: a logic of gathering, interpreting and weighing evidence. It includes a hierarchy of evidence, with the randomized controlled trial (RCT) at the top. EBM has, for all intents and purposes, become the dominant perspective in biomedicine, prescribing what type of evidence is most important, most credible and most sought after. However, in 2011, a group of nutrition scientists challenged the EBM consensus with reference to the European health claims debate (Biesalski et al., 2011). They made a plea for more methodological diversity, and for a different conceptualisation of health, and propose to replace EBM with “evidence based nutrition” (EBN) in the context of nutrition science.

EBN can be considered a strategic programme that serves both academic and industry-related nutrition science. EBN is politically analogous to EBM, in that it identifies its own experts and methods to reach consensus and formalise scientific judgment. The most important distinction EBN proponents forward between EBN and EBM deals with a line drawn between what pertains to “the medical” and “the nutritional.” “The medical” is actively externalised from nutritional research.

The RCT has become emblematic of EBM. The RCT is also the preferred form of evidence for EFSA's NDA panel, if available. When adherents of EBN propose an alternative to EBM, they are questioning the role of the RCT for research into the diet-health relationship, and they are questioning the hierarchy of evidence for the assessment of health claims. Their arguments range from pragmatic ones (ethical permissibility, cost and duration of RCTs) to fundamental questions about the suitability of RCTs. They argue that RCTs are not suitable to identify subtle effects and synergies amongst nutrients that are fundamental to human health, but not measurable in a single-point clinical outcome.

This leads to a situation where nutrition scientists in the EFSA panels are being reproached by peers for “wrongly” adhering to the EBM paradigm. To some, EBN is the only valid way of weighing nutritional evidence—the only conceptualisation of proper science, able to get to the dietary “truth.”

The insistence on a singular “proper science,” however, can lead to conflating two different scientific practices and types of knowledge: experimental practice and regulatory practice. Experimental practice ideally creates new knowledge by posing further questions and by changing experimental setups. In contrast, regulatory practice proposes a fixed setup in order to make judgments and put further questions (temporarily) to a halt. In both cases, “knowledge” does not refer to the same thing, and it does not fulfill the same function. Both EBM and EBN propose a set of criteria to make judgments, rather than scientific hypotheses, meaning that both are regulatory instruments, and thus political.

The singular notion of “proper science” makes it impossible to accommodate the different meanings and roles that knowledge, evidence, and “truth” represent in an arena where actors with different agendas and different practices meet. A possible consequence is that one is left with the impression that there is only good science and bad science, both morally and methodologically. Under such circumstances, the question "where does the good science come from?" resurfaces over and over again.

Instead, proper nutrition science it is about conceptualisations of origin, knowledge, evidence, and truth that fit the local context. The challenge lies in the continuous evaluation of that fit, and all the inclusions and exclusions it may invoke.

Note: A full length version has been submitted for publication.


Selected References

CS Barrow and JW Conrad Jr., “Assessing the Reliability and Credibility of Industry Science and Scientists,” Environmental Health Perspectives 114 (2006): 153-155.

HK Biesalski , PJ Aggett, R Anton et al., “Scientific Substantiation of Health Claims: Evidence-Based Nutrition,” 26th Hohenheim Consensus Conference, September 11, 2010, Nutrition 27 (2011): S1-S20.

MB Katan, “Does Industry Sponsorship Undermine the Integrity of Nutrition Research?” PLoS Medicine 4 (2007): e6.

AS Kesselheim, CT Robertson, JA Myers et al., “A Randomized Study of How Physicians Interpret Research Funding Disclosures,” New England Journal of Medicine 367 (2012): 1119-1127.

LI Lesser, CB Ebbeling, M Goozner et al., “Relationship between Funding Source and Conclusion among Nutrition-Related Scientific Articles,” PLoS Medicine 4 (2007): e5.

JH Marks, “What’s the Big Deal? The Ethics of Public-Private Partnerships Related to Food and Health,” Edmond J. Safra Research Lab Working Papers, No. 11 (2013), http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2268079.

MB Mathioudakis, “Seminar Contribution,” DG Health and Consumers, European Commission, in Updates on Health and Nutrition Claims (2013). Seminar Organised by Lexxion, Brussels.

B Scholtens, “Hopborsten-pillen,” De Volkskrant (8 II 1997), http://www.volkskrant.nl/vk/nl/2672/Wetenschap-Gezondheid/archief/article/detail/484171/1997/02/08/Hopborsten-pillen.dhtml.