Proper experimental design is the foundation of any scientific publication. However, a study is not so easy to plan, particularly when it includes methods that are expensive or that use tools that are hard to find. To make things more complicated, many studies are performed as part of a Master’s or Doctoral thesis, and the investigator gains skills and knowledge throughout the course of the experiment. By the time the study is done, the investigator sees parts she would have done differently.
Studies that involve animals are especially complex, since you cannot “redo” a failed experiment as easily as you can with in vitro or in silico assays. Criticisms by reviewers and editors can seldom be addressed during the peer review process: if an editor or reviewer identifies a flaw in an animal feeding study, it often cannot be redone due to resource constraints.
Poorly designed GMO feeding studies abound, quite possibly due to these difficulties in performing any animal feeding study. Such studies are often used by people who claim GMO are dangerous. It can be difficult to determine if a study has been properly designed and performed. We’ve put together a list to help you navigate through the messy world of GMO feeding studies.
Evaluating GMO Feeding Studies
- Feed Analysis
- The nutritional content of feed given to both control and treatment animals must be analyzed to determine if there are any differences other than the GM trait. If the feeds aren’t as identical as possible, any difference observed between the treated animals and controls cannot be attributed exclusively to the GM trait.
- Many papers have shown that the environment has a strong impact on nutrient and mineral content in crops, so a failure to perform this analysis is a critical flaw in any GM feeding study. Anti-nutrient content, and toxin-producing fungi and bacteria must be analyzed as well.
- For example, the paper “The Comparative Effects of Genetically Modified Maize and Conventional Maize on Rats” observed differences in organ size and other parameters between the rats fed a diet with GMOs and controls, however, without analysis of the feed we don’t know if the differences are due to the GM trait. Maize has natural variation in sugar content, protein content and other nutrients which could have given rise to the observed differences, rather than the Bt-trait to which the authors attributed the observed differences.
- Feed Source
- The feed that is provided to control and treated animals must be as similar as possible and should be isogenic. This means that the GM feed is the same variety as the control, with the exception of the introduction of the genetically engineered trait.
- A well designed study will have the control and GM feed grown in the same field and in the same year, to minimize variability caused by the environment.
- Often times, a failure to use similar feed sources can be a fatal flaw, such as in the paper “Feeding Study with Bt Corn (MON810: Ajeeb YG) on Rats: Biochemical Analysis and Liver Histopathology“, where the authors had identified nutritional differences in the GM feed but do not describe normalization of the nutrients in the feed provided to the animals nor do they provide information on how the crops were grown, if pesticides were used, or other important factors. Consequently, the observed differences between the control rats and the GM-feed rats cannot be attributed exclusively to the transgenic protein in the diet.
- The control and treated animals must be kept and treated in the exact same way, with the exception of the presence of the transgenic protein and gene in the feed. This ensures that any differences observed between the animals can be attributed exclusively to the feed, not the environment or the conditions under which the animals were kept.
- For example, the paper “Biological impact of feeding rats with a genetically modified-based diet” identifies tissue changes in rats fed a diet of GM soy and corn, however the controls were fed a diet of wheat when they should have been fed a diet of non-GM soy and/or corn. Therefore, the two groups were not treated equivalently.
- The proper statistical tests should be used throughout the study. For example, in the paper “A Comparison of the Effects of Three GM Corn Varieties on Mammalian Health“, the authors jump from one statistical test to another without explaining why a new statistical test is being used, suggesting that the authors may have been fishing for significance.
- A well designed study must consider statistical power during the design phase. The authors must consider in advance what metrics they will be measuring and how much the measurement fluctuates in healthy subjects to determine how many animals they need in their study. For example, Seralini’s study examining the toxicity of GM-maize and Round-Up was found not to have a large enough sample size to make relevant conclusions.
- There’s natural variation in any species. For any given trait, there’s a range for what’s considered “normal”. The types of mice and rats used for feeding studies have less variation because they are inbred, but there’s still variation for most traits. As such, any difference observed between the controls and the animals given GM feed must be explained within the context of natural variation for the species.
- Statistically significant differences are not necessarily biologically relevant. This point is intertwined with statistical power. If you take two groups of animals and take enough measurements, you are bound to find a measurement that is different between the two groups. As such, it is important to address the question: is the measured difference biologically relevant?
- The European Food Safety Authority defines biological relevance as “an effect considered by expert judgement as important and meaningful for human, animal, plant or environmental health. It therefore implies a change that may alter how decisions for a specific problem are taken.” The EFSA also points out that the magnitude of the effect must be considered when examining biological relevance.
- As an example, the paper “A three generation study with genetically modified Bt corn in rats: Biochemical and histopathological investigation” finds “minimal” differences between GE-fed and control animals in several measurements. The paper concludes that despite these minor differences, “long-term consumption of transgenic Bt corn throughout three generation did not cause severe health concerns on rats.” However, the findings from this paper are often taken out of context as an example of harm.
- If a study finds a difference between GM-fed and control animals, studies that repeat the experiment should observe the same difference. In contrast, if a similar study has been done in the past and didn’t see the observed difference, the authors should address the discrepancy and propose a hypothesis on why their results are different.
- For example, three different studies have examined the impact of Round-Up Ready Soy feed on goats and their kids. None of their conclusions are the same (see here, here, and here – note that one of these has been retracted).
When are animal feeding studies useful?
Several years ago, the European Commission funded the GRACE project (GMO Risk Assessment and Communication of Evidence). The goal of the project is to review the literature to find evidence of benefits and harms of GM crops, and to determine which types of studies are best suited for GMO risk assessments. Their most recent publication investigates whether animal feeding studies are useful.
In contrast with reviews that found 90-day feeding studies to be sufficient, the GRACE project concluded that in most cases 90-day feeding studies do not provide any additional information over non-animal testing. “GRACE data support the scientific reasoning that only in case a trigger is available from the initial molecular, compositional, phenotypic and/or agronomic analyses, feeding trials with whole food/feed may provide an added scientific value for the risk assessment of GM crops. Thus, feeding trials might be considered, provided that the study design can be tailored to the posed safety concern.”
GRACE’s conclusions are similar to the idea that most healthy people do not need to undergo regular medical testing. When large numbers of people (or animals) are tested, some differences will be detected simply due to random chance – even when there is no underlying concern. To avoid the risk of false positives, there must be some initial concern to trigger the additional tests. For example, doctors only recommend that pregnant women undergo amniocentesis if there is some risk factor, such as concerning results from another test. Similarly, GRACE recommends that animal feeding studies are not necessary unless there is some risk factor, such as a compositional difference.