This post introduces our infographic about GMO feeding studies.
Proper experimental design is the foundation of any scientific publication. However, a study is not so easy to plan, particularly when it includes methods that are expensive or that use tools that are hard to find. To make things more complicated, many studies are performed as part of a Master’s or Doctoral thesis, and the investigator gains skills and knowledge throughout the course of the experiment. By the time the study is done, the investigator sees parts she would have done differently.
Studies that involve animals are especially complex, since you cannot “redo” a failed experiment as easily as you can with in vitro or in silico assays. Criticisms by reviewers and editors can seldom be addressed during the peer review process: if an editor or reviewer identifies a flaw in an animal feeding study, it often cannot be redone due to resource constraints.
Poorly designed GMO feeding studies abound, quite possibly due to these difficulties in performing any animal feeding study. Such studies are often used by people who claim GMO are dangerous. It can be difficult to determine if a study has been properly designed and performed. We’ve put together a list to help you navigate through the messy world of GMO feeding studies.
- Feed Analysis
- The nutritional content of feed given to both control and treatment animals must be analyzed to determine if there are any differences other than the GM trait. If the feeds aren’t as identical as possible, any difference observed between the treated animals and controls cannot be attributed exclusively to the GM trait.
- Many papers have shown that the environment has a strong impact on nutrient and mineral content in crops, so a failure to perform this analysis is a critical flaw in any GM feeding study. Anti-nutrient content, and toxin-producing fungi and bacteria must be analyzed as well.
- For example, the paper “The Comparative Effects of Genetically Modified Maize and Conventional Maize on Rats” observed differences in organ size and other parameters between the rats fed a diet with GMOs and controls, however, without analysis of the feed we don’t know if the differences are due to the GM trait. Maize has natural variation in sugar content, protein content and other nutrients which could have given rise to the observed differences, rather than the Bt-trait to which the authors attributed the observed differences.
- Feed Source
- The feed that is provided to control and treated animals must be as similar as possible and should be isogenic. This means that the GM feed is the same variety as the control, with the exception of the introduction of the genetically engineered trait.
- A well designed study will have the control and GM feed grown in the same field and in the same year, to minimize variability caused by the environment.
- Often times, a failure to use similar feed sources can be a fatal flaw, such as in the paper “Feeding Study with Bt Corn (MON810: Ajeeb YG) on Rats: Biochemical Analysis and Liver Histopathology“, where the authors had identified nutritional differences in the GM feed but do not describe normalization of the nutrients in the feed provided to the animals nor do they provide information on how the crops were grown, if pesticides were used, or other important factors. Consequently, the observed differences between the control rats and the GM-feed rats cannot be attributed exclusively to the transgenic protein in the diet.
- Controls
- The control and treated animals must be kept and treated in the exact same way, with the exception of the presence of the transgenic protein and gene in the feed. This ensures that any differences observed between the animals can be attributed exclusively to the feed, not the environment or the conditions under which the animals were kept.
- For example, the paper “Biological impact of feeding rats with a genetically modified-based diet” identifies tissue changes in rats fed a diet of GM soy and corn, however the controls were fed a diet of wheat when they should have been fed a diet of non-GM soy and/or corn. Therefore, the two groups were not treated equivalently.
- Statistics
- The proper statistical tests should be used throughout the study. For example, in the paper “A Comparison of the Effects of Three GM Corn Varieties on Mammalian Health“, the authors jump from one statistical test to another without explaining why a new statistical test is being used, suggesting that the authors may have been fishing for significance.
- A well designed study must consider statistical power during the design phase. The authors must consider in advance what metrics they will be measuring and how much the measurement fluctuates in healthy subjects to determine how many animals they need in their study. For example, Seralini’s study examining the toxicity of GM-maize and Round-Up was found not to have a large enough sample size to make relevant conclusions.
- Relevance
- There’s natural variation in any species. For any given trait, there’s a range for what’s considered “normal”. The types of mice and rats used for feeding studies have less variation because they are inbred, but there’s still variation for most traits. As such, any difference observed between the controls and the animals given GM feed must be explained within the context of natural variation for the species.
- Statistically significant differences are not necessarily biologically relevant. This point is intertwined with statistical power. If you take two groups of animals and take enough measurements, you are bound to find a measurement that is different between the two groups. As such, it is important to address the question: is the measured difference biologically relevant?
- The European Food Safety Authority defines biological relevance as “an effect considered by expert judgement as important and meaningful for human, animal, plant or environmental health. It therefore implies a change that may alter how decisions for a specific problem are taken.” The EFSA also points out that the magnitude of the effect must be considered when examining biological relevance.
- As an example, the paper “A three generation study with genetically modified Bt corn in rats: Biochemical and histopathological investigation” finds “minimal” differences between GE-fed and control animals in several measurements. The paper concludes that despite these minor differences, “long-term consumption of transgenic Bt corn throughout three generation did not cause severe health concerns on rats.” However, the findings from this paper are often taken out of context as an example of harm.
- Reproducibility
- If a study finds a difference between GM-fed and control animals, studies that repeat the experiment should observe the same difference. In contrast, if a similar study has been done in the past and didn’t see the observed difference, the authors should address the discrepancy and propose a hypothesis on why their results are different.
- For example, three different studies have examined the impact of Round-Up Ready Soy feed on goats and their kids. None of their conclusions are the same (see here, here, and here – note that one of these has been retracted).
When are animal feeding studies useful?
Several years ago, the European Commission funded the GRACE project (GMO Risk Assessment and Communication of Evidence). The goal of the project is to review the literature to find evidence of benefits and harms of GM crops, and to determine which types of studies are best suited for GMO risk assessments. Their most recent publication investigates whether animal feeding studies are useful.
In contrast with reviews that found 90-day feeding studies to be sufficient, the GRACE project concluded that in most cases 90-day feeding studies do not provide any additional information over non-animal testing. “GRACE data support the scientific reasoning that only in case a trigger is available from the initial molecular, compositional, phenotypic and/or agronomic analyses, feeding trials with whole food/feed may provide an added scientific value for the risk assessment of GM crops. Thus, feeding trials might be considered, provided that the study design can be tailored to the posed safety concern.”
GRACE’s conclusions are similar to the idea that most healthy people do not need to undergo regular medical testing. When large numbers of people (or animals) are tested, some differences will be detected simply due to random chance – even when there is no underlying concern. To avoid the risk of false positives, there must be some initial concern to trigger the additional tests. For example, doctors only recommend that pregnant women undergo amniocentesis if there is some risk factor, such as concerning results from another test. Similarly, GRACE recommends that animal feeding studies are not necessary unless there is some risk factor, such as a compositional difference.
Animal Feeding Study – pdf – high quality
Animal Feeding Study – pdf – low quality
Permissions
Biology Fortified is making these infographics available under a Creative Commons Attribution-NonCommercial-NoDerivatives License. Everyone is free to download, republish, and use these infographics (images, slides) in their original form for nonprofit purposes. This infographic was made by Layla Katiraee together with Anastasia Bodnar.
Downloads
We are providing these graphics for non-profit educational use by anyone, in multiple formats. Please attribute them to us when you use them, and do not modify or translate them without the permission of Biology Fortified, Inc.
Biofortified 
I would like to highlight the importance of the final paragraph of this article about when they’re important.
It is a waste of time, resources and animal life to run well designed (moreso if the design isn’t good) feeding studies unless one has good reason to expect differences. If you don’t have a defensible hypothesis as to why a difference should be observed then you shouldn’t be performing the tests at all (from both an ethical and resource stance).
From what I know (which could be wrong, but most likely isn’t) both the arctic apple and the aquavantage salmon achieved regulatory approval without feeding studies being conducted, I think this deserves a degree of applause.
LikeLike
A number of the Bt strains were done without feeding trials.
Syngenta did what I thought was quite interesting.
Since the amount of Bt expressed in the corn seed is so low, they extracted and fed large chronic doses to a relatively small sample of rats (compared to the often 400+ rats in feeding trials). This way they were able to feed the rats far more of the Cry protein than the rats could possibly consume in a lifetime, in a relatively short time. The fact that it caused no harm was pretty convincing evidence that the incredibly tiny quantities in the actual corn was not harmful.
LikeLike
Rather than quite interesting though, that trial remains pointless. There was no valid hypothesis, and all evidence already suggested there’d be no impact. The trial was a waste of time, money and animals.
LikeLike
There is still rational for doing studies to confirm that there is no effect.
Logic and reasoning can only take you so far.
LikeLike
There is rationale, but it is weak, weak enough that one can write off the resources used as wasted, likewise the lives lost as pointless.
LikeLike
The resources are quite minor and verifying that there isn’t a down-side risk, even though the risk is quite small, is still worth that modest effort.
I like rats, but still, they don’t live much longer than the tests are run anyway. I certainly wouldn’t condone tests done on chimps for instance.
LikeLike
Quite minor?
Click to access rovida_hartung_altex_3_09.PDF
Has EU numbers on costs for the various studies.
28 day studies cost between 50,000 – 100,000 Euros and 40 animal lives. 90 day studies 115,000 – 250,000 Euros and 80 animal lives.
The only ones where one might argue resource use is low is acute toxicity where the cost is maybe 1,000 to 11,000 Euro and lives lost 12-45.
I’d assume, based on how corporate science goes, that you can probably multiply these numbers by a factor of 2 or more to land on a cost (corporate science, for one thing, tends to pay scientists a lot better)
Sure, in the grand scheme of things, not a huge amount of money. But it remains, in my mind, a complete waste of time of an individual who could have been doing something more meaningful, a complete waste of the resources, and a complete waste of animal life (better to never have existed than to exist merely to answer a question to which the answer is blindingly obvious)
LikeLike
One of the trials gave 5,000 mg/kg in bolus dose to mice with no apparent effects. to give a 100 pound human an equivalent dose, you would have to feed them 150,000 pounds of maize.
The “experiment” shows that the LD50 of that particular Cry protein is pretty high.
LikeLike
I considered including the animal welfare aspect in this post, but it seemed out of place. I am glad you have brought it up. To me, one of the most upsetting things about the proliferation of feeding studies that do not meet basic standards (as described in this post) is the waste of animal life. I mean, the waste of dollars and time that could go to useful things is bad enough, but some of these lab animals are basically tortured for flawed data. It’s not acceptable to keep animals that have huge tumors alive just so you can take gruesome photos, for example.
LikeLike
What to me is one of the most compelling “feeding studies” is this:
https://dl.sciencesocieties.org/publications/jas/articles/92/10/4255?highlight&search-result=1
It looks at the entire population of farm animals in the US. For the past number of years, virtually all animal feed has been GMO in origin. Before GMOs were introduced, the GMO content of feed was zero. During the transition from zero GMOs to mostly GMOs, there has been no observed detrimental effect on animal health.
The number of animals involved exceeds 100 billion. If there was a large adverse effect of GMOs, it would be obvious. If there was a tiny but consistent, adverse effect, animals fed GMOs would not be getting healthier over time (as the data indicates).
LikeLike
This is also true of every mouse from Jackson Lab or Charles River. Or any rat, rabbit, or other experimental animal. Every “control” animal photo or data since the 1990s was not getting “organic” chow in the US, I’m quite sure. And trained animal technicians monitor these animal colonies specifically looking for problems.
If all the research animal colonies collapsed from infertility and giant tumors, we’d have heard of it.
LikeLike
Starting in the late 90s I was working in a lab with a breeding colony, a “regular” population around 5K Sprague-Dawley and Long-Evans rats (standing population, with replacements), and about 50 p53 KO mice. Many of the rats were maintained for long periods, over 3 years. The mouse colony was specifically maintained to the equivalent of senescence, because it was an Alzheimers-related study.
Not a single one of the animals, including the neonates and the aged ones, developed tumours (even though some would have been more likely to do so than other breeds or models), during my time there. All told, they were extremely healthy animals.
Our monkeys, all very old ladies, were very healthy too, on both diets. They loved Teletubbies, bless their hearts.
/anecdata
LikeLike
No, really. We all got together and , as concerned lab technicians decided to post to this one media release and corroborate each other. Although it’s never happened before in 30 years of posting on this subject, you can trust us.
LikeLike
I would say that that feed source is a nice to have, but very hard to have. Think about it this way, you would have to start your study nearly a year before (upping the cost and delaying the findings) and you would need to find a farmer willing to grow both. From the studies I’ve seen, many have used the commercial lab chow diets and had the maker substitute in the transgenic crop, in the same quantity as the conventional crop. But they weren’t necessarily grown in adjacent fields. I wouldn’t find a study to be designed incorrectly simply because the transgenic crop wasn’t grown in the same place as the conventional crop. In fact the only study that I would expect that to be done on might be in the initial studies by the company developing the trait. I guess what I’m saying is I wouldn’t put that burden on say Seralini and say if that wasn’t done, his study wouldn’t have been done correctly. It does though point out the importance of the first point, feed analysis. Even if not grown in the same place, they should have a very similar nutrient profile.
LikeLike
Thanks Robert
LikeLike
Thank you Mark for continuing to demonstrate this level of discourse from those who oppose GE crop technology.. It is very telling to those who watch.
LikeLike
I guess were both telling? I speak for children and the population. You speak for anyone that will pay you.
LikeLike
It’s true that more planning is required in order to have an appropriate feed comparison. Having the feed be isogenic and grown in identical conditions is the ideal situation, as we state above. But if that is not possible, then feed analysis is even more important. Even with feed analysis, we can not be sure that any detected difference is due to the biotech trait and not to other genetic or environmental differences – so additional care must be taken when making conclusions from the results.
LikeLike
Excellent resource
LikeLike
Damn Robert, you got almost all your buddies from work to post on this one. Or are they all you under different accounts again?
LikeLike
Is everything a conspiracy to you? I have zip to do with what other people post. Now would you care to actually address their questions or demonstrate where they got the science wrong? I’ll wait.
LikeLike
You work for an orgamnization that hired you to post on this one subject day in and day out for years. You have no other income and you expect to be treated like a upstanding citizen. Your masquarades purpose is only to spead misinformation. That’s not a conspiracy. Robert, you and I have gone around for years. You’ve been busted in lies dozens of times. You are a paid pumper.
LikeLike
Hello Mark, please review our comment policy. https://biofortified.org/blog/comment-policy/
LikeLike
Sorry. Interesting we are talking about poisoning billions of people and you use good manners as an excuse to silence my concerns. Which have yet to be addressed. You have several members known for being in the employ of the industry posting freely. Robert Wagner (Not Wager) Wagner was banned on too many blogs for posting proven misinformation., and Arthur Doucette are widely known. Yet you encourage their input, despite their ties to AstroTurf. Anastassia, I look forward to your posts on this subject on other blogs. As a geneticist you may have a unique perspective.
LikeLike
You are definitely welcome to engage in conversation here, but ad homs will be deleted. This is your last warning on that topic. This is an evidence based forum. Anyone making specific claims should be ready to back them up, no matter what their option is.
LikeLike
I’d like to add – we encourage everyone to participate in discussions here, even including industry scientists, large scale farmers, organic co-op employees, and everyone in between. It doesn’t matter who you are – it matters what you bring to the conversation.
LikeLike
In French here:
http://seppi.over-blog.com/2016/01/les-etudes-d-alimentation-avec-des-ogm.html
The three goat studies are from Federico Infascelli, who is being accused of fraud. One of them has (already) been retracted.
LikeLike
Hello Wackes, thanks for the tip. We have covered the alleged fraud here: https://biofortified.org/2016/01/italian-research-group-subject-of-data-fabrication-probe/
What’s really ironic about this is that even with potentially fabricated data, the same research group couldn’t come to the same conclusions on the impact of Round-Up Ready Soy feed on goats and their kids. Just goes to show how important reproducibility is and why we need it before we can really rely on a single paper.
LikeLike
Ya, the hundreds of scientists that demanded a retraction were all very publicly rallied from the ranks of the GM industry. The studies have never been disproved by any independent lab. And the GM industry has never recreated the experiment in public or published any papers disproving those studies that were open for debate. So, industry millions spent suppressing a small study that was not commissioned to find GM problems but did. No studies to prove that the products are safe. That sounds outlandishly anti-science.
LikeLike
Mark, you have already been provided with many links to independent studies. We welcome you to participate in conversation here, but it’d be helpful if you’d follow through and either (or both) read the evidence provided to you or provide your own evidence.
LikeLike
Every study that’s been done and allowed out in the public is bought and paid for by corporations that developed the seed. NO long term studies have been done past 90 days excepting those done outside the country. When these are released the entirety of the GMO anti-science industry is leveled on them to immediately discredit them. Those country’s that get those studies immediately ban these foods as poison. Garbage science and outright anti-science is the only thing allowed to be published by, you guessed it, corporate American media. Trusting the media or people that say ridiculous statements like “If there was anything wrong, we’d have heard about it” is foolish. These are EXACTLY the same statements made by the tobacco industry for 70 years , while tens of millions died of cancers while Big Tobacco swore their products were completely safe.
Every independent study shows that these foods cause cancer, renal failure, kidney damage and mental development issues. ANYONE that says different is yelled down by an army of paid advertising AstroTurf posters.
LikeLike
Hi Mark, regarding studies done outside of the country, I find it odd to demand that scientific studies be done in a particular nation. Do you think the results of a well-conducted study done in EU should not apply in a different country? For that matter, what would you have changed to the GRACE study?
LikeLike
I believed you would. But most people in this country are very much aware of the intense pressure the GM industry has gone to suppress published results. The control they hold at teh FDA and the Chamber of Commerce. The government has been called upon for 30 years to do an impartial and long range study. The FDA refuses and the government removes funding to Universities that try to. These tests can only be done impartially, outside and away from the influence of these companies.
If you would care to address my points, I would assuredly address yours.
LikeLike
I’d be happy to address your points:
-“Every study that’s been done and allowed out in the public is bought and paid for by corporations that developed the seed.”
This is false. See the GRACE study which I cited in the piece. See GENERA where you can sort according to funding source/type and according to study outcome (http://genera.biofortified.org/). See this infographic that was generated using data from Genera: https://www.geneticliteracyproject.org/2014/11/26/glp-infographic-all-gmo-research-is-industry-funded-biofortified-analysis-sets-record-straight/
-“NO long term studies have been done past 90 days excepting those done outside the country.”
You provided your opinion on why you think this important in your comment above. Please provide evidence.
-“Those country’s that get those studies immediately ban these foods as poison.”
Please provide evidence.
-“Every independent study shows that these foods cause cancer, renal failure, kidney damage and mental development issues.”
Again, the GRACE study which is independent found no evidence of harm. So your statement is false.
LikeLike
“The control they hold at teh FDA and the Chamber of Commerce.”
So, what you are claiming is that an industry whose main player has net sales of approximately $15Bn can control the scientific consensus and control major arms of government?
Can you explain why it is that the same cannot be said for the fossil fuel industry? Exxon has net sales in excess of $300Bn per year.
How have we found ourselves in a situation where a relative minnow can sway the scientific consensus on a topic that is easily provable in a lab setting, whereas a behemoth is incapable of suppressing the science on a vastly complex topic (tin foil hatters not withstanding)?
To me this seems pretty irreconcilable with anything approaching a realistic view of the world. How do you manage such a reconciliation?
LikeLike
Hi Mark, you seem to have missed the many studies that are longer than 90 days, and the many studies that have no corporate funding. I know it can be difficult to find scientific journal articles. You may want to try PubMed or Google Scholar, searching for genetic engineering and biotechnology rather than “GMO”, as the term GMO doesn’t appear in many scientific journal articles.
Here’s a nice overview of long studies http://www.skepticink.com/smilodonsretreat/2012/10/24/a-survey-of-long-term-gm-food-studies/
And here is a discussion of how the grand majority of independent studies match the conclusions found in corporate studies https://biofortified.org/2014/02/industry-funded-gmo-studies/
LikeLike
I’m afraid none have come to light, and you have produced none here. If there are many, please produce a link and show that is impartial. The one you pasted is from a publication funded by the GM industry.You may as well post from the Genetic Literacy Project. There ARE hundreds of studies done in other counties. Those are freely available in a pro-scientific manner. They are put out for discourse and debate. Scrutiny is welcomed and the results are open to interpretation. None in the United States have thus far been produced under those pro-science conditions. Saying they are there does not prove they are there. Science is not something that can be treated with bias, unless you are simply in it to sell us something. With consistent proof provided of suppression of information, misinformation for hire being used on all fronts and consistent results outside the sphere of GM industry influence showing a clear and consistent record that indicates that these foods have severe problems, I would question anyone in the scientific community that has a positive opinion of the products. Far too many questions have been raised for any scientist to endorse them.
LikeLike
You’re making very broad claims yet providing no evidence. Did you check the funding sources of every study listed on the Smilodon’s Retreat post? Are you saying that the scientist behind Smilodon’s Retreat is industry funded? If yes, provide proof. Did you read the Biofortified Blog post? Are you saying that Marc Brazeau and Biology Fortified are industry funded? If yes, provide proof.
You’re right, there are hundreds of studies. And the majority of them show no concerns from biotech crops (as described in Marc’s post). The conclusions of industry funded studies generally match the conclusions from independent studies. That leaves us two options – either there’s a vast conspiracy, or the science is fairly consistent. Occam’s razor says it’s probably the second. If you have evidence of a conspiracy, then bring it forward.
LikeLike
Dr. Bodnar, have you assessed these studies to see that they meet Biofortified’s own guidelines for good controls? Are we able to see the actual results to decide for ourselves whether or not they are biologically relevant?
The very first study referenced in your link to Biofortified’s page on “those industry studies” is the Snell et al metastudy. I’ve pointed out in my comment to Jonathan Latham (above) that a large number of those studies fail the criteria the OP says are important.
You can’t claim that the feeding trials that show harm are flawed, but that those which show safety aren’t when they both fail to use good controls.
Besides, claiming that gmos are safe is unscientific, just like claiming that drugs or cars are safe. Every event is unique. And many traits are unique. And the way they affect human physiology is unique. I’d like to see us get away from these generalizations and start to try to take a critical stance on the individual products. It’s problematic for those who want to condemn or promote the technology wholesale – but I think such an approach better reflects an appropriate application of science to regulation.
LikeLike
There is quite a bit I disagree with in the article above. This statement stood out in particular because regulators use such arguments too: “Statistically significant differences are not necessarily biologically relevant. This point is intertwined with statistical power. If you take
two groups of animals and take enough measurements, you are bound to
find a measurement that is different between the two groups. As such, it
is important to address the question: is the measured difference
biologically relevant?”
Except that:
1) if the result is highly statistically significant (as opposed to borderline) then what you have written becomes irrelevant (and not “bound” to occur).
2) and that “biological relevance” is a bogus issue for 2 reasons. One is that no one ever really knows what is biologically relevant, and secondly if the authors really think an unexpected difference is biologically irrelevant (which argument I see a lot) then why did they measure the parameter in the first place? To count a parameter only if the outcome suits your argument is a double standard yet this argument is routine in EU GMO risk assessments.
LikeLike
In fact this statement is not correct. If you make 20 observations, then roughly 1 in 20 will show a statistically significant effect at 0.05, even if that effect is not real. If you make 100 observations then roughly 1 in 100 will show a statistically significant effect at 0.01 even if that effect is not real, and if you make 1000 observations then roughly 1 in 1000 will show a statistically significant effect at 0.001 even if that effect is not real.
In fact it is worse than this. The calculations are usually based on the number of tests done and the probability of false positives. So for 300 measurements, the probability of finding one or more false positives is virtually 100% at p = 0.05, 95% at p = 0.001 and 26% at p = 0.0001. So you don’t even need to have close to the number of tests to have an inflated probability of detecting false positives for even highly statistically significant values.
This argument is just so much semantics.
In whole food feeding studies a whole range of values are measured, because they are essentially fishing expeditions. At the beginning of the study, there is no knowing what might turn up to be interesting. If you were to take a proper view of GM whole food feeding studies, you wouldn’t bother doing them if your standard was a prioiri biological relevance.
The dismissal of an apparently significant effect as not biologically relevant in well conducted GM whole food feeding studies comes about either because there is no dose response for that item. That is the significant value occurs only at a lower concentration of food and not a higher one. Or because the values obtain fit within the expected set of values across a variety of foods of that type. That is a significant value in the comparison between the GM and non-GM corn isolines occurs, but values greater or lower than that occur with feeding other types of non-GM corn.
LikeLike
“In whole food feeding studies a whole range of values are measured,
because they are essentially fishing expeditions…If
you were to take a proper view of GM whole food feeding studies, you
wouldn’t bother doing them if your standard was a prioiri biological
relevance.”
This opinion dismisses, out of hand, the best means we have available for measuring real-life effects of engineered genetics and new traits in the food supply. The values measured typically reflect on the health of the individual: liver and kidney function, blood and tissue histology,etc. “A whole range of values” are what’s needed to make a comprehensive comparison.
That doesn’t mean there aren’t problems with these studies. As the article shows, those with a pro-industry bent can find a reason to dismiss the findings of probably any study they choose to, while the industry can find ways to dismiss the results of their own studies and claim “trade secrets” on the raw data.
The US doesn’t require feeding studies. As we begin to engineer new traits for direct human consumption, we ought to be calling for mandatory, improved and standardized feeding trials – not throwing them out the window.
LikeLike
As described in this textbook chapter on multiple comparisons, if a positive result is found in a fishing expedition type of experiment, that means we need to use caution and test some more to determine if there really is an effect. http://www.biostathandbook.com/multiplecomparisons.html
The problem I see in a lot of these studies that test a ton of factors is that they find something significant and claim it’s proof of risk, even when there’s no dose response.
It would be best to have a plausible risk hypothesis first, then test the hypothesis. This is what GRACE advocates – they want to find evidence of risk before getting into a feeding study that has high risk of false positives.
If there is no plausible risk hypothesis, and the researchers want to do a feeding study anyway, then they should use caution. That means correcting for multiple comparisons and/or following up any significant results with more carefully designed studies.
LikeLike
I don’t disagree with anything you’re saying here Dr. Bodnar. I’m just troubled by the fact that the industry itself doesn’t use good controls and also doesn’t follow up on positive and statistically significant results. And usually has no purpose in parameters that would reflect on human safety. So, to dismiss feeding trials as “fishing expeditions” is disingenuous on the part of the industry. You dont’ find what you don’t look for.
LikeLike
When one measures a whole bunch of stuff and gets a couple of statistically significant differences one does not “follow up” on them, because one understands statistics.
This is why massive multiple measure experiments are called ‘fishing expeditions’ – because if one measures enough variables you will find statistically significant differences simply because of how statistics works.
The difference between a fishing expedition type of experiment, and one that is not, is that the former makes a fanfare of the handful of statistically significant differences it finds, whereas the properly analyzed experiment will only make noise if there is a coherent signal of something actually going on.
LikeLike
Therefore the famous ‘reanalyzed’ Monsanto study (where Seralini made a fanfare about finding differences) was an appropriately designed experiment with appropriate analysis, whereas Seralini’s take was a fishing trip on the same data designed to pop up a few more significant values (through non-standard methodology) and make a damned circus out of science based on most people’s inability to misunderstand what statistical significance means, and what it doesn’t.
LikeLike
Do you want to go into details about the NK603 studies? Show us the raw data so we can decide whether what you’re saying about positive results not being statistically significant is factual.
If the Monsanto study was appropriately designed and analyzed, why did Monsanto scramble to dig up some historical data in order to re-frame the results of the experiment?
LikeLike
Do I want to?
No, I really don’t. I’ve gone round and round and round on the bad reanalysis of the NK603 papers (After 7+ years of arguing the same stale points and getting back the same level of misunderstanding the level of desire to fire up again and explain things only to be responded to by sheer ignorance or bloody mindedness literally holds no pull for me whatsoever), and have at no point suggested that something wasn’t statistically significant (my suggestion is that things that are statistically significant aren’t necessarily biologically relevant or indeed real, and if you have issue with this then this simply illustrates that you do not have a sound understanding of statistical significance and what it means (I’d argue that most people fall into this category))
Not getting this is literally down to not understanding statistics. That’s fine. Carry on not understanding statistics. Carry on going against what any remotely qualified scientist gets drilled into them over and over – one does not go chasing p-values simply because one gets a few statistically significant data points, for that way madness lies.
LikeLike
By what rationale did Monsanto dismiss statistically significant results as being biologically irrelevant? I believe it was liver and kidney lesions. Since you work at Monsanto, maybe you know?
LikeLike
It’s not physics. If you’re assessing the health of an animal, you need to check those parameters that reflect on the health of the animals. And if you’re trying to find out whether or not something the animal is eating might be bad for humans to eat, you need to choose a species that is physiologically similar to human. Liver and kidney histology/function test results aren’t affected by the results of gut histology analysis or endocrine function – although at a certain point in a disease process these things are inter-related. When there are statistically significant results, it’s important to follow-up.
LikeLike
“When there are statistically significant results, it’s important to follow-up.”
Not when they are not biologically significant. No. You don’t. This is standard practice accepted by all regulatory agencies globally dealing with materials where animal feeding studies are used to assess toxicology.
You don’t simply get to assert that one has to follow up on every last statistically significant result – there has to be sound reason to do so, if all the liver results were off, if a dose response was obvious, etc etc. A couple of ‘lucky/unlucky’ p-values aren’t worthy of note, as essentially any experimental scientist in any discipline at all will tell you.
LikeLike
Massive multiple measures? Huh?
LikeLike
Massive (experiment size)
Multiple measures (multiple measurements are taken in the experiment)
Not technical jargon, just me abusing the English language.
LikeLike
What massive multiple measurements are you talking about? Are you still talking about feeding trials? With NK603 Monsanto measured 10 rats per group. Same as Seralini. And in the case of Monsanto’s study on its own NK603, it decided that a statistically significant number of liver and kidney lesions were biologically irrelevant. I concede that liver and kidney lesions don’t necessarily mean the GMO was bad. But they’d have to be accounted for somehow. They don’t just show up as random chance on top of non-random physiology.
LikeLike
Here’s the problem I see with this approach – we already have a hypothesis based on how some engineering techniques affect the genetics and metabolism of the plant. I think this approach would definitely let some plants off the hook, but it’s too broad to say “we need a hypothesis” – because it’s already obvious that even highly educated scientists don’t agree on whether or not there’s evidence of risk.
Also, I don’t think you meant to say that if there’s no dose-response, it’s unlikely there’s evidence of risk? Some effects of toxicity aren’t proportionate to dose.
LikeLike
This statement is completely incorrect. A significant problem with whole food feeding studies is their lack of statistical power. They have a high probability of not finding something even if it is there and are highly likely to throw up false positives.
The problems lie at the beginning. Whole food is an extremely complex substance and any difference of toxicological concern is likely to be present in very low concentrations. That means high amounts of the food need to be fed (often 1/3 of the diet). This risks the results being compounded by failure to obtain nutritional balance. Not so bad for corn, but potatoes and other foods can be tough on rats. It also means there is plenty of opportunity for confounders to appear. You need isolines of the variety, grown in the same field, under identical conditions and with the same nutrition to avoid having other differences present in the food.
Of far more value is to test the toxicological properties of the introduced proteins directly. These can be concentrated to high concentrations that avoid the problems with diet matching and can be fed at concentrations much higher than people would ever encounter in the food. One can also test for known toxins produced by that species to see whether the modification or the new way of growing the crop (if relevant) has affected these and increased the risk.
There is good reason why many regulatory agencies do not require animal feeding studies.
LikeLike
From your own link:
“…we do acknowledge that there may be future GM foods where these studies may be useful.”
I agree.
…The expert panel concluded that we should continue our case-by-case assessment of GM foods based on the best available science.”
As I’ve always agreed, and which is contrary to your contention that we ought to just not do them because they’re fishing expeditions and can’t possibly relate any actual data.
LikeLike
So do I. However, the situation as pointed out is that at the moment these studies are not.
It would be remiss of a regulator to completely dismiss forever a type of trial that may be useful in the future due to changes in technology or a different type of application. That doesn’t mean it has value now.
The expert panel concluded that the best available science did not include whole food feeding studies. This is why they are still not required. It is in fact reasonably simple to determine whether a study has a reasonable hypothesis or is simply a fishing expedition. When a whole food feeding study is presented that contains a reasonable hypothesis and whole food is the best way of addressing that hypothesis, I would accept that. Until that time occurs, whole food feeding studies are not the best way of determining safety of GM foods and in fact have virtually no value.
LikeLike
We need qualified and unbiased scientists to determine whether or not a study is a “fishing expedition” (derogatory) or has a reasonable hypothesis. Some genetic engineering can cause changes that would make it possible for deleterious molecules to enter the food supply. The fact that we don’t do feeding studies now is due to the regulations that were set up when GMOs first came on the market and the difficulty of changing such regulations. Also, the fact that 99.9% of this food isn’t eaten directly, and, as pointed out by the above post – we really don’t have any good feeding trials to look at. The Snell study, which industry advocates reference as showing “long-term” studies, suffers from the very same problems enumerated in this post. Instead of shining a critical light on that meta-study, the OP says that the studies that Snell et al reviewed shouldn’t even be done at all!
So – let’s review. There are no good feeding trials out there that would reflect on human safety. Even if there were, every GMO is a new event offering a new opportunity for molecular changes. And while on the one hand the industry likes to claim that there are hundreds of studies that “reflect” on human safety – the truth is that the studies they’re talking about generally reflect on very basic safety for meat animals. That is, they’re short-term and look at gross characteristics like weight or litter size – parameters that are important to the meat industry. Now, while the industry says that there are lots of safety studies, they simultaneously point out all the flaws of every feeding trial that shows negative results, or hide the negative results of their own studies.(Monsanto NK603)
Instead of setting some standards and deciding which events we ought to do feeding trials on based on the risk of ending up with some altered and potentially harmful metabolite in the human body – we apparently have decided that feeding trials should not be done at all.
To me this is just the industry’s latest go at a free-for-all as they plan a move into engineering more of our food supply – foods directly consumed by humans. Every industry that’s employed a technology to make money (oil, auto, nuclear, etc) will function like artificial intelligence. It’s programmed to make money (as it should be). Part of making money is avoiding regulatory oversight. The industry will always work to loosen regulations and will push the limits of safety in the interest of profit. What this means is that those who support the industry find themselves advocating less regulation. And that’s why we have disasters like the BP Gulf spill, the methane disaster in Southern California, 3-mile island, etc. With GMOs, the companies we’re talking about each have their own histories of environmental and/or customer harm. They’re chemical companies that have morphed into seed companies, and there safety records are public knowledge. If you don’t learn from the past you’re bound to repeat it.
If we don’t do feeding trials as part of the approval process, I don’t understand why there’s a push to stop doing feeding trials. Scientists ought to be able to do research on the safety of GE foods for human consumption. And studies done by the industry ought to be available for public review.
LikeLike
No. This is an experimental design question. In these feeding studies there is no a priori hypothesis that X item in GM food will cause Y impact. Therefore, they are fishing expeditions. Indeed in regulatory parlance these are known as ‘safety assurance studies’.
There is also nothing derogatory about describing this type of study as a fishing expedition, although the more normal usage is ‘hypothesis generating’, but it means much the same.
How exactly?
The most likely way is that the protein produced by the change is itself toxic. This is already tested in much better quality toxicity testing than the whole food feeding studies done.
Another possibility is that the process of tissue culture results in a mutation that increases the concentration of a toxic molecule already present in the species. This is also explicitly tested for.
What else?
What negative results exactly? Hammond et al. 2004 reported no statistically significant difference between rats fed up to 33% GM corn compared to those fed normal corn. Actually, if you are going to have whole food feeding studies, Hammond et al. 2004 is a textbook way of analysing such studies,
LikeLike
Chris, the facts of the technology provide the hypothesis for X causing Y in GMOs. I tried to explain this in our conversation about tryptophan. That conversation has been all but deleted and I’m reluctant to go over this all again. “Fishing expedition” is a derogatory term. When Monsanto does its own feeding studies to see if anything deleterious has occurred – are those “fishing expeditions”?
You don’t seem to be familiar with what regulations actually require, and how regulations wouldn’t catch various kinds of problems that can occur – especially with the kind of plants that the industry hopes to begin to engineer, in order to capture a larger share within one sector of the food production industry. You’re claiming some things that don’t happen. I would ask you to provide evidence that these plants are tested at all beyond gross characteristics and toxins common to that plant. We test the trait – if it’s a protein – separate from the plant. What is your guarantee that the plant without the protein and the engineered plant are the same, except for the addition of the protein? Where is the research?
LikeLike
“When Monsanto does its own feeding studies to see if anything deleterious has occurred – are those “fishing expeditions”?”
A fishing expedition, as I’m pretty sure has already been pointed out (I at least vaguely remember typing similar) is not characterized by the experimental design (look at lots of things) but by the experimental design followed up by then making a big deal about a handful of unrelated non-dose dependant responses which don’t have the hallmarks of an actual reaction.
One cannot, without knowing the actors, designate any given experimental design as a fishing expedition until one does a post-hoc analysis of the design plus the conclusions.
LikeLike
So apparently you and Chris don’t agree on what a fishing expedition is.
LikeLike
I disagree with you. An experimental design where there is no a priori hypothesis and you are only looking to see what might turn up if x is done is a fishing expedition. The normal practice is to then set up a test to explicitly test the relationship between treatment x and outcome y.
Making a hullabaloo about x causing y without the follow up testing is extremely poor practice.
LikeLike
You may disagree. That’s fine.
Click to access 41753317.pdf
The authority on feeding studies, however, doesn’t agree with you.
I’d generally rather be on the side of the authority on the matter.
I’m not convinced good argument can be made that the OECD guidelines are not scientifically rigorous.
LikeLike
You need to be more specific than that. What specific impact do you propose GM food will cause that will not occur with non-GM food. What is the base evidence that supports such an effect might occur?
EMS was largely the result of high tryptophan ingestion and had nothing to do with the source. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3848710/ It just turned out that Showa Denko was a common source in the US at the time.
Monsanto like to call the safety assurance studies, but they really are only fishing expeditions.
I am actually very familiar with what the regulations require. I am occasionally consulted by regulators and submit comments to regulators during the comment period that the regulator actually responds to in detail.
What else exactly would be worth testing and why? Where is the evidence that anything else might be a problem?
Some can be found here
LikeLike
“I am actually very familiar with what the regulations require. I am
occasionally consulted by regulators and submit comments to regulators
during the comment period that the regulator actually responds to in
detail.”
If you don’t mind my asking: in what capacity to regulators consult with you? And what are your qualifications in that area?
The paper you linked to illustrates the kind of changes that can occur and which aren’t tested for as part of regulatory approval process.
We still don’t understand the etiology of EMS. I’d like to be able to just link you back to
the last time we went ’round on this – but my comments and links on that
page have been deleted, so whatever. Likewise, have offered
explanations on X causing Y.
LikeLike
Your argument would be stronger if many studies did measure so many parameters. The study Seralini extracted from Monsanto (which from memory was on NK603 corn) did but most studies I have seen only measure a few.
LikeLike
I agree. The meta-review from Snell et al
Click to access Snell%20GM_feed_review_Food_Chem_Toxicol_50_1134%202012.pdf
shows that the studies that industry advocates cite as showing safety, have the same problems that the OP describes – no isogenic controls, and few relevant parameters. The OP says that the GRACE project “concluded that in most cases 90-day feeding studies do not provide any additional information over non-animal testing”.
Monsanto’s study on their own NK603 revealed statistically significant liver and kidney changes. Monsanto had dismissed those results as not significant.
LikeLike
You may want to look into the concept of false positives when testing multiple comparisons.
Here is a good overview of the topic, including some examples of how to correct for multiple comparisons. http://www.biostathandbook.com/multiplecomparisons.html
LikeLike
How can you compare this sort of statistical analysis with that of histological and chemical analysis on organisms in control groups within a feeding study?
LikeLike
Thanks for stopping by, Jonathan. One detail that did not make it into the infographic but bears mentioning is that you need to weed out false positives by doing multiple comparisons. This is a basic issue that researchers from Pusztai to Seralini seem to have a problem with doing. A single statistically significant finding usually becomes non-significant when you factor in the multiple comparisons being made. Even then, it may not indicate a problem.
http://www.tandfonline.com/doi/full/10.3109/07388551.2015.1130684
Determining biological relevance is highly relevant. While an infographic does not have the space to go into this, it is a necessary step before you can claim harm or disease. Diseases don’t just impact one biomarker – they have profiles of impacted systems in the organism. Finding a statistically significant difference in one marker but not in others that are related suggests random chance and not a pattern of disease. If you find, say, a pattern of differences in liver parameters that indicate impaired function then you have the kind of biological relevance that one liver parameter does not.
LikeLike
Feeding trials aren’t looking for disease. They’re simply making comparisons between control groups and compared to what’s considered normal.
LikeLike
“Determining biological relevance is highly relevant.”
That’s why they call it relevance. (sorry, couldn’t resist)
“A single statistically significant finding usually becomes non-significant when you factor in the multiple comparisons being made.”
Can you please explain this in the context of a feeding trial? for instance, if multiple animals show the same kind of changes in the liver compared to the control which shows none or a significant number less – how would that become insignificant? While it may be valid to say that diseases impact numerous “biomarkers” – it isn’t valid to say that if the animals aren’t “diseased” as a result of eating a gmo for a few weeks, then any physiological changes are irrelevant. If you have good controls, the differences between the groups are what’s important.
In short-term studies, if you found a consistent change in one liver parameter – wouldn’t you have to justify applying “random chance” to biological systems in order to dismiss such a finding? Physiology isn’t random. It’s highly non-random.
LikeLike
” Physiology isn’t random. It’s highly non-random.”
One assumes you have never participated in an experiment that involved measuring physiology.
LikeLike
For giggles I once measured the nitrate uptake of 3 groups of plants. (well, I was also measuring the inherent noise in the system that we had in order to properly power the test, but giggles were had too)
Group 1 had a statistically significant increase in uptake as compared to group 2, which had a statistically significant increase in uptake as compared to group 3.
This potentially would have been an exciting result – corn lines with inherently better nitrate uptake, what a hit. Right?
Only downside is that they were all genetically identical, the only difference between the lines was that I had arbitrarily assigned group numbers.
LikeLike
And this relates to rats in a feeding trial,,, how?
LikeLike
It establishes that physiology has a random component that can throw an experiment and give statistically significant results which have absolutely no biological relevance.
LikeLike
Can you give an example?
LikeLike
Can I give an example of an instance where physiology has a random component that can throw an experiment and give statistically significant results with no biological relevance?
I literally just did… (albeit my own anecdotal)
Here’s a report from the literature where one sees the same thing going on
Click to access Hammond_Study_Bt_corn_ECB.pdf
LikeLike
Ewan – I consider you to be a strictly honest and scientifically-minded person. I have no problem with your anecdotes. The problem is that in this context it would be impossible for you to tell us everything we’d need to know about your experiment before we could say take it as an illustration of your idea “There is a random component on top of non-random responses”. And I think I would likewise need to have you explain how the Monsanto paper illustrates the same.
Here’s my hypothetical in response to your anecdotal:
You go to your doc for an annual check-up. You are poked, pried, palpated and generally violated for your own sake. Lastly, there’s a blood draw and a urine sample. Everything looks good, but one of your blood values is off. I can tell you, this would typically be ignored and most patients never see the results. However, if it’s off – that’s statistically significant. Because we have established a range of normal in each case.
If something’s off – it’s may or may not be biologically relevant. But you don’t get to just dismiss it as irrelevant without determining the cause. Of course there are plenty of other explanations for why it might be off other than something wrong with you. And if several other apparently healthy people had their blood drawn and analyzed that same day from the same place and they all had the same “off” – then the biological significance becomes less. Hence, the purpose of controls. But if all of you live in the same apartment complex and eat the same food, then it becomes relevant again. But you don’t just say it’s biologically irrelevant because it’s only one value or it’s only off a little. You repeat the draw at a later date. Then if it’s still off, you do some more diagnostics.
It is absolutely possible for a perfectly healthy person to have abnormal test results. But no one authoritatively ignores something physiologically abnormal. Because, again, we’ve statistically established those ranges, and if one is actually off after re-test, then there is something abnormal. Not necessarily disease – not necessarily a problem. But not irrelevant. Good scientists don’t just say “must have been something wrong at the lab because everything else is fine”.
LikeLike
Coincidentally, I have a new product to sell that is labelled 100% as “group 1 corn.” It has been scientifically proven to have increased nitrogen uptake.
LikeLike
That’s funny.
LikeLike
One assumes that another doesn’t necessarily have any education in physiology.
LikeLike
There is a random component on top of non-random responses.
I do not suggest that physiology is entirely random, just random enough that the noise can, will, and does (as anyone who has done physiology on any species will attest) throw up statistically significant differences with absolutely no meaning whatsoever at a rate that one would expect based on how statistics works.
Hence something like, say, nitrate uptake (corrected for root dry weight) amongst 3 groups of genetically identical plants, can vary so much by random chance between groups as to give statistically significant differences between the groups, when no biological relevance can be assigned. Or, how something like, say, one single liver enzyme level, can vary so much by random chance between 3 groups of rats that statistically significant differences are observed where there is no biological relevance.
LikeLike
That’s ridiculous. If you’ve got results that show difference in a liver enzyme and you’ve set good controls and numbers – you do it again. You don’t just say “oh – must be random chance”. What’s the point of the research if you’re going to ignore the results? If your experiment is well designed, you’ve allowed for random chance – and differences that are random won’t be significant. Of course it’s possible for the universe to play a joke on you and create results that don’t actually reflect facts. That’s the importance of follow-up, or critical review of your research to make sure you didn’t overlook something.
LikeLike
No, you design the experiment so that if something is actually going on then you see it, and accept that there will be some degree of noise in the data.
Results are not being ignored here. It is not the fault of scientific researchers that you lack the basic scientific or statistical knowledge (or at least pretend to) to know which results one follows up on and which results one does not.
These results have been looked at by multiple actual experts in toxicology and the like (ie those at every regulatory agency globally, for a start) and have been accepted as just fine.
LikeLike
It’s a bit strange pointing to Seralini and complaining about the power. The time to complain is when you think the authors want to conclude the null hypothesis (no effect) is true, and use small sample sizes in order to fail to reject the null. Seralini group is presumably highly motivated to want to find a significant effect, not hide it.
LikeLike