Making sense of lists of studies

posted in: Science | 31
Wouldn’t it be great if these were the signs at protests? by IFeakingLoveScience

The global discussion of genetically engineered crops has been heating up, and people are looking for answers to their questions. Are they safe to eat? Do they harm the environment? Does it make the foods radically different from what they were like before? And is there independent research besides the industry-funded science? These are all good questions, and we’ve been doing what we can to help people understand the issues through blogging, interviewing, and hosting discussions on our site. The biotech industry has started up their own GMO Answers site to address the common questions that they get, but we at Biology Fortified have gotten a fair bit of attention for a little list we started years ago. That list, what it means, and what it is becoming, will be the subject of this exercise.

As scientists, we here are familiar with the scientific literature of our field, and know how to look up and understand information in the scientific literature. The peer-reviewed scientific literature is written by scientists for scientists, and are not generally intended to communicate the results to the general public, as the terms, methods, and results are complicated and require background knowledge to understand. For the public, the scientific literature is generally not accessible. So it comes as no surprise that in our journey to help explain the science to the public and engage on this topic, the claim that there is no science, hardly any science, or only industry science conducted on genetically engineered crops kept coming up. Some would say that the few independent studies always found problems.

Dr. David Tribe started to assemble a list of studies to refute these claims, and to show the studies that support the safety of these crops for consumption, the environment, etc. Over time, this started to climb into the low hundreds. Based on this list, we created our own here, and started adding more studies to the list as we found them. We also created a second sub-list that contained studies indicating that they were funded by sources independent of industry. In discussions, when people would make these various claims, we and others started plugging the links to these lists in to help people see that at least those basic claims were false.

The Failure of Mere Lists

listSuch lists as communication tools are not very useful. I recall one discussion where someone kept aggressively demanding for us to find out if there was any research on a particular genetically engineered crop. We suggested they take a look at the list and see if they can find anything. They declared that there was nothing of use in that list and no research on that crop at all. Again, we entreated them to just give it a try, and search for words that describe that crop and they will probably find something. Again, they howled that it was not in there and we were taking them for a ride.

The research they were looking for was number 4 in that list.

There have been similar instances that demonstrated to us that simply providing a list would not help people understand the science. Indeed, it can be a barrier to learning because of how intimidating it is. Lists like this are really only useful for scientists, so we started thinking about how we could turn this list into something far more useful for our target audience – everyone else. That’s when we conceived of making it into an actual database that told people about the research itself. And rather than fill it with only research that appeared to support one conclusion or another – we decided that it must include all of the relevant research to be a useful resource. We wanted to write plain-language summaries of each study, and list their outcomes and funding sources on the sidebar, and use crowd-sourcing to do it. The first concept of the GENetic Engineering Risk Atlas (GENERA) was born.


With a little fanfare, we announced in 2010 that we were starting the GENERA project, complete with how-to guides for volunteers to help us organize and describe this research. But, this task of writing original summaries for what was then only 300 studies proved too daunting of a task to do on a voluntary basis. The next year, we applied for grant funding to make this possible, but didn’t get it. This started a process of thinking and reconsidering our approach.

Scientific papers have their own summaries already written – the abstract. You can get the general idea about the scope of a paper by reading this, and most people would probably not bother reading even the abstract if we had the simple sidebar study breakdown like we had envisioned. For most people, seeing the funding, results, and category of research at-a-glance would be enough. Last year, we re-applied for a grant with the American Society of Plant Biologists to fund the project, and we got it.

At the time, we then had about 350 studies in the list, but when we started to systematically search the literature for more studies, the number grew – and fast. Before we knew it, we had 600 citations in our list. Since we were busy turning this into a database, but people were still looking for the basic list in discussions online, we set it up to display the first 600 citations, organized alphabetically, and no longer edited it to add or remove citations so we could concentrate on the project at hand. Though 600 were listed, the number has grown considerably more in the past few months.

As discussed in this post, we searched long and hard for a system to make managing this project possible in the long term, and we settled on a reference manager that allowed us to keep track of and tag each PDF with useful information, and export it into a spreadsheet to instantly turn it into a database we could upload to the site. Our call for volunteers was answered by some of our readers and fans, who have been helping us track down, download, and enter the citation information for each and every study, and search the reference sections for additional studies to add.

Our growing library of studies. The number is real.

It has been working out great, and of the original 600 citations you can see online, our volunteers have already gone through and picked out all the publicly-available studies, and have moved on to combing other people’s lists of citations, including those on anti-GMO activist websites. They have helped us identify duplicate citations, as well as citations that were to conference abstracts instead of peer-reviewed studies. These sorts of things can happen when you cobble together multiple lists and search results, particularly when there are minor spelling differences between citations that make them harder to spot.

I am slowly crawling down our list to get to the hard-to-reach studies by contacting the authors, reaching through the toughest paywalls, and even scanning the studies from library collections if need be. I’m only as far down as the letter D for this process right now. There is a lot more to find, and indeed, as I will demonstrate, the number of studies that will be included may very well reach 1,000 before we’re done digging up past research.

Now comes the difficult task of assessing the outcomes and funding sources of each and every study. Each study may have experiments that specifically address questions of safety for consumption, safety for the environment, efficacy of using genetic engineering for a desired outcome, and analyses of the equivalence or differences between genetically engineered crops and their non-GE counterparts. We are not interpreting the quality of the methods, analysis, or conclusion of each study, but are simply rating the outcomes as reported by the study authors. Similarly for funding sources, we are categorizing them according to industry, government, competing industry, and various categories of NGOs, and for those without such information, we are contacting the authors.

The end result, we hope, will be the most comprehensive and useful database of research that focuses on the relative risks of genetically engineered crops.

To think like a Scientist

There is a bit of a mental shift that has occurred, which is akin to the change in perspective that you can get when you become a scientist. When talking about the safety of genetically engineered crops, it can be tempting to dismiss the studies that reach conclusions that you disagree with, or not include them for consideration on the basis of your assessment of the science or the scientists conducting it. People cherry-pick all the time to win arguments. The original list of studies was a response to cherry-picking from critics of genetic engineering who try to ignore the vast array of science that overshadow the few critical studies they can find. But just responding with only studies that support your own point of view – even if it is the consensus view, is presenting less than 100% of the science.

Officer Barbrady does not want you to read the NAS NRC consensus reports

I once heard a politician go on the radio, responding to me mentioning our GENERA project earlier in the program, and regarded it as merely saying “My science is bigger than your science.” He regarded the few cherry-picked studies he had in hand as being enough for him to ignore the rest of the literature. This is a fundamental misunderstanding of how science works. There is no “your science” and “my science” – there is just “science.” When you collect it all together, you derive a consensus view based on repeatable and predictable observations. There will be papers that dispute the consensus – this is predicted by statistics – but you only arrive at a complete view of what we know when you take it all into account.

This week, a group in Europe called ENSSER published a political signing statement that there is no consensus on the safety of genetically engineered crops. They list a few cherry-picked studies that find problems, and on this basis declare that there is no consensus. They also reference David Tribe’s version of our list of studies, but avoided mentioning it by name in the body of the text. What is funny is that they call it an “Internet website” – as opposed to non-internet websites? They totally ignore all the National Academy of Sciences consensus reports, and misrepresent studies as finding “Toxic effects” though the studies themselves do not claim to find them. Their intent is to dismiss a large body of research and focus on unrepeated studies with methodological problems that disagree with the rest of the literature. This is not thinking like scientists, but like ideologues and politicians.

When I see people link to their favorite unrepeated studies claiming harm from genetically engineered crops, instead of getting annoyed I get excited. “Excellent – more studies for GENERA!” This is a healthy perspective, one that reflects a truth-seeking attitude that I hope more will come to appreciate. So naturally, I combed through the references in the signing statement for any new studies they knew about that were not in our database. The only studies that were new to me actually had conclusions that were positive about GE crops. We’ve been grabbing lists of studies from groups claiming that they have research that shows problems with genetic engineering, and harvesting them like this for every last citation they have. All will be assimilated. Because Science – and GENERA – are not about cherry-picking to reach a predetermined conclusion. Science does not operate on the basis of political signing-statements, but on the weight of evidence. This is what it means to think like a scientist.

Unclear on the Concept

We’ve been very public and clear on the aims and scope of this project, however, communicating this to others has met some challenges. Claire Robinson of GM Watch, was unaware that we were including all research, not just one “side” as she had assumed. I talked to several activists and people skeptical of GE foods who seemed to also have difficulty with understanding this. Some of it could be a consequence of remembering only what the list used to be, but much of it I think reflects how people typically approach this topic. I have said many a time that “we’re including all of the relevant research” to hear people process at and respond: “so you’re only including what you agree with?” – at which point I have to repeat myself. But I have come out of conversations with anti-GMO activists who were really excited when they learned that we were going to be comprehensive with the Atlas. No other organization deeply embedded in this debate has ever tried to do this.

Some have expressed shock that we would include research that reaches negative conclusions in our project. How can a database of studies demonstrate a low risk for GE crops while there are a few studies that disagree with that conclusion? Simply by showing that they are a minority of the studies, and that their results are not found in studies done by other researchers. The same is true for one-off studies claiming that climate change isn’t real, when the vast majority of the literature shows that it is, and it is repeatable. This is a communication challenge that we’re hoping to address, because it cuts to the core about how science works, as I have explained above. Plus, by including all of these studies, we may end up finding patterns that point to risks that no one had considered. Science does that.

A more recent example of misunderstanding is a series of blog posts by Madeleine Love formerly of MADGE Australia, in which she prematurely reaches the conclusion that all the many studies in our GENERA project are irrelevant, misrepresented, or over-reported. I would normally not highlight this, however, several prominent activist sites, pundits, and one scientist who is a vocal critic of genetic engineering have been promoting it as if it was a cogent and insightful analysis of our project. It is not.

Ms. Love has rightly pointed out that there are duplicates within the list, and ironically at the very top of the list, so she started subtracting from the total of 600. Her goal is to try to prove that this project and the research outlined in it can be dismissed. She gives the impression that we’ve overstated how much research there is, or what its conclusions are. A twitter conversation with her revealed that she blames us for misunderstanding what our project was about, but places the blame on others, never once pointing out an error of ours, or taking responsibility for her own misunderstanding.

She is engaging in a citation-by-citation exploration of the studies in our list, which is a process that I wholly endorse. She is demonstrating through this process what we already know about the uselessness of such lists as a communication tool, and the contents of our own list that we have already figured out months ago.

GENERA-success-boy300However, she engages in a doctrinaire and unscientific approach in her posts right from the beginning. For instance, the first citation was a conference abstract for a paper that was later published. Caine et al. (2007) looked at pigs fed four diets: one containing roundup-ready canola, one with the nearest isoline, and two reference diets with different varieties of canola. The conclusions of the researchers was that the diets were nutritionally equivalent, and that there was no differences due to the addition of the genetically engineered trait. They looked mostly at parameters of meat and carcass quality, however, they also examined growth characteristics, organ weights, and other parameters that gave insight into development and metabolism. They found that the major differences between groups were due to the differences in glucosinolates between the varieties of canola – the consequences of breeding, and not between the genetically engineered canola and its similar parent.

Ms. Love tried to minimize their conclusion about the equivalence of the genetically engineered canola, avoided mentioning the measurement of non-meat-quality parameters, and yet somehow, twisted the study to the creative conclusion that any canola from Monsanto is bad to eat because the variety was different from the comparison varieties. This is also trying to have it both ways – calling it both a study irrelevant to health and trying to reach conclusions about the healthfulness of different varieties of canola based on it. Ms. Love claimed that the well-described issue with glucosinolates somehow “raise[s] doubt about the quality and relevance of the findings,” which it does not.

In other posts about studies in the list, she reaches similar dismissive conclusions apparently by only reading the abstract and not reviewing the full studies at all.

We are actually reading them all.

Her dismissal of this study as being in any way relevant for human health can be shown to be doctrinaire with a simple thought question. If the study had, in reverse, found that the pigs had enlarged or deformed organs, lower muscle mass, and altered metabolism as a result of eating GE canola, would Ms. Love still consider it to have no implications for human health? We would all agree in that case that it would be cause for concern, including her. So therefore in absence of such harms, it is reasonable to conclude that this study provides some assurance, albeit not very detailed, that GE canola would be unlikely to have a negative (or positive) impact compared to conventional canola.

Because of the duplication, Ms. Love gleefully announced that we had only 599 studies, not 600. As we go through the Atlas building process systematically, we identify studies that are missing as well as those that are duplicated or otherwise incorrect. In the process of reviewing Caine et al. (2007), we removed one abstract from our spreadsheet, however, we identified 4 additional studies that were not included. Our math disagrees with hers because we’re trying to build a resource, while she is trying to make people ignore it. We don’t just have 603 studies, either. There’s a lot more science to catalog than that.

Like Counting Stars in the Sky

Milky Way Galaxy appears over Ontario Credit: Kerry-Ann Lecky Hepburn

I said above that I would demonstrate why I think there could be well over 1,000 studies that will end up in GENERA. Here is the current status of how many studies we have as of today. We have downloaded 572 PDF files of peer-reviewed scientific studies, and entered them into our library. These are mostly the easily available ones, and our volunteers have identified 127 studies that are behind paywalls that academics with access will have to retrieve. There are another 51 studies newly added from references and searches that have not been examined yet for access, and 38 more that have been flagged for administrators to figure out where they are. If we just add up the ones that we know are peer-reviewed scientific studies, then we get 572 + 127 + 51 = 750.

This does not include the studies from other lists we are currently sifting through, the dozens being sent to us by other scientists, or new searches of journals, nor citations yet to be harvested from the reference sections of each PDF we currently have, and if only one out of three of these gave us a new one to include, we would exceed 1,000 studies easily. The rate at which we are adding new studies to our database suggests that this will be the case. It could always slow down as we get more duplicates, which is why I could not say it would definitely have 1,000 studies. That was just a projection that was spoken, but not written down. But earlier this month, something changed that will make this a more definite reality.

A review study was published in the peer-reviewed scientific journal, Critical Reviews in Biotechnology, which searched for and cataloged all the peer-reviewed research on genetically engineered crops in the last decade. The scope of Nicolia et al. is broader than GENERA, as it included research on coexistence and traceability in addition to research on the safety of consumption or off-target effects in the environment. Here are the category breakdowns and how many studies they found in each:

  • General: 167
  • Biodiversity: 580
  • Coexistence: 96
  • Wild Relatives: 111
  • Horizontal Gene Transfer in Soil: 59
  • Non-targeted assessment: 107
  • Equivalence: 46
  • Consumption: 313
  • Traceability: 305

If we were to add up just the general, biodiversity, non-targeted assessment, equivalence, and consumption studies together from this list alone, that would total 1,213. We will go through all of these to add relevant ones to our database, so maybe the number could go down, but it very likely could go up (keep in mind, this is only research from the past 10 years and we have studies dating back to 1994). 1,200+ would be a conservative estimate. Jon Entine at the Genetic Literacy Project nicely juxtaposes the sheer number of studies done on GE crops with statements from activists saying that there’s almost none at all. However, to say that there are 2,000 that confirm safety is optimistic. As you can see, about 400 of the studies in the Nicolia review are about coexistence and traceability, which isn’t a safety issue so much as a cultural and political one. But it would be entirely accurate to say that these ~2,000 studies all show together that genetic engineering is a highly scrutinized and well-studied technology.

Where we are right now

Frank needs an army of GENERA volunteers. There are perks!

The review process for outcomes and funding is going on right now, and a fair pace. We’re still just in the double-digits on this part, but as we also work on how we will display the Atlas on our site, we will prepare the current collection of studies in our database to begin some alpha and beta testing, with an expected public display of the contents of GENERA by the end of this year. The sheer number of studies that we are finding means that we have an ever-present need for additional volunteers, and are searching for more sources of funding to finish the analysis of the study outcomes, which takes the most time. Please feel free to give us suggestions, submit studies, or help us out by volunteering for the project. When we get this information online and searchable for everyone, it will be a fantastic resource, and we can get beyond talking about how many studies there are and more about what information those studies contain.

GENERA will continue to evolve as we learn more about research there is out there, and we will be challenged by studies that don’t fit into the neat categories we try to put them in. We’re going to find errors in our approach and refine it to address them and improve the database. We’re also going to encounter criticisms, both constructive and dismissive. Constructive comments will help us identify errors we can fix, and information we can add to the database. Efforts to dismiss GENERA will ultimately be seen as a result of a narrow view of the world, threatened by the vast amounts of knowledge that science constantly generates. Science marches on, and its our duty to help people find and understand the discoveries and conclusions made by scientists around the world.

Editor’s note: I incorrectly indicated that Madeleine Love was currently with MADGE Australia, and she has informed me that she is no longer a part of the organization, so I have updated the post to reflect that.

Follow Karl Haro von Mogel:
Karl earned his Ph.D. in Plant Breeding and Plant Genetics at UW-Madison, with a minor in Life Science Communication. His dissertation was on both the genetics of sweet corn and plant genetics outreach. He recently moved back to his home state of California. His favorite produce might just be squash.