If you’re unsure about something, there is probably a non-profit organization (aka special interest group, aka non-governmental organization) just waiting to tell you what the “facts” are. These organizations all claim to present the most accurate information in their non-peer-reviewed reports, but can we trust them?
Let’s take Research Shines Light on Gulf of Mexico Hypoxic Zone (full paper) as an example. This report by the National Corn Growers association was, to be blunt, biased to the point of falsehood. I explain how in Rotten Corn. The organization has an agenda to put corn farming practices in the best possible light, which means every report we see from them will have some degree of spin. We should expect some degree of spin from any of these groups, but sometimes they overstep the line.
Spin can be frustrating, particularly when we have specific evidence that contradicts what the special interest group said. What happens when the bias isn’t as obvious as in the NCG’s hypoxia report? Sometimes these reports seem 100% legitimate, especially when we agree with the agenda of the group, and especially when we don’t have the prerequisite knowledge to judge them. Even worse, there are many situations where two groups will put out directly opposing reports. Each group claims to have the “real” information, sometimes even calling out opposing reports.
The most recent example of this is the Organic Center’s Impacts of Genetically Engineered Crops on Pesticide Use: The First Thirteen Years. It directly contradicts the year old GM crops: global socio-economic and environmental impacts 1996- 2006 (pdf) by PG Economics. I covered some of the specific differences in Does using GMOs really increase pesticide use? a few days ago. In researching for the post, I made the decision to include the PG Economics report as an opposing viewpoint because the sources of the data are solid and the conclusions they make in the paper are well supported by peer-reviewed research. The Organic Center’s report leaves out a lot of data that is readily available, and doesn’t explain why – which is enough to make me question the conclusions in the report (along with glaring problems like lumping all biotech traits as “GMOs” with only a passing mention of how Bt and glyphosate resistant crops are different).
I mention these two opposing reports on GMOs and pesticide use to show that it is possible to evaluate “spun” reports when we consider them with a critical eye and a reasonable familiarity with peer-reviewed research on the subject. Why peer-reviewed? To paraphrase Winston Churchill, “Peer review is the worst form of quality control for scientific research except for all those others that have been tried.” Mistakes do, famously, get through, but don’t matter as they are either ignored (not cited by other scientists) or directly contradicted by new research.
These reports, scientifically sound or not, bypass the peer-review process. They aren’t screened by other scientists before they are published, and sometimes they are written by people who aren’t even in the field they are writing about. They can be good sources of information, but only if we take it to the next level and seek out the peer-reviewed research behind the reports as well as opposing viewpoints to help us get the big picture.
When I say peer-reviewed research, I’m not just talking about one paper. Instead, I mean multiple papers, preferably by different authors from different institutions, and different funding agencies. The papers should use different data sets and different experimental designs that ask similar questions.
Imagine that the entire body of peer-reviewed research for a subject area is a deck of cards that we’ve placed on the table, 52 card pickup style. Each card is a paper that is related to some of the other papers. Some papers cover very similar areas, totally overlapping. Others are only slightly related, with just a tip overlapping. Any one of those cards won’t tell us that much about what’s really happening, but when we look at the whole pile, particularly the overlapping areas, we can start to understand what’s really happening.
For more on the benefits and downfalls of peer review, see Nature’s peer review debate (accessible without login!).