It is not surprising that you are skeptical of some scientific results as there have been some high profile examples of fraudulent research in the recent past. One particular example that has caused much public debate was a study published in The Lancet that linked the measles, mumps, and rubella vaccination with autism. After the study was published, major flaws were found in the study as was a conflict of interest with the study researcher. The journal retracted the study, but to this day, some still defend it. This leaves one asking what studies to trust and the simple answer is that there this is not a simple thing to evaluate.
Testing done by Industry has little room for manipulation because most of the testing strategy and protocols are based on regulations and guidelines developed by international agencies like the OECD. These protocols were developed by international experts who evaluated the types of data that were needed to establish the safety of GM crops. In my scientific career I’ve been an ad hoc reviewer for several journals, have been on the editorial board as a reviewer, and an editor. Because I am usually looking at studies with the scientific method in mind, there are some criteria that I typically use to judge the quality of a published study:
- Does the study have a testable hypothesis and does the study design adequately test that hypothesis?
- In an animal study, was the selection of type of test animal proper? For instance, if one wanted to measure the impact of a treatment that is designed to alter cellulose digestion then a chicken would not be an appropriate animal – they don’t have cellulose digesting enzymes.
- Is there a control? Is the control an appropriate control for the question being asked?
- Were there enough experimental units? This is especially important in an animal study because regardless of your position on use of animals in studies we should all agree that using too few animals leads to an inadequate test and this is a wasteful use of animals. There is a way to calculate the appropriate number or, for some science disciplines, there are standard protocols for defining adequate numbers.
- Where is it published? At a minimum it should be peer-reviewed, but I want to know more. Is the impact factor of the journal high and is the editorial board qualified to review the subject matter? Impact factor is a measure of citations and is not a perfect method of assessing quality of the peer review but many have come to the conclusion there is nothing better (http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001675 ). Likewise the editorial board’s qualifications are not informative of quality of the peer review, but the reviewers and editors should have some proven knowledge of the subject matter. There’s no doubt that peer review is not infallible – if you think so, read this from Science Magazine (http://www.sciencemag.org/content/342/6154/60.full ).
- Is it repeatable? This is one of the basic rules of research. Consider that there are hundreds of studies across numerous disciplines (molecular biology, toxicology, nutrition, agronomy, etc) that conclude efficacy and safety of GM crops. Recently there was a paper published on GM crops containing >1700 references (http://informahealthcare.com/doi/abs/10.3109/07388551.2013.823595?prevSearch=allfield%253A%2528nicolia%2529&searchHistoryKey= ) and the Federation of Animal Sciences Society maintains a database of more than 400 studies in which livestock/poultry were fed GM-derived feeds.
- Are the results sensational? Completely unexpected results are not a common occurrence. Sensational results that go against the opinions of most credible studies and credible science organizations can occur, but they are very rare. Brown (2012; EMBO Reports, 13: 964) put it well, “Fluke science discoveries and surprising results are often considered newsworthy, even if they end up being false positives. The general media, including blogs and newspapers, will of course focus on what is curious, funny, controversial, and so on”.
- Does the statistics analysis use a proper experimental unit (what the treatment was applied to; i.e., individual animals or pens of animals) and does it account for the way experimental units are randomized to the study?
- To your point about resources, feeding studies can be expensive, but it is a shame when one uncovers a finding with animals that more detailed information could have explained/validated gross findings (i.e., weight gain change with no feed consumption results or gross necropsy findings without histopathology). In summary, here is my experience: Science studies that don’t conform to the scientific method will eventually be rejected by the scientific community. Science studies that do conform to the scientific method will be repeatable by other scientist and confirmed to be valid.