Brian S. Hooker, Ph.D.
June 23, 2017
Very early this year, a research group from the insurance giant Kaiser Permanente published a paper concluding no evidence of harm in administering prenatal influenza vaccines. The study authors asserted that there was no relationship between those who received the flu shot during pregnancy and later autism spectrum disorder (ASD) diagnosis in the child. However, that proclamation was not consistent with the study’s results. Specifically, women who received the vaccine during their first trimester of pregnancy showed a 20% greater risk of having the child later develop ASD. This was based on a sampling of 13,477 women who received the maternal flu shot in the first trimester, resulting in 260 ASD cases, versus 151,698 “control” women who received no flu shot during pregnancy, resulting in 2,338 ASD cases. This result was statistically significant with a p value of 0.01, which in this case means that the possibility that this is a “chance” finding and not a “true” association was just 1%. In other words, the chances of this being a “true” association are 99%.
In statistics, the gold standard “cut-off” to determine statistical significance is actually a higher p value of 0.05, meaning that the possibility of a chance association is less than 5%. Thus, the first trimester flu shot – ASD relationship should have been deemed statistically significant, with p=0.01, and accordingly a policy change should have been made to suspend use of that vaccine, at least in the first trimester of pregnancy.
However, the study authors instead reached into their statistical “bag of tricks” and trotted out what is termed the “Bonferroni” adjustment. This adjustment is applied in statistics only under very specific instances, when multiple, unrelated statistical evaluations are made using a single data sampling. In this adjustment, simply, the p value is adjusted by multiplying its original value with the number of “independent” evaluations completed in the study of that single data set (Bland et al. 1995 BMJ 310:170). In the case of Zerbo et al. 2017, there were 8 evaluations completed (4 evaluations regarding the flu shot and 4 evaluations regarding women who actually contracted the flu during pregnancy) and thus the original p value of 0.01 was adjusted to 0.08, above the “cut off” value used for deeming “statistical significance.” The Zerbo et al. authors rounded the result up to p=0.1, further moving the result away from the “magic” 0.05 cut-off level, causing the significant result to disappear.
There’s a huge problem here, however, which I pointed out in my letter to the editor of the journal (Hooker 2017 JAMA Pediatrics 171:600) published in their June 2, 2017 edition. The Bonferroni adjustment, among other corrections for multiple, independent comparisons, should not be applied to statistics when there is any interdependence within the different evaluations completed within the data sample. In this case, 4 of the evaluations completed dealt specifically with the timing of the maternal flu shot (first, second and third trimesters, as well as overall risk at any point in pregnancy) and subsequent ASD incidence. So, not only were these four trials all focused on an ASD outcome, but they all dealt with different phases of pregnancy, which were then summed to develop an “overall” risk at any phase of pregnancy. By definition, these trials were anything but statistically independent. An example of an independent evalution would be polling different groups of college students for their tastes in music, food, art, and perhaps health care practices, where none of the preferences could be empirically tied to the next.
My rebuttal letter was not the only objection raised regarding the Zerbo et al. (2017) statistical error. A group of researchers from the Health Protection Agency of Milan, Italy (Donzelli et al. 2017 JAMA Pediatrics 171:601) published a similar letter, which went even further and recommended that medical practitioners “apply the precautionary principle and refrain from vaccinating pregnant women or at least to avoid vaccination in the first trimester of pregnancy.”
In the interest of fairness, the Zerbo et al. (2017) study authors had the opportunity to reply to both rebuttal letters and stated in their reply, “We agree with Donzelli et al and Hooker that the 3 trimesters are not independent of the entire pregnancy period. However, a less-conservative adjustment for multiple testing, accounting for the dependence of the entire pregnancy on the trimesters, would still yield a P value of .07 or higher, which should not change interpretations of our findings.” This response is wrong on many levels.
First, the point of the discussion regarding multiple, dependent comparisons is that NO correction factors should be applied to said p values. The Zerbo authors’ approach of using a less aggressive correction factor (the Family Pairwise Error Rate correction method) was wrong-headed at best. The literature is clear, even in the citations that the Zerbo group provided as justification in their reply (Greenland et al. 2008 Int J Epidemiol 37:430, Rothman 1990 Epidemiol 1:43, Stacey et al. 2012 Invest Ophthalmol Vis Sci 53:5955), the p values should stand as originally calculated, regardless of whether you like or dislike the outcome. The scientific group responsible for this paper, which, as I stated previously, is housed in insurance giant Kaiser Permanente’s research division, obviously needs to deal with bias towards vaccination at all costs, including the fact that since 2004, they have recommended that their pregnant patients receive the flu shot in any trimester of pregnancy. They might be hesitant to admit any adverse relationship with ASD; if they did, they would be liable for damages caused to their patients and their children. In addition, one of the coauthors of the study had financial ties to vaccine manufacturers Sanofi Pasteur, GlaxoSmithKline and Merck, among several others. Shear optics would dictate that these researchers would err on the side of caution and avoid the additional risk of ASD that first trimester flu shots afford. They did not.
In addition, there are similar indications in open scientific literature of ASD risk in pregnancy due to flu shot exposure, specifically in a study completed by the CDC. In their paper Price et al. 2010, published in the journal Pediatrics, the increased risk of regressive ASD due to prenatal thimerosal exposure is 86% as opposed to those women not exposed to thimerosal at all during pregnancy. Since thimerosal, the mercury-containing vaccine preservative, is still in flu shots given to pregnant women, this result ties directly to the Zerbo et al. 2017 study. Like the Zerbo group, the CDC dismissed this finding because the p value was 0.073, just slightly above the cut-off for significance at 0.05. However, the background report which served as the basis of the CDC’s Price et al. 2010 paper showed that more representative versions of the statistical model used to obtain this result were actually highly significant with p values as low as 0.009.
Where the rubber meets the road: When it’s your own child who you are potentially putting at risk for ASD, do you really want to split hairs with p values? Whether something is “marginally statistically significant” or “highly statistically significant”, there is still a risk.
As the father of a wonderful and special ASD young man, I would not want to wish this on any other individual. ASD frustrates my son to no end, given that he cannot do things (like speak and communicate) that he desperately wants to do. It is high time that these researchers put the shoe on the other foot and get their heads out of the numbers. Their actions have consequences, and putting children at a highly established, very real risk of ASD should never be taken lightly.
Stay Informed. . . Stay Healthy!
Join the Conversation
your thoughts matter
Brian S. Hooker, Ph.D., P.E.
Science Adviser, Focus For Health
Brian has been a member of the Focus for Health Team since 2012 and has more recently joined the Board. He is an Assistant Professor of Biology at Simpson University in Redding, California, where he specializes in chemistry and biology coursework. Additionally, Hooker is the Senior Process Consultant at ARES Corporation, working closely on process design for the environment restoration industry. His design efforts focus on industrial biotechnology and chemical engineering principles. He has a teenage son with autism and has been active in the autism community since 2004.