Significance versus explanation

Next time I have to teach the difference between explanation and significance, this will be my go-to example: “Scientific aptitude better explains poor responses to teaching of evolution than psychological conflicts” by Mead et al. in Nature Ecology & Evolution.

The article has several figures that look like this:

The P-value looks amazing (9 x 10^-15), but the spread is equally large. I would not even want to guess what the R-square of that relationship is. The authors in the figure legend also specify that:
“The regression line is the best-fit line of y predicted by x. However, as assumptions of linear regression are not fully met it is provided for illustrative purposes alone to indicate the trend.”
Good point, because without that line it would be impossible to tell what the relationship would be, which is always a dire sign. But those two sentences should be part of a master class in scientific writing: Necessary, to the point, anticipating a reader’s needs and confusion, packaged in a short and succinct statement.

This study works perfectly well in a hypothetico-deductive framework with well-laid out hypotheses, logical predictions, and strong and convincing statistical tests. It also leaves the reader (or some of the readers, for instance me) with this “but is it really important?” question. I could not find any mention of this variability issue, and only lots of highly significant p-values.  What is actually the best explanation of these students’ evolutionary understanding?

Karl Cottenie
Associate Professor in Community Ecology

I am a community ecologist with a broad interest in data analysis.