Why we fall for bogus academic research
Even when you have a larger sample, however, the groups are not going to match the average of the whole population every time; by blind luck, sometimes the group will be exceptionally tall, sometimes exceptionally short. Statisticians understand this. But journal editors and journalists do not necessarily exercise appropriate caution. That’s not because journal editors are dumb and don’t get statistics, but because scientific journals are looking for novel and interesting results, not “We did a study and look, we found exactly what you’d have expected before you’d plowed through our four pages of analysis.” This “publication bias” means that journals are basically selecting for outliers. In other words, they are in the business of publishing papers that, for no failure of method but simply from sheer dumb luck, happened to get an unusual sample. They are going to select for those papers more than they should – especially in fields that study humans, who are expensive and reluctant to sit still for your experiment, rather than something like bacteria, which can be studied in numbers ending in lots of zeroes.