16.5 Design calculations after the data have been collected

• Return to the Beauty and sex ratio example (from 9.4-9.5).

• Study claimed that very attractive parents were 8% +- 3% more likely to have girls.

• Should we give this much credence?

  • Unusual things do happen, so this could be one of those 5% cases. Also the researchers had degrees of freedom to manipulate…

  • Previous studies suggest any such effect is probably (much) less then 0.5%. This study would only be able to detect an effect that was 12 times this! This suggests a type M error (or even type S).

• Lessons learned?

  • It is well known that with large samples, even a small estimate can be statistically significant (but not practically significant)

  • It should be also remember that only large effects will be significant with small samples. So any significant effect is likely to also strong! So with small samples, you must be wary, or at least except that you have only weak evidence of an effect that are suggestive and not definitive.