Lmst
Login

#MultipleComparisons

Dr Mircea Zloteanu ☀️ 🌊🌴mzloteanu
2025-07-01

#377 To adjust, or not to adjust, for multiple comparisons

Thoughts: Not all-encompassing, but it does cover some relevant notions about multiplicity adjustments.

jclinepi.com/article/S0895-435

Mark RubinMarkRubin@fediscience.org
2023-03-23

Multiple Testing:

New article discusses the “use and misuse of corrections for multiple testing.”

“In general, avoid corrections for multiple testing if statistical claims are to be made for each individual test...”

doi.org/10.1016/j.metip.2023.1

#Stats
#Statistics
#MultipleTesting
#MultipleComparisons
#NHST

Current psychological research addresses multifaceted questions demanding multiple analyses of data. Statistical analyses regarded as instances of multiple testing are often subjected to alpha adjustments to guard against inflation of Type-I errors. A review of papers published in the last two years in two major psychology journals shows inconsistent and discretionary use of alpha adjustments in a broad diversity of statistical analyses that are formally identical across papers. Authoritative sources also do not clarify the circumstances in which alpha adjustments should or should not be used. This paper describes the workings of Bonferroni and false-discovery rate adjustments, showing that they only control the Type-I error rate for an (omnibus) hypothesis stating that all its individual (surrogate) nulls are true. For individual nulls, alpha adjustment only has the trivial consequences of the use of a lower alpha level, without reducing the occurrence of Type-I errors or Type-II errors below their expected rates. In practice, then, corrections for multiple testing only come down to testing individual hypotheses at a lower alpha level without preventing the rejection of true nulls and without favoring the rejection of false nulls. Thus, use of alpha adjustments is only justifiable for inferences about an omnibus null for which a one-shot statistical test does not exist and which must instead be tested piecewise via several surrogates ...

Client Info

Server: https://mastodon.social
Version: 2025.04
Repository: https://github.com/cyevgeniy/lmst