Lmst

🧵
> ... unethical behaviour during the report of results is.. P hacking... frequent in research.. [of a] clinical nature... two main reasons.. First, scientists are often evaluated by the number and quality of publications, and sometimes this pressure to get significant results makes some scientists cherry-pick their results. Second (and more frequent), some inexperienced analysts are unaware of the importance of #MultipleTesting and think this is
OK. But it is not! #PHacking
@bsmall2@fedibird.com

“Statisticism refers to an overemphasis on abstract statistical principles at the expense of context-specific nuance and caveats (e.g., Boring, 1919; Brower, 1949; Proulx & Morey, 2021). Statisticism may help to explain the unthinking statistical ritualism that has been noted by some commentators (Gigerenzer, 2004, 2018; Proulx & Morey, 2021).” #statistics #stats #MultipleTesting #Statisticism

RE: https://fediscience.org/@MarkRubin/111804619588671451

Multiple Testing:

New article discusses the “use and misuse of corrections for multiple testing.”

“In general, avoid corrections for multiple testing if statistical claims are to be made for each individual test...”

https://doi.org/10.1016/j.metip.2023.100120

#Stats
#Statistics
#MultipleTesting
#MultipleComparisons
#NHST

Current psychological research addresses multifaceted questions demanding multiple analyses of data. Statistical analyses regarded as instances of multiple testing are often subjected to alpha adjustments to guard against inflation of Type-I errors. A review of papers published in the last two years in two major psychology journals shows inconsistent and discretionary use of alpha adjustments in a broad diversity of statistical analyses that are formally identical across papers. Authoritative sources also do not clarify the circumstances in which alpha adjustments should or should not be used. This paper describes the workings of Bonferroni and false-discovery rate adjustments, showing that they only control the Type-I error rate for an (omnibus) hypothesis stating that all its individual (surrogate) nulls are true. For individual nulls, alpha adjustment only has the trivial consequences of the use of a lower alpha level, without reducing the occurrence of Type-I errors or Type-II errors below their expected rates. In practice, then, corrections for multiple testing only come down to testing individual hypotheses at a lower alpha level without preventing the rejection of true nulls and without favoring the rejection of false nulls. Thus, use of alpha adjustments is only justifiable for inferences about an omnibus null for which a one-shot statistical test does not exist and which must instead be tested piecewise via several surrogates ...

New paper provides a history of “voodoo science,” which discusses the controversy surrounding Vul et al.’s (2009) controversial article “Puzzlingly High Correlations in FMRI Studies of Emotion, Personality, and Social Cognition.”

Five quotes follow: 🧵👉

🔓 https://doi.org/10.3390/socsci12010015

#MetaScience
#Neuroscience
#Neuroimaging
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#Fmri
#VoodooCorrelations
#UseNovelty
#MultipleTesting

'Voodoo” Science in Neuroimaging: How a Controversy

Transformed into a Crisis

Abstract: Since the 1990s, functional magnetic resonance imaging (fMRI) techniques have continued to advance, which has led researchers and non specialists alike to regard this technique as infallible. However, at the end of 2008, a scientific controversy and the related media coverage called functional neuroimaging practices into question and cast doubt on the capacity of fMRI studies to produce reliable results. The purpose of this article is to retrace the history of this contemporary controversy and its treatment in the media. Then, the study stands at the intersection of the history of science, the epistemology of statistics, and the epistemology of science. Arguments involving actors (researchers, the media) and the chronology of events are presented. Finally, the article reveals that three groups fought through different arguments (false positives, statistical power, sample size, etc.), reaffirming the current scientific norms that separate the true from the false. Replication, forming this boundary, takes the place of the most persuasive argument. This is how the voodoo controversy joined the replication crisis.

Useful recommendations when to use corrections for #MultipleTesting and when not:

Rubin, M. (2021). When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing. Synthese, https://doi.org/10.1007/s11229-021-03276-4

From the abstract:
"It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis."

#MultipleTesting

Client Info