#statstab #456 Bestiary of Questionable Research Practices in Psychology

Happy new year! ✨

Thoughts: Starting the year with a resource to make you more cynical in 2026.

#QRPs #phacking #metascience #research #ethics

https://journals.sagepub.com/doi/10.1177/25152459251348431

#statstab #363 p-checker The one-for-all p-value analyzer

Thoughts: Easy way to check for publication bias using some current tools.

#shiny #pvalue #phacking #QRPs #zcurve #bias #pcurve #rindex

https://shinyapps.org/apps/p-checker/

#statstab #358 What are some of the problems with stepwise regression?

Thoughts: Model selection is not an easy task, but maybe don't naively try step wise reg.

#stepwise #regression #QRPs #issues #phacking #modelselection #bias

https://www.stata.com/support/faqs/statistics/stepwise-regression-problems/

Il P-hacking è la pratica di modificare l'analisi o i dati per ottenere un risultato statisticamente significativo.
Si cerca di ottenere un risultato desiderabile e si riportano solo i risultati ottenuti, ignorando tutte le volte in cui non si è ottenuto nulla.
Potrebbe portare a una pubblicazione a breve termine, ma il P-hacking contribuisce alla crisi di riproducibilità e replicabilità nella scienza, riempiendo la letteratura scientifica di conclusioni dubbie o infondate.
#science #statistic #phacking
https://www.nature.com/articles/d41586-025-01246-1

#phacking #LLM performance:

"The researchers found that big players like Meta, Google, OpenAI, and Amazon are given special privileges to privately test multiple versions of their models and only publish the best results. This hidden practice allows them to inflate their rankings by cherry picking data, making their models appear stronger than they actually are."

https://arxiv.org/abs/2504.20879?utm_source=beehiiv&utm_medium=newsletter&utm_campaign=mediamobilize&_bhlid=045dede5ad6eedcb96ec953f2fbac11a166a2243

via Sabine Hossenfelder's newsletter

#statstab #330 Encourage Playing with Data and Discourage Questionable Reporting Practices

Thoughts: What are and aren't "Questionable Research Practices"? Where is the "grey area"? Interesting opinion piece.

#QRPs #exploratory #EDA #posthoc #phacking

https://link.springer.com/article/10.1007/s11336-015-9445-1

> 2011: Joseph Simmons, Leif Nelson, and Uri Simonsohn publish a paper, “False-positive psychology,” in Psychological Science introducing the useful term “researcher degrees of freedom.” Later they come up with the term p-hacking, and Eric Loken and I speak of the garden of forking paths to describe the processes by which researcher degrees of freedom are employed to attain statistical significance.
https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/
#PHacking #DegreesOfFreedom
@bsmall2@mstdn.jp

🧵
> ... unethical behaviour during the report of results is.. P hacking... frequent in research.. [of a] clinical nature... two main reasons.. First, scientists are often evaluated by the number and quality of publications, and sometimes this pressure to get sig­nificant results makes some scientists cherry-pick their results. Second (and more frequent), some inexperienced analysts are unaware of the importance of #MultipleTesting and think this is
OK. But it is not! #PHacking
@bsmall2@fedibird.com

#statstab #168 Large P-Values Cannot be Explained by Power Analysis

Thoughts: "Researchers cannot “aim” for p = .05, not even with a careful, perfectly accurate, power analysis."

#research #nhst #pvalues #power #QRPs #phacking

https://quentinandre.net/post/large-pvalues-and-power-analysis/

#statstab #135 {p-checker} The one-for-all p-value analyzer

Thoughts: Why not try your hand at some tools for detecting publication bias (mileage may vary). Useful teaching demo tools.

#shinyapp #r #rstats #phacking #pvalues #NHST #education #edu

https://shinyapps.org/apps/p-checker/

Fun class class survey of other undergrads, current N=55. I'm doing #irresponsible #DataAnalysis bc not actual #research.

Still #WTF?

ghost_recv_log = How many times have you been ghosted? (log-transformed)

mosi = Misperception of others' sexual interest

bjw = Belief in a just world

swls = Satisfaction with life

csei = College self-efficacy

High self-esteem assholery? IDK.

OH WAIT. Gender!

Shit. We didn't ask.

#NotResearch #Datafishing #phacking but it's #OK I'm a #professional #oops

Table 1 of the "False-positive psychology" paper (Simmons, Nelson & Simonsohn 2011, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2916240) estimate the false-positive rate of some questionable #ResearchPractices, both alone and in combination (see attached figure).

I remember reading somewhere that the authors later stated that the numbers were inaccurate and should have been somewhat higher. Does anyone have a reference for this? #FalsePositivePsychology #QRP #phacking #HARKing #ReproducibiliTea

[en] Cheating in Science: Harvard "Honesty Scholar" May Have Been Caught in Dishonesty

"... dishonesty can lead to creativity" - an interesting and somewhat amusing read.

The New York Times: "Questions about a widely cited paper are the latest to be raised about methods used in #behavioral research."

https://datacolada.org/111

#ResearchHighlights #honesty #dishonesty #phacking #harking #dredging #gino #harvard #fraud #cheating #academic #datacolada

“It’s a bit like seeing a rabbit shape in the clouds and then testing whether all clouds look like rabbits… using the same cloud. I hope you appreciate that you’re going to need some new clouds to test your theory.

Any datapoint you use to inspire a theory or question can’t be used to test that same theory.”

#phacking #statistics #rstats

https://towardsdatascience.com/the-most-powerful-idea-in-data-science-78b9cd451e72

@sociology @communicationscholars @rstats

"Do not chase better metrics. Chase better models."

Excellent lighting talk on how spatial sampling affects the quality of data driven models by Luísa Lucchese

https://youtu.be/q9dy2dQuTak

Chasing better #metrics is like #phacking

#GeoAI #gischat #datascience #machinelearning

If you’re interested in #statistics, #Metascience, #openScience, #pHacking, etc., this is quite a good read.

https://joebakcoleman.com/blog/2023/pcurve/

In 2012, Noam Chomsky mentioned a Lisp-language guy, #PatWinston. Winston said that, in the AI^1 field, people were directed away from "original questions." That sent me to #Chomsky's #PowersAndProspects(1996). _Prospects_ reminded me of Alex Carey's _Taking the Risk Out of Democracy_ and I now see #Mayo's #HawthorneStudies as 1927-1932 #PHacking. #GeorgeEltonMayo! in textbooks.
^1 AI, short for SALAMI: Systematic Approaches to Learning Algorithms and Machine Inferences #AISalami

> Psychologists who specialize in exercise music have quantified.. [that] listening to songs with high tempos motivates us to run faster, and the swifter we move, the quicker we prefer our music. Likewise, when drivers hear loud, fast music, they unconsciously step a bit harder on the gas pedal. Walking at our own pace creates an unadulterated feedback loop between the rhythm of our bodies and our mental state that we cannot experience as easily.. during any other.. locomotion.
#PHacking?

> OUTRIGHT FAKERY IS CLEARLY more common in.. sciences than we’d like to believe. But it may not be the biggest threat to their credibility... #MichaelKinsley once said of wrongdoing in Washington, so too in the lab: “The scandal is what’s legal.” The kind of manipulation that went into the “When I’m Sixty-Four” paper, for instance, is “nearly universally common,” #Simonsohn says. It is called “p-hacking,” or, more colorfully, “torturing the data until it confesses.”
#PHacking #OnBullshit?

No more #PHacking or vague #HealthClaims for products to obscure the fact that they lack solid evidence from #RCTs for such claims. So says the FTC. We say that's definitely a good call.
https://conscienhealth.org/2023/01/ftc-asks-for-rcts-not-vague-health-claims/