"The researchers found that big players like Meta, Google, OpenAI, and Amazon are given special privileges to privately test multiple versions of their models and only publish the best results. This hidden practice allows them to inflate their rankings by cherry picking data, making their models appear stronger than they actually are."
via Sabine Hossenfelder's newsletter
> 2011: Joseph Simmons, Leif Nelson, and Uri Simonsohn publish a paper, “False-positive psychology,” in Psychological Science introducing the useful term “researcher degrees of freedom.” Later they come up with the term p-hacking, and Eric Loken and I speak of the garden of forking paths to describe the processes by which researcher degrees of freedom are employed to attain statistical significance.
https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/
#PHacking #DegreesOfFreedom
@bsmall2@mstdn.jp
🧵
> ... unethical behaviour during the report of results is.. P hacking... frequent in research.. [of a] clinical nature... two main reasons.. First, scientists are often evaluated by the number and quality of publications, and sometimes this pressure to get significant results makes some scientists cherry-pick their results. Second (and more frequent), some inexperienced analysts are unaware of the importance of #MultipleTesting and think this is
OK. But it is not! #PHacking
@bsmall2@fedibird.com
#statstab #168 Large P-Values Cannot be Explained by Power Analysis
Thoughts: "Researchers cannot “aim” for p = .05, not even with a careful, perfectly accurate, power analysis."
#research #nhst #pvalues #power #QRPs #phacking
https://quentinandre.net/post/large-pvalues-and-power-analysis/
Fun class class survey of other undergrads, current N=55. I'm doing #irresponsible #DataAnalysis bc not actual #research.
Still #WTF?
ghost_recv_log = How many times have you been ghosted? (log-transformed)
mosi = Misperception of others' sexual interest
bjw = Belief in a just world
swls = Satisfaction with life
csei = College self-efficacy
High self-esteem assholery? IDK.
OH WAIT. Gender!
Shit. We didn't ask.
#NotResearch #Datafishing #phacking but it's #OK I'm a #professional #oops
@ALLEA No time to read the full framework, but is there any mention of enforcing best scientific practices and ensuring scientific integrity?
#scientists #replicability #scientificmisconduct #reproducibilityofresearch #reproducibilitea #publishorperish #phacking
Table 1 of the "False-positive psychology" paper (Simmons, Nelson & Simonsohn 2011, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2916240) estimate the false-positive rate of some questionable #ResearchPractices, both alone and in combination (see attached figure).
I remember reading somewhere that the authors later stated that the numbers were inaccurate and should have been somewhat higher. Does anyone have a reference for this? #FalsePositivePsychology #QRP #phacking #HARKing #ReproducibiliTea
A nice hatchet job from the New Yorker!
https://www.newyorker.com/magazine/2023/10/09/they-studied-dishonesty-was-their-work-a-lie
The story of morally and intellectually bankrupt superstar researchers who elevated what is/was/has always been a somewhat semi-decent, hardly scientific enterprise called behavioral economics (within social psychology) into a juggernaut of "just-so stories" that swayed nations, industries and academias to form absolutely horrendous worldviews and make decisions likewise.
The field: The poor need to be nudged (read, extra inconvenience) to make rational decisions with marginally any value to them. Meanwhile, dear corporations, here are some tax write-offs and subsidies; you can also engage in open thievery. That's our way to nudge you to do the right thing, i.e., fudging with the accounts. This is how we social engineer (All this, based on absolutely moronic and morally dubious "studies". I am reminded of the evo-psych psychopaths).
As if economics as it stands isn't dismal and unscientific enough, whether you legitimize with a couple Nobels or not, this is like "polishing a turd".
Of course not much is likely to change in academia across fields as well. It will continue to seek and hire more such people who will bring the money, fame, and storytelling at the expense of hard work and reality. Knowledge building and intellectual work... what's that?
#Dishonesty #Academia #BehavioralEconomics #Management #BusinessSchools #Economics #Nudge #pHacking #SocialPsychology
[en] Cheating in Science: Harvard "Honesty Scholar" May Have Been Caught in Dishonesty
"... dishonesty can lead to creativity" - an interesting and somewhat amusing read.
The New York Times: "Questions about a widely cited paper are the latest to be raised about methods used in #behavioral research."
#ResearchHighlights #honesty #dishonesty #phacking #harking #dredging #gino #harvard #fraud #cheating #academic #datacolada
“It’s a bit like seeing a rabbit shape in the clouds and then testing whether all clouds look like rabbits… using the same cloud. I hope you appreciate that you’re going to need some new clouds to test your theory.
Any datapoint you use to inspire a theory or question can’t be used to test that same theory.”
https://towardsdatascience.com/the-most-powerful-idea-in-data-science-78b9cd451e72
If you’re interested in #statistics, #Metascience, #openScience, #pHacking, etc., this is quite a good read.
In 2012, Noam Chomsky mentioned a Lisp-language guy, #PatWinston. Winston said that, in the AI^1 field, people were directed away from "original questions." That sent me to #Chomsky's #PowersAndProspects(1996). _Prospects_ reminded me of Alex Carey's _Taking the Risk Out of Democracy_ and I now see #Mayo's #HawthorneStudies as 1927-1932 #PHacking. #GeorgeEltonMayo! in textbooks.
^1 AI, short for SALAMI: Systematic Approaches to Learning Algorithms and Machine Inferences #AISalami
> Psychologists who specialize in exercise music have quantified.. [that] listening to songs with high tempos motivates us to run faster, and the swifter we move, the quicker we prefer our music. Likewise, when drivers hear loud, fast music, they unconsciously step a bit harder on the gas pedal. Walking at our own pace creates an unadulterated feedback loop between the rhythm of our bodies and our mental state that we cannot experience as easily.. during any other.. locomotion.
#PHacking?
> OUTRIGHT FAKERY IS CLEARLY more common in.. sciences than we’d like to believe. But it may not be the biggest threat to their credibility... #MichaelKinsley once said of wrongdoing in Washington, so too in the lab: “The scandal is what’s legal.” The kind of manipulation that went into the “When I’m Sixty-Four” paper, for instance, is “nearly universally common,” #Simonsohn says. It is called “p-hacking,” or, more colorfully, “torturing the data until it confesses.”
#PHacking #OnBullshit?
No more #PHacking or vague #HealthClaims for products to obscure the fact that they lack solid evidence from #RCTs for such claims. So says the FTC. We say that's definitely a good call.
https://conscienhealth.org/2023/01/ftc-asks-for-rcts-not-vague-health-claims/
