Lmst

Simon Porter will be at #ICSSI this week showcasing a significant step forward in ORCID data accessibility! His poster, "Enhancing the Accessibility of ORCID Public Data via Google BigQuery," introduces a new way to access our public data file. Exploratory data analysis is easier than ever. This beta service, generously sponsored by Digital Science, opens up innovative use cases. If you're at #ICSSI, stop by Simon's poster or download it here:

https://orcid.filecamp.com/s/d/zXA9Up5rvHVvkNBs

I am at the International Conference on the Science of Science and Innovation in Copenhagen

Good schedule! https://icssi.org/schedule/

#ICSSI

I didn't manage to take slides of @mollymking's ICSSI talk as I was backlogged with slides from other lightning talks, but I liked her talk on 'mesearch' - research on yourself. Molly has done mesearch, so this was meta-mesearch. The finding that some demographics get a citation boost for certain kinds of mesearch, but others don't, was intriguing.

#ICSSI #ScienceofScience #Mesearch #DEIA

"Elite" peer review was studied by Daniel Larremore and colleagues using an anonymised dataset from AAAS of 112,000 papers submitted to Science and Science Advances. A few more desk rejects for women authors, but no difference in peer review. Clear preference for work from prestigious institutions: biased or just better? Big teams also do better.
#ICSSI #ScienceofScience #PeerReview #ScienceMagazine #ScienceAdvances #DanielLarremore #AAAS

Elite peer review is qualitatively different

Difficulty
📚 Extraordinarily high volume of submissions
💡 Complex, multidisciplinary, and groundbreaking work

Commitment
👱 Professional editors, with substantial discretion
🔍 Highly engaged reviewers

Stakes
☄️ Huge potential positive career impacts for authors
⚠️ Downside reputational risk for journals
🌎 Outcomes shape the allocation of attention across new discoveries and directions of scientific progress

Data: 6y of editorial records at Science and Science Advances

68,047 Submissions to Science

17%
11,845 Papers Sent to Review

6.1% Accepted

35% of papers sent to review
4,181 Papers Accepted

83%
56,202 Submissions Desk Rejected

65%
7,664 Papers Rejected After Review

Forthcoming Data Release:
• Journal and topic cluster for all papers
• Gender: authors, editors, advisors, reviewers
• Prestige: authors, reviewers
• Country/region: authors
• BORE advisor rating & confidence

Terms of the data use agreement:
• Review sentiment, length, trajectory
data will be anonymized and
publicly released
• Final outcome (desk reject, reject, accept)
• n=68K Science, n=44k Science Advances

Preview: associations in the Science data

Logistic Regressions

Separate models for (i) editor's desk and (ii) peer review. Covariates: topic cluster, # authors, CA prestige, CA gender, FA gender, DE gender; Reviewer gender, prestige, length, sentiment.

Gender
Papers with women CAs show increased odds of desk rejection (OR 1.075). Among reviewed papers, no significant gender associations.

Prestige
Papers from the most prestigious ~20% show big increase in going out to review (OR 3.00) and a moderate increase in acceptance given review (OR 1.23).*

Team Size
Compared to 1-5 author papers, those with 6-9 authors and 10+ authors are much more likely to go out to review (ORs: 1.65, 2.55), and be accepted after review (ORs 1.17, 1.61).

Corresponding Author. FA: First Author. DE: Deputy Editor, OR: odds ralio.

Ref group: unknown prestige. At editor's desk, 2nd quintile also sees a bump (OR 1.92) as does 3rd (OR 1.43), But no effects after review for these quintiles.

How do different values affect peer review and how do these influence editorial decisions, asks Daniel Scott Smith? Very different reviews can make decisions uncertain.

Across STEM, they find accuracy and novelty are the most important values in peer review reports, social factors (biases) have little effect, and dissensus increases editorial gatekeeping. It doesn't matter if reviewers disagree.

#ICSSI #ScienceofScience #PeerReview #Values #DecisionMaking #JournalEditing

Given a manuscript,
a reviewer's:
Rating =
Reviewer Value Judgments
Novelty Accuracy
Simplicity Consistency
Replicability
Thoroughness

+
Author-Reviewer Biases

FINDINGS

Engineering & life sciences are strongly aligned

More
influential:
Novelty
Accuracy
Consistency
Clarity
Replicability
Thoroughness

1 Great alignment in STEM
2 Accuracy & Novelty are key
3 Little to no bias
4 Uncertainty increases gatekeeping

Peer review in these venues appears to be
✅fair
✅consequential

Plurality and dissensus in human judgment strengthen gatekeeping

Breakaway and Zombie journals: what happens after mass editorial board resignations. @KyleSiler looks into what happens when a journal and its community separate.

Glossa vs Lingua & QSS vs Journal of Informetrics. Lingua suffered, JoI didn't - but both 'Zombies' published more Chinese studies whereas the European and North American researchers moved to the OA breakaways.

#ICSSI #ScienceofScience #JournalPublication #EditorialIndependence #OpenAccess

Kyle Siler at the National Academies wooden podium. He's a white man with light brown hair, wearing a light blue shirt and a grey checked tie.

Juan Mateos-Garcia of Google DeepMind wants equitable use of their tools like AlphaFold. They solved protein structure prediction in 2020 and this is helpful for drug discovery, but researchers in LMIC have barriers to adoption.

Used OpenAlex & PDB to study 20.6K papers citing AlphaFold. LMIC researchers are underrepresented, but study of diseases affecting LMICs is overrepresented - esp. work on natural products.

#ICSSI #ScienceofScience #AItools #DeepMind #AlphaFold #StructuralBiology

Equitable adoption

There is evidence of an adoption gap

LMICs and LMLICs seem underrepresented in AlphaFold vs. baseline

Also visible when we focus on PDB research

Equitable impact

AF disease research more focused on LMIC diseases 20% AF papers focus on diseases (vs. 34% in baseline)

Share of AF in diseases with big LMIC impact 36% higher than baseline

User interviews

We interviewed 10 LMIC AlphaFold users in India, Pakistan, South Africa, Tunisia, Ghana and Brazil.

They focus on diseases with big LMIC impacts including Malaria, infectious diseases, Antimicrobial Resistance.

AlphaFold impacts:
Lowers barriers to entry
Accelerates research
Attracts talent to structural biology

Barriers to adoption:
Accessibility (IT infrastructure and skills)
Lack of information about AF/its relevance
Weak research ecosystem

"So it reduces reliance on costly experiments too. Often in a laboratory setting, there's a lot of troubleshooting, a lot of experiments that fail. And using an online capability such as AlphaFold for this deep understanding would definitely be beneficial in reducing the reliance on costly experiments"

"Indian Universities are not teaching AlphaFold because it's a very new concept and they spend the whole curriculum on teaching the basics."

"I think one important factor is to make sure that people understand that AlphaFold can be like used like a database, which was not clear for a lot of people"

Corporate research is dominating AI science, says Nur Ahmed, but they're not engaging with responsible AI research. They're choosing speed over safety.

They used supervised machine learning on 6 million papers and found few firms were working on responsible AI. A handful attended responsible AI conferences vs hundreds at other AI conferences.

When they do engage, it's on explainability not societal impacts and human rights.

#ICSSI #ScienceofScience #AItools #CorporateScience #ResponsibleAI

Sai Koneru says rapid growth of the literature makes it hard to keep up with relevant work. Can LLMs discern the hypotheses reported in abstracts? No - currently it's tough for natural language models to do this.

#ICSSI #ScienceofScience #AItools #LLMs #LiteratureReviewTips

"Let's *delve* into this topic..."

Science of AI-mediated science - Shahan Ali Memon gave a very engaging talk on how AI is being used in science, how it changes how we do and think about science, and how these AI tools are different to those we've used before.

He was too modest to advise a representative of the UK government in the audience on how much of his metascience budget should pivot to AI.

#ICSSI #ScienceofScience #AItools

Data exploration
InsightPilot

Writing
jenni
Paperpal

Thematic analysis
atlas.ti

Peer reviewing
MARG

Data visualization and accessibility
Chat2VIS
FIGURA11Y

Making slides
BEAUTIFUL AI

Creating speaker notes
ChatGPT

Did Sean do "science?"

What is the difference between our traditional model of science vs. the new Al-mediated model of science?

What did Al actually change in Science?

How is Al different from other tools, technologies, and revolutions of the past? Is it different from statistics? Or computing? Or the web? Or the availability of big data? Or calculator?

Is this a paradigm shift?

Let's delve into this topic..

"Sean" is an AI-generated cartoon who looks like Shahan.

What is AI changing in science?

Our basic model of science, and the scientific process or the method hasn't quite changed.

What has changed → a possible revolution in the way individual components of the scientific process are executed

Writing: mechanics and structure → substance

Coding: logic building → prompt engineering + debugging + code review + verification

Takeaways/Questions

What is the difference between our traditional model of science vs. the new Al-mediated model of science?

What is Al changing in Science?

How is it different from other tools, technologies, and revolutions of the past? Is it different from statistics? Or computing? Or the web? Or the availability of big data?

Is this a paradigm shift?

Survey: tinyurl.com/SciencePlusAlSurvey Twitter/X: @SciencePlusAl

Slides: tinyurl.com/ICSSI-SciencePlusAl

samemon@uw.edu | @shahanmemon | samemon.github.io jevinw@uw.edu | @jevinwest | jevinwest.org

Elle O'Brien at UMich says there are barriers to data science in academia. Does generative AI break this down?

Many researchers are using LLMs for tips on using software even when good documentation exists. They're not software literate.

They wrongly think chatbots do web search, can code, have databases, can calculate, and give faithful explanations. Does that make their use more risky?

#ICSSI #DataScience #LLMs #AItools #CoPilot #ResearchCode #ScienceofScience #ChatGPT #InformationLiteracy

Elle O'Brien speaking at ICSSI. She's a white woman with brown hair in a bun, wearing a dark blue dress.

Do misconceptions about LLMs lead to risky code practices?

In what contexts are scientific code errors introduced by LLMs most and least problematic?

Will we need new strategies for verifying scientific code?

The real reason I attend conferences is the swag. Here's a box of sprinkle-covered Oreos I received as thanks for being a panel speaker at ICSSI (icssi.org).

#ConferenceSwag #ICSSI #SprinkleCookies

A rectangular white box with a ribbon bow, containing multicoloured sprinkle-covered Oreos. It's on a swirly gold and grey tablecloth, with a cup of coffee and a conference booklet next to it.

I really enjoyed speaking on the ICSSI conference panel (icssi.org) on junk, fraud, retractions, and paper mills; we got lots of great questions and could have kept going all morning! Thanks to Daniel Larremore for the invite and especially Daniel Acuna for organising and chairing.

🧵

#PaperMills #ICSSI #ScienceofScience #PublicationEthics #JournalPublication #ScientificFraud #AcademicPublishing #Retractions

Here's slides for my plenary talk "Imagining the space of knowledge" at #ICSSI

https://yongyeol.com/slides/20240701_ICSSI_yyahn.pdf

The choice of chariots of fire as the lightning session bell is genius 😂 #icssi

A study of "beauty bias" in scientific careers presented at #ICSSI made me uneasy. Scraping faces from Google Scholar, business school websites, and job applicants and using machine learning to rate attractiveness without their consent seems disrespectful, esp. as the names of the "beautiful" institutions (which correlates with school ranking) are released. A small business school might not appreciate being labelled as ugly! 🧵
#ScienceofScience #Attractiveness #CareerProgression #ScienceCareers

There's a class gap in career progression in US academia, find Anna Stansbury and Kyra Rodriguez of MIT. #ICSSI #ScienceofScience #ClassDisparities #ClassDivide

Concluding: The Class Gap in Career Progression

Conditional on PhD program attended first gen college grads are less likely to end up tenured at high-ranked research-intensive schools than Dieu PhD classmates

This "class gap in career progression".

• Exists between first-gen college grads and people with a parent with a non-PhD graduate degree (aka is not only driven by academia-specific advantages).

• Exists conditional on race/ethnicity, birth region, and gender, and is as big as analogous race and gender gaps.

• Emerges at the PhD to tenure-track job juncture, and widens at the point of getting tenure. It is not driven by differential selection out of academia

• Is associated with a class gap in earnings and job satisfaction.

Our mechanisms analysis suggests that the class gap:

• is driven in part by differences in research output, but large gaps remain even conditional on very detailed measures of research quantity and quality

• may relate to differences in networks

• seems unlikely to be explained by differential preferences or constraints

Here's an ICSSI talk close to my interests: @sjcporter of @digitalscience explains how to find networks of faking by authorship selling schemes.

They're sold on Telegram and Facebook etc. Shout out to @nickwizzo who keeps track of these adverts and the final published papers.

🧵

#Authorship #PublicationEthics #ScientificFraud #PaperMills #ICSSI #ScienceofScience

#ICSSI

Client Info