Lecture 3 - The reform movement in psychological science

The Social Brain: Critical Perspectives on Science, Society and Neurodiversity

Richard Ramsey

Today


Part 1

  • The reform movement in psychological science


Part 2

  • Read articles and discuss



Recap last week

Where are we now?

(Munafò et al., 2017)

Is psychological science in a crisis?


Crisis implies we are at a loss for solutions, when in fact we have identified many ways to improve science’s credibility

Simine Vazire (2018)

What is the reform movement?

What is the reform movement?

  • At a broad level, it is a response to the replication crisis, which involves a group of academics and organisations that aim to improve the quality of psychological science. There are too many to mention, so I’ll just outline one to give you the general idea.

  • Society for the Improvement of Psychological Science (SIPS) https://improvingpsych.org/

SIPS

Redemption

The method of science, as stodgy and grumpy as it may seem, is far more important than the findings of science.

Carl Sagan, 1995

(Chambers, 2017)

Redemption

Solving the Sins of…

  • Bias and Hidden Flexibility
  • Unreliability
  • Data Hoarding
  • Corruptibility
  • Internment
  • Bean Counting

Solving the Sins of Bias and Hidden Flexibility

  • Biases:

    • confirmation bias
    • publication bias
    • hindsight bias

  • Hidden flexibility: p-hacking until something turns up

  • Solution: pre-registration and Registered Reports.

Solving Bias and Hidden Flexibility

Normal publishing cycle:

  • Do all the work first (plan, data collection, analysis, write-up) and THEN submit to a journal

Open Science Framework

Solving Bias and Hidden Flexibility

Pre-registration is different:

  • You first write down and publish in a time-stamped manner EXACTLY what you are going to do. Then you do it.

Open Science Framework

Solving Bias and Hidden Flexibility


Solving Bias and Hidden Flexibility

Pre-registration

Pre-print

Solving Bias and Hidden Flexibility

Registered Reports are even better:

Open Science Framework

Solving Bias and Hidden Flexibility

Stage 1 Review:

  • The significance of the research questions
  • The logic, rationale, plausibility of the proposed hypotheses.
  • The soundness and feasibility of the methodology and analaysis pipeline (including statistical power analysis).
  • Whether clarity and degree of methodological detail would be sufficient to replicate exactly the proposed experimental procedures and analysis pipeline.
  • Whether the authors provide a sufficiently clear and detailed description of the methods to prevent undisclosed flexibility in the experimental procedures or analysis pipeline.
  • Whether the authors have considered sufficient outcome-neutral conditions (e.g., absence of floor or ceiling effects; positive controls) for ensuring that the results are able to test the stated hypotheses.

Solving Bias and Hidden Flexibility

Stage 2 Review:

  • Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor or ceiling effects).
  • Whether the introduction, rationale and stated hypotheses are the same as the approved Stage 1 submission (required).
  • Whether the authors adhered precisely to the registered experimental procedures
  • Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative.
  • Whether the authors’ conclusions are justified given the data

Solving the Sin of Unreliability

Unreliability: lack of replication, lack of power, vague and nonreplicable methods, misuse of statistics, failure to retract papers.

Solutions:

  • Registered Reports
  • Reproducibility Index
  • Pottery Barn rule by journals and funders
  • Regular multi-centre replication initiatives

Solving the Sin of Data Hoarding

Data hoarding: a widespread lack of data sharing

Solutions:

  • Peer Reviewers’ Openness (PRO) initiative (https://www.opennessinitiative.org/)
  • Transparency and Openness Promotion (TOP) guidelines (https://www.cos.io/initiatives/top-guidelines)
  • Centre for Open Science is a very useful platform (https://youtu.be/9YuNGB3vNOw)
  • Git and GitHub (and other version control systems and code repositories)

TOP guidelines

Solving the Sin of Data Hoarding

Open Science Framework

Solving the Sin of Data Hoarding

Open Science Framework

Solving the Sin of Data Hoarding

Git and GitHub

Solving the Sin of Corruptibility

Fraud

Solutions:

  • Required data and code sharing
  • Data audits and “data detectives”
  • Profiling researchers (not sure about this one!)
  • Criminalisation of academic fraud
  • Protection and support of whistle-blowers

Solving the Sin of Corruptibility

A recent example

[Wall Street Journal]

Solving the Sin of Corruptibility

A recent example

Solving the Sin of Internment

Internment: research is “published” behind paywalls making it inaccessible to the public.

Solutions:

Solving the Sin of Bean Counting

Bean counting: an obsession with superficial metrics to evaluate science and scientists

Solutions:

  • Formal CRediT system (rather than opague authorship order). https://credit.niso.org/

  • More generally, new hiring and promotion guidelines are BADLY needed

Solutions that my lab has been trying

  • See my blog post on open science here.

A list of resources

  • Useful resources can be found here.

Summary

We need saving from ourselves

The first principle is that you must not fool yourself - and you are the easiest person to fool

Richard Feynman (1974)

  • Each belt, shackle and guard may represent:
    • Pre-registration.
    • Statistical power and sample size.
    • Replication.
    • Meta-analysis.
    • Open data, materials and code.
    • Pre-prints.
    • Making more modest claims.

Scientific reform is difficult

It is overwhelming

Changing behaviour is hard


Take a break

Part 2 - Read and discuss

Discussion material


  • break into small groups (~ 5 per group)
  • discuss aspects of the lecture: – which solutions do you consider to be most important and why?
  • discuss aspects of the journal article: https://www.nature.com/articles/s41562-016-0021
    • can you think of any other solutions?
    • are there any barriers to widespread uptake?

References

Chambers, C. D. (2017). The Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice. Princeton University Press.
Feynman, R. P. (1974). Cargo Cult Science. Engineering and Science, 37(7), 10–13.
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
Vazire, S. (2018). Implications of the Credibility Revolution for Productivity, Creativity, and Progress. Perspectives on Psychological Science, 13(4), 411–417. https://doi.org/10.1177/1745691617751884

Acknowledgements