Methods and Meta-science

rich-ramsey.github.io/talks/sbs-retreat-24/

Richard Ramsey
www.rich-ramsey.com

Roadmap

  1. Problems.
  2. Data science.
  3. Planning.
  4. Modelling.

Problems

Problems

Open Science Collaboration, 2015

Problems

Open Science Collaboration, 2015




Problems

  1. Bias
  2. Hidden Flexibility
  3. Unreliability
  4. Data Hoarding
  5. Corruptibility
  6. Internment
  7. Bean Counting

Problems

Munafò et al., 2017

Problems

  • An example: publication bias (or the “file-drawer effect”).
  • Incentives for significant effects can create so-called zombie research

Problems

Roadmap

  1. Problems.
  2. Data science.
  3. Planning.
  4. Modelling.

Data science

Data science

Data science

https://ajgoldstein.com/2017/11/12/deconstructing-data-science/

Data science


  • R for Data Science
  • A free, online book: https://r4ds.hadley.nz/
  • Or buy the hard copy
  • This should be your data science bible

Data science

Data science

Data science

  • No Excel sheets !
  • Instead:
  • A raw data file (i.e., trial level data with no exclusions).
  • An analysis script (or set of scripts)

Roadmap

  1. Problems.
  2. Data science.
  3. Planning.
  4. Modelling.

Planning

Planning

  • What are the scientifc goals of the research project?
  • Are there practical or more applied goals?
  • What kind of data?
  • What kind of research design?
  • How much data per participant per condition?
  • How many participants?
  • How many experiments?
  • What kind of data analysis will be performed?
  • What kind of resources are available?

Planning

Before testing hypotheses, researchers should spend more time (Scheel, 2021):

  • forming concepts
  • developing valid measures
  • establishing the causal relationships between concepts
  • identify boundary conditions and auxiliary assumptions

Planning

Planning

Planning

Planning

  • Effect sizes (Funder & Ozer, 2019)
  • Respect your chosen approach (strengths and weaknesses) before AND after you run your study
  • Pre-registration
  • Avoid the Cult of Single Isolated Study (Nelder, 1986. Tong, 2019)

Planning

Planning


Planning


  • Pre-registration is your friend.
  • It helps you to push back against reviewers in a principled way
  • “It is a plan, not a prison” (DeHaven, 2017).

Roadmap

  1. Problems.
  2. Data science.
  3. Planning.
  4. Modelling.

Modelling

Modelling

Image from: https://danawanzer.github.io/stats-with-jamovi/

Modelling

https://lindeloev.github.io/tests-as-linear/

Modelling

summary / aggregated data

trial-level data

Modelling

  • Multi-level modelling takes advantage of partial pooling or shrinkage

https://blog.conductrics.com/prediction-pooling-and-shrinkage/

Modelling

  • Parameters in models rather than p-values/Bayes factors

Modelling

  • One general, powerful and flexible approach, rather than a separate “test” for each design

Final thoughts

Final thoughts

https://www.eaton.com/gb/en-gb/markets/machine-building/service-and-support-machine-building-moem-service-eaton/blogs/Two-hand-control-for-machinery-blog-functional-safety.html
  • Data science, methods reform and meta-science approaches can help provide protection against the scientific environment

Final thoughts

  • The first principle is that you must not fool yourself - and you are the easiest person to fool (Richard Feynman, 1974).
  • Each belt, shackle and guard may represent:
    • Pre-registration.
    • Statistical power and sample size.
    • Replication.
    • Meta-analysis.
    • Open data, materials and code.
    • Pre-prints.
    • Make more modest claims.

Final thoughts

  • All of this is overwhelming

  • Changing behaviour is hard

Final thoughts

The method of science, as stodgy and grumpy as it may seem, is far more important than the findings of science.

Carl Sagan, 1995

https://www.nature.com/articles/d41586-023-03240-x. Credit: Science History Images/Alamy

Some relevant resources

GitHub tutorials: