Lecture 4 - A problem in theory?

The Social Brain: Critical Perspectives on Science, Society and Neurodiversity

Richard Ramsey

Today


Part 1

  • A problem in theory?


Part 2

  • Read articles and discuss



Context

Unreliable methods, but what about theory?

(Munafò et al., 2017)

A problem in theory

Without a sufficiently well-developed theory, then it is hard to interpret results in a meaningful way.

  • Anything that turns up as statistically significant can be used to tell a story or sell newspapers.

  • At most, only 0.4% of adolescent wellbeing is related to screen use - which only slightly surpasses the negative effect of regularly eating potatoes.

(Orben & Przybylski, 2019)

A problem in theory

Predictions are often impossible to falsify because of infinite wriggle room…

Data

HARKing

Wriggle room

Theory as a blueprint for science


Evidence / Facts / Experiments

Theory

Oragnised knowledge

(Gray, 2017)

Theory before the test

Are psychological scientists ready to test hypotheses?

Inputs to informative hypothesis tests:

  • Concept formation
  • Developing measures
  • Establishing relationships between concepts
  • Boundary conditions and auxiliary assumptions
  • Deriving statistical predictions

(Scheel et al., 2021)

Concept formation

What do we mean by terms such as screen time, intrinsic motivation or depression?

Why is this important?

  • Lots of time and money is spent having pointless debate and argument about reproducibility, when key terms are not adequately defined

Measurement

Measurement schmeasurement

We demonstrate that psychology is plagued by a measurement schmeasurement attitude: questionable measurement practices are common, hide a stunning source of researcher degrees of freedom, pose a serious threat to cumulative psychological science, but are largely ignored.

(Flake & Fried, 2020)

Measurement


  • Are the measures valid and reliable?
  • And are they measuring the concept of interest?

(Flake & Fried, 2020)

RMET - Example

(Baron-Cohen et al., 2001)

RMET - Problems

[[some stuff here describing the problems]]

(Higgins et al., 2022)

Relationships between concepts

Once concepts are defined, we need a causal model of how they relate to each other.

Boundary conditions

A good theory is clear about its boundary conditions – where does the theory apply and where is beyond its scope?

Basic vision vs. social cognition

Individual differences

Cultural variation

Why is there a lack of good quality theory?


  • OK, these seem like sensible ideas. But they are not new ideas?

  • Paul Meehl was writing about this in (1967).

Question for the class:

  • Why don’t researchers do more “theory” before the test?

How can we improve theory building?

How can we improve theory building?


  • This is hard and it is generally undervalued

I’ll provide two examples:

  • Theory Mapping
  • Formal theory

Theory mapping

(Gray, 2017)

Formal theory

  • Psychological theory tends to be narrative

  • Predictions are ordinal

  • Formal theory is a mathematical description that can give rise to quantitative predictions

(Farrell & Lewandowsky, 2018; Guest & Martin, 2021; Yarkoni, 2022)

What are the benefits of formal theories?


It forces researchers to be explicit about parts of the system under investigation and how they are linked together (e.g., \(E = mc^2\)).

(Hintzman, 1991)

Formal theory from my lab

(Ward & Ramsey, 2024)

Formal theory from my lab

(Ward & Ramsey, 2024)

Formal theory from my lab

(Ward & Ramsey, 2024)

Formal theory from my lab

(Ward & Ramsey, 2024)

Formal theory from my lab


  • The benefit of formal theory here is not that our model is “correct”, but that it is explicit and it can therefore be more easily be falsified.

  • Formal theory reduces verbal wriggle room

Today


Part 1

  • A problem in theory?


Part 2

  • Read articles and discuss



Take a break

Part 2 - Read and discuss

Discussion material


References

Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The “Reading the Mind in the Eyes” Test Revised Version: A Study with Normal Adults, and Adults with Asperger Syndrome or High-functioning Autism. Journal of Child Psychology and Psychiatry, 42(2), 241–251. https://doi.org/10.1111/1469-7610.00715
Farrell, S., & Lewandowsky, S. (2018). Computational Modeling of Cognition and Behavior: (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781316272503
Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3, 456–465. https://doi.org/10.1177/2515245920952393
Gray, K. (2017). How to Map Theory: Reliable Methods Are Fruitless Without Rigorous Theory. Perspect Psychol Sci, 12(5), 731–741. https://doi.org/10.1177/1745691617691949
Guest, O., & Martin, A. E. (2021). How Computational Modeling Can Force Theory Building in Psychological Science. Perspectives on Psychological Science, 1745691620970585. https://doi.org/10.1177/1745691620970585
Higgins, W. C., Ross, R. M., Langdon, R., & Polito, V. (2022). The “Reading the Mind in the Eyes” Test Shows Poor Psychometric Properties in a Large, Demographically Representative U.S. Sample. Assessment, 10731911221124342. https://doi.org/10.1177/10731911221124342
Hintzman, D. L. (1991). Why are formal models useful in psychology? In Relating theory and data: Essays on human memory in honor of Bennet B. Murdock. (pp. 39–56). Lawrence Erlbaum Associates, Inc.
Meehl, P. E. (1967). Theory-Testing in Psychology and Physics: A Methodological Paradox. Philosophy of Science, 34(2), 103–115. https://doi.org/10.1086/288135
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
Orben, A., & Przybylski, A. K. (2019). The association between adolescent well-being and digital technology use. Nature Human Behaviour, 3(2), 173–182. https://doi.org/10.1038/s41562-018-0506-1
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021). Why hypothesis testers should spend less time testing hypotheses. Perspectives on Psychological Science, 16, 744–755. https://doi.org/10.1177/1745691620966795
Ward, R., & Ramsey, R. (2024). Integrating Social Cognition Into Domain-General Control: Interactive Activation and Competition for the Control of Action (ICON). Cognitive Science, 48(2), e13415. https://doi.org/10.1111/cogs.13415
Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, e1. https://doi.org/10.1017/S0140525X20001685

Acknowledgements