@bpaassen Sure, but they are *not* recommending we use those abilities to simulate experimental participants *instead of collecting behavioural data*. The only use case they discuss that is remotely like that is using that possibility to create an artificial pilot in order to fine tune experimental designs.
Discussion
Loading...
@bpaassen you start the thread by saying: “Our starting point is a trend in psychology: Recruiting human participants for psychological studies is a lot of effort and expensive. Wouldn't it be great if we could just use LLMs as participants, instead? N = 1000 for just a few dollars. (2/12)”
that is *not at all* what Binz et al are doing, advocating, or even interested in
@UlrikeHahn Right, but the immediate next post is "LLMs as participants can only work if LLMs really respond like humans to psychological questions. Do they?" And only then do I lead over to Dillion and Binz - which I think is fair because that is a claim both papers are making. And their claim is of immediate relevance to the starting point - hence I think this is a valid line of argument.
@bpaassen Benjamin, your paper (which I have now read and otherwise very much enjoyed) concludes: "Still, LLMs may be useful in other ways in psychological research, for example as tools for brainstorming, pilot testing, and refining experimental materials, perhaps even automating single, well-validated steps of data annotation. " (pg 14).
Binz et al. conclude: "It may, for instance, be used for in silico prototyping of experimental studies. In this context,one could use the model to figure out which designs lead to the largest effect sizes, how to design a study to reduce the number of required participants or to estimate the power of an effect." pg 7
your positions seem indistinguishable to me!
@UlrikeHahn In these recommendations, that may very well be. The point where we sharply depart (I think) is that we claim LLMs do not respond like humans (reliably) and therefore we should not rely on them but always check. They agree that one should check (which is appreciated) but still their work could be used as justification for synthetic participants because they also claim (in tension with their own recommendations) that LLMs respond like humans would.
@UlrikeHahn And we argue against this latter point.
Open Science
We are a network of scientists, developers and organizations building the next generation of digital spaces for open science.
Federation disabled