computational cognitive science he/him
Berkeley, CA
- Tweets 7,449
- Following 2,530
- Followers 10,018
- Likes 17,581
Did you know that there are psychologists who study "stereotype accuracy"? I've always wondered what the hell, so I've been reading it recently. For the record, it’s exactly as bad as it sounds.
Here’s a thread.
37
481
3,334
The ideas in the @Harpers letter were destroyed on Twitter yesterday. Here's a thread/meta-thread.
49
912
2,249
it's official: a reviewer has commented on my use of "accordion" in a paper and said to choose "some other more reasonable instrument"
73
93
1,754
Large language models change everything for linguistics, starting with Chomsky.
Featuring: LLMs as scientific theories, response to prior takes, "why" questions in language, acquisition... and how the field should have seen this coming.
Paper is here: lingbuzz.net/lingbuzz/007180
94
392
1,945
877,256
I work on mathematical models of cognition at UC Berkeley, and also research the origins of math. All of your questions are wonderful. Here are some shots at answers.
10
108
1,464
everyone works to make figures that are easy to understand, but this one took me a minute (I thought it was a graphics mistake) and that made the point even stronger
11
369
1,335
Instead of h-index, what about computing how likely someone is to work with you again after working with you once?
24
107
1,288
185,799
I am sooooooo excited for this paper. We've spent years developing a super fast program induction library. We use it to learn key pieces of language structure.
So much of what Chomskyan linguists say about learnability is totally wrong.
🧵
pnas.org/content/119/5/e2021…
27
297
1,357
Christian Ebbesen wrote a damning paper critique psyarxiv.com/tzr8c (which the figure just above is from)
6
71
789
(It’s actually not clear that people without printed materials understand pictures in the same way we do. Here's Tepilit Ole Saitoti, from his autobiography, The Worlds of a Maasai Warrior)
6
78
705
They often aren't interested in doing our tasks---or doing them how we expected---and why would they be? They're foraging-farmers who live a traditional lifestyle, and few can read and write, and so even the idea of a test, much less experiment, is foreign.
3
25
694
In our own tasks, when we find low scores, we usually discover it’s due to lack of clarity by experimenters (or translators), not evidence of inability. We address this by having controls to ensure participants understood the task. But IQ tasks don’t have a control condition.
2
42
652
For a “culture free” IQ test, Cattell even suggested that people could be given mazes. I wonder if he thought about the fact that some indigenous people have never been in a hallway.
6
39
631
But ethnobotanical knowledge, or thinking about causal relations in natural kinds, or training dogs to hunt, or surviving alone in the rain forest aren't tested in IQ tests. What is tested instead in "culture fair" IQ tests are visual relations, geometric shapes, patterns, etc
11
45
620
A recent paper in @PsychScience relied on national differences in IQ that held that the average IQ of some countries is below the threshold for intellectual disability.
10
66
586
I once saw a researcher present results that an indigneous group I work with, the Tsimane’, have an average IQ of around 70. But I know from leading lots of experiments with them that they approach written tests very differently because they don't have years of testing in schools
3
31
577
I am so happy to report the official news that doing this for over a decade has not prevented @UCBerkeley from awarding me tenure.
36
13
573
It is an amazing time to work in the cognitive science of language. Here are a few remarkable recent results, many of which highlight ways in which the critiques of LLMs (especially from generative linguistics!) have totally fallen to pieces.
8
141
628
198,474
Nobody who knows the Tsimane’ would think they are intellectually handicapped--they have, for instance, extensive ethnobotanical knowledge that would surpass most US adults.
science.sciencemag.org/conte…
2
27
553
But cultures differ in, for example, whether they have words for shapes, spatial relations, etc. These differences influence how people think about and use categories @glupyan (sapir.psych.wisc.edu/papers/… )
3
34
525
You know, you can't predict if you'll win your next roulette spin, but you can predict the long-term average. Same for many chaotic systems.
Exxon itself predicted the climate average shockingly well decades ago science.org/doi/10.1126/scie…
3
5
498
15,058
The editor resigned after the publisher noted "a significant breakdown in our editing processes" including, the editor reportedly not reading the piece before it was published. He wasn’t fired for being controversial; he was fired for doing a bad job.
nytimes.com/2020/06/07/busin…
2
32
448
4yo's best linguistic innovations to date:
"hafted" (had to do something in the past)
"nopesolutely" (definitely not)
"as hot as ice is cold" (very hot)
16
33
486
Such differences were highlighted decades ago with examples that show the cultural baggage inherent in creating any test. If you construct the right IQ test, as in "The Black Intelligence Test of Cultural Homogeneity", black children score higher than white children.
6
36
452
This isn’t the deterioration of society. A student had the confidence to speak about language that even another professor agreed was inappropriate. The student raised the issue for formalized examination and debate. Why aren’t they applauding her care and vigilance?
7
17
399
Every "philosophical" problem is just people failing to recognize that words don't have precise definitions, and words get used in different ways.
Once we give words precise definitions, "philosophy" turns into just ordinary mathematics.
100
39
486
469,340
We are happy to share our new commentary on why using language inclusively for transgender people is perfectly appropriate biologically, socially, and scientifically.
Here is a 🧵 on this new commentary by @AndyPerfors, @celestekidd, and myself.
nature.com/articles/s41562-0…
3
144
427
105,340
We've found that many Tsimane’, for instance, don't know shape labels. Then why should anyone *ever* run tests that could be influenced by knowledge of shapes with the Tsimane’?
2
17
433
Large language models probably have meanings that work like our own and are far from autocompleting parrots: new paper with @FelixHill84 @DeepMind spans AI, cognitive science, and philosophy to address the "octopus test" and find meaning without reference.
arxiv.org/abs/2208.02957
17
93
484
So, IQ tests in part measure how much effort you're willing to exert, not ability. What's worse, we don't know how large an effect motivation *could* have w/ optimal intervention because the field has not considered it important to figure out the dose-response curve.
4
33
428
Overall, these findings reflect a primary failing of IQ research: it *equates* ability with outcome on a standardized test. This assumption is so ingrained in research assumptions, it’s rarely questioned.
8
32
398
We could keep going through the studies Jussim reviews, but I think a fair generalization is that it’s basically garbage.
3
8
369
If they really had some convincing cases, it sure would have helped to point to them. Without examples, it’s as though they think the argument can be made just based on the names of the signers.
At Current Affairs we have a note we put on drafts called "Please Buttress With Examples" that asks the writer to provide citations and specifics before making broad sweeping generalizations. I see Harper's does not have the same policy.
3
26
355
The GameStop $GME news is too funny not to write a thread about.
It's also the most interesting intersection of psychology and politics going on.
6
94
404
Tired of over-fitting your data with neural network models that have millions of parameters? Now you can over-fit with just one parameter: newly accepted paper derives a simple, one-parameter equation that can fit any scatter plot to a given precision colala.bcs.rochester.edu/pap…
11
154
424
But, unless you're deeply racist, you'll know that ethnicity is NOT the causal factor at play (as opposed to, e.g. environmental or cultural factors). So any stereotype that the differences are *about* ethnicity is wrong from the get-go.
15
9
357
This is the most remarkable line because it simultaneously says they want to protect THEIR speech, but censor YOUR speech—stop your "call” if it's asking for consequences.
6
46
352
Beyond this one study, international IQ results put researchers in an awkward position. Even if we acknowledge their samples weren’t representative of these nations, there’s still the question of why the original studies found so many children with such low IQ in these countries
2
17
377
The cross-cultural results on IQ aren’t an error or a fluke. They’re a canary. They show us the baggage that IQ tests bring. Once you see that these measures don’t do what they claim across cultures, we have to doubt that they measure what they claim within cultures too.
23
61
369
New perspective in @NatRevPsych: human intelligence is a matter of scale of information processing, not genetic changes to one domain. Implications for AI, evolution, and development. - with @CantlonLab
rdcu.be/dDoBt
10
106
429
108,783
Thinking about large cultural differences between people who live far away and speak different languages (non-WEIRD-vs-WEIRD) should motivate us to think about potential cultural and experience-based differences between people who speak the same language and live nearby.
1
25
360
Better research has documented that people are actually terrible--I mean, really terrible--at estimating these group differences. They severely underestimate levels of inequality. @mwkraus
journals.sagepub.com/doi/pdf…
2
25
349
One big one is that performance on them is sensitive to factors that nobody would consider "ability." For instance, if you pay people more, they will do better. The size of the effect across studies averages to about 0.5 SD @angeladuckw pnas.org/content/108/19/7716
3
18
362
Cognitive science proposes plausible explanations of behavior. But did you know that some behavioral patterns logically imply that certain neural representations *must* exist? New paper w/ Randy Gallistel. Implications for neuroscience and machine learning onlinelibrary.wiley.com/doi/…
13
92
405
82,367
Rebecca Sear @RebeccaSear wrote a damning thread.
mobile.twitter.com/RebeccaSe…
Do 'national IQ' datasets present accurate and unbiased data on average IQs in nation-states worldwide?
A thread 👇
(Spoiler alert: NO)
1/n
1
18
356
The photo helped employees also raise other issues of racial discrimination at the magazine. Was a "clumsy mistake" what made Bon Appétit only pay white editors for video appearances under his leadership?
2
17
334
This actually highlights a real demarcation problem for this field: Is any belief about people a stereotype? Vegetarians don’t eat meat? Redheads have hair? Doesn’t the scope of what you include determine whether stereotypes are generally accurate? More on that later.
3
6
344
Everyone seems to think it's absurd that large language models (or something similar) could show anything like human intelligence and meaning. But it doesn’t seem so crazy to me. Here's a dissenting 🧵 from cognitive science.
14
74
394
Motivation? Not measured. Expectations? Not measured. Test familiarity? Not measured. Interest in the task? Not measured. Attention? Not measured. Confidence? Not measured. Wandering thoughts about financial insecurity? Not measured.
2
29
327
A week ago, the paper was retracted from @PsychScience.
nitter.app/CousinAmygdala/status/…
1
16
326
Failing to name the examples is not innocent.
This is not an innocent rhetorical strategy. It's likely I would agree with the signatories about most or all the examples alluded to. But by erasing the substance of the controversies, we are offered a simple morality play instead of real engagement. (5/n)
2
19
301
How easily do you understand what experimenters want? Not measured. How much do you like doing what others want? Not measured. (@cantlonlab told me about a kid in an IQ test who said "mouth" when asked to name a container for cereal. It was a good answer, marked incorrect.)
3
23
324
But, it’s a fiction. These authors haven’t studied the class of all “stereotypes.” They haven’t randomly sampled stereotypes. As a result, the field has NO ability to evaluate whether the *typical* stereotype is true or not.
3
12
291
Nick Brown @sTeamTraen found the results had been “pre”-registered after it was submitted.
Here's the preregistration. It was "preregistered" on June 3, 2019. That's 130 days ***after the manuscript was submitted***. It describes a preregistered analysis of... the possible effect of adding an extra explanatory variable to the models that are already in the paper. /7
1
13
288
Jussim et Al. cite McCauley & Stitt, a tiny, narrow study from the 1970s. This field is so empirically weak that 40 years later, people are still writing about what 69 kids in a junior college in PA thought about Germans. Really.
2
9
288
But @celestekidd (et al.) realized that these studies were confounded by environmental reliability: for instance, maybe poor kids *should not* delay gratification because, based on their experiences, promised rewards are less likely to materialize.
3
27
294
Stress at home? Not measured. Hunger because you couldn't buy breakfast? Not measured. How much do you care about making the experimenter or teacher happy? Not measured. Rapport with your teachers or experimenters? Not measured.
1
22
292
If there aren't good actual examples to illustrate their point, what's going on? I think @ParkerMolloy's take is exactly spot on here:
nitter.app/ParkerMolloy/status/12…
2
25
267
And bias in psychology is not just in IQ testing.
mobile.twitter.com/duane_g_w…
2
15
296
All it takes is a single tweet for @jk_rowling to push discourse to the level of calling her lawyers to stop people who say things she doesn't like.
6
16
276
Indeed, academic conservatives like @JonHaidt claim not only that stereotypes are accurate, but that denial of "stereotype accuracy" is one of the left's anti-science positions. The equivalent of anti-vaxxers and young-earth creationists.
piped.video/watch?v=rBNtOCCS…
9
11
275
I am so excited that this new paper with @samisaguy is out. We explain how humans perceive number by mathematically deriving an optimal perceptual system with bounded information processing. Four N=100 pre-registered experiments match the predictions.
nature.com/articles/s41562-0…
4
81
302
… ignoring *everything else* in history. I wonder if "stereotype accuracy" researchers think that women's suffrage really did ruin men and families? Or were stereotypes not accurate then but are now? What happened?
9
5
272
This letter is about writers wanting their own words to never have consequences: "We need to preserve the possibility of good-faith disagreement without dire professional consequences." That must mean you're supposed to keep buying their books no... matter... what?
3
24
272
Thanks, part of my background is dynamical systems and what you are saying is nonsense. There is an entire field about predicting statistical averages of dynamical systems ("ergodic theory"). But whether you can predict climate is clear: Exxon did accurately predict decades ago
1
3
284
3,902
It's about prominent intellectuals not wanting everyone to have a say because they themselves don't like being criticized online.
1
20
263
Demonstration of effects of extrinsic motivation need to make us worry about individual variation. The problem is that individuals and groups almost certainly vary in motivation--people in extreme situations or poverty or other countries may not care much about doing your test.
3
11
266
So, do the Harper's signers then attest to the new book's accuracy? Or is the argument that publishers shouldn’t consider the accuracy of the work in their publishing decisions?
independent.co.uk/arts-enter…
2
11
242
Linguistic confidence in the test language? Not measured. Native dialect? Not measured. Belief you'll get the promised incentive? Not measured. Knowledge of test taking strategies? Not measured. Confidence you understood instructions? Not measured.
1
17
267
A new paper on (i) how to connect high-level cognitive theories to neuroscience / neural nets, and (ii) how learners can construct new concepts like number or logic without presupposing them. "Church encoding" is the metaphor we need.
link.springer.com/article/10…
5
69
285
Don’t believe that LLMs are good at solving math problems. New work led by @NasimBorazjani develops a set of simple problems that GPT fails miserably.
However, interfacing an LLM with a reliable symbolic system (Prolog) raises performance near ceiling.
arxiv.org/abs/2407.11373
5
49
316
38,858
It’s not that kids *can’t* wait. It’s not “ability.” It's a rational judgement about a reward's expected value. And the effect is huge---ceiling and floor. (It was recently replicated by an independent lab: sciencedirect.com/science/ar…)
2
21
272
For people who complain about “a vogue for public shaming”, the Harper’s signers sure seem to do it a lot.
nitter.app/CantlonLab/status/1280…
4
27
236
Arsonist here. I always advise against smoke detectors unless required by building codes.
Publisher here. In the social sciences I always advise against submitting to / publishing in any OA journal unless required by a funder. But if you're already in this pickle, asking the publisher for a discount or the editor for a waiver can work.
33
263
For this data, Ashton & Esses artificially shifted participant ratings to equal the true average (so the values reflect only relative differences). I wonder what the data looked like before that. Stereotypes must not be accurate enough without a little guiding hand.
4
7
244
I started with @PsychRabble et al's review paper, "Stereotype accuracy: one of the largest and most replicable effects in all of social psychology." It claims the correlation between stereotypes and reality is stronger than most effects in psychology.
gwern.net/docs/psychology/20…
5
10
250
Many papers on "stereotype accuracy" seem to begin by complaining that everyone assumes stereotypes are inaccurate (e.g. the word "stereotype" is *defined* to mean inaccurate) without evidence.
3
4
245
But in other areas of psychology, this assumption is common enough it has a name: “the fundamental attribution error”---presupposing behavior (test performance) is driven by internal ("ability") rather than external factors.
en.wikipedia.org/wiki/Fundam…
1
20
253
This editorial's tone, call for violence, and distortions of facts, led to an unheard of protest from staffers and mass cancellations of subscriptions. Who would want to support an organization that promoted that?
slate.com/news-and-politics/…
5
14
224
Moreover, as motivation would predict, the effect of reward in IQ test performance appears sensitive to the size of reward.
3
8
249
Or it could have been a reference to James Watson, an unscrupulous former scientist, current intellectual fossil, who was outed from leading Cold Spring Harbor lab for his unsupported and outrageous racial claims. Do they think he should be reinstituted?
nytimes.com/2019/01/01/scien…
6
12
229
"books are withdrawn for alleged inauthenticity" might be a reference to Naomi Wolf, where the publisher decided to cancel its U.S. release due to concerns about accuracy, which have long followed her work
nytimes.com/2019/06/05/books…
7
14
227
It’s the same Fox News trick used to silence #MeToo: attempt to convince everyone that allegations are levied for minor missteps. See, for instance, @celestekidd’s @NeurIPS comments on what this distortion does to the “climate for men”: piped.video/watch?v=FqWoS1I8…
1
21
239
Besides the unavoidable culture-ladenness of completing a written exam, there are other good reasons to doubt IQ tests meaningfully measure ability.
1
11
238
