Psychological Scientist at TU Eindhoven. Co-host of Nullius in Verba. Improving Stats Textbook lakens.github.io/statistical… Omnia probate 🇪🇺
Netherlands
- Tweets 66,582
- Following 1,583
- Followers 30,667
- Likes 43,663
Pinned Tweet
Together with my co-host @smreeteemehta we've started a new podcast: Nullius in Verba. A podcast about science - what it is, and what it could be. The introduction episode is up now!
Apple:
podcasts.apple.com/us/podcas…
Spotify:
open.spotify.com/show/0tw0yV…
RSS:
feed.podbean.com/nulliusinve…
6
26
141
108,464
I’ve written a textbook on improving your statistical inferences. It consists of the content I teach in workshops, and combines material from my MOOC’s, blog, and published work. It’s a free open educational resource: You can read it here: lakens.github.io/statistical…
25
647
2,072
Plot of a p-value, calculated for a simple t-test, for sample sizes of 10 to 2010, when the null hypothesis is true. Because p-values are uniformly distributed, the p-value just wanders randomly between 0 and 1. This is why repeatedly analyzing data as it comes in is not good!
22
565
1,761
An abbreviation (ABB) in a journal article (JA) is rarely worth the words it saves. Every ABB requires cognitive resources (CR) and at my age by the time I'm halfway through a JA I no longer have the CR to remember what your ABB stood for.
29
472
1,364
Common mistake, even among statisticians, to think p-values are measures of evidence. I explain why that is not a coherent way of thinking about p-values here osf.io/preprints/psyarxiv/7n…
>A p-value quantifies the "strength of evidence" in an experiment."
: |
I'm just going to stop correcting people unless they ask me
12
317
1,373
240,233
New World View article out in Nature, where I argue for methodological review of research proposals before data collection. nature.com/articles/d41586-0…
30
270
1,302
435,244
One of the great pleasures of learning statistics is that you replace the nagging uncertainty about whether you are doing the right analysis with the nagging uncertainty about whether there is such a thing as the right analysis.
11
184
969
New preprint: Sample Size Justification. psyarxiv.com/9d3yf/ It tries to help you to write an honest sample size justification for your next study. This means thinking about how to deal with resource limitations and uncertainty.
12
280
814
My new MOOC "Improving Your Statistical Questions" has launched! coursera.org/learn/improving…. There are 15 videos and 13 assignments, all freely available. I hope you'll like it! An overview of the contents and a thank you to all who helped in this blog post: daniellakens.blogspot.com/20…
14
256
789
The labor of the thousands of scientists who did research on a COVID vaccine that didn't work was just as essential as the work of the small subset that was involved in a vaccine that did work. At the start, we didn't know who would succeed. That's how science works.
4
128
694
My wife went to art school, which sounds like it was 4 years of learning to believe in your own work despite constant criticism. My PhD felt like it was 4 years of learning how to follow the norms to prevent being criticized. Maybe we need a bit more art school in academia.
16
125
666
96% of published findings in psychology are statistically significant. There are 2 options. 1) We study effects with more than 90% power, and more than 90% probability of being true. 2) There is MASSIVE publication bias.
Hint: the answer is 2
journals.sagepub.com/doi/10.…
14
187
690
65,146
Please: Don't refer to papers as 'a paper by X' where X is the best know co-author. The correct reference is '1st author et al'. It is disrespectful to the first author who did most of the work, and ignores how papers are a team effort. Remember: Prestige is the death of science.
18
110
601
Young researchers often feel they failed or did something wrong when they find a non-significant result. But if you think it is just as likely that your hypothesis is true as that it is false, have 80% power and a 5% alpha, a true negative is actually the most likely outcome!
10
188
587
It has been interesting to see this study go viral - as the authors expected when starting 3 years ago. They asked us to organize a Red Team to make sure their work was criticized internally at each step, to try to catch any flaws. So far, despite the controversial > 1/2
Meta-analysis of field studies on gender bias in hiring: Three key findings
1. In male-dominated/gender-balanced fields, male applicants were favoured before 2009, but since then, there’s been no consistent bias or even a weak pro-female bias.
9
121
594
296,092
When writing a scientific paper, here is a mistake almost everyone (including me) makes: The goal of a theoretical introduction is not to explain how a theory predicts the observed data. The goal is to explain how without the theory the observed data would not be predicted.
15
119
564
My paper 'The Practical Alternative to the P-value is the Correctly Used P-value' is now in press at Perspectives on Psychological Science. Accepted version is at psyarxiv.com/shm8v
8
141
552
I generally recommend people to compare themselves to their past selves (i.e., have mastery goals). But if you *must* compare yourself to others, do it well and enter resources, social support, and random luck as covariates in your model.
15
143
479
Please don't use 'power analysis' when you mean 'sample size justification'. Power analysis is only one approach to sample size justification (online.ucpress.edu/collabra/…) - accuracy and feasibility (!) are often more in line with the actual inferential goal of a study.
5
105
504
51,184
When I write a paper on how to improve science, I always find it useful to remember that at least 20 people already said the same thing. Half of them said it better. Half of those said it before 1970.
18
45
469
Dutch universities are moving to a system where researchers will get a permanent contract after 18 months. Glad to see this. So much healthier for early career researchers, and will make it easier to settle early in life, with less moving countries, less uncertainty.
12
66
446
145,904
Confidence intervals are confusing intervals. The reason is because they tell us something about what happens across many confidence intervals, not in any single confidence interval. In the long run, 95% of 95% CI contain a population parameter. 1/n
15
102
531
72,006
I wish academics would be as worried about the money and time that is wasted because isolated groups are working in secrecy, trying to be better than others, making no coordinated effort to generate knowledge, as they are about getting scooped when doing open science.
11
132
458
People sometimes wonder if pre-registration is worth the effort. I show them Kaplan & Irvin's 2015 (journals.plos.org/plosone/ar…) findings on the likelihood of null-effects before and after pre-reg became required in medicine, and ask if they want a drug discovered in 1998 or 2002.
7
215
468
I sometimes fear we are training a generation of graduate students who worry more about calculating a p-value or a Bayes factor than about how to ask a good research question and how to design a solid experiment.
17
111
412
Not every study needs to have a power analysis, but every study needs a sample size justification. I discuss 6 approaches, and 6 ways to think about which effect sizes are of interest in the study you are planning.
online.ucpress.edu/collabra/…
5
101
487
33,211
New Preprint: The practical alternative to the p-value is the correctly used p-value. psyarxiv.com/shm8v
12
158
415
Understanding p-values is not difficult, but it does take more time (say 2 hours initially, rehearsing it 4 times for 30 minutes) than people invest, which is the only barrier to getting people to understand them. This material should be all you need. lakens.github.io/statistical…
7
78
408
73,967
If your new year resolution is to update your stats knowledge, and you want to learn about p-values, Bayesian stats, power and error control, equivalence testing, p-curve analysis, pre-registration, etc., maybe you'd like to try out my free online course: coursera.org/learn/statistic…
5
134
384
If you like more guidance through the steps of a sample size justification for your next study, my Sample Size Justification paper comes with a Shiny app: shiny.ieis.tue.nl/sample_siz… Complete the steps, and you can download the justification as a PDF to include in a preregistration.
4
90
385
50,204
The Max Planck Society (with its 14.000 scientists) has ended its subscription to Elsevier journals (following Sweden and Germany). the-scientist.com/news-opini…
4
176
412
My bachelor students are working on a project where they are coding how often scientists misinterpret the results of statistical tests in published papers. Not surprisingly, they are finding many mistakes. Just to repeat: My bachelor students. In published papers.
20
67
398
I like this recommendation to report confidence intervals not as 3.5 [-2.0, 8.7] but as subscripts ₋₂.₀ 3.5 ₈.₇
academic.oup.com/biostatisti…
Not the biggest thing, jut but seems a tiny bit easier to process.
14
43
404
35,776
If you are preparing your bachelor statistics course and would like to add optional material for students to better understand statistics on a conceptual level (see topics in the screenshot) my free textbook provides a state of the art overview. lakens.github.io/statistical…
4
104
455
23,254
Too often exploratory work is forced into a hypothesis testing mold. We need journals to accept exploratory research. Other fields than psychology do this systematically. There are many widely cited papers titled 'An exploratory study....' in other fields. We need more of them.
7
89
367
This is why people need basic training in logic (in this case about the positive predictive value). After selecting patients on 'needing to go on a ventilator' many will die, whatever you do. Same flaw as 'hospitals have equal numbers of vaccinated and unvaccinated patients'.
80% of Covid patients put on ventilators died. This is probably most shocking revelation unknown to the public.
13
70
387
58,716
Being a Dungeon Master as a teenager prepares you surprisingly well for chairing academic meetings.
14
25
350
I strongly believe that the widespread norm that it is acceptable to not share non-significant results, even though we know how hugely problematic publication bias is for reliable knowledge generation, will be seen as the greatest ethical lapse of scientists in my generation.
19
97
374
My textbook now includes an updated figure from Carter & McCullough, 2014. It's now generated from the raw data. It's a striking image. The uncorrected effect size is d = 0.62 based on 198 studies. Now, large replications later, we know the true effect size is 0. What a waste.
11
70
373
171,501
The benchmarks for ‘small’, ‘medium’, and ‘large’ effects for Cohen’s d are d = 0.2, d = 0.5, and d = 0.8, and for a correlation are r = 0.1, r = 0.3, and r = 0.5. Except they don't match. Converting d to r actually gives r's of .10, .24, and .37. The benchmarks are incoherent.
11
50
367
Statisticians are often trained in how to compute numbers, but not in why to compute numbers. But the goal of science is not to compute numbers, but to generate knowledge. If your approach to statistics is not grounded in a philosophy of science, you are just computing numbers.
7
55
371
33,193
My wife came home yesterday. She said : 'I have a gift. You will soon be recording your next MOOC. Your audience has expectations' ❤
18
5
361
If you are a young social scientist, thinking about what your next research project should be, here's a suggestion: study what the smallest effect size of interest is in your specific research area. Here are a few reasons why.🧵
9
71
360
I vividly remember teaching p-curve analysis in 2014 to PhD students when a student in the back loudly swore. I asked 'what's wrong?'. She said 'I just p-curved the studies I have been trying to replicate for the first 1.5 years of my PhD, and now I see why I could not.'
6
35
373
120,203
My wife, who is planning to pursue a PhD, had been searching for literature today. There were a lot of interesting papers she could not read because the school she teaches at does not have access. Seeing her response after learning Sci-hub exists is pretty much priceless.
6
21
353
I've made a Shiny app form that guides you through the recommendations in my Sample Size Justification preprint (psyarxiv.com/9d3yf/). Find the app at shiny.ieis.tue.nl/sample_siz…. It's a first version, so I appreciate feedback (link to form in app) for bugs/improvements.
5
94
347
Finally got around to incorporating all Github feedback on my open access textbook, making around 20 minor improvements, and updating some references. 17 chapters of state of the art stats and methods education, freely available for any course you teach. lakens.github.io/statistical…
5
102
400
21,978
I really look forward to not buying this. A resource like this should be available for free #openaccess
Do you use @APA_Style for your writing? Get ready for some changes! We’re pleased to announce that the 7th edition of our Publication Manual will go on sale in October.
Get a sneak peek at some of the latest updates, 10 years in the making: on.apa.org/2GRrSCR
6
47
326
I have remade my open textbook 'Improving Your Statistical Inferences' in Quarto. lakens.github.io/statistical… All 'Test Yourself' questions at the bottom of each chapter are now interactive thanks to the 'webexercises' package (thanks @LisaDeBruine!).
3
94
352
67,777
I find the way social media make it possible to communicate across academic boundaries incredibly valuable. I've learned so much from people on here who work in medicine, philosophy, statistics, economics, etc. My knowledge would be much more limited without Twitter.
7
45
359
Did you know my free online textbook comes with 140 interactive practice questions for your students to test their understanding of p-values, Type 1 and 2 errors, effect sizes, confidence intervals, sample size justification, etc? lakens.github.io/statistical…
2
84
328
32,039
If you are a mediocre programmer like I am, programming in R is like a variable ratio reinforcement schedule: You randomly try a number of different commands until you are unpredictably rewarded by code that works. Super addictive.
15
42
311
New comment in Nature Human Behavior with my former bachelor student Eline Ensinck: "Make abandoned research publicly available". We argue it is time to be honest about the content of our file drawers, and the varied reasons studies remain unshared. nature.com/articles/s41562-0… >
5
100
324
32,080
When you are explaining equivalence tests for the first time #statsTikTok
7
32
312
If every scientific paper you have written would be magically erased and had never existed, what would be the consequences?
Good night, sleep well.
24
32
309
I've rarely seen worse advice than trying to interpret p-values directly as measures of evidence. This does not work because of Lindley's paradox, and this graph is misleading (a p = 0.04 can be evidence *against* a hypothesis). If you want to talk evidence, use likelihoods.
New p-values paper from some of us in Trondheim: Rewriting results sections in the language of evidence @chloe_nater
@eb_nilsen @StefanieMuff
@CBD_NTNU sciencedirect.com/science/ar…
17
56
304
Perhaps the first description in the scientific literature of p-hacking (or 'Cooking') by (the) Charles Babbage, from 1830 (!), in 'Reflections on the Decline of Science in England: And on Some of Its Causes'.
4
169
310
Dear stats teachers: The fact that p-values are uniformly distributed when the null is true, and skewed as a function of power, is essential to understanding what p-values are. It should be taught in the first year intro stats lecture where you say what a p-value is.
11
99
310
It's great to see many of you seem to like this. The graph is part of my free online stats course on Coursera - and it isn't even the most interesting thing in the course ;) Check it out: coursera.org/learn/statistic…
1
25
300
Here's your regularly reminder that 99% of the papers you have written would have been improved if you would follow the APA JARS guidelines. Show me a discussion section that checks these boxes. apastyle.apa.org/jars
8
70
301
On a medium timeline I believe solid training in mixed methods (qualitative and quantitative) will become a highly valued skill as big data starts to realize how much and how quickly you can learn things by first simply talking to the people you are studying.
8
70
275
This seems to hve struck a chord. The whole presentation is here: surfdrive.surf.nl/files/inde…. The main theme was that due to individualistic ways of working, lack of time to learn all skills we need, and science becoming open, we will need a culture to openly talk about mistakes.
7
51
291
Why is it difficult to interpret null results in underpowered studies? Below, you see a study with 50% power for an effect of d = 0.5. Let’s say the observed effect is d = 0.3, so p > 0.05. What do we do?
3
80
299
82,036
Pilot studies are your friend. They will inform you about how participants interpret your instructions and questions, and identify where you need to adjust your analysis plan. It's very difficult to write a good preregistration without a pilot study.
5
36
289
44,698
The strongest possible sample size justification is based on an explicit statement of the smallest effect size that is considered interesting. You can't know what the 'true effect size' is - so design a study to detect effects that matter lakens.github.io/statistical…
5
53
286
'The Practical Alternative to the p Value Is the Correctly Used p Value' is now published in Perspectives on Psychological Science: journals.sagepub.com/doi/pdf…
68
274
New blog post: Effect Sizes and Power for Interactions in ANOVA Designs daniellakens.blogspot.com/20… I explain how effect sizes for interactions depend on the pattern of means, and why it might be worthwhile to design studies predicting crossover interactions where possible.
6
78
278
Dear early career researchers, it is always sensible to ask senior researchers how things in science are done. Just remember that for science to improve you are expected to at regular times ignore their opinion and follow your own principles instead.
2
43
262
New blog post: Requiring high-powered studies from scientists with resource constraints daniellakens.blogspot.com/20…
12
129
280
My paper 'Sample Size Justification' is now published @CollabraOA: online.ucpress.edu/collabra/…. The accompanying Shiny app to complete your sample size justification: shiny.ieis.tue.nl/sample_siz… Audio recording of the paper: soundcloud.com/lakens/sample…
4
78
282
You know how they say "stop saying 'statistically significant'" but you didn't know what to say instead? Glad that is solved! I am now following each p<.05 test result with: "There is no way that anyone can believe that there's this much statistical impossiblity happening".
We need a full audit of each and every vote cast in Pennsylvania. There is no way that anyone can believe that there’s this much statistical impossibility happening everywhere.
8
39
279
Dear Science, if you can't keep people like @jamesheathers in, you have a really, really big problem. medium.com/@jamesheathers/i-…
5
44
277
In the history of science we have only identified one intervention that is associated with an increase of null results in the published literature: Registered Reports. Deciding to accept a paper irrespective of the results. doi.org/10.1177/251524592110…
The idea that publication decisions should be based only on design and not on findings is not sensible. Most things in the world don't work. Given this prior, papers that demonstrate effectiveness are all else equal much more informative.
4
48
283
63,845
This is a common mistake. If you want to learn how to not be like this reviewer, read lakens.github.io/statistical…
And insists that if two CIs overlap then there can't be a significant difference. Really, just don't comment on things outside your area of expertise. I regularly plead ignorance to certain things in reviews.
8
54
289
89,939
UCLA urges staff to decline reviewing for El$evier until the company proposes a deal that is more sustainable for the university.
5
134
262
I've received a VIDI grant to hire 2 PhD's & a postdoc on my "Increasing the Reliability and Efficiency of Psychological Science" project :)
24
54
263
Just because a finding is statistically significant does not mean it is practically significant.
Even fortune cookies know you need to think about your smallest effect size of interest!
4
51
266
Francesca Gino's defense as to why she did not commit fraud is to put the bar to be found guilty *incredibly* high. She wants uni's to have *irrefutable proof* that she *intentionally* messed with the data. She knows: With the bar this high, everyone will get away with fraud.
9
31
284
71,972
My Intro Psych course includes an assignment where randomly paired students complete the 'Fast Friends Procedure', designed to generate closeness between strangers. Today two former students came by to tell me that they met through this class assignment and just got engaged ❤️
13
10
279
18,798
This paper makes 2 very important points. First, for within designs G*Power has weird defaults that lead to large mistakes. Use our Superpower app instead: arcaldwell49.shinyapps.io/an… Second, as I stress in my paper on sample size justification, make power analyses reproducible.
new preprint by Thibault et al examineds widespread use (and misuse) of G*power by researchers
medrxiv.org/content/10.1101/…
1
67
261
40,741
Very complete and accessible tutorial article by Marc Brysbaert on power analysis, full of useful information and recommendations, including sample size recommendations for Bayes factors and equivalence tests. Great reading material for a masters course! journalofcognition.org/artic…
5
91
256
The more experience I have with scientific publications as an author, reviewer, and editor, the more convinced I have become that we need *open* peer review if we want to rely on peer review as a tool for quality control.
16
46
254
When justifying sample sizes, PhD students have feasibility (time, money) constraints, yet the literature mainly discusses sample size justifications that ignore feasibility (e.g. power analysis). Is there any good work out there on doing research with sample size constraints?
26
57
257
If you prefer reading journal lay-out versions of articles, the paper "Why Hypothesis Testers Should Spend Less Time Testing Hypotheses" by @annemscheel, @LeonidTiokhin @peder_isager and me is now available in print at PoPS: journals.sagepub.com/doi/pdf… #openaccess
5
62
253
I would like to express my gratitude to all people who dedicated their time in 2020 on diamond open access journals (free to publish, free to read, mostly financed by governments). We all know this is what scientific journals should look like. Thanks for moving us to the future!
4
47
241
Many of the problems we are addressing in science emerged because people mindlessly followed norms. We are not going to solve any of the underlying problems if people mindlessly adopt new norms. This is why I think education, throughout your entire career, is so important.
9
71
246
Thanks for citing my 2022 Sample Size Justification 1000 times. online.ucpress.edu/collabra/…
This is a good moment to tell you that you (in general) suck at citing my paper. I made a whole app to help you write a solid justification. Please use it.
shiny.ieis.tue.nl/sample_siz….
4
38
263
40,449
It is sad to see we still need to explain to professors what the problems are with publishing findings based on whether results are significant or not. Accepting counterintuitive claims in top journals only when significant messes up the incentive structure in science.
The idea that publication decisions should be based only on design and not on findings is not sensible. Most things in the world don't work. Given this prior, papers that demonstrate effectiveness are all else equal much more informative.
15
33
248
85,443
I'm grateful for receiving an Ammodo science award for fundamental research in the social sciences. ammodo-science-award.org/en/… I look forward to using the prize to expand my team! And with that I am going back to teaching a workshop on good research practices at Zurich University 😃
14
17
235
19,435
Preregistering a one-sided test when you test a directional prediction increases statistical power with on average ~11% at no cost whatsoever compared to doing a (logically incoherent) two-sided test. You are welcome.
12
35
246
62,797
After people do my MOOC, they often say 'everyone who uses p-values should learn this'. And this is true. Doing week 1 will take you 3 hours and will change the way you use p-values for the rest of you life. It's free. Schedule it in. It will be worth it.
Mind boggling: When you have very high power (e.g. >98%) observing a p-value of 0.045 indicates that the data are more likely under the null hypothesis than under the alternative hypothesis. Still learning every day, thanks to @lakens for his excellent @coursera class!
5
43
237
Fisher, in his book The Design of Experiments (1935), explaining why a single study is never enough to conclude we have discovered something.
3
95
256
As part of every scientists' education you should try to enter the data required to perform a meta analysis for let's say 10 studies in your field. There is nothing like it to make you realize how badly we currently share our results.
Doing a large-scale meta-analysis for my 1st time. I now realize how much time could be spared if data were systematically available online.
8
63
232
Statisticians should be less like priests and more plumbers. I don't care what you personally believe is the right way to do things - if I have a specific problem, I want to know all possible solutions that might fix it, what their limitations are, and how much each would cost.
17
28
249
54,384
New paper shows that even in large randomized controlled trials in economics a pre-specified analysis plan reduces p-hacking. Kind of obvious but good to demonstrate. This effect is likely even bigger in small easily performed studies in psychology. papers.ssrn.com/sol3/papers.…
4
66
227
As I'm moving most of my writing from Word to R Markdown, stumbling across @writage made my life a lot easier! A Word plugin that allows you to save any old word file as a markdown file! Moving text from word to markdown couldn't be easier! writage.com/
9
49
234