Some people weren't sure what I meant by superintelligence not being a thing so maybe an analogy would help.
Say you've got an college physics major, and every month you chart her performance on a test of physics understanding against the number of textbook pages she's read. Throughout her undergraduate years, you'd see a pretty consistent relationship where more pages read leads to better performance. Someone could extrapolate this curve out and predict that if she keeps reading physics textbooks, she'll be the next Einstein some time in the 2030s.
But what will actually happen is that some time early in grad school she'll reach the frontiers of physics knowledge and switch from learning about other people's physics discoveries in textbooks to trying to make her own. And at this point the progress of her understanding of physics will, in some sense, slow way down.
You could say that she "ran out of training data" but the problem wouldn't be that there are no more textbooks she can read. Rather, it would be that she already knows most of what's in the textbook. What she ran out of is new knowledge that can be readily assimilated.
By the same token, we might reach a point where LLMs have achieved human levels of understanding of a wide range of topics. At this point, providing more words of training data might become unhelpful because while there might still be plenty of new words available, they might just be different ways of saying stuff the LLM already knows (like a physics major reading her 20th textbook).
It seems like one reason Bostrom-style superintelligence might turn out not to be a thing is that we "run out" of training data. Intuitively LLMs plateau in performance once they've "used up" the information in their training set.