Eh, you are rationalizing. The phrase "traumatically over-X" is extremely rare. Any problem is easy after you've seen the solution. :) The solution "traumatically over-trained LLM" to the problem "What description best fits karpathy's description?" is certainly not easy to find. Connecting RL, poor LLMs, extreme fear, and welfare to excess training and severe lasting emotional pain is pretty darn impressive. E.g., I know exactly what situation karpathy describes is, but I couldn't in a million years put it into writing as succinctly and as precisely as the LLM.
> The phrase "traumatically over-X" is extremely rare.
There are plenty of "over-x" phrases in English associated with trauma or harm. Do a web search in quotes for "traumatic over{extension/exertion/stimulation}" (off the top of my head) and you'll get direct hits. And this isn't a Markov chain—its doesn't have to pull n-grams directly from its training material. That it could glue trauma and training into "traumatic over-training" is deeply unsurprising to me.
> I couldn't in a million years put it into writing as succinctly and as precisely as the LLM.
If that's the case, then (with respect) that may be down to your skills as a writer. The LLM puts it decently enough, but it's not very expressive and it doesn't add anything.
> Connecting RL, poor LLMs, extreme fear, and welfare to excess training and severe lasting emotional pain is pretty darn impressive
Is it? Really, we're just analogizing it to an abused pet. You over-train your dog, so it gets traumatized. The LLM connects the ideas and then synthesizes a lukewarm sentence to capture that connection at the cost of losing a degree of precision, because LLMs aren't animals. Models are good at those vector-embedding-style conceptual connections—I won't begrudge them that. Expressive use of language and fine-grained reasoning, though? Not so much.
