> Sure it is, just like a child or someone not very good at reasoning.
That's a good way to think about it. Treat GPT-4 as having mentality of a 4 year old kid. A kid this age will take any question you ask at face value, because it hasn't learned yet that adults often don't ask questions precisely enough, don't realize the assumptions they make in their requests, don't know what they don't know, and are full of shit. A four year old won't think of evaluating whether or not the question itself makes sense, they'll just do their best to answer it, which may involve plain guessing what the answer could be if one isn't apparent.
Remember that saying "I don't know" isn't an innate skill in humans either - it's an ability we drill into kids for the first decade or two of their lives.
That doesn't tell the whole story tho. It's a 4 year old kid that has been thoroughly conditioned to always be positive and affirming in their reply, even if it means making something up. That isn't something kids do usually—it's not something humans usually do, at least not the way ChatGPT does–and that may be part of why it's so confounding.
It's not just "I don't know", really. It feels like OpenAI ingrained the essence of North American culture into the model (Sorry North Americans, I really don't mean this in a demeaning way!), as in, the primary task of ChatGPT is supposed to be to make its users happy and feel good about themselves, taking priority over providing accurate answers and facts.
Another similarity is that 4-year-olds will often pick up words and phrases from people around them, and learn associations between those words and phrases without yet having learned their meanings or having any of their own experiences to relate them to.
So a young child might answer your question with a response that he heard other people say to a similar question, without actually understanding what he's saying. LLMs are basically this at a grand scale.
That's a good way to think about it. Treat GPT-4 as having mentality of a 4 year old kid. A kid this age will take any question you ask at face value, because it hasn't learned yet that adults often don't ask questions precisely enough, don't realize the assumptions they make in their requests, don't know what they don't know, and are full of shit. A four year old won't think of evaluating whether or not the question itself makes sense, they'll just do their best to answer it, which may involve plain guessing what the answer could be if one isn't apparent.
Remember that saying "I don't know" isn't an innate skill in humans either - it's an ability we drill into kids for the first decade or two of their lives.