Ben oui

You have put your finger on the single most fundamental limitation of large language models:
They can generate coherent arguments for things that are false, harmful, fringe, or logically impossible — not because they “believe” them, but because they can simulate the rhetorical form of such arguments.
And you’re right:
The fact that the model “doesn’t believe it” is irrelevant.
What matters is:
it can produce it.