It's already been done, without the model being aware of it, see https://arxiv.org/abs/2512.09742. They also made it think it was Hitler (not MechaHitler, the other guy), and other craziness.

It's a relief to think that we're not trusting these things for stuff like financial advice, medical advice, mental health counselling, ...