Or just stop after the first sentence.

Have you tried saying, “Please don’t ask me that anymore”?

That will address the exact problem without being rude, without offending him, and without opening it up for more questions. You don’t owe him an explanation, so don’t leave an opening for one. Just say: “Please don’t ask me that anymore.”

If he asks why, you say, “Doesn’t matter. Please don’t ask me that anymore.”

If he offers an explanation for why he’s asking you that, you say, “Ok. Please don’t ask me that anymore.”

Neat and easy. No unintended consequences.