I've been saying no to visits because I thought the only downside was not having a chance to talk to them, but now I just got attacked and i absolutely would not have said no if I thought it could have meant war.
Opinions are everything in this game.
I had a noble ask to join my kingdom. I said sure, so his former kingdom attacked me. But it kinda makes sense because the other kingdom is mad that they lost a noble to me.
In your case, they are mad because you won't let them satisfy their curiosity. But allowing them to stay also means they sometimes help you with tasks, like storing items that are just laying around, defending against attacking kingdoms (if they are also at war with them) or healing the wounds of your nobles that are hurt during an attack.
Maybe a warning from the person themselves?