Raider keeps attacking door after desertion//Repeat of desertion-message/sound

Turned a raid away and like 10 people of them deserted (which gave repeatedly the alert sound and the screen message, latter ok, first anoying :D ), but one of the deserted raiders kept attacking my doors