In the book "Astral Dynamics" Robert Bruce describes his theory of the "mind split" that does not allow other spirits to enter your body while you are away.To me this theory is convincing as I tend to be bi-local in OBE's and even some lucid dreams. I mean I experience two perspectives simultaniously. Thus I know my body is not empty at all when I am gone. A part of my mind is still inside and noone can enter.

As to being annoyed, I would I would trust my instincts in this. When you felt annoyed you had a reason for it. Memory is a problem for most persons, we do not recall all sensorical and other aspects of these experiences. If you felt annoyed you probably knew something during the experience you do not remember any more.

I never experienced re-entry problems myself but I read that this happens when the part of your mind that is still inside your body is in deep sleep. You have to wait until it wakes up or enters dream-state.