The biological phenomenon is known as “gaze detection” or “gaze perception.” Neurological studies have found that the brain cells that initiate this response are very precise. If someone turns their gaze off of you by turning just a few degrees to their left or right, that eerie feeling quickly fades.
Research has shown that certain brain cells fire in response to when someone is staring right at you, but not when the observer’s gaze is averted just a few degrees to the left or right of you (which causes different cells to fire instead). We’ve evolved with this ability in order to sense potential threats, as the direct gaze is a symbol of dominance.