Enode has a good suggestion, or, you could make X stop listening on that port by editing

/etc/X11/xdm/Xservers

and changing the line which reads

:0 local /usr/X11R6/bin/X

to read

:0 local /usr/X11R6/bin/X -nolisten tcp

good luck.

Oops, forgot to add

You will need to restart X after that.