Generally no. That said, what OS do you use? Certain OSes have a remote administrative "feature" in them and if you haven't removed it or disabled it they could gain access.

Alternatively, if someone knows your phone, they then can find out your name. From your name they can get your email and then can send you a message that might be infected with a trojan. Do you have AntiVirus software, up-to-date and running?

FYI: I deleted your second thread of this. No need to post twice. Once is enough. Someone is bound to respond but you need to have patience.