First of all, your manager having you and others to give your password to an assistant is a no no. This is a bad way to ensure email from customers and clients can be accessed. The preferred method is to create a distribution list and give the distribution lists email address to the customers/clients. Then there will be no need to give out passwords to access your email. Also, asking the assistant if he/she is accessing your mail is your duty. You need to find out who's doing it or change this troublesome policy your department has.

If I were you, I woulld talk your manager into changing the policy on password sharing, change yours and get the email admin to create distribution list for you guys. Situations like this will come back to bite you in the long run.