Well, not completely...you need to remember that when you send a request for a web page, it is handled by the server--and it is the server that will need to be able to read/execute the document so it can send it back to you. Most (if not all) apache setups are run with the user "nobody" and under the group "nogroup". Thus, if you own the files, then the user nobody becomes an other.

I'm not sure how your hosting provider handles all this. Unless your provider makes you part of nogroup, you could have some problems down the line if you rely soley on file permissions to allow access, unless you can change file ownership as well as permissions. Thus, I would advise that you either employ some other access control over files you don't want every one to see (password authentication, allow by ip) or just not upload them in the first place.