As I understand it, most of the objects in the "asteroid" realm of things aren't really easy to spot when they're coming toward us... but once they pass us, it gets our attention. If I recall, it has something to due with the way in which the light reflects off the asteroid (I'll have to find the link again somewhere).

<edit>
And, if they're coming from us from the same direction as the sun, well... ever try catching a fast moving baseball while looking in to the sun? It usually doesn't work too well...
</edit>