Printable View
I'm looking for a good crawler...
I've used MS visio in the past for diagrams, wget for grabbing the actual files... I'm looking for something that can handle logins and such... that will stop and ask me to fill in forms and whatnot.
I've given the burp suite and webscarab both shots, I wasn't impressed by either.
Any recommendations?
I don't know a whole lot about running spiders or crawlers, but for the sake of technicalities, wget can "handle" logins in terms of usernames/passwords ( .htaccess ). Are you looking for a spider that will index and when it hits something that requires an account and bring it to your attention and wait until give it an account? I've never heard of something fancy like that.
Sorry.
Rasem
Have you tried WinHTtrack.?
it has a setting that will let you configure a username and psswrd.
so if your trying to crawl a site that needs a usrnme or psswrd then it will automagically ;) use the credentials given and continue.
I got so annoyed with trying to find an effective spider that did what I wanted that I gave up and wrote my own in Python, I would recommend it as it does provide you a useful insight.
I have tried WinHTtrack before, and it was good for grabbing an entire pages worth of images or something such as this, theres a freeware/shareware/crimpleware version to that might be worth a shot.
What exactly is it your trying to do?? spider or crawler to me conjure up images of finding links on a page and following them. I'd saw HTtrack is more of a copier... but im prehaps being pedantic.
i2c