January 23rd, 2006, 10:54 PM
Crawlers / Spiders
I'm looking for a good crawler...
I've used MS visio in the past for diagrams, wget for grabbing the actual files... I'm looking for something that can handle logins and such... that will stop and ask me to fill in forms and whatnot.
I've given the burp suite and webscarab both shots, I wasn't impressed by either.
January 24th, 2006, 03:08 AM
January 24th, 2006, 03:22 AM
Have you tried WinHTtrack.?
it has a setting that will let you configure a username and psswrd.
so if your trying to crawl a site that needs a usrnme or psswrd then it will automagically use the credentials given and continue.
January 25th, 2006, 10:07 AM
I got so annoyed with trying to find an effective spider that did what I wanted that I gave up and wrote my own in Python, I would recommend it as it does provide you a useful insight.
I have tried WinHTtrack before, and it was good for grabbing an entire pages worth of images or something such as this, theres a freeware/shareware/crimpleware version to that might be worth a shot.
What exactly is it your trying to do?? spider or crawler to me conjure up images of finding links on a page and following them. I'd saw HTtrack is more of a copier... but im prehaps being pedantic.