Monitoring Website Changes

Background:

I have recently changed jobs and moved to a Betting Company as head of their IT Department. They have a need to constantly monitor competitor's websites for price changes. So if my company has a football game priced at 10/1 and a competitor also has the game priced at 10/1 then all is good. But if this competitor company suddenly changes it's price to 2/1 then they must have got some inside information. In order for my company not to take a loss, we need to assess this price change and act accordingly.

At present there is just one person monitoring the website for changes which is obviously not perfect.

What I need is an application that monitors the competitor's website and alerts my company via whatever method I decide.

My question is: Can I bot be written with this functionality in mind?

Now I know Bots are dreadful things and can be quite annoying, but forgetting about their regular use, could a bot be written to do what I need.

If so, could you please point me in the general direction. I don't wish to be spoon fed, but my knowledge as of yet on the subject is limited. I've no problem coding in whatever language it takes.

Thanks for your help.

Wolfe.

As long as the layout of their pages don't change you could probably use curl and grep (or your favoured alternatives) to pull down a copy if the riavls page in HTML, a copy of your page and display the differences in an output that you desire.

That's my initial thought anyway. It's quite an interesting task to undertake though, will give it a bit more thought.

Did you have a specific platform in mind?

Perl? http://baruch.ev-en.org/proj/websec/
Specto? http://specto.sourceforge.net/
Firefox Add on? https://addons.mozilla.org/en-US/firefox/addon/3491
Website Watcher? http://www.aignes.com/

Cheers for the links PP but most of them only check a single webpage.

The competitor's website is dynamic and driven by events so monitoring one webpage at a time is not plausible as when an event has passed it is removed and events are added/deleted on a regular basis.

Not too sure how this is going to be achieved!

For something like this I would:

Create a site in php that opens up each site in separate frames and using regular expressions take out all the other parts of their site. Basically you will end up with frames on a single page of each of your competitors with only say the table of information you need from each of their pages... If that makes sense, if not I can try to explain better...

Yeah I was thinking along the lines of a script that is automated that would strip all information up to the start of the events/odds table. Then read each line of the table and store in a database format.

Run the script every few mins and compare to that of the stored database.

The problem I see here is that the pages would have to be fairly similar otherwise I would need to code for each page.

Plus, don't forget that you're going to show up in their usage stats. The user agent, the refer if there is one, etc. Before you had a human doing it. It might have looked like normal activity? It would be fairly simple for them to watch for your activities and block you?

If you're looking to script it. You may want to look into httrack. That will probably have most of the features you're looking for. It's both a GUI and command line... http://www.httrack.com/html/fcguide.html

Will check it out - thanks!

thank you

It would be fairly simple for them to block you, even if you disguise your user agent. However, instead of blocking you, they could simply send back rubbish data to your bot, causing you to make erroneous business decisions.

Writing such a bot is nontrivial and error-prone, but if you choose to do it, there are plenty of methods to do it.

Perl and Python have their own APIs, .NET and Java also have HTTP clients and HTML parsers in them.

Using regular expressions is not recommended as they don't actually work for parsing HTML.

Use a proper HTML parser if you do it.

Mark

Cheers for the info Mark, but alas I only stayed in that job for a week due to the company not approving my development budget - they wanted to cut corners etc so I told them I don't do that and left.

In a better job now with a 4 day week for better money so all is good.