Monitoring Website Changes
Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Monitoring Website Changes

  1. #1
    Senior Member WolfeTone's Avatar
    Join Date
    Jun 2007
    Location
    Ireland
    Posts
    197

    Monitoring Website Changes

    Background:

    I have recently changed jobs and moved to a Betting Company as head of their IT Department. They have a need to constantly monitor competitor's websites for price changes. So if my company has a football game priced at 10/1 and a competitor also has the game priced at 10/1 then all is good. But if this competitor company suddenly changes it's price to 2/1 then they must have got some inside information. In order for my company not to take a loss, we need to assess this price change and act accordingly.

    At present there is just one person monitoring the website for changes which is obviously not perfect.

    What I need is an application that monitors the competitor's website and alerts my company via whatever method I decide.

    My question is: Can I bot be written with this functionality in mind?

    Now I know Bots are dreadful things and can be quite annoying, but forgetting about their regular use, could a bot be written to do what I need.

    If so, could you please point me in the general direction. I don't wish to be spoon fed, but my knowledge as of yet on the subject is limited. I've no problem coding in whatever language it takes.

    Thanks for your help.

    Wolfe.

  2. #2
    Right turn Clyde Nokia's Avatar
    Join Date
    Aug 2003
    Location
    Button Moon
    Posts
    1,696
    As long as the layout of their pages don't change you could probably use curl and grep (or your favoured alternatives) to pull down a copy if the riavls page in HTML, a copy of your page and display the differences in an output that you desire.

    That's my initial thought anyway. It's quite an interesting task to undertake though, will give it a bit more thought.
    Drugs have taught an entire generation of kids the metric system.

    http://tazforum.**********.com/

  3. #3
    AO übergeek phishphreek's Avatar
    Join Date
    Jan 2002
    Posts
    4,324
    Did you have a specific platform in mind?

    Perl? http://baruch.ev-en.org/proj/websec/
    Specto? http://specto.sourceforge.net/
    Firefox Add on? https://addons.mozilla.org/en-US/firefox/addon/3491
    Website Watcher? http://www.aignes.com/
    Quitmzilla is a firefox extension that gives you stats on how long you have quit smoking, how much money you\'ve saved, how much you haven\'t smoked and recent milestones. Very helpful for people who quit smoking and used to smoke at their computers... Helps out with the urges.

  4. #4
    Senior Member WolfeTone's Avatar
    Join Date
    Jun 2007
    Location
    Ireland
    Posts
    197
    Cheers for the links PP but most of them only check a single webpage.

    The competitor's website is dynamic and driven by events so monitoring one webpage at a time is not plausible as when an event has passed it is removed and events are added/deleted on a regular basis.

    Not too sure how this is going to be achieved!

  5. #5
    Senior Member
    Join Date
    Oct 2003
    Location
    MA
    Posts
    1,053
    For something like this I would:

    Create a site in php that opens up each site in separate frames and using regular expressions take out all the other parts of their site. Basically you will end up with frames on a single page of each of your competitors with only say the table of information you need from each of their pages... If that makes sense, if not I can try to explain better...

  6. #6
    Senior Member WolfeTone's Avatar
    Join Date
    Jun 2007
    Location
    Ireland
    Posts
    197
    Yeah I was thinking along the lines of a script that is automated that would strip all information up to the start of the events/odds table. Then read each line of the table and store in a database format.

    Run the script every few mins and compare to that of the stored database.

    The problem I see here is that the pages would have to be fairly similar otherwise I would need to code for each page.

  7. #7
    AO übergeek phishphreek's Avatar
    Join Date
    Jan 2002
    Posts
    4,324
    Plus, don't forget that you're going to show up in their usage stats. The user agent, the refer if there is one, etc. Before you had a human doing it. It might have looked like normal activity? It would be fairly simple for them to watch for your activities and block you?

    If you're looking to script it. You may want to look into httrack. That will probably have most of the features you're looking for. It's both a GUI and command line... http://www.httrack.com/html/fcguide.html
    Last edited by phishphreek; November 8th, 2007 at 01:46 PM.
    Quitmzilla is a firefox extension that gives you stats on how long you have quit smoking, how much money you\'ve saved, how much you haven\'t smoked and recent milestones. Very helpful for people who quit smoking and used to smoke at their computers... Helps out with the urges.

  8. #8
    Senior Member WolfeTone's Avatar
    Join Date
    Jun 2007
    Location
    Ireland
    Posts
    197
    Will check it out - thanks!

  9. #9
    Junior Member
    Join Date
    Dec 2007
    Posts
    1
    thank you

  10. #10
    Senior Member
    Join Date
    Jan 2002
    Posts
    1,207
    It would be fairly simple for them to block you, even if you disguise your user agent. However, instead of blocking you, they could simply send back rubbish data to your bot, causing you to make erroneous business decisions.

    Writing such a bot is nontrivial and error-prone, but if you choose to do it, there are plenty of methods to do it.

    Perl and Python have their own APIs, .NET and Java also have HTTP clients and HTML parsers in them.

    Using regular expressions is not recommended as they don't actually work for parsing HTML.

    Use a proper HTML parser if you do it.

    Mark

Similar Threads

  1. Website Administration
    By jethro in forum The Security Tutorials Forum
    Replies: 4
    Last Post: August 9th, 2006, 11:13 AM
  2. Website to website malware scanning
    By Aspman in forum Spyware / Adware
    Replies: 20
    Last Post: November 21st, 2005, 09:07 AM
  3. Security management - setting up audit and account
    By tenzenryu in forum The Security Tutorials Forum
    Replies: 7
    Last Post: July 11th, 2005, 04:57 AM
  4. Tcp/ip
    By gore in forum Newbie Security Questions
    Replies: 11
    Last Post: December 29th, 2003, 08:01 AM
  5. Central Internet monitoring?!?!
    By phishphreek in forum Cosmos
    Replies: 7
    Last Post: December 21st, 2002, 12:32 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •