Home > IT Answers > Development > Website capture and comparison
Boardpig
90 pts.
 Website capture and comparison
Website Testing, Offline Backup, Content management applications
We have an interesting project in that we would like to be able to capture a web site for off line viewing (I realise there are many apps that can do this) but then we would also like to be able to re-capture the content at a later date and have an automatic process compare the initial capture with the re-capture and produce a report of the changes (if any)?? Been asking around and it seems this might be a difficult thing to do. Any ideas would be appreciated.
ASKED: May 19, 2010  1:44 AM GMT
UPDATED: June 7, 2010  3:58:46 PM GMT
RELATED QUESTIONS
Chippy088
4,625 pts.
Answer Wiki:
In that case it is a subject I can support . Unwanted web content is something that bugs me personally, and professionaly in the work space.

To my knowledge, there is no easy way of doing this. I have used the off line browsing capability on my browser to automatically upload a single page, if it changed at some point after my last visit, but your task is far more complex. I do not cross check pages in the quantity you will be doing.

The functions that spring to mind are serious applications to download data, save and comparrison checking. As I am sure you are aware of in your quest to provide a solution.

For instance, some pages of my website have dates and times associated with them. e.g. todays date, user last visited date and time, which will have changed each time it was visited. So, that will be a change that will have to be taken into consideration when automatically checking the saved pages for differences.

I use a web update function in excel, to download a page of latest lottery results, each time I open the work book. Macros format, copy and sort the data into the respective sheets. Maybe Word, or Publisher could be used in the same way using VBA.

This is out of my league, but my train of thought is running along the lines of down loading web pages into publisher or word, on a regular basis, and then comparing the changes in them using file comparison s/w. If it is only 1 or 2 errant pages you need to check, scripting may be just as good.

Maybe someone with better scripting skills could come up with a automated script to do this. It does, however, bring up the fact that, maybe, at sometime a human will have to look at the text of the page and decide if the change is a valid content change, or just a change like you would get from my site (last date visited etc)

Good luck with this project, let me know if baldness sets in.
Last Wiki Answer Submitted:  Jun 7, 2010  3:52 PM (GMT)  by  Chippy088   4,625 pts.
To see other answers submitted to the Answer Wiki View Answer History.
RSS - Discussions Help
Discuss This Question:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _



 

Nobody has any ideas on this?

Boardpig
 90 pts.
 

Any ideas about what, specifically ?

I have never done something like that, but If you know how to download the site’s files, you could then open each file and compare it to the newer version, line by line, or with the help of some other tool like the Linux diff command.

Carlosdl
 60,255 pts.
 

Most Web browsers can be set to download the page again, if it changes. Maybe that is the way to go. Content of the pages for public viewing are not sensitive or they wouldn’t be there.

You can view the code in some web browsers, but there are private files which you will not have access to, held on the ISP server.

The problem with downloading a site, is that if you are trying to copy it through the internet, you will not get all the files on the site, unless you are an admin of the site. In that case you wouldn’t be asking this question, so I assume you are not.

So if the site collects data from visitors, for whatever reason, it would be breaking the Data Protection Act if you could see this type of information.

As to why you want to do this, or what information you want to compare, you don’t say. Therefore, I think I would be very cautious in the help given.

Chippy088
 4,625 pts.
 

Hello,

Yeah I understand this could be sensitive. The organisation I am working with are in the business of tracking and classifying unsuitable material on web sites and are looking for an automated way of re-checking the websites after classification. (without the human going back to look at them).
Ie when thew website has been classfied, the hoster may makes changes to the site to alter the classification. They are looking for an automated way of detecting the change.

Boardpig
 90 pts.
 

I have just noticed I entered my reply in the wiki section. (I just started typing where my mug shot was)

I do not mean to imply this is the only answer. In fact, I hope some one does come up with the a more informed answer, as it has aroused my interest too.

Chippy088
 4,625 pts.