Comments on: Developing a Web application

By: nigelmcfarlane

nigelmcfarlane — Fri, 10 Dec 2004 05:39:48 +0000

If your web page is to be a good global citizen, then you shouldn’t scan a site for pages if the site’s robots.txt tells you not to.

An excellent client-side web spidering library can be found here: http://www.bclary.com.

However: you’re pretty stuck client-side because
of security.
You can’t script into a web page that belongs to a foreign site (The “Same Origin Policy”). That means you can’t follow all the links in the other site’s loaded pages, to disciver the size of the site.

That means your code that accepts the form submission
has to do it. You might as well use the Google API (it’s SOAP I think) and ask Google to do the word search for you. Then you don’t need to count or
scan pages at all – just use the Google results.

- N.

By: riverwind

riverwind — Wed, 08 Dec 2004 22:55:16 +0000

It depends on how you design your web application. At server level you can use software like ‘spider’ to monitor the usage. I think that the most suitable method is to develop the controls and logs on the web application itself, i.e., the application level of control. However, it is quite tough for it.

By: bobkberg

bobkberg — Wed, 08 Dec 2004 17:46:50 +0000

John Brandt already provided the basic answer.

The only other caveat I’d offer is that a spider utility will only show you the web pages that are linked through the index (or other known) page. Many sites have “landing” pages specifically designed for advertising click-through that are never referenced through anything but ads, and some pages which are meant to remain private.

The other place you can look is to do a search on that domain for robots.txt, since (if present) that specifies the files/pages/directories which a spider should or should not consider fair game for indexing, and may provide you with the information you need.

Bob