Re: [Summary-Talk] WebFountain...

[Jason Linhart <jason@xxxxxxxxxxx>]

a.h.s. boy (lists) wrote:
> 
> http://www.almaden.ibm.com/cs/crawler
> 
> But as far as I can tell, there's no indication of it in the Summary
> statistics...is it not in the database of robots and crawlers? Am I
> missing something, or can it be added?
Sender: owner-summary-talk@lists.summary.net
Precedence: bulk
Reply-To: summary-talk@lists.summary.net

This robot is not in Summary's list of known robots, but it will get
detected as a likely robot. It shows up in the browser report as "http:
www.almaden.ibm.com cs crawler [XXXX".

> Are robots still supposed to temper the speed of their requesets, or is
> that an archaic guideline? This thing hits 1000s of my pages at a time,
> about 1 per second...

Yes, robots are still expected to be respectful of servers. There is no
hard and fast standard, but most people these days consider one per
second to be acceptably slow.

Jason

-- 
Jason@Summary.Net
--
Dr. Seuss books . . . can be read and enjoyed on several levels. For
example, 'One Fish Two Fish, Red Fish Blue Fish' can be deconstructed
as a searing indictment of the narrow-minded binary counting system.
   -- Peter van der Linden, Expert C Programming, Deep C Secrets

-------------
Go to <http://summary.net/list.html> to update subscription info.