[rdfweb-dev] FOAF-a-matic

Edd Dumbill edd at u...
Thu Jul 11 01:58:10 UTC 2002


On Wed, 2002-07-10 at 21:12, Dan Brickley wrote:
> > does your harvester go asking for .asc's of the files it's harvesting?
> > Is there any chance of having some sort of robots.rdf for giving rules to
> > such harvesters - monitoring 404's is beginning to be a pain...
> 
> That's Edd's harvester. 

I've since changed the behaviour of my spider to be less agressive, and
include my email address in the user agent string.

-- Edd

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.usefulinc.com/pipermail/foaf-dev/attachments/20020711/6d3e51bc/attachment-0001.pgp


More information about the foaf-dev mailing list