[hunchentoot-devel] googlebot revisitation rate excessive?

[Zach Beane]

On Sun, Jun 29, 2008 at 04:31:47PM -0700, Jeff Cunningham wrote:
> 
> Has anyone else noticed excessive visitation of pages served dynamically 
> by Hunchentoot? On a couple of small sites virtually every page gets hit 
> every five minutes by http://www.google.com/bot.html, regular as clockwork.
> 
> Normally, I setup robots.txt to allow google to walk these sites but 
> unless I can think of a way to tell them to back off to once a week or 
> something I'm about to Disallow them. They're just little private sites 
> and Google ends up being 99% of the traffic. It's silly.

FWIW, I haven't had any trouble like that with two different
Hunchentoot sites.

Zach