[hunchentoot-devel] googlebot revisitation rate excessive?

[Jeff Cunningham]

Has anyone else noticed excessive visitation of pages served dynamically 
by Hunchentoot? On a couple of small sites virtually every page gets hit 
every five minutes by http://www.google.com/bot.html, regular as clockwork.

Normally, I setup robots.txt to allow google to walk these sites but 
unless I can think of a way to tell them to back off to once a week or 
something I'm about to Disallow them. They're just little private sites 
and Google ends up being 99% of the traffic. It's silly.

--Jeff