[hunchentoot-devel] googlebot revisitation rate excessive?

[Jeff Cunningham]

Hans Hübner wrote:
> I'd set HUNCHENTOOT:*HEADER-STREAM* to *STANDARD-OUTPUT* and
> *BREAK-ON-SIGNALS* to 'WARNING, then wait for the Googlebot request to
> come in.  The headers printed to the console may give you a clue what
> the request looks like and maybe a way to initiate such a failing
> request yourself, maybe with Drakma or wget.  You'll may also be able
> to get a clue from looking at the backtrace in a debugger.
>
>   
Good suggestion, Hans. I found a contraction inside a meta content 
string which was demarked with single quotes.
> I find it curious that Google retries every five minutes.  Did you
> verify that the request is coming from a Google IP address?  It may
> also be a prankster's script gone wild, in which case I'd block the IP
> address.
>   
It is curious. I'd checked it right off - it is a Google IP address. 
Seems like a terrible waste of their bandwidth.

--Jeff