[hunchentoot-devel] googlebot revisitation rate excessive?

Fri Jul 4 15:22:30 UTC 2008

I've cleaned up a couple small problems using page validators but the 
net result has been that the Googlebot has gone from visiting my site 
every minute or so to every second or so. No kidding.

Before I block them altogether, there is one thing I don't understand 
that I'm hoping someone can explain to me. What does it mean exactly 
when I get a "No session for session identifier" INFO message in my 
error_log? There is one of these for each of the Googlebot hits.

--Jeff

Here is a snippet of the access_log (followed by the corresponding 
error_log snippet):

127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:32] "GET 
/zippy.html?hunchentoot-session=482%3A1424DD49911CD85
E35CE168E602686DC HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; 
Googlebot/2.1; +http://www.google.com/bot
.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:32] "GET 
/robogames-2008.html?hunchentoot-session=483%3AFBDE26
FEB3370A5DC63A565E2611DD5A HTTP/1.1" 200 12724 "-" "Mozilla/5.0 
(compatible; Googlebot/2.1; +http://www.goog
le.com/bot.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:33] "GET 
/robogames-2008.html?hunchentoot-session=482%3A1424DD
49911CD85E35CE168E602686DC HTTP/1.1" 200 12724 "-" "Mozilla/5.0 
(compatible; Googlebot/2.1; +http://www.goog
le.com/bot.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:34] "GET 
/zippy.html?hunchentoot-session=485%3A97280661B72EBFC
07C48B8F100FCB60C HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; 
Googlebot/2.1; +http://www.google.com/bot
.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:34] "GET 
/robogames-2008.html?hunchentoot-session=456%3A211AA2
04AD4D3A3703E0FA80DF3D0D51 HTTP/1.1" 200 12724 "-" "Mozilla/5.0 
(compatible; Googlebot/2.1; +http://www.goog
le.com/bot.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:35] "GET 
/zippy.html?hunchentoot-session=289%3A8676A2453C327D6
10DDB4F79A48A61F6 HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; 
Googlebot/2.1; +http://www.google.com/bot
.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:36] "GET 
/zippy.html?hunchentoot-session=177%3A6EF66F58D2E0A59
19AACB700DDB5D435 HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; 
Googlebot/2.1; +http://www.google.com/bot
.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:37] "GET 
/robogames-2008.html?hunchentoot-session=464%3AC5740E
8F90480BCF319EFCD9C7A8EE88 HTTP/1.1" 200 12724 "-" "Mozilla/5.0 
(compatible; Googlebot/2.1; +http://www.goog
le.com/bot.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:42:39] "GET 
/zippy.html?hunchentoot-session=483%3AFBDE26FEB3370A5
DC63A565E2611DD5A HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; 
Googlebot/2.1; +http://www.google.com/bot
.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:46:28] "GET 
/zippy.html?hunchentoot-session=478%3AB00C2E319A15E56
58D72E232C36F30DD HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; 
Googlebot/2.1; +http://www.google.com/bot
.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:50:18] "GET 
/zippy.html?hunchentoot-session=230%3A85588DC64DDDBF9
219B71F4DFA746B50 HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; 
Googlebot/2.1; +http://www.google.com/bot
.html)"
127.0.0.1 (66.249.66.195) - [2008-07-04 07:54:08] "GET 
/zippy.html?hunchentoot-session=145%3A4C7B146A126FC14
ACFC7BFA8100C07D8 HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; 
Googlebot/2.1; +http://www.google.com/bot
.html)"

HERE'S THE CORRESPONDING ERROR LOG ENTRIES:

[2008-07-04 07:39:32 [INFO]] No session for session identifier 
'483:FBDE26FEB3370A5DC63A565E2611DD5A' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:39:33 [INFO]] No session for session identifier 
'482:1424DD49911CD85E35CE168E602686DC' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:39:34 [INFO]] No session for session identifier 
'485:97280661B72EBFC07C48B8F100FCB60C' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:39:34 [INFO]] No session for session identifier 
'456:211AA204AD4D3A3703E0FA80DF3D0D51' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:39:35 [INFO]] No session for session identifier 
'289:8676A2453C327D610DDB4F79A48A61F6' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:39:36 [INFO]] No session for session identifier 
'177:6EF66F58D2E0A5919AACB700DDB5D435' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:39:37 [INFO]] No session for session identifier 
'464:C5740E8F90480BCF319EFCD9C7A8EE88' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:39:57 [ERROR]] Error while processing connection: I/O 
timeout reading #<SB-SYS:FD-STREAM for
"a socket" {133B61D9}>.
[2008-07-04 07:42:38 [INFO]] No session for session identifier 
'483:FBDE26FEB3370A5DC63A565E2611DD5A' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:42:59 [ERROR]] Error while processing connection: I/O 
timeout reading #<SB-SYS:FD-STREAM for
"a socket" {13C504E1}>.
[2008-07-04 07:46:28 [INFO]] No session for session identifier 
'478:B00C2E319A15E5658D72E232C36F30DD' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:46:48 [ERROR]] Error while processing connection: I/O 
timeout reading #<SB-SYS:FD-STREAM for
"a socket" {13C740D1}>.
[2008-07-04 07:50:18 [INFO]] No session for session identifier 
'230:85588DC64DDDBF9219B71F4DFA746B50' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:50:38 [ERROR]] Error while processing connection: I/O 
timeout reading #<SB-SYS:FD-STREAM for
"a socket" {13C50F59}>.
[2008-07-04 07:54:08 [INFO]] No session for session identifier 
'145:4C7B146A126FC14ACFC7BFA8100C07D8' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:54:28 [ERROR]] Error while processing connection: I/O 
timeout reading #<SB-SYS:FD-STREAM for
"a socket" {13C84E89}>.
[2008-07-04 07:57:59 [INFO]] No session for session identifier 
'478:B00C2E319A15E5658D72E232C36F30DD' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 07:58:19 [ERROR]] Error while processing connection: I/O 
timeout reading #<SB-SYS:FD-STREAM for
"a socket" {13C5BF61}>.
[2008-07-04 08:01:49 [INFO]] No session for session identifier 
'493:640F2D5F184258D62527F8BD4AA0B91B' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 08:02:09 [ERROR]] Error while processing connection: I/O 
timeout reading #<SB-SYS:FD-STREAM for
"a socket" {13C735C9}>.
[2008-07-04 08:05:40 [INFO]] No session for session identifier 
'308:FA7CD1A4C4592F69DA33BE6826383D46' (User-
Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)', IP: '127.0.0.1')
[2008-07-04 08:06:00 [ERROR]] Error while processing connection: I/O 
timeout reading #<SB-SYS:FD-STREAM for
"a socket" {13C624A9}>.