From kcrosbie at ravenpack.com Wed Nov 7 18:39:15 2007 From: kcrosbie at ravenpack.com (Kevin Crosbie) Date: Wed, 07 Nov 2007 19:39:15 +0100 Subject: [drakma-devel] HTTPS via Proxy Server Message-ID: <47320653.3090703@ravenpack.com> Hi, I was wondering how Drakma supports talking to a proxy server when using HTTPS connections. AllegroServe seems to make a HTTP connection to the proxy server then if the proxy server supports SSL Forwarding (For Apache, say, the conf file will read: SSLProxyEngine On) then the proxy server will make your HTTPS connection for you. This however seems to fail when SSLProxyEngine is not set to On. I've seen using WireShark that modern browsers implement a different protocol for making SSL connections via Proxy Servers... ftp://ftp.isi.edu/in-notes/rfc2817.txt Possibly: http://www.ietf.org/rfc/rfc2818.txt It seems that they first send a CONNECT request, asking for the connection to be upgraded to TLS. From then on they negotiate the connection via a tunnel directly to the destination server. So my questions are: 1. Does Drakma support this 2. Does anybody know of any Lisp HTTP Client that does? 3. Does anybody have any suggestions about how to go about doing this? For instance, I could use a Java or C library that supports this and make native calls or I could extend whatever HTTP Client I choose to support this. Best Regards, Kevin Crosbie Ravenpack International S.L. From edi at agharta.de Thu Nov 8 20:12:31 2007 From: edi at agharta.de (Edi Weitz) Date: Thu, 08 Nov 2007 21:12:31 +0100 Subject: [drakma-devel] HTTPS via Proxy Server In-Reply-To: <47320653.3090703@ravenpack.com> (Kevin Crosbie's message of "Wed, 07 Nov 2007 19:39:15 +0100") References: <47320653.3090703@ravenpack.com> Message-ID: On Wed, 07 Nov 2007 19:39:15 +0100, Kevin Crosbie wrote: > 1. Does Drakma support this No, not yet. > 2. Does anybody know of any Lisp HTTP Client that does? I don't. > 3. Does anybody have any suggestions about how to go about doing > this? I'd be happy to accept patches which integrate this cleanly into Drakma. http://weitz.de/patches.html Cheers, Edi. From lispercat at gmail.com Mon Nov 19 03:41:02 2007 From: lispercat at gmail.com (Andrei Stebakov) Date: Sun, 18 Nov 2007 22:41:02 -0500 Subject: [drakma-devel] Strange response message calling http-request Message-ID: Sometimes I get the response like "Your browser sent a request that this server could not understand". I am scratching my head where this response might be coming from. Probably not from the server I am sending a http-request to. >From what I could find on the net, this message might be coming from an Apache server. I am using hunchentoot behind apache mod_proxy, but drakma has nothing to do with it. I'd appreciate any ideas where this message might be coming from or what would be the best way to debug the problem. Thank you, Andrew From lispercat at gmail.com Mon Nov 19 04:22:41 2007 From: lispercat at gmail.com (Andrei Stebakov) Date: Sun, 18 Nov 2007 23:22:41 -0500 Subject: [drakma-devel] Re: Strange response message calling http-request In-Reply-To: References: Message-ID: To be more precise, that's the response (some AkamaiGHost is mentioned here, could it be the culprit?): HTTP/1.0 400 Bad Request Server: AkamaiGHost Mime-Version: 1.0 Content-Type: text/html Content-Length: 215 Expires: Mon, 19 Nov 2007 04:16:16 GMT Date: Mon, 19 Nov 2007 04:16:16 GMT Connection: close " Bad Request

Bad Request

Your browser sent a request that this server could not understand.

Reference #7.ab0dd58.1195445776.0 On Nov 18, 2007 10:41 PM, Andrei Stebakov wrote: > Sometimes I get the response like "Your browser sent a request that > this server could not understand". I am scratching my head where this > response might be coming from. Probably not from the server I am > sending a http-request to. > From what I could find on the net, this message might be coming from > an Apache server. I am using hunchentoot behind apache mod_proxy, but > drakma has nothing to do with it. > I'd appreciate any ideas where this message might be coming from or > what would be the best way to debug the problem. > > Thank you, > Andrew > From lispercat at gmail.com Mon Nov 19 04:53:53 2007 From: lispercat at gmail.com (Andrei Stebakov) Date: Sun, 18 Nov 2007 23:53:53 -0500 Subject: [drakma-devel] Re: Strange response message calling http-request In-Reply-To: References: Message-ID: After some investigation I starting to understand what's probably going on. The server I am sending requests to is hosted by the Akamai service and because my GET request sends a lot of data (I am sending SVG glyphs information) they may consider it as some kind of DoS attack and try to block it. That's just my guessing... Andrew On Nov 18, 2007 11:22 PM, Andrei Stebakov wrote: > To be more precise, that's the response (some AkamaiGHost is mentioned > here, could it be the culprit?): > > HTTP/1.0 400 Bad Request > Server: AkamaiGHost > Mime-Version: 1.0 > Content-Type: text/html > Content-Length: 215 > Expires: Mon, 19 Nov 2007 04:16:16 GMT > Date: Mon, 19 Nov 2007 04:16:16 GMT > Connection: close > > " > Bad Request > >

Bad Request

> Your browser sent a request that this server could not understand.

> Reference #7.ab0dd58.1195445776.0 > > > > > On Nov 18, 2007 10:41 PM, Andrei Stebakov wrote: > > Sometimes I get the response like "Your browser sent a request that > > this server could not understand". I am scratching my head where this > > response might be coming from. Probably not from the server I am > > sending a http-request to. > > From what I could find on the net, this message might be coming from > > an Apache server. I am using hunchentoot behind apache mod_proxy, but > > drakma has nothing to do with it. > > I'd appreciate any ideas where this message might be coming from or > > what would be the best way to debug the problem. > > > > Thank you, > > Andrew > > > From edi at agharta.de Mon Nov 19 07:32:19 2007 From: edi at agharta.de (Edi Weitz) Date: Mon, 19 Nov 2007 08:32:19 +0100 Subject: [drakma-devel] Strange response message calling http-request In-Reply-To: (Andrei Stebakov's message of "Sun, 18 Nov 2007 22:41:02 -0500") References: Message-ID: On Sun, 18 Nov 2007 22:41:02 -0500, "Andrei Stebakov" wrote: > I'd appreciate any ideas where this message might be coming from or > what would be the best way to debug the problem. You should try this as a first step: http://weitz.de/drakma/#*header-stream* If that doesn't help, you'll have to look at tools like Wireshark. This is assuming that you're trying to figure out if Drakma does the right thing. Of course, you can't really debug the server from your Lisp image... :) Edi. From lispercat at gmail.com Tue Nov 20 22:44:25 2007 From: lispercat at gmail.com (Andrei Stebakov) Date: Tue, 20 Nov 2007 17:44:25 -0500 Subject: [drakma-devel] http-request error... In-Reply-To: References: Message-ID: I am using usocket-0.3.5 but if you give the request some fake domain it still has the same exception: The value 0 is not of type (OR (SINGLE-FLOAT (0.0)) (DOUBLE-FLOAT (0.0d0)) (INTEGER 1)). [Condition of type TYPE-ERROR] With the new usocket-3.5 I started to get a new type of error (which happens rarely and I don't know how to reproduce it). It reads: Condition: Couldn't write to #: Broken pipe and it happens right after the drakma:http-request. Is it something related to the server I am trying to connect to? It may be related to drakma or usocket as I changed both of them as well as updated the dramka dependencies like chunga. It those things never happened to you, guys I am starting think that maybe hardware (linksys router or NICs) may cause some problems. Any ideas? Thank you, Andrew On Oct 12, 2007 2:37 AM, Erik Huelsmann wrote: > > On 10/12/07, Andrei Stebakov wrote: > > I run across this error time after time but it's really hard to reproduce > > but I can replicate exactly the same backtrace of the problem if instead of > > some valid http domain I choose any fake one. > > If I run: > > (drakma:http-request > > "http://some-fake-domain.com" > > :method :get > > :parameters `(("param1" . "text"))) > > > > The backtrace will be: > > > > The value 0 > > is not of type > > (OR (SINGLE-FLOAT (0.0)) (DOUBLE-FLOAT (0.0d0)) (INTEGER 1)). > > [Condition of type TYPE-ERROR] > > > > Restarts: > > 0: [ABORT] Return to SLIME's top level. > > 1: [ABORT] Exit debugger, returning to top level. > > > > Backtrace: > > 0: (RANDOM 0) > > 1: (USOCKET::GET-RANDOM-HOST-BY-NAME > > "some-fake-domain.com") > > 2: (USOCKET:SOCKET-CONNECT " some-fake-domain.com" 80) > > 3: (DRAKMA:HTTP-REQUEST "http://some-fake-domain.com") > > 4: (SB-INT:SIMPLE-EVAL-IN-LEXENV > > (DRAKMA:HTTP-REQUEST " http://some-fake-domain.com" :METHOD :GET > > :PARAMETERS '(("param1" . "text"))) > > #) > > 5: ((LAMBDA ())) > > > > In my case it looks like sometimes even for the real domain it calls (RANDOM > > 0) which gives this log. > > Even if the domain is wrong do you think it should throw an exception? > > I use SBCL 1.0.5. with the latest drakma and usocket ( usocket-0.3.2). > > I'll have a look tonight, but I think I fixed this problem in newer > usocket releases already. The newest usocket is 0.3.5. [After looking > at the list of releases at the bottom of > http://common-lisp.net/project/usocket/] I think your problem may have > been fixed in usocket 0.3.3. (But again, I'll look at it later - > tonight) > > bye, > > Erik. > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > From lispercat at gmail.com Sat Nov 24 01:38:49 2007 From: lispercat at gmail.com (Andrei Stebakov) Date: Fri, 23 Nov 2007 20:38:49 -0500 Subject: [drakma-devel] http-request hangs on utf-8 sites Message-ID: Calling (drakma:http-request "http://www.cl.cam.ac.uk/~mgk25/ucs/examples/digraphs.txt") just hangs. Requests to other sites works. I am using drakma-0.10.2, chunga-0.4.0, usocket-0.3.5 and SBCL 1.0.11 on Linux. Sometimes I get "broken pipe" exception, but it may be not relevant to this case. Andrew From lispercat at gmail.com Sat Nov 24 02:05:49 2007 From: lispercat at gmail.com (Andrei Stebakov) Date: Fri, 23 Nov 2007 21:05:49 -0500 Subject: [drakma-devel] Re: http-request hangs on utf-8 sites In-Reply-To: References: Message-ID: Probably it just happens because my terminal can't handle utf-8 itself. Andrew On Nov 23, 2007 8:38 PM, Andrei Stebakov wrote: > Calling (drakma:http-request > "http://www.cl.cam.ac.uk/~mgk25/ucs/examples/digraphs.txt") just > hangs. Requests to other sites works. > I am using drakma-0.10.2, chunga-0.4.0, usocket-0.3.5 and SBCL 1.0.11 on Linux. > Sometimes I get "broken pipe" exception, but it may be not relevant to > this case. > > Andrew > From edi at agharta.de Sat Nov 24 10:34:34 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 24 Nov 2007 11:34:34 +0100 Subject: [drakma-devel] http-request hangs on utf-8 sites In-Reply-To: (Andrei Stebakov's message of "Fri, 23 Nov 2007 20:38:49 -0500") References: Message-ID: On Fri, 23 Nov 2007 20:38:49 -0500, "Andrei Stebakov" wrote: > Calling (drakma:http-request > "http://www.cl.cam.ac.uk/~mgk25/ucs/examples/digraphs.txt") just > hangs. Works for me (LWW 5.0.2). From rsynnott at gmail.com Sat Nov 24 14:38:27 2007 From: rsynnott at gmail.com (Robert Synnott) Date: Sat, 24 Nov 2007 14:38:27 +0000 Subject: [drakma-devel] http-request hangs on utf-8 sites In-Reply-To: References: Message-ID: <24f203480711240638t49d18e49ic6fc8e9fd5f2bd5@mail.gmail.com> On Nov 24, 2007 10:34 AM, Edi Weitz wrote: > On Fri, 23 Nov 2007 20:38:49 -0500, "Andrei Stebakov" wrote: > > > Calling (drakma:http-request > > "http://www.cl.cam.ac.uk/~mgk25/ucs/examples/digraphs.txt") just > > hangs. > If you're using Slime, you'll have to (a) use Emacs22, and (b) add (set-language-environment "UTF-8") (setq slime-net-coding-system 'utf-8-unix) to your .emacs. Rob -- Robert Synnott http://myblog.rsynnott.com MSN: rsynnott at gmail.com Jabber: rsynnott at gmail.com From vilsonv at gmail.com Tue Nov 27 10:48:40 2007 From: vilsonv at gmail.com (Vilson Vieira) Date: Tue, 27 Nov 2007 08:48:40 -0200 Subject: [drakma-devel] Problems with a specific URI Message-ID: <2618ae6e0711270248u3091f64fg8801066d9ed95ef1@mail.gmail.com> Hi, first congratulations about Drakma, Weitz! I'm using it to web crawler at my work and it's running like a charm. Well, I'm getting a strange error when trying to request patents from USPTO website: (http-request "http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=0&f=S&l=50&TERM1=active&FIELD1=&co1=AND&TERM2=noise&FIELD2=&d=PTXT") And the error is something like that: end of file on # [Condition of type END-OF-FILE] Thanks for any suggestions. -- Vilson Vieira vilsonv at gmail.com From rosssd at gmail.com Tue Nov 27 13:30:19 2007 From: rosssd at gmail.com (Sean Ross) Date: Tue, 27 Nov 2007 13:30:19 +0000 Subject: [drakma-devel] Problems with a specific URI In-Reply-To: <2618ae6e0711270248u3091f64fg8801066d9ed95ef1@mail.gmail.com> References: <2618ae6e0711270248u3091f64fg8801066d9ed95ef1@mail.gmail.com> Message-ID: <5bef28df0711270530r3f2fa63m233dc34173f08a75@mail.gmail.com> On 11/27/07, Vilson Vieira wrote: > end of file on # > [Condition of type END-OF-FILE] An EOF error is generally a symptom of invalid EOL's from the server. You can work around this by binding CHUNGA:*ACCEPT-BOGUS-EOLS* to T eg. (let ((chunga:*accept-bogus-eols* t)) (drakma:http-request *aforementioned-url*)) sean . From vilsonv at gmail.com Tue Nov 27 13:55:03 2007 From: vilsonv at gmail.com (Vilson Vieira) Date: Tue, 27 Nov 2007 11:55:03 -0200 Subject: [drakma-devel] Problems with a specific URI In-Reply-To: <5bef28df0711270530r3f2fa63m233dc34173f08a75@mail.gmail.com> References: <2618ae6e0711270248u3091f64fg8801066d9ed95ef1@mail.gmail.com> <5bef28df0711270530r3f2fa63m233dc34173f08a75@mail.gmail.com> Message-ID: <2618ae6e0711270555s625451e4l467f853d76bcf73f@mail.gmail.com> 2007/11/27, Sean Ross : > An EOF error is generally a symptom of invalid EOL's from the server. > You can work around this by binding CHUNGA:*ACCEPT-BOGUS-EOLS* to T Now it runs perfect Sean! Thanks a lot! -- Vilson Vieira vilsonv at gmail.com