From asf at boinkor.net Tue Mar 7 23:43:12 2006 From: asf at boinkor.net (Andreas Fuchs) Date: Wed, 08 Mar 2006 00:43:12 +0100 Subject: [flexi-streams-devel] in-memory streams have no way to set external-format? Message-ID: <87r75dhldb.wl%asf@boinkor.net> Hi all, I would like to teach cl-irc about the goodness of external-formats, and because irc doesn't let users specify which external format to use, I need something as flexible as flexi-streams (-: So, once I've read an irc command (as a "line" of latin-1 characters), I'd like to convert it to text that looks the least insane, which is why my code tries several external-formats in a row and returns as soon as it found one that doesn't throw an error: (defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (flexi-streams:with-input-from-sequence (in line) (setf (flexi-streams:flexi-stream-external-format in) external-format) (read-line in)))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded))) Try it with: (try-decode-line "foo" '((:UTF-8 :EOL-STYLE :LF) (:LATIN-1 :EOL-STYLE :LF))) (alternatively, (map 'vector #'char-code "foo") and with (list (make-external-format...)) calls) And for each of the external formats, I get: There is no applicable method for the generic function # when called with arguments ((:UTF-8 :EOL-STYLE :LF) #). Hrmpf! Am I abusing flexi-streams too much or is that a bug? How should one read externally-formatted data from an in-memory stream, anyway? And are string-backed in-memory streams even allowed? How to specify the internal external format for them? (-: Thanks for your time and for developing flexi-streams. In return, I hope to be able to buy you a beverage of your choice in Hamburg (-: Cheers, -- Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs From edi at agharta.de Wed Mar 8 00:48:52 2006 From: edi at agharta.de (Edi Weitz) Date: Wed, 08 Mar 2006 01:48:52 +0100 Subject: [flexi-streams-devel] New version 0.5.3 (Was: in-memory streams have no way to set external-format?) In-Reply-To: <87r75dhldb.wl%asf@boinkor.net> (Andreas Fuchs's message of "Wed, 08 Mar 2006 00:43:12 +0100") References: <87r75dhldb.wl%asf@boinkor.net> Message-ID: Hi! On Wed, 08 Mar 2006 00:43:12 +0100, Andreas Fuchs wrote: > I would like to teach cl-irc about the goodness of external-formats, > and because irc doesn't let users specify which external format to > use, I need something as flexible as flexi-streams (-: > > So, once I've read an irc command (as a "line" of latin-1 > characters), I'd like to convert it to text that looks the least > insane, which is why my code tries several external-formats in a row > and returns as soon as it found one that doesn't throw an error: > > (defun try-decode-line (line external-formats) > (loop for external-format in external-formats > for decoded = nil > for error = nil > do (multiple-value-setq (decoded error) > (ignore-errors > (flexi-streams:with-input-from-sequence (in line) > (setf (flexi-streams:flexi-stream-external-format in) > external-format) > (read-line in)))) > do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded > error) > if decoded > do (return decoded))) If LINE is a string you want this: (defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (with-input-from-string (in line) (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) (read-line flexi))))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded))) But actually I think you want LINE to be a sequence of octets, so this is what you want: (defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (flexi-streams:with-input-from-sequence (in line) (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) (read-line flexi))))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded))) Result: CL-USER 11 > (try-decode-line '(228 246 252) '((:utf-8 :eol-style :lf) (:latin-1 :eol-style :lf))) tried (:UTF-8 :EOL-STYLE :LF): NIL error: Unexpected value #xF6 in UTF-8 sequence. tried (:LATIN-1 :EOL-STYLE :LF): "???" error: T "???" But you'll need version 0.5.3 to see that because there was a typo in the code which generated the error messages. > Hrmpf! Am I abusing flexi-streams too much or is that a bug? How > should one read externally-formatted data from an in-memory stream, > anyway? You forgot that you have to create a flexi stream first - in-memory streams happen to be provided by the same library but they're something different. Use MAKE-FLEXI-STREAM to turn them into flexi streams. > And are string-backed in-memory streams even allowed? No. CL already has WITH-INPUT-FROM-STRING... :) > Thanks for your time and for developing flexi-streams. In return, I > hope to be able to buy you a beverage of your choice in Hamburg (-: Nice. Looking forward to seeing you there! Cheers, Edi. From asf at boinkor.net Wed Mar 8 14:01:39 2006 From: asf at boinkor.net (Andreas Fuchs) Date: Wed, 08 Mar 2006 15:01:39 +0100 Subject: [flexi-streams-devel] Re: New version 0.5.3 (Was: in-memory streams have no way to set external-format?) In-Reply-To: References: <87r75dhldb.wl%asf@boinkor.net> Message-ID: <87k6b5ghmk.wl%asf@boinkor.net> Today, Edi Weitz wrote: > Hi! Hi there! > On Wed, 08 Mar 2006 00:43:12 +0100, Andreas Fuchs > wrote: >> So, once I've read an irc command (as a "line" of latin-1 >> characters), I'd like to convert it to text that looks the least >> insane, which is why my code tries several external-formats in a >> row and returns as soon as it found one that doesn't throw an >> error: > > If LINE is a string you want this: > > (defun try-decode-line (line external-formats) > (loop for external-format in external-formats > for decoded = nil > for error = nil > do (multiple-value-setq (decoded error) > (ignore-errors > (with-input-from-string (in line) > (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) > (read-line flexi))))) > do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded > error) > if decoded > do (return decoded))) This didn't exactly work; I get errors stating that the in-memory stream isn't a binary stream. > But actually I think you want LINE to be a sequence of octets, so > this is what you want: And this does work perfectly. Thanks so much! (: My current scheme is (horrible and) as follows: I read a line in the latin-1 external format, then create a vector from the char-codes, and decode that with try-decode-line. Why latin-1, you ask? Because it has a 1:1 code point mapping to bytes, as far as irc is concerned, and I don't have to implement my own buffering, which would really suck - IRC operates on lines, whereas all binary streams need fixed-width buffers. 54It seems like the easiest way to interoperate with broken protocols is to break a little inside, just like their authors once did. (: > But you'll need version 0.5.3 to see that because there was a typo > in the code which generated the error messages. Heh. I just upgraded. >> Hrmpf! Am I abusing flexi-streams too much or is that a bug? How >> should one read externally-formatted data from an in-memory stream, >> anyway? > > You forgot that you have to create a flexi stream first - in-memory > streams happen to be provided by the same library but they're > something different. Use MAKE-FLEXI-STREAM to turn them into > flexi streams. Oh. I misread the phrase 'These streams can obviously be used as the underlying streams for flexi streams.' as 'They are the base class of flexi-streams, and they work just like flexi-streams do.' -- I blame late-night hacking. (: >> And are string-backed in-memory streams even allowed? > > No. CL already has WITH-INPUT-FROM-STRING... :) > >> Thanks for your time and for developing flexi-streams. In return, I >> hope to be able to buy you a beverage of your choice in Hamburg (-: > > Nice. Looking forward to seeing you there! Me too (: Thanks again, -- Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs From edi at agharta.de Wed Mar 8 14:06:04 2006 From: edi at agharta.de (Edi Weitz) Date: Wed, 08 Mar 2006 15:06:04 +0100 Subject: [flexi-streams-devel] Re: New version 0.5.3 In-Reply-To: <87k6b5ghmk.wl%asf@boinkor.net> (Andreas Fuchs's message of "Wed, 08 Mar 2006 15:01:39 +0100") References: <87r75dhldb.wl%asf@boinkor.net> <87k6b5ghmk.wl%asf@boinkor.net> Message-ID: On Wed, 08 Mar 2006 15:01:39 +0100, Andreas Fuchs wrote: > This didn't exactly work; I get errors stating that the in-memory > stream isn't a binary stream. OK. I only tested on LispWorks. But as we know now this wasn't what you wanted anyway... :) > My current scheme is (horrible and) as follows: I read a line in the > latin-1 external format, then create a vector from the char-codes, > and decode that with try-decode-line. > > Why latin-1, you ask? Because it has a 1:1 code point mapping to > bytes, as far as irc is concerned, and I don't have to implement my > own buffering, which would really suck - IRC operates on lines, > whereas all binary streams need fixed-width buffers. Seems reasonable, although not cute... :)