[flexi-streams-devel] Re: note on flexi-streams

[rpgoldman at real-time.com]

>>>>> "EW" == Edi Weitz <edi at agharta.de> writes:

[...snip...]

    EW> On Thu, 1 Jun 2006 14:35:26 -0500, "Robert P. Goldman"
    EW> <rpgoldman at sift.info> wrote:

    >> I just pulled a copy of flexi-streams, using asdf-install, and
    >> had a couple of minor problems:
    >>

[...snip...]

    >> 2.  Making a flexible stream with an external format blows up
    >>     for me
    >> on ACL.  E.g., cl-irc tries to make a flexi-io-stream with
    >> external format (:utf-8 :eof-type :crlf)
    >>
    >> This value is stuffed into the flexi-stream slot
    >> EXTERNAL-FORMAT.  Unfortunately, IIUC, this also triggers an
    >> Allegro-specific method as follows:
    >>
    >> (METHOD (SETF STREAM-EXTERNAL-FORMAT) :BEFORE (T STREAM))
    >>
    >> This method checks to make sure that the value you are setting
    >> as a stream-external-format is an excl:external-format object.
    >> In this case, it clearly isn't, so the :before method hurls an
    >> error.
    >>
    >> I believe that this error is triggered because a
    >> flexi-io-stream ISA fundamental-binary-input-stream which ISA
    >> stream.
    >>
    >> I confess I'm not really sure what to do about this problem....

    EW> Hmm, at first glance I'd say that this is either an AllegroCL
    EW> bug or it is not related to FLEXI-STREAMS at all.  The slot
    EW> you're talking about above is called
    EW> FLEXI-STREAMS::EXTERNAL-FORMAT, its name is not exported from
    EW> the package.  I can't see why setting the value of this slot
    EW> should trigger an Allegro-specific method.

    EW> (Note that on AllegroCL EXCL:EXTERNAL-FORMAT is accessible in
    EW> various packages, including CL-USER.  Maybe you're seeing a
    EW> package conflict?)

The problem is not the FLEXI-STREAMS::EXTERNAL-FORMAT symbol, but the
use of :external-format as an initarg, IIUC.  See the (relatively)
simple test case below for more details.

    EW> It'd be nice if you could provide a simple test case which
    EW> provokes this error message, so we can see what's really going
    EW> on.

OK.  Understand that I don't really know how flexi-streams are
supposed to work --- I just stumbled into this because I use BEIRC,
which uses CL-IRC, which (newly) uses FLEXI-STREAMS.  So I'm pretty
far from understanding things, and some of the things that I give may
not be bugs; they may be me using the package wrong.  

My first shot at causing an error is the following:

(defpackage :flexi-test (:use :common-lisp :flex))
(in-package :flexi-test)
(setf foo (open "/tmp/foo" :direction :io))
(setf bar (make-flexi-stream foo))
Error: :DEFAULT is not known to be a name for an external format.

Restart actions (select using :continue):
 0: Return to Top Level (an "abort" restart).
 1: Abort entirely from this (lisp) process.
[1] FLEXI-TEST(18): :bt
Evaluation stack:

FLEX::NORMALIZE-EXTERNAL-FORMAT-NAME <-
  FLEX::MAKE-EXTERNAL-FORMAT% <- MAKE-EXTERNAL-FORMAT <-
  FLEX::MAYBE-CONVERT-EXTERNAL-FORMAT <-
  (METHOD INITIALIZE-INSTANCE :AFTER ...) <-
  (:INTERNAL (:EFFECTIVE-METHOD 1 T ...) 0) <-
  (METHOD MAKE-INSTANCE (CLASS)) <- (METHOD MAKE-INSTANCE (SYMBOL)) <-
  MAKE-FLEXI-STREAM <- [... EXCL::%EVAL ] <- LET* <- [... EXCL::%EVAL ] <-
  EVAL <- TPL:TOP-LEVEL-READ-EVAL-PRINT-LOOP <- TPL:START-INTERACTIVE-TOP-LEVEL

[I'm honestly not sure where the :default value comes from, and it may
be that I am simply committing an error here and that the
:external-format keyword is mandatory, rather than optional, as I have
treated it.]

So I tried to do what cl-irc does:

(defun external-format-fixup (format)
  (let ((new-format (copy-list format)))
    (setf (getf (cdr new-format) :eol-style) :crlf)
    new-format))
(defun mock-connect (&key (server "irc.freenode.net")
					  (port 6667)
					  (connection-type 'connection)
                     (logging-stream t))
		  (let* ((stream (socket-connect server port))
			 (connection (make-connection :connection-type connection-type
                                      :network-stream stream
                                      :client-stream logging-stream
                                      :server-name server)))
		    (values connection stream)))
(defun make-connection (&key (connection-type 'connection)
                             (user nil)
                             (password nil)
                             (server-name "")
                             (server-port nil)
                             (network-stream nil)
                             (outgoing-external-format *default-outgoing-external-format*)
                             (client-stream t)
                             (hooks nil))
  (let ((output-stream (flexi-streams:make-flexi-stream
                         network-stream
                         :element-type 'character
                         :external-format (external-format-fixup outgoing-external-format))))
	output-stream))
;; overlooked this dependency....	
(asdf:oos 'asdf:load-op :trivial-sockets)
(defun socket-connect (server port)
  "Create a socket connected to `server':`port' and return stream for it."
  (trivial-sockets:open-stream server port :element-type '(unsigned-byte 8)))    
(setf *default-outgoing-external-format* '(:utf8))
That gets me the error:
FLEXI-TEST(62): (mock-connect)
Error: `:EOL-STYLE' does not name an external-format.
  [condition type: NO-EXTERNAL-FORMAT-ERROR]

Restart actions (select using :continue):
 0: Return to Top Level (an "abort" restart).
 1: Abort entirely from this (lisp) process.
[1] FLEXI-TEST(63): :bt
Evaluation stack:

(METHOD (SETF STREAM-EXTERNAL-FORMAT) :BEFORE ...) <-
  (:INTERNAL (:EFFECTIVE-METHOD 2 NIL ...) 0) <-
  (:INTERNAL (:EFFECTIVE-METHOD 2 T ...) 0) <-
  (METHOD INITIALIZE-INSTANCE (STANDARD-OBJECT)) <-
  (:INTERNAL (:EFFECTIVE-METHOD 1 T ...) 0) <-
  (METHOD MAKE-INSTANCE (CLASS)) <- (METHOD MAKE-INSTANCE (SYMBOL)) <-
  MAKE-FLEXI-STREAM <- LET <- MAKE-CONNECTION <- LET* <- MOCK-CONNECT <-
  EVAL <- TPL:TOP-LEVEL-READ-EVAL-PRINT-LOOP <- TPL:START-INTERACTIVE-TOP-LEVEL

The error happens when making the flexi-io-stream class.  It triggers
this ACL :before method that checks the value of the
stream-external-format, and which seems to be seeing the 
'(:utf8 :eol-style :crlf) value that cl-irc is stuffing into the
flexi-stream constructor.

ACL has a method that evidently checks the assignment to
stream-external-format for every stream to make sure that it is a
legitimate excl-external-format, IIUC.  A flexi-stream ISA stream, so
somehow this is triggered.  Your email seems to suggest that you did
NOT expect the flexi-stream-value to become the
stream-external-format, right?  But this is the make-instance call:

((METHOD MAKE-INSTANCE (CLASS)) #<STANDARD-CLASS FLEXI-IO-STREAM> :STREAM
 #<MULTIVALENT stream socket connected from 192.168.1.18/41332 to
   kornbluth.freenode.net/6667 @ #x71dc7412>
 :ELEMENT-TYPE CHARACTER :EXTERNAL-FORMAT (:UTF8 :EOL-STYLE :CRLF))

So even though flexi-streams::external-format is not exported,
flexi-streams is using the :external-format keyword in its
make-instance call, and that is triggering the ACL code.  Would it be
enough to fix things to change the :initarg to
":flexi-stream-external-format"?  That seems cumbersome, though.  I
don't know another way to keep ACL from grabbing up the
:external-format keyword arg, though...  But again, I have not thought
deeply about this.