Re: [flexi-streams-devel] New release 0.11.1

[Vodonosov Anton]

Hi!

Edi, as far as I understand external format 
with multibyte encodings and cr-lf style newlines
are not optimized because its difficult to predict
number of characters that will fit into buffer.

We can solve it if we will always have few reserved
bytes in buffer. 20 will be sufficient for any encoding.
I.e. loop while we have at least 20 free bytes in buffer.

This solution is implemented in the attached patch (against 0.11.2).

The patch also contains some additions in tests. As for
separate tests for READ-SEQUENCE/WRITE-SEQUENCE,
maybe thay are useless - I noticed that, at least in CLISP, 
WRITE-LINE is implemented using WRITE-SEQUENCE. So in 
case of errors in WRITE-SEQUENCE both WRITE-LINE and 
WRITE-SEQUENCE tests fail.

In regard to the way I reused existing WRITE-CHAR code
in STREAM-WRITE-SEQUENCE. I do not like mach working 
throught temporary stream, for example we have redundant 
slot access in WRITE-BYTE*. I've tried to keep changes small 
and not disturb other code. Maybe with some refactoring 
it will be possible to have more clean and efficient code.

For example WRITE-CHAR code will not use WRITE-BYTE* directly
but use instead some BYTE-WRITER-FUN passed as a parameter.
Also external format may be distinguished as a separate 
entity responsible for byte/character conversions.

BTW, I've started changing STREAM-WRITE-SEQUENCE with the 
version provided below. It is more efficient, but it isn't
thread safe.

(defmacro dyna-let-f-global (symbol func-to-bind &body body)
  "Something similar to FLET for global functions but with dynamic extent"
  `(let ((old-fdef (fdefinition ,symbol)))
    (unwind-protect (progn 
              (setf (fdefinition ,symbol) ,func-to-bind)
              , at body)
      (setf (fdefinition ,symbol) old-fdef))))

(defmethod stream-write-sequence ((stream flexi-output-stream) (sequence string) start end &key)
  (declare (optimize speed) (type (integer 0 *) start end fill-pointer src-ptr))
; (declare (optimize speed) (type (integer 0 *) start end))
  (let ((buffer (make-array (+ +buffer-size+ 20) :element-type 'octet)))
    ;; use repeated calls to WRITE-SEQUENCE for arrays of octets
    (loop with src-ptr = start 
      while (< src-ptr end)
      do
      (let ((fill-pointer 0))
        (dyna-let-f-global 'write-byte* (lambda (byte stream) 
                          (declare (ignore stream) (type (integer 0 *) fill-pointer) (type octet byte)) 
                          (setf (aref buffer fill-pointer) byte) (incf fill-pointer))
          (loop while (and (< src-ptr end) (< fill-pointer +buffer-size+))
            do (stream-write-char stream (aref sequence src-ptr)) (incf src-ptr)))
        (write-sequence buffer (flexi-stream-stream stream)
                :start 0 :end fill-pointer))))
  sequence)

Here are some performance tests. 
Tested in CLISP on Windows XP.

(asdf:operate 'asdf:load-op :flexi-streams)
(asdf:oos 'asdf:test-op :flexi-streams)

(defparameter long-str "")
;; populate it with some Russian characters
(dotimes (i 10)  
  (setf long-str 
	(concatenate 'string 
		     long-str 
		     "Достаточно длинная строка на русском языке. 
Ее длина составляет порядка ста символов")))

(defun time-test(external-format)
  (with-open-file (stream "/cygdrive/c/usr/projects/flexi-dev/test-output.txt" 
			  :direction :output 
			  :element-type '(unsigned-byte 8) 
			  :buffered nil) ; :buffered is a CLISP extension
    (with-open-stream (fstream (flex:make-flexi-stream stream 
						       :external-format external-format))
      (loop for i from 0 below 200 do
	    (stream-write-sequence long-str fstream)))))


;; original flexi-streams-0.11.2 
;; =============================

(time (time-test :utf-8))
;; Real time: 12.178 sec.
;; Run time: 12.093 sec.
;; Space: 430660 Bytes
;; GC: 1, GC time: 0.015 sec.

(time (time-test :koi8-r))
;; Real time: 0.246 sec.
;; Run time: 0.25 sec.
;; Space: 1820584 Bytes
;; GC: 2, GC time: 0.048 sec.

;; with temp flexi stream
;; ======================

(time (time-test :utf-8))
;; Real time: 1.08 sec.
;; Run time: 1.079 sec.
;; Space: 1791632 Bytes
;; GC: 2, GC time: 0.063 sec.

(time (time-test :koi8-r))
;; Real time: 0.861 sec.
;; Run time: 0.875 sec.
;; Space: 1795636 Bytes
;; GC: 2, GC time: 0.048 sec.

;; with dyna-let-f-global
;; ======================

(time (time-test :utf-8))
;; Real time: 0.639 sec.
;; Run time: 0.641 sec.
;; Space: 1734832 Bytes
;; GC: 2, GC time: 0.062 sec.

(time (time-test :koi8-r))
;; Real time: 0.567 sec.
;; Run time: 0.563 sec.
;; Space: 1734836 Bytes
;; GC: 2, GC time: 0.062 sec.

Best regards,
-Anton

-----Original Message-----
From: Edi Weitz <edi at agharta.de>
To: flexi-streams-devel at common-lisp.net
Date: Thu, 22 Mar 2007 22:58:27 +0100
Subject: [flexi-streams-devel] New release 0.11.1

> 
> ChangeLog:
> 
>   Version 0.11.1
>   2007-03-22
>   More ugliness for a bit of output performance in special cases
> 
> Download:
> 
>   http://weitz.de/files/flexi-streams.tar.gz
> 
> Cheers,
> Edi.
> _______________________________________________
> flexi-streams-devel mailing list
> flexi-streams-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/flexi-streams-devel
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: flexi-patch.diff
Type: application/octet-stream
Size: 15340 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/flexi-streams-devel/attachments/20070421/d1572c25/attachment.obj>