[Ecls-list] Memory problems during IO

[Juan Jose Garcia-Ripoll]

On Tue, Dec 1, 2009 at 4:59 AM, Waldek Hebisch <hebisch at math.uni.wroc.pl> wrote:
> The file has 122 MB, I compiled ECL without Unicode support, so
> the strings should take similare space in memory as in a file.
> So why there is so much consing?  For comparison sbcl needs
> 5.216 sec but conses only 13,360,144 bytes.  And for reading
> sbcl is much faster, 2.559 sec and 492,137,056 bytes consed
> (note that sbcl uses unicode strings, so the amount consed
> is essentially the same as space needed to store strings).

Things SBCL does that we did not:

1- It lets the input buffers grow without bounds. That means the first
time you read one of the very long strings in your example, then the
buffer is large enough not to cons any extra memory.

2- It does not compact strings: the output is always a general string.
ECL takes some extra effort to simplify the string so that it uses
less memory.

3- It has a more compact representation of unicode characters, using less bytes.

4- It open codes the access to ANSI streams.

The 9.12.2 patch for ECL will contain changes for 1 and 2 which
decrease consing, but contrary to what you would expect they do not
change the speed very much.

> (load "iotst")
[...]
writing
real time : 15.008 secs
run time  : 6.002 secs
gc count  : 1 times
consed    : 12309120 bytes
reading
real time : 10.880 secs
run time  : 5.796 secs
gc count  : 13 times
consed    : 470267008 bytes
> (load "iotst")
[...]
writing
real time : 12.731 secs
run time  : 5.972 secs
gc count  : 1 times
consed    : 24986880 bytes
reading
real time : 10.717 secs
run time  : 5.752 secs
gc count  : 4 times
consed    : 492315904 bytes

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com