[Ecls-list] Memory problems during IO

[Waldek Hebisch]

> On Sun, Nov 29, 2009 at 9:50 PM, Waldek Hebisch
> <hebisch at math.uni.wroc.pl> wrote:
> > I have problem reading modertely large (76 MB) input file
> > using FriCAS. ?I am geting ?the following message:
> >
> > Memory limit reached. Please jump to an outer pointer, quit program and
> > enlarge the memory limits before executing the program again.
> >
> > I am using CVS version from about 3 weeks ago. ?The machine is Core
> > 2 with 8 GB of ram running 64-bit Fedora 9.
> 
> The problem is not with I/O, but rather with the fact that you hit the
> memory limits creating a file with about 200Mb of non-eq strings. The
> memory eaten by a 64-bits system in that case is much larger than the
> one required for a 32-bits system. I just increased the limits to 1Gb
> for 32 bits and 4 Gb for 64 bits.
>
 
Thanks for increased limit.  But IO may be also to blame, loading
the test file from previous message a few times I see:

ECL (Embeddable Common-Lisp) 9.12.1
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2000 Juan J. Garcia-Ripoll
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.  
Top level.
> (load "iotst.lisp")

;;; Loading "/mnt/a5/fricas/axp19/pp1/pp12/iotst.lisp"
;;; Loading #P"/mnt/a5/fricas/usrt5/lib/ecl-9.12.1/cmp.fas"
;;; Loading #P"/mnt/a5/fricas/usrt5/lib/ecl-9.12.1/sysfun.lsp"
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=0
;;; End of Pass 1.
;;; Note: Invoking external command:
;;;   gcc "-I/mnt/a5/fricas/usrt5/include/"  -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -g -O2 -fPIC  -Dlinux -O -w -c
"/tmp/ECL001TzwKkp.c" -o "/tmp/ECL001TzwKkp.o"
;;; Note: Invoking external command:
;;;   gcc -o "/tmp/ECL001TzwKkp.fas" -L"/mnt/a5/fricas/usrt5/lib/"
"/tmp/ECL001TzwKkp.o"  "-Wl,--rpath,/mnt/a5/fricas/usrt5/lib/" -shared
-lecl  -lgmp -lgc -ldl  -lm writing
real time : 18.398 secs
run time  : 5.381 secs
gc count  : 1 times
consed    : 20210640 bytes
reading
real time : 6.934 secs
run time  : 6.738 secs
gc count  : 31 times
consed    : 7000862856 bytes
#P"/mnt/a5/fricas/axp19/pp1/pp12/iotst.lisp"
> (load "iotst.lisp")

;;; Loading "/mnt/a5/fricas/axp19/pp1/pp12/iotst.lisp"
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=0
;;; End of Pass 1.
;;; Note: Invoking external command:
;;;   gcc "-I/mnt/a5/fricas/usrt5/include/"  -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -g -O2 -fPIC  -Dlinux -O -w -c
"/tmp/ECL002rcMjSP.c" -o "/tmp/ECL002rcMjSP.o"
;;; Note: Invoking external command:
;;;   gcc -o "/tmp/ECL002rcMjSP.fas" -L"/mnt/a5/fricas/usrt5/lib/"
"/tmp/ECL002rcMjSP.o"  "-Wl,--rpath,/mnt/a5/fricas/usrt5/lib/" -shared
-lecl  -lgmp -lgc -ldl  -lm writing
real time : 5.513 secs
run time  : 5.383 secs
gc count  : 1 times
consed    : 608093808 bytes
reading
real time : 6.902 secs
run time  : 6.719 secs
gc count  : 10 times
consed    : 9296469976 bytes
#P"/mnt/a5/fricas/axp19/pp1/pp12/iotst.lisp"
> (load "iotst.lisp")

;;; Loading "/mnt/a5/fricas/axp19/pp1/pp12/iotst.lisp"
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=0
;;; End of Pass 1.
;;; Note: Invoking external command:
;;;   gcc "-I/mnt/a5/fricas/usrt5/include/"  -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -g -O2 -fPIC  -Dlinux -O -w -c
"/tmp/ECL003U2xdXZ.c" -o "/tmp/ECL003U2xdXZ.o"
;;; Note: Invoking external command:
;;;   gcc -o "/tmp/ECL003U2xdXZ.fas" -L"/mnt/a5/fricas/usrt5/lib/"
"/tmp/ECL003U2xdXZ.o"  "-Wl,--rpath,/mnt/a5/fricas/usrt5/lib/" -shared
-lecl  -lgmp -lgc -ldl  -lm writing
real time : 5.525 secs
run time  : 5.392 secs
gc count  : 1 times
consed    : 1195973600 bytes
reading
real time : 6.864 secs
run time  : 6.758 secs
gc count  : 7 times
consed    : 10723562480 bytes
#P"/mnt/a5/fricas/axp19/pp1/pp12/iotst.lisp"

The file has 122 MB, I compiled ECL without Unicode support, so
the strings should take similare space in memory as in a file.
So why there is so much consing?  For comparison sbcl needs
5.216 sec but conses only 13,360,144 bytes.  And for reading
sbcl is much faster, 2.559 sec and 492,137,056 bytes consed
(note that sbcl uses unicode strings, so the amount consed
is essentially the same as space needed to store strings).

-- 
                              Waldek Hebisch
hebisch at math.uni.wroc.pl