[elephant-devel] 'not of type FIXNUM' type-error

[Ian Eslick]

Geesh...I chased this down to a design decision inside the SBCL sb- 
alien implementation as I couldn't reproduce this on Allegro.

One design decision in Elephant was to serialize all lisp values to a  
foreign byte array so they can be directly submitted to the Berkeley  
DB internal pages as a single foreign byte array (required by BDB  
API).  As values are serialized, the foreign array is resized to  
accommodate longer and longer values so the total serialized size of  
your object has to not violate any of the various fixnum related  
limits in the system.

It turns out that SBCL has an internal limitation that it computes  
foreign array sizes by first computing the # of bits.  Thus when you  
try to allocate a buffer of 80M bytes, it becomes 640M bytes which is  
greater than the SBCL positive fixnum limit of ~512MB (30-bit  
signed).  Perhaps SBCL 64 doesn't have this same limitation, but when  
you allocate extremely large objects you'll eventually run into these  
kinds of limitations.

Elephant itself is currently limited to 2 billion objects (indexes  
and persistent indexes) but there is no fixed limit on # of values  
other than the ultimate depth of the btree which will exhaust your  
locks on a write.

If you chose to store large arrays in a special index by row you  
should have no problem:

;; psuedo code - you should get the idea
(defun store-large-array (array index)
   (setf (get-value 'columns index) (num-array-columns array))
   (setf (get-value 'rows index) (num-array-rows array))
   (loop for i from 0 upto (num-array-rows array) do
        (setf (get-value i index) (array-row array i))))

(defun load-large-array (index)
    (let* ((columns (get-value 'columns index))
  	  (rows (get-value 'rows index))
           (array (make-array (list rows columns))))
       (loop for i from 0 upto rows do
           (write-row (get-value i index) array))))

Cheers,
Ian


On Feb 14, 2007, at 8:45 AM, Dmitry V. Gorbatovsky wrote:

> Hello , here is debian testing with sbcl 1.02
> and cvs version of elephant. Intel 32-bit machine with
> 1gb of ram.
>
> The code I am try to run is pretty straitforward,
> it runs with 2000x2000 array, but returns type-error
> with anything bigger.
>
> (let ((big-array (make-array '(5000 5000)
>
>                   			:element-type 'float
>                   			:initial-element 0.0)))
>
>   (time (ele:with-open-store
>             (*test-bdb-spec*)
>           (ele:add-to-root 'big-array big-array))))
>
> And here is output from the debugger:
>
> The value 1000000640 is not of type FIXNUM.
>    [Condition of type TYPE-ERROR]
>
> Restarts:
>   0: [ABORT-REQUEST] Abort handling SLIME request.
>   1: [TERMINATE-THREAD] Terminate this thread (#<THREAD  
> "worker" {B2CBE61}>)
>
> Backtrace:
>   0: (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM #<unavailable lambda  
> list>)
>       Locals:
>   1: ((LABELS ELEPHANT-SERIALIZER2::%SERIALIZE) #<unavailable  
> lambda list>)
>       Locals:
>   2: ((LABELS ELEPHANT-SERIALIZER2::%SERIALIZE) #<unavailable  
> lambda list>)
>       Locals:
>   3: (ELEPHANT-SERIALIZER2::SERIALIZE #<unavailable lambda list>)
>       Locals:
>   4: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T DB-BDB::BDB-BTREE))
> #<unavailable lambda list>)
>       Locals:
>   5: ((LAMBDA NIL))
>       Locals:
>   6: (SB-IMPL::%TIME #<CLOSURE (LAMBDA NIL) {A90C005}>)
>       Locals:
>         SB-DEBUG::ARG-0 = #<CLOSURE (LAMBDA NIL) {A90C005}>
>   7: (SB-INT:SIMPLE-EVAL-IN-LEXENV (LET ((BIG-ARRAY #)) (TIME
>  (WITH-OPEN-STORE # #))) #<NULL-LEXENV>)
>       Locals:
>         SB-DEBUG::ARG-0 = (LET ((BIG-ARRAY (MAKE-ARRAY # :ELEMENT-TYPE
> # :INITIAL-ELEMENT 0.0))) (TIME (WITH-OPEN-STORE (*TEST-BDB-SPEC*)
> (ADD-TO-ROOT # BIG-ARRAY))))
>         SB-DEBUG::ARG-1 = #<NULL-LEXENV>
>   8: ((LAMBDA NIL))
>       Locals:
>   9: ((LAMBDA (SWANK-BACKEND::FN)) #<CLOSURE (LAMBDA NIL) {B2CD03D}>)
>
> I would highly appreciate any help.
> Regards, Dmitry
>
> -- 
> Dmitry V. Gorbatovsky
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel