From ctdean at sokitomi.com  Wed Jan 17 00:45:03 2007
From: ctdean at sokitomi.com (Chris Dean)
Date: Tue, 16 Jan 2007 16:45:03 -0800
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
Message-ID: <m2lkk2iir4.fsf@tungsten.kyluka.com>


I have an application where I need to convert an arbitrary Lisp object
into a C function that expects an array of characters.  What I do now
is use flexi-streams to convert to a Lisp vector, and then use some
FFI code to copy that vector to C array.  (The C code is the Berkeley
DB library.)

The flexi-streams form is called WITH-OUTPUT-TO-SEQUENCE, so the code
is very simple:

    (defun to-storage-vector (obj)
      (with-output-to-sequence (out) 
        (cl-store:store obj out)))

    (to-storage-vector 123) => #(67 108 115 84 24 0 1 123)  

Is there a way to do this same sort of operation using cl-store
directly and bypassing WITH-OUTPUT-TO-SEQUENCE ?  

Cheers,
Chris Dean


From rosssd at gmail.com  Wed Jan 17 09:54:21 2007
From: rosssd at gmail.com (Sean Ross)
Date: Wed, 17 Jan 2007 09:54:21 +0000
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
In-Reply-To: <m2lkk2iir4.fsf@tungsten.kyluka.com>
References: <m2lkk2iir4.fsf@tungsten.kyluka.com>
Message-ID: <5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>

On 1/17/07, Chris Dean <ctdean at sokitomi.com> wrote:
> The flexi-streams form is called WITH-OUTPUT-TO-SEQUENCE, so the code
> is very simple:
>
>     (defun to-storage-vector (obj)
>       (with-output-to-sequence (out)
>         (cl-store:store obj out)))
>
>     (to-storage-vector 123) => #(67 108 115 84 24 0 1 123)
>
> Is there a way to do this same sort of operation using cl-store
> directly and bypassing WITH-OUTPUT-TO-SEQUENCE ?

I'm afraid not, cl-store just knows how to serialize objects to streams.
Are you looking to bypass WITH-OUTPUT-TO-SEQUENCE
for performance reasons or aesthetic ones?

Sean.


From ctdean at sokitomi.com  Wed Jan 17 19:01:07 2007
From: ctdean at sokitomi.com (Chris Dean)
Date: Wed, 17 Jan 2007 11:01:07 -0800
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
In-Reply-To: <5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>
	(Sean Ross's message of "Wed, 17 Jan 2007 09:54:21 +0000")
References: <m2lkk2iir4.fsf@tungsten.kyluka.com>
	<5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>
Message-ID: <m2d55dfpfw.fsf@tungsten.kyluka.com>


"Sean Ross" <rosssd at gmail.com> writes:
> I'm afraid not, cl-store just knows how to serialize objects to
> streams.  Are you looking to bypass WITH-OUTPUT-TO-SEQUENCE for
> performance reasons or aesthetic ones?

For performance reasons (in general I think that flexi-streams is very
nice).  I'm creating many intermediate objects that need to be garbage
collected and I'd like to reduce the amount of garbage.

Cheers,
Chris Dean


From rosssd at gmail.com  Thu Jan 18 10:32:08 2007
From: rosssd at gmail.com (Sean Ross)
Date: Thu, 18 Jan 2007 10:32:08 +0000
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
In-Reply-To: <m2d55dfpfw.fsf@tungsten.kyluka.com>
References: <m2lkk2iir4.fsf@tungsten.kyluka.com>
	<5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>
	<m2d55dfpfw.fsf@tungsten.kyluka.com>
Message-ID: <5bef28df0701180232p25ca38ebw1796c4fd7e58d84d@mail.gmail.com>

On 1/17/07, Chris Dean <ctdean at sokitomi.com> wrote:
> For performance reasons (in general I think that flexi-streams is very
> nice).  I'm creating many intermediate objects that need to be garbage
> collected and I'd like to reduce the amount of garbage.

I'm not so sure that flexi-streams creating the garbage.

Currently cl-store is pretty suboptimal when it comes to serializing a large
number of values one after the other, this is probably something that should be
looked at. Currently a new hash-table is created on every call to
cl-store:store
which is used to track which objects have been serialized already.

If you are very sure that you aren't going to be serializing circular
objects then
you can bind cl-store:*check-for-circs* to nil which will prevent said
hash-tables
from being created.

cl-store could do with some optimization re consing.

Cheers,
 Sean.


From ml13 at onlinehome.de  Sun Jan 21 10:49:56 2007
From: ml13 at onlinehome.de (Kilian Sprotte)
Date: Sun, 21 Jan 2007 11:49:56 +0100
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
In-Reply-To: <m2d55dfpfw.fsf@tungsten.kyluka.com>
References: <m2lkk2iir4.fsf@tungsten.kyluka.com>
	<5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>
	<m2d55dfpfw.fsf@tungsten.kyluka.com>
Message-ID: <9117734F-D75A-43C9-9DC1-5C74780C001E@onlinehome.de>

Hi Chris,

I am almost sure you have already seen this, but just in case:

You took a look at http://common-lisp.net/project/bdb?

I think there is quite some infrastructure for what you want to do,
so it might be of interest (I did not take a very close look, however).

Finally it boils down to:
(setf (symbol-function 'buf-writer)
       (make-cbuffer-writer #'cl-store:store))

Cheers,
   Kilian

Am 17.01.2007 um 20:01 schrieb Chris Dean:

>
> "Sean Ross" <rosssd at gmail.com> writes:
>> I'm afraid not, cl-store just knows how to serialize objects to
>> streams.  Are you looking to bypass WITH-OUTPUT-TO-SEQUENCE for
>> performance reasons or aesthetic ones?
>
> For performance reasons (in general I think that flexi-streams is very
> nice).  I'm creating many intermediate objects that need to be garbage
> collected and I'd like to reduce the amount of garbage.
>
> Cheers,
> Chris Dean
> _______________________________________________
> cl-store-devel mailing list
> cl-store-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/cl-store-devel



From ml13 at onlinehome.de  Sun Jan 21 10:51:56 2007
From: ml13 at onlinehome.de (Kilian Sprotte)
Date: Sun, 21 Jan 2007 11:51:56 +0100
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
In-Reply-To: <5bef28df0701180232p25ca38ebw1796c4fd7e58d84d@mail.gmail.com>
References: <m2lkk2iir4.fsf@tungsten.kyluka.com>
	<5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>
	<m2d55dfpfw.fsf@tungsten.kyluka.com>
	<5bef28df0701180232p25ca38ebw1796c4fd7e58d84d@mail.gmail.com>
Message-ID: <B8D50A0F-2072-42BF-879E-2B1C4227A289@onlinehome.de>


Am 18.01.2007 um 11:32 schrieb Sean Ross:
> Currently cl-store is pretty suboptimal when it comes to  
> serializing a large
> number of values one after the other, this is probably something  
> that should be
> looked at. Currently a new hash-table is created on every call to
> cl-store:store
> which is used to track which objects have been serialized already.

A little question: Is it safer to create a new hash-table, than doing  
clrhash ?

_k


From ml13 at onlinehome.de  Sun Jan 21 11:00:05 2007
From: ml13 at onlinehome.de (Kilian Sprotte)
Date: Sun, 21 Jan 2007 12:00:05 +0100
Subject: [cl-store-devel] cl-clone ?
Message-ID: <8F4CBAB2-49D1-45D0-83A3-00797AB31452@onlinehome.de>

Hi again,

this is really interesting me, so lots of mails... :)

I just wanted to tell you that cl-store is not only important for me  
to store data, but also to implement a deep copying, which I find  
extremely useful.

Before flexi-streams existed, I had to use a file buffer, but now I  
am using in-memory buffers as well, although it seems to be only  
worthwhile for little data.

Of course a deep equal can be done as well and probably some other  
things...

What do you think, would it be worth to add a lib cl-clone on cl.net?
(I could provide an /initial/ codebase... :)

Cheers,
   Kilian




From gwking at metabang.com  Sun Jan 21 13:27:55 2007
From: gwking at metabang.com (Gary King)
Date: Sun, 21 Jan 2007 08:27:55 -0500
Subject: [cl-store-devel] cl-clone ?
In-Reply-To: <8F4CBAB2-49D1-45D0-83A3-00797AB31452@onlinehome.de>
References: <8F4CBAB2-49D1-45D0-83A3-00797AB31452@onlinehome.de>
Message-ID: <1DB68203-1F69-40D4-AA05-4AF571A3D76A@metabang.com>

Hi Kilian,

There is metacopy at common-lisp.net; I expect that the cl-clone  
solution is 'deeper' and perhaps more useful / extensible but, in the  
interest of shared interests <smile>, perhaps you and I could merge  
your code into metabang and make it bigger and better rather than  
having two similar yet dissimilar projects...

whatdayathink?

On Jan 21, 2007, at 6:00 AM, Kilian Sprotte wrote:

> Hi again,
>
> this is really interesting me, so lots of mails... :)
>
> I just wanted to tell you that cl-store is not only important for  
> me to store data, but also to implement a deep copying, which I  
> find extremely useful.
>
> Before flexi-streams existed, I had to use a file buffer, but now I  
> am using in-memory buffers as well, although it seems to be only  
> worthwhile for little data.
>
> Of course a deep equal can be done as well and probably some other  
> things...
>
> What do you think, would it be worth to add a lib cl-clone on cl.net?
> (I could provide an /initial/ codebase... :)
>
> Cheers,
>   Kilian
>
>
> _______________________________________________
> cl-store-devel mailing list
> cl-store-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/cl-store-devel

--
Gary Warren King, metabang.com
Cell: (413) 885 9127
Fax: (206) 338-4052
gwkkwg on Skype * garethsan on AIM






From ctdean at sokitomi.com  Sun Jan 21 19:41:41 2007
From: ctdean at sokitomi.com (Chris Dean)
Date: Sun, 21 Jan 2007 11:41:41 -0800
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
In-Reply-To: <9117734F-D75A-43C9-9DC1-5C74780C001E@onlinehome.de> (Kilian
	Sprotte's message of "Sun, 21 Jan 2007 11:49:56 +0100")
References: <m2lkk2iir4.fsf@tungsten.kyluka.com>
	<5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>
	<m2d55dfpfw.fsf@tungsten.kyluka.com>
	<9117734F-D75A-43C9-9DC1-5C74780C001E@onlinehome.de>
Message-ID: <m2vej0upze.fsf@jacaranda.local>


Kilian Sprotte <ml13 at onlinehome.de> writes:
> I am almost sure you have already seen this, but just in case:
>
> You took a look at http://common-lisp.net/project/bdb?

Thanks, I'll take a look.

Cheers,
Chris Dean



From ctdean at sokitomi.com  Sun Jan 21 19:50:56 2007
From: ctdean at sokitomi.com (Chris Dean)
Date: Sun, 21 Jan 2007 11:50:56 -0800
Subject: [cl-store-devel] cl-clone ?
In-Reply-To: <8F4CBAB2-49D1-45D0-83A3-00797AB31452@onlinehome.de> (Kilian
	Sprotte's message of "Sun, 21 Jan 2007 12:00:05 +0100")
References: <8F4CBAB2-49D1-45D0-83A3-00797AB31452@onlinehome.de>
Message-ID: <m2k5zgupjz.fsf@jacaranda.local>


Kilian Sprotte <ml13 at onlinehome.de> writes:
> Of course a deep equal can be done as well and probably some other
> things...

I have some internal code that I use to do deep equal and deep hash.
The difficulty comes in walking complex objects in a portable way,
which is easy for me since I'm all LispWorks all the time.

    (defgeneric object-equal? (a b))

    (defmethod object-equal? ((a null) (b null))
      t)

    (defmethod object-equal? ((a symbol) (b symbol))
      (eq a b))

    (defmethod object-equal? ((a number) (b number))
      (= a b))

    ;; etc ..

The object-hash method is similar.

Cheers,
Chris Dean



From rosssd at gmail.com  Mon Jan 22 09:21:19 2007
From: rosssd at gmail.com (Sean Ross)
Date: Mon, 22 Jan 2007 09:21:19 +0000
Subject: [cl-store-devel] cl-clone ?
In-Reply-To: <1DB68203-1F69-40D4-AA05-4AF571A3D76A@metabang.com>
References: <8F4CBAB2-49D1-45D0-83A3-00797AB31452@onlinehome.de>
	<1DB68203-1F69-40D4-AA05-4AF571A3D76A@metabang.com>
Message-ID: <5bef28df0701220121k726c9410u349a5cc74d173b48@mail.gmail.com>

On 1/21/07, Gary King <gwking at metabang.com> wrote:
> Hi Kilian,
>
> There is metacopy at common-lisp.net; I expect that the cl-clone
> solution is 'deeper' and perhaps more useful / extensible but, in the
> interest of shared interests <smile>, perhaps you and I could merge
> your code into metabang and make it bigger and better rather than
> having two similar yet dissimilar projects...

I'd say, given that metacopy exists, it would be a better idea
to contribute to an existing code base rather than creating a new
project for it.

#.sean


From ch-lisp at bobobeach.com  Thu Jan 25 18:30:07 2007
From: ch-lisp at bobobeach.com (Cyrus Harmon)
Date: Thu, 25 Jan 2007 10:30:07 -0800
Subject: [cl-store-devel] problems storing big integers on x86-64
Message-ID: <5A95FB98-ECE4-4787-A8A7-2FAF38C2EE0E@bobobeach.com>


On x86 (sbcl/macos),

(cl-store:store #x100000000 #p"/tmp/lame")

works fine, but on x86-64 (sbcl/linux),

(cl-store:store #x100000000 #p"/tmp/lame")

fails with an error message of "The value 4294967296 is not of type  
(UNSIGNED-BYTE 32)" in cl-store::store-arbitrary-integer.

I'm trying to dig into store-arbitrary-integer to see where this is  
going wrong.

Thanks,

Cyrus
  
   


From rosssd at gmail.com  Fri Jan 26 10:21:00 2007
From: rosssd at gmail.com (Sean Ross)
Date: Fri, 26 Jan 2007 10:21:00 +0000
Subject: [cl-store-devel] problems storing big integers on x86-64
In-Reply-To: <5A95FB98-ECE4-4787-A8A7-2FAF38C2EE0E@bobobeach.com>
References: <5A95FB98-ECE4-4787-A8A7-2FAF38C2EE0E@bobobeach.com>
Message-ID: <5bef28df0701260221p30b83dd4wcf0f865e8939b3df@mail.gmail.com>

could you send me a backtrace of this.

On 1/25/07, Cyrus Harmon <ch-lisp at bobobeach.com> wrote:
>
> On x86 (sbcl/macos),
>
> (cl-store:store #x100000000 #p"/tmp/lame")
>
> works fine, but on x86-64 (sbcl/linux),
>
> (cl-store:store #x100000000 #p"/tmp/lame")
>
> fails with an error message of "The value 4294967296 is not of type
> (UNSIGNED-BYTE 32)" in cl-store::store-arbitrary-integer.
>
> I'm trying to dig into store-arbitrary-integer to see where this is
> going wrong.
>
> Thanks,
>
> Cyrus
>
>
> _______________________________________________
> cl-store-devel mailing list
> cl-store-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/cl-store-devel
>


From rosssd at gmail.com  Fri Jan 26 15:02:44 2007
From: rosssd at gmail.com (Sean Ross)
Date: Fri, 26 Jan 2007 15:02:44 +0000
Subject: [cl-store-devel] problems storing big integers on x86-64
In-Reply-To: <5A95FB98-ECE4-4787-A8A7-2FAF38C2EE0E@bobobeach.com>
References: <5A95FB98-ECE4-4787-A8A7-2FAF38C2EE0E@bobobeach.com>
Message-ID: <5bef28df0701260702i75d89681i405b05a493b59cae@mail.gmail.com>

> fails with an error message of "The value 4294967296 is not of type
> (UNSIGNED-BYTE 32)" in cl-store::store-arbitrary-integer.

As it turns out store-arbitrary-integer was quite broken, it would quite
happily pass non ub32 integers along. I don't know how it
ever actually worked, sorry about that.

I've just committed a fix for this and added it to the tests.

Thanks for the report.

#.sean.


From rosssd at gmail.com  Fri Jan 26 15:17:39 2007
From: rosssd at gmail.com (Sean Ross)
Date: Fri, 26 Jan 2007 15:17:39 +0000
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
In-Reply-To: <B8D50A0F-2072-42BF-879E-2B1C4227A289@onlinehome.de>
References: <m2lkk2iir4.fsf@tungsten.kyluka.com>
	<5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>
	<m2d55dfpfw.fsf@tungsten.kyluka.com>
	<5bef28df0701180232p25ca38ebw1796c4fd7e58d84d@mail.gmail.com>
	<B8D50A0F-2072-42BF-879E-2B1C4227A289@onlinehome.de>
Message-ID: <5bef28df0701260717p3e97273dm17a991f542faa849@mail.gmail.com>

I've just committed  a quick change to cl-store which should help to
reduce the amount of consing which cl-store does.

Theres no concrete documentation yet (help is always welcome)
but you can now reuse the hash-tables which are created.

This can be accomplished by using

(cl-store:with-serialization-unit ()
  <lots of cl-store:store calls>)

This will reuse the circularity tracking hash-tables for scope of the call.
Using this in combination with a stream which is already open
will greatly reduce the amount of garbage which cl-store will create.


Cheers,
 Sean.


From ctdean at sokitomi.com  Sat Jan 27 02:09:11 2007
From: ctdean at sokitomi.com (Chris Dean)
Date: Fri, 26 Jan 2007 18:09:11 -0800
Subject: [cl-store-devel] Fastest way to convert an object to a vector?
In-Reply-To: <5bef28df0701260717p3e97273dm17a991f542faa849@mail.gmail.com>
	(Sean Ross's message of "Fri, 26 Jan 2007 15:17:39 +0000")
References: <m2lkk2iir4.fsf@tungsten.kyluka.com>
	<5bef28df0701170154o5f0e8be0jf87e5d5791f43227@mail.gmail.com>
	<m2d55dfpfw.fsf@tungsten.kyluka.com>
	<5bef28df0701180232p25ca38ebw1796c4fd7e58d84d@mail.gmail.com>
	<B8D50A0F-2072-42BF-879E-2B1C4227A289@onlinehome.de>
	<5bef28df0701260717p3e97273dm17a991f542faa849@mail.gmail.com>
Message-ID: <m27iv92pbs.fsf@jacaranda.kyluka.com>


"Sean Ross" <rosssd at gmail.com> writes:
> I've just committed  a quick change to cl-store which should help to
> reduce the amount of consing which cl-store does.

Great, thanks.  I'll check it out.

Cheers,
Chris Dean