From helmut at cybernetic-systems.de  Fri Oct  3 10:40:38 2008
From: helmut at cybernetic-systems.de (Helmut G. Enders)
Date: Fri, 03 Oct 2008 12:40:38 +0200
Subject: [rucksack-devel] SB-SYS:FD-STREAM for "file /tmp/rs-tute/objects"
	{B1CC241}> is closed
Message-ID: <48E5F6A6.10409@cybernetic-systems.de>

I try to store an instance with  a  persistent list of persistent objects as a slot-value.
E.g.  an invoice with a list of invoice items.

Do I have to save a list of unique-ids and then fetch all objects with these ids (sql like),
or is my attempt simply the wrong approach.

Helmut




(defpackage :rucksack-tutorial
  (:nicknames :rs-tute)
  (:use :cl :rucksack))
(in-package :rucksack-tutorial)
(defvar *rs-tute-directory* #p"/tmp/rs-tute/")

(with-rucksack (rs *rs-tute-directory* :if-exists :supersede)
   (with-transaction ()
     (defclass contact-details ()
        ((unique-id    :initarg :unique-id :accessor unique-id-of
		     :index :number-index
                      :unique t
		     :documentation "A unique number for each contact in our DB")
          (name       :initarg :name :accessor name-of
		     :index :case-insensitive-string-index
	             :documentation "The full name of the contact")

         (list-of-friends
                      :initarg :friends :accessor friends
		     :documentation "All the friends."))
       (:metaclass persistent-class)
       (:index t))
     ))


(defun make-my-friends ()
  (declare (optimize (debug 3)))
    (with-rucksack (rs *rs-tute-directory*)
     (with-transaction ()
       (loop with fs = nil
             for x from 1 upto 1000
             do  (setf fs (p-cons (make-instance 'contact-details
                                                  :unique-id x
                                                  :name (format nil "Friend~D" x)
                                                  :friends nil) fs))
             finally    (make-instance 'contact-details :name "Helmut" :friends fs)))))


(defun find-contact-by-name (name)
   (with-rucksack (rs *rs-tute-directory*)
     (with-transaction ()
       (rucksack-map-slot rs 'contact-details 'name
			 (lambda (contact)
			   (return-from find-contact-by-name contact))
			 :equal name)))
   nil)

(defun list-my-friends ()
  (declare (optimize (debug 3)))
  (with-rucksack (rs *rs-tute-directory*)
     (with-transaction ()
        (loop with me =  (find-contact-by-name "Helmut")
              for f in (unwrap-persistent-list (friends me))
              do (print (name-of f))))))

(defun test ()
   (make-my-friends)
   (list-my-friends))



From senatorzergling at gmail.com  Sat Oct 11 13:56:22 2008
From: senatorzergling at gmail.com (szergling)
Date: Sun, 12 Oct 2008 02:56:22 +1300
Subject: [rucksack-devel] Fwd: My initial experiences
In-Reply-To: <8a4b88390810071118y6ebd77ecq124c6b079ca8e15d@mail.gmail.com>
References: <8a4b88390810071118y6ebd77ecq124c6b079ca8e15d@mail.gmail.com>
Message-ID: <8a4b88390810110656q6cfe81ffn5a6628d2a6cee714@mail.gmail.com>

Apologies if this is a duplicate. I think my first email disappeared.
The common-lisp.net mailing lists seemed flaky when I sent this.


---------- Forwarded message ----------
From: szergling <senatorzergling at gmail.com>
Date: Wed, Oct 8, 2008 at 7:18 AM
Subject: My initial experiences
To: rucksack-devel at common-lisp.net


Hi all,

In the spirit of sharing experiences with rucksack (man, love that
ECLM talk, cool stuff), here's my evaluation. Apologies if it is a bit
rambling (still exploring) or sounds a bit whiny. Like all Lispers,
all I want is a perfect world (is that too much to ask for? (: ).

I was writing/toying with a small program, and was looking for a
simple way to switch between persisted vs non-persisted version of the
application (maybe my data is just left in RAM for normal REPL usage,
maybe there's a separate development (in RAM) vs release branch
(persisted)). Regardless of the specific reasons, for general
modularity alone, I was hoping to be able to trivially flip a switch,
persistent, flip switch off, and get non-persistent behaviour. I
wasn't sure how much work would be required to get this behaviour.

I started with Brad Beveridge's tutorial, which went smoothly, no
issues (good job!). It seems to me that having *rucksack-dir*, instead
of some first class *some-rucksack*, is unlispy, though it is obvious
in hindsight that rucksacks 'identities' are given by pathname, not by
'eq'ness.

My first try at the 'persistence switch' involves a slight change to
defclass. Not quite invisible persistence, but close enough for now:

(defmacro defpclass (name supers slots &rest others)
 (assert (equal (rucksack-directory (current-rucksack)) *rucksack-dir*))
 `(with-transaction ()
   (defclass ,name ,supers ,slots
     (:index t)
     (:metaclass persistent-class)
     ,@(remove-properties others '(:metaclass)))))

Kinda assumes that there's already a *rucksack* open. Maybe it's not a
good idea, and an additional (with-rucksack ...) around the outside is
better.

Aside from that, it's simply defclass -> defpclass with no other
changes required. But if I close-rucksack, I often forget to
re-evaluate the defpclass forms with a new rucksack.

How come closing a rucksack (close-rucksack) doesn't cause *rucksack*
to become nil? I don't have the right mental model for what's going on
I guess. Reading the source suggests that close-rucksack commits
transactions and flushes the cache.



It is obvious in hindsight that this must be so, but object 'identity'
is only maintained within a

(with-rucksack ...
 (with-transaction ...
  --XXX--))

form. Across separate invocations, the same object is not 'eq'.

One always need to do the (with-rucksack ... (with-transaction ...))
wrapping. Since it's a transaction, the body to be wrapped should be
minimised, so everywhere we use rucksack, we have to think on a 'fine
scale' on how we want to wrap it. Other emails recently in this list
have also mentioned the need to wrap even reads with a transaction. I
have also observed overuse of transactions leading to issues with
nested transactions. Anyhoo... for now, I use a convenience utility
like this for REPL explorations.

(defmacro rs ((var) &body body)
 `(with-rucksack (,var *rucksack-dir*)
   (with-transaction ()
     , at body)))

Later I found out that ordinary lists do not seem to be handled
transparently/cleanly/conveniently. After storing one, we may recover
something like:

(#<RUCKSACK::PROXY {BECAF61}>)

The solution is to prefix p- to everything, p-list, p-cons, p-cdr,
p-length, p-mapcar, etc to use the persistent versions of
cons. However, any form/function thus prefixed must now be called
within a wrapping with-rucksack form. This doen't seem too different
from elephant's persistent set abstraction, pset. It does increase the
book-keeping required for the switch/toggle between persistent vs
non-persistent versions of a collection. I might explore an abstracted
collection for this...

It is important to remember the rule that anytime we access a
persistent object, we need with-transaction (is that right?). This
includes all methods (print-object, with-accessors,
etc). Unfortunately, that mean something as innocent as this
can cause subtle problems:

;; This method accesses the slots of a persistent object outside
;; a transaction.
(defmethod print-object ((e account) stream)
 (print-unreadable-object (e stream :type t)
   (format stream "~a (~a)"
           (account-type e)
           (+ (p-length (account-debits e))
              (p-length (account-credits e))))))

All of a sudden, basic things break.

;; This works ok.
(rs (r)
 (print (p-car (ledger-entries (main-ledger r)))))

;; This doesn't work. Get an error, or worse, Slime issues.
(rs (r)
 (p-car (ledger-entries (main-ledger r))))

Why? Where we do the printing (inside or outside a transaction) seem
to matter. So now, it seems we're more likely to get errors even with
'basic' functions. More wrapping required? Eg, something like this
handler-case?

(defmethod print-object ((obj test) stream)
 (print-unreadable-object (obj stream :type t)
   (handler-case
       (format stream "~a" (slot-a-of obj))
     (error (e)
       (declare (ignore e))
       (format stream "Error printing object (class ~a)"
               (find-class 'test))))))

Does rucksack (wait, maybe it's Slime) do this automatically? I might
have seen something like this:

#<ACCOUNT <<error printing object>>

It turns out that unhandled printing errors can also interact badly
with Slime (objects in the stack may throw errors when printed in the
stack trace). So occasionally, no slime debugger window is available,
and the following can be seen in *standard-output*,

debugger invoked on a SIMPLE-ERROR in thread #<THREAD "worker"
{B5F3639}>: There is no applicable method for the generic function
#<STANDARD-GENERIC-FUNCTION TRANSACTION-ID (3)>when called with
arguments (NIL).

Or sometimes, I'll get confusing error messages when I forgot to use
(with-transaction ()...).

Unable to display error condition
  [Condition of type SIMPLE-ERROR]

Restarts:
 0: [ABORT-REQUEST] Abort handling SLIME request.
 1: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
{ADDBC31}>)

Backtrace:
 0: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) #<error printing object>)
 1: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) #<error printing object>)
 2: ((SB-PCL::FAST-METHOD CACHE-CREATE-OBJECT (T STANDARD-CACHE))
#<error printing object>)
 3: (RUCKSACK::MAKE-PERSISTENT-DATA PERSISTENT-CONS (3)
#<RUCKSACK::SERIAL-TRANSACTION-RUCKSACK in
#P"/home/tyc20/hda/code/lisp/accounting/rucksack1/" with 1 root
{AA77559}>)

Inspecting the error

The object is a CONDITION of type SIMPLE-ERROR.
 [type: SIMPLE-ERROR]
--------------------
FORMAT-CONTROL: "~@<There is no applicable method for the generic
function ~2I~_~S~
         ~I~_when called with arguments ~2I~_~S.~:>"
FORMAT-ARGUMENTS: #<CONS <<error printing object>> {BBDA7AB}>

A proper list.
 [type: CONS]
--------------------
Elements:
0: #<STANDARD-GENERIC-FUNCTION RUCKSACK::TRANSACTION-TOUCH-OBJECT (1)>
1: #2=#<CONS <<error printing object>> {BBDA793}>


Notice how stack traces show up badly, more examples:
 7: (PRIN1 #<error printing object>)
 8: (PRINT #<error printing object>)

The reasons for these errors (see the example above, printing) took a
while for me to discover :(

At the risk of possibly screwing up with nested transactions, maybe
some automated way could make things 'more invisible' with accessing
persistent objects... I'm still thinking/playing around.

(defmethod slot-value-using-class :around (.. (class persistent-class) .. slot)
 (with-rucksack (*some-default*)
   (with-transaction ()
     (call-next-method))))

Anyway, that's where I'm up to in my random wanderings...

I hope my experience is helpful or moderately interesting. I look
forward to comments/tips from the gentle folk on this list on using
rucksack and also await future releases that might polish away some of
the arkwardness with transactions. Cheers.



From helmut at cybernetic-systems.de  Mon Oct 13 08:13:40 2008
From: helmut at cybernetic-systems.de (Helmut G. Enders)
Date: Mon, 13 Oct 2008 10:13:40 +0200
Subject: [rucksack-devel] Fwd: My initial experiences
In-Reply-To: <8a4b88390810110656q6cfe81ffn5a6628d2a6cee714@mail.gmail.com>
References: <8a4b88390810071118y6ebd77ecq124c6b079ca8e15d@mail.gmail.com>
	<8a4b88390810110656q6cfe81ffn5a6628d2a6cee714@mail.gmail.com>
Message-ID: <48F30334.1070702@cybernetic-systems.de>

szergling wrote:
>
> 
> Hi all,
> .....  and was looking for a
> simple way to switch between persisted vs non-persisted version of the
> application (maybe my data is just left in RAM for normal REPL usage,
> maybe there's a separate development (in RAM) vs release branch
> (persisted)). Regardless of the specific reasons, for general
> modularity alone, I was hoping to be able to trivially flip a switch,
> persistent, flip switch off, and get non-persistent behaviour. I
> wasn't sure how much work would be required to get this behaviour.

> (#<RUCKSACK::PROXY {BECAF61}>)
> 
> The solution is to prefix p- to everything, p-list, p-cons, p-cdr,
> p-length, p-mapcar, etc to use the persistent versions 


To "switch between persisted vs non-persisted version of the
application" implied that you have no problems to hold *all*
data in ram. Then I do not need rucksack, or rucksack is not
much more that cl-store, save-image etc.

E.g. An instance of email-archive with slots like
   - year,
   - list-of-persons
      - with list-of-emails
or an instance of invoices-2007 with list of these where
each holds a list of articles.

It is very important when accessing the
instance email-archive from rucksack, that there is a
<Proxy> in that slot and not a list of 170 TByte of emails.
And there is a loop construct to access the persistant list,
element by element.

If the price for this is p-..., I am pleased to pay. :-)

Helmut