[rucksack-devel] Fwd: My initial experiences

[szergling]

Apologies if this is a duplicate. I think my first email disappeared.
The common-lisp.net mailing lists seemed flaky when I sent this.


---------- Forwarded message ----------
From: szergling <senatorzergling at gmail.com>
Date: Wed, Oct 8, 2008 at 7:18 AM
Subject: My initial experiences
To: rucksack-devel at common-lisp.net


Hi all,

In the spirit of sharing experiences with rucksack (man, love that
ECLM talk, cool stuff), here's my evaluation. Apologies if it is a bit
rambling (still exploring) or sounds a bit whiny. Like all Lispers,
all I want is a perfect world (is that too much to ask for? (: ).

I was writing/toying with a small program, and was looking for a
simple way to switch between persisted vs non-persisted version of the
application (maybe my data is just left in RAM for normal REPL usage,
maybe there's a separate development (in RAM) vs release branch
(persisted)). Regardless of the specific reasons, for general
modularity alone, I was hoping to be able to trivially flip a switch,
persistent, flip switch off, and get non-persistent behaviour. I
wasn't sure how much work would be required to get this behaviour.

I started with Brad Beveridge's tutorial, which went smoothly, no
issues (good job!). It seems to me that having *rucksack-dir*, instead
of some first class *some-rucksack*, is unlispy, though it is obvious
in hindsight that rucksacks 'identities' are given by pathname, not by
'eq'ness.

My first try at the 'persistence switch' involves a slight change to
defclass. Not quite invisible persistence, but close enough for now:

(defmacro defpclass (name supers slots &rest others)
 (assert (equal (rucksack-directory (current-rucksack)) *rucksack-dir*))
 `(with-transaction ()
   (defclass ,name ,supers ,slots
     (:index t)
     (:metaclass persistent-class)
     ,@(remove-properties others '(:metaclass)))))

Kinda assumes that there's already a *rucksack* open. Maybe it's not a
good idea, and an additional (with-rucksack ...) around the outside is
better.

Aside from that, it's simply defclass -> defpclass with no other
changes required. But if I close-rucksack, I often forget to
re-evaluate the defpclass forms with a new rucksack.

How come closing a rucksack (close-rucksack) doesn't cause *rucksack*
to become nil? I don't have the right mental model for what's going on
I guess. Reading the source suggests that close-rucksack commits
transactions and flushes the cache.



It is obvious in hindsight that this must be so, but object 'identity'
is only maintained within a

(with-rucksack ...
 (with-transaction ...
  --XXX--))

form. Across separate invocations, the same object is not 'eq'.

One always need to do the (with-rucksack ... (with-transaction ...))
wrapping. Since it's a transaction, the body to be wrapped should be
minimised, so everywhere we use rucksack, we have to think on a 'fine
scale' on how we want to wrap it. Other emails recently in this list
have also mentioned the need to wrap even reads with a transaction. I
have also observed overuse of transactions leading to issues with
nested transactions. Anyhoo... for now, I use a convenience utility
like this for REPL explorations.

(defmacro rs ((var) &body body)
 `(with-rucksack (,var *rucksack-dir*)
   (with-transaction ()
     , at body)))

Later I found out that ordinary lists do not seem to be handled
transparently/cleanly/conveniently. After storing one, we may recover
something like:

(#<RUCKSACK::PROXY {BECAF61}>)

The solution is to prefix p- to everything, p-list, p-cons, p-cdr,
p-length, p-mapcar, etc to use the persistent versions of
cons. However, any form/function thus prefixed must now be called
within a wrapping with-rucksack form. This doen't seem too different
from elephant's persistent set abstraction, pset. It does increase the
book-keeping required for the switch/toggle between persistent vs
non-persistent versions of a collection. I might explore an abstracted
collection for this...

It is important to remember the rule that anytime we access a
persistent object, we need with-transaction (is that right?). This
includes all methods (print-object, with-accessors,
etc). Unfortunately, that mean something as innocent as this
can cause subtle problems:

;; This method accesses the slots of a persistent object outside
;; a transaction.
(defmethod print-object ((e account) stream)
 (print-unreadable-object (e stream :type t)
   (format stream "~a (~a)"
           (account-type e)
           (+ (p-length (account-debits e))
              (p-length (account-credits e))))))

All of a sudden, basic things break.

;; This works ok.
(rs (r)
 (print (p-car (ledger-entries (main-ledger r)))))

;; This doesn't work. Get an error, or worse, Slime issues.
(rs (r)
 (p-car (ledger-entries (main-ledger r))))

Why? Where we do the printing (inside or outside a transaction) seem
to matter. So now, it seems we're more likely to get errors even with
'basic' functions. More wrapping required? Eg, something like this
handler-case?

(defmethod print-object ((obj test) stream)
 (print-unreadable-object (obj stream :type t)
   (handler-case
       (format stream "~a" (slot-a-of obj))
     (error (e)
       (declare (ignore e))
       (format stream "Error printing object (class ~a)"
               (find-class 'test))))))

Does rucksack (wait, maybe it's Slime) do this automatically? I might
have seen something like this:

#<ACCOUNT <<error printing object>>

It turns out that unhandled printing errors can also interact badly
with Slime (objects in the stack may throw errors when printed in the
stack trace). So occasionally, no slime debugger window is available,
and the following can be seen in *standard-output*,

debugger invoked on a SIMPLE-ERROR in thread #<THREAD "worker"
{B5F3639}>: There is no applicable method for the generic function
#<STANDARD-GENERIC-FUNCTION TRANSACTION-ID (3)>when called with
arguments (NIL).

Or sometimes, I'll get confusing error messages when I forgot to use
(with-transaction ()...).

Unable to display error condition
  [Condition of type SIMPLE-ERROR]

Restarts:
 0: [ABORT-REQUEST] Abort handling SLIME request.
 1: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
{ADDBC31}>)

Backtrace:
 0: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) #<error printing object>)
 1: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) #<error printing object>)
 2: ((SB-PCL::FAST-METHOD CACHE-CREATE-OBJECT (T STANDARD-CACHE))
#<error printing object>)
 3: (RUCKSACK::MAKE-PERSISTENT-DATA PERSISTENT-CONS (3)
#<RUCKSACK::SERIAL-TRANSACTION-RUCKSACK in
#P"/home/tyc20/hda/code/lisp/accounting/rucksack1/" with 1 root
{AA77559}>)

Inspecting the error

The object is a CONDITION of type SIMPLE-ERROR.
 [type: SIMPLE-ERROR]
--------------------
FORMAT-CONTROL: "~@<There is no applicable method for the generic
function ~2I~_~S~
         ~I~_when called with arguments ~2I~_~S.~:>"
FORMAT-ARGUMENTS: #<CONS <<error printing object>> {BBDA7AB}>

A proper list.
 [type: CONS]
--------------------
Elements:
0: #<STANDARD-GENERIC-FUNCTION RUCKSACK::TRANSACTION-TOUCH-OBJECT (1)>
1: #2=#<CONS <<error printing object>> {BBDA793}>


Notice how stack traces show up badly, more examples:
 7: (PRIN1 #<error printing object>)
 8: (PRINT #<error printing object>)

The reasons for these errors (see the example above, printing) took a
while for me to discover :(

At the risk of possibly screwing up with nested transactions, maybe
some automated way could make things 'more invisible' with accessing
persistent objects... I'm still thinking/playing around.

(defmethod slot-value-using-class :around (.. (class persistent-class) .. slot)
 (with-rucksack (*some-default*)
   (with-transaction ()
     (call-next-method))))

Anyway, that's where I'm up to in my random wanderings...

I hope my experience is helpful or moderately interesting. I look
forward to comments/tips from the gentle folk on this list on using
rucksack and also await future releases that might polish away some of
the arkwardness with transactions. Cheers.