From hans.huebner at gmail.com  Wed Aug  1 06:17:57 2012
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Wed, 1 Aug 2012 08:17:57 +0200
Subject: [bknr-devel] bknr.datastore store lockfile
In-Reply-To: <CAD_S9D9zFxQQth9oQUPw14xkgWHapO0jf3L-KMKDcqJvnaJdTg@mail.gmail.com>
References: <CAD_S9D9zFxQQth9oQUPw14xkgWHapO0jf3L-KMKDcqJvnaJdTg@mail.gmail.com>
Message-ID: <CABj0bwK=OE34zEec17KY8gYBaX5A1_cMQ3DAe_L3=JxWnrverQ@mail.gmail.com>

Hi Vladimir,

On Wed, Aug 1, 2012 at 12:44 AM, Vladimir Sedach <vsedach at gmail.com> wrote:
> Are there any plans to add a lockfile to bknr.datastore stores? Right
> now it seems possible for multiple Lisp processes to open the same
> store, and I'm not sure the resulting behavior is desirable.

No, there are no plans to do that, but I'm open to patches that add
the functionality.  I did not ever see the lack of a lock file as a
problem, but I see how it is easily possible to seriously mess up a
store by trying to open it from multiple processes.

> Another problem I noticed (at least with SBCL on Linux) is that taking
> a store whose directory permissions are owned by some user and opening
> it as a process running as root (for example, if running a daemon that
> opens the store before doing setuid) causes that store to be erased
> for some reason.

Again, this is a problem I have not encountered, but I'm open to
patches correcting the problem.  Do you have an isolated test case
that exposes the problem?

Thanks,
Hans



From vsedach at gmail.com  Thu Aug  2 00:59:35 2012
From: vsedach at gmail.com (Vladimir Sedach)
Date: Wed, 1 Aug 2012 20:59:35 -0400
Subject: [bknr-devel] bknr.datastore store lockfile
In-Reply-To: <CABj0bwK=OE34zEec17KY8gYBaX5A1_cMQ3DAe_L3=JxWnrverQ@mail.gmail.com>
References: <CAD_S9D9zFxQQth9oQUPw14xkgWHapO0jf3L-KMKDcqJvnaJdTg@mail.gmail.com>
	<CABj0bwK=OE34zEec17KY8gYBaX5A1_cMQ3DAe_L3=JxWnrverQ@mail.gmail.com>
Message-ID: <CAD_S9D8r4gAshhNnMFkDaMBU83idpeocEATt3MgRJT7yjp1L7Q@mail.gmail.com>

Hi Hans,

> No, there are no plans to do that, but I'm open to patches that add
> the functionality.  I did not ever see the lack of a lock file as a
> problem, but I see how it is easily possible to seriously mess up a
> store by trying to open it from multiple processes.

One thing I found in building a lock file mechanism for my application
is that there's no way to do it portably in CL, which makes sense
since some details are OS dependent (and even on Unix it might not
work on NFS, etc.). So I don't know of the right way of doing it
(maybe a lock file is not actually the right way), and I'd like to
figure one out before putting any mechanisms in a library.

> Again, this is a problem I have not encountered, but I'm open to
> patches correcting the problem.  Do you have an isolated test case
> that exposes the problem?

I don't really have a test case for this beyond "make a store as
non-root user, put some data in the log (and maybe a snapshot), then
try to open that store as root."

Thank you,
Vladimir



From hans.huebner at gmail.com  Thu Aug  2 06:47:06 2012
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Thu, 2 Aug 2012 08:47:06 +0200
Subject: [bknr-devel] bknr.datastore store lockfile
In-Reply-To: <CAD_S9D8r4gAshhNnMFkDaMBU83idpeocEATt3MgRJT7yjp1L7Q@mail.gmail.com>
References: <CAD_S9D9zFxQQth9oQUPw14xkgWHapO0jf3L-KMKDcqJvnaJdTg@mail.gmail.com>
	<CABj0bwK=OE34zEec17KY8gYBaX5A1_cMQ3DAe_L3=JxWnrverQ@mail.gmail.com>
	<CAD_S9D8r4gAshhNnMFkDaMBU83idpeocEATt3MgRJT7yjp1L7Q@mail.gmail.com>
Message-ID: <CABj0bwJYxfYmV0fSVvL1Aka+wp6VkiGcJ8rmdFa5Vop+g-NXEA@mail.gmail.com>

On Thu, Aug 2, 2012 at 2:59 AM, Vladimir Sedach <vsedach at gmail.com> wrote:
> Hi Hans,
>
>> No, there are no plans to do that, but I'm open to patches that add
>> the functionality.  I did not ever see the lack of a lock file as a
>> problem, but I see how it is easily possible to seriously mess up a
>> store by trying to open it from multiple processes.
>
> One thing I found in building a lock file mechanism for my application
> is that there's no way to do it portably in CL, which makes sense
> since some details are OS dependent (and even on Unix it might not
> work on NFS, etc.). So I don't know of the right way of doing it
> (maybe a lock file is not actually the right way), and I'd like to
> figure one out before putting any mechanisms in a library.

I think the best way, moving forward, would be to depend on iolib and
use a simple POSIX open call with O_EXCL on a lock file placed in the
store directory.  It would be good to make the lock file
implementation optional so that compiling without it is possible (i.e.
on Window or Lisp implementations not supported by iolib).

>> Again, this is a problem I have not encountered, but I'm open to
>> patches correcting the problem.  Do you have an isolated test case
>> that exposes the problem?
>
> I don't really have a test case for this beyond "make a store as
> non-root user, put some data in the log (and maybe a snapshot), then
> try to open that store as root."

I tried the below and cannot reproduce the problem on OSX:

(ql:quickload :bknr.datastore)

(defpackage :store-user
  (:use :cl)
  (:export :test))

(in-package :store-user)

(defclass test-object (bknr.datastore:store-object)
  ((a :initarg :a :reader a))
  (:metaclass bknr.datastore:persistent-class))

(defun test ()
  (make-instance 'bknr.datastore:mp-store :directory "/tmp/object-store/"
                                          :subsystems (list
(make-instance 'bknr.datastore:store-object-subsystem)))
  (make-instance 'test-object :a 1)
  (bknr.datastore:snapshot)
  (print (bknr.datastore:all-store-objects))
  (bknr.datastore:close-store))

(eval (read-from-string "(progn (store-user:test) (sb-ext:exit))"))

panadol 1019_% sbcl --load reproduce-bug.lisp
This is SBCL 1.0.56.73-29fe19a, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
; adding components under ~/lisp-libs/ to asdf registry
To load "bknr.datastore":
  Load 1 ASDF system:
    bknr.datastore
; Loading "bknr.datastore"
..........
initializing store random state
restoring #<MP-STORE DIR: "/tmp/object-store/">

(#<TEST-OBJECT ID: 0>)
panadol 1020_% sudo sbcl --load reproduce-bug.lisp
This is SBCL 1.0.56.73-29fe19a, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
; adding components under ~/lisp-libs/ to asdf registry
To load "bknr.datastore":
  Load 1 ASDF system:
    bknr.datastore
; Loading "bknr.datastore"
..........
restoring #<MP-STORE DIR: "/tmp/object-store/">
; loading snapshot file
/tmp/object-store/current/store-object-subsystem-snapshot
loading transaction log /tmp/object-store/current/transaction-log

(#<TEST-OBJECT ID: 0> #<TEST-OBJECT ID: 1>)

Can you determine some more details about the problem using this test?

Thanks,
-Hans