In CLOS, instance remorphing considered useless in practice?

[Duane Rettig]

--------
    Date:  Tue, 08 Dec 2020 22:17:10 +0100
    From:  "Svante Carl v. Erichsen" <svante.v.erichsen at web.de>

    Hi!

[Sorry to be coming late to the game.]

    It is used all the time while working on a live system (image).
    
    If you want raw performance, you can often substitute structs, if you
    can live with the downsides: all objects from a class must be thrown
    away on redefinition (which might be OK if you want to treat e. g. a
    production system as unchangeable except through re-deployment), you
    only get single inheritance (which is often not needed anyway), and most
    of the MOP goodness is gone, starting with that you can't have a
    different metaclass than structure-class.  Also, because of the
    mentioned inflexibility, you have to dismiss several warnings if you do
    re-define a struct.

The only reason why structs must be thrown away after a change in
definition is because the objects have identity, and thus cannot be
modified internally and under-the-table.  CLOS objects, on the other
hand, have base objects that are tiny and their guts can be modified
without losing their identity, because these guts have been indirected
through the base objects that provide that identity.  I'm guessing
that most CL implementations use similar techniques, because it would
be hard to implement CLOS semantics without having this single
indirection.  A struct-like CLOS reference is therefore possible with
one more indirection than a struct, and that extra register load need
not be per-reference - the indirection could be pre-loaded for any
sections of code which can afford to (or which must) treat the object
consistently.  Conversely, a struct, if implemented with an
indirection, could be set up to be lazily updated, just like CLOS
instances, if the implementation so chose to pay the extra load penalty.
    
    On the other hand, structs have a lot of other, more low-level
    convenience tooling, e. g. automatic and portable serialization and
    deserialization.

Yes, structs have an implied make-load-form, although m-l-f is itself
much more powerful than the implicit one for structs.  In almost every
way, structs are a subset of CLOS semantically, and we can get fairly
close to struct performance with CLOS objects.

    I _think_ that one might be able to create a new macro (let's say
    def-struct-class) that has exactly the same semantics as defstruct, but
    in development mode uses defclass underneath, while being an alias for
    defstruct in production mode.  It might get a bit hairy to adhere to
    :type :list or :vector, though.  The result is a bit limited, but just
    might be what you need.

Bringing out defstruct semantics as a wrapper for a CLOS implementation
seems counterproductive; the (CLOS) implementation would always be
fighting the semantic limitations of structs, whereas voluntary
restrictions on CLOS semantics would allow the full CLOS semantics to
be brought out and extensions would be trivial; instead of inventing
new extensions on defstruct semantics, a relaxing of the voluntary
restrictions would bring back the CLOS smantics as needed.

    The folk at ITA also did quite some work on compile-time metaprogramming
    in order to get fast objects etc., so maybe they can chime in.
    
    Anyway, my stance is that the incredibly flexible MOP classes provide
    general semantic usefulness, but Common Lisp does offer more constrained
    options that trade general usefulness for speed.

Elsewhere in this thread, both inlining and sealing have been
mentioned.  In fact, they are different parts of the same elephant,
which may look different but which can all be placed under the
category of early vs late binding.  That is, where do you package up a
code base to the point where the api is solid enough to perform
optimizations on that code?  Often, it is the difference between
interpreted code and compiled code - the binding occurs at
compile-time when macros are expanded once and then commited to
assembler code (whereas interpreted code tends to macroexpand for each
invocation).  In the case of CLOS, the commitment never need be
permanent; even after an object has been created, it can be modified
without destroying its identity, thanks to AMOP.

Allegro CL provides a different cross-section of binding-time; a CLOS
slot can be given a "fixed-index" attribute, which holds a slot to a
particular index.  It is one of those restrictons I mentioned that can
be used or not - a class can specify fixed-index and/or regular
(non-fixed-index) slots.  The semantics of fixed-index slots are the
same as regular slots, except that trying to define or mixin two slots
with the same index and a different name will cause an error.  This
has a few remifications, but is usable in practice.   A fixed-index
slot is used exactly as a regular slot, but a separate macro,
slot-value-using-class-name, provides syntax for defining extremely
fast accesses (one or two instructions, depending on how it is used).

This feature has been around for many years, but was mostly
undocumented until recently.  For the documentation and examples, see
https://franz.com/support/documentation/current/doc/implementation.htm#ef-slot-value-1

Note that in fixed-index (and especially slot-value-using-class-name)
semantics, the binding time is macroexpand/compile time, whuch means
that redefinition will require recompilation, but _only_ if the actual
fixed-index values have changed; slots that have unchanged fixed index
locations will remain at the same place, allowing the accessors to
remain in place, and other slot-value usages for the non-fixed slots
will be taken care of lazily by the
update-instance-for-{different,redefined}-class mechanisms.

Duane Rettig
    
    Jean-Claude Beaudoin writes:
    
    > Hello Pros of Common Lisp,
    >
    > Here is my attempt at starting a significant (and hopefully useful) debat
   e
    > on a subject squarely about Common Lisp and its internals.
    >
    > My main stance here is to state that I have yet to see, in a significant
    > application, any use of functions #'cl:make-instance-obsolete and
    > #'cl:update-instance-for-redefined-class and of the underlying machinery
    > that support the remorphing of instances following a class redefinition
    > (through a subsequent cl:defclass most likely). I can state the same thin
   g
    > about #'cl:update-instance-for-different-class and #'cl:change-class.
    >
    > The machinery required of any CL implementation to properly support those
    > functions (mentioned here above) is quite significant in its complexity a
   nd
    > usually imposes a sizeable performance penalty on the speed of any instan
   ce
    > slot access.
    >
    > A different choice of class redefinition semantics can lead to an
    > implementation of CLOS with much reduced instance slot access overhead.
    >
    > The necessity of supporting instance remorphing should therefore be well
    > motivated by very significant gains in application code performance (in
    > speed and/or size and/or expressiveness power).
    >
    > Can anyone of you point me to some evidence of such application level
    > usefulness?
    > Is there any notorious usecase of this (instance remorphing) since its
    > inclusion into ANSI CL?
    >
    >
    > By the way, at the bottom of the entry about #'cl:change-class in the tex
   t
    > of the CL standard, one can find a "Notes:" section that starts with this
    > sentence: "The generic function change-class has several semantic
    > difficulties." And this was written in the context of a single-threaded
    > implementation, as ANSI-CL limited itself. I bet that almost all currentl
   y
    > significant CL implementations are now multi-threaded, therefore the
    > "several semantic difficulties" are greatly and gravely compounded. In my
    > opinion, this makes proper motivation of usefulness all the more imperiou
   s.
    > What do you think?
--------