[pro] Modularity for subclassing in Common Lisp

[Daniel Weinreb]

(This is a new topic.)

Many times, people have presented a major problem with object-oriented
programming as it is often/idiomatically used.  The good thing is that
OOP, when done right (in Lisp, that means, among other things, always
using defgeneric explicitly), provides a great abstarct interface to
the caller.  It's good for modularity, because if you want to changes
the implementation of the class, you know what you can change easily,
and what you cannot because callers depend on it.

If all of the callers are under your control, you can easily make
incompatible changes, but if you're providing a general-purpose
library, that people all over the world are using, then it's a lot
harder.

I call the set of defgenerics (plus the factory functions) the
"protocol".  The word "type" is sort of right but carries a lot of
connotations and freight that I'd rather avoid.

Often all of the CLOS classes that implement a given protocol inherit
from a common abstract base class.  There are two reasons to do this.
First, a common base class can provide implementations of some of the
generic functions all by itself.  My favorite simple example is an
"output stream" protocol, that has a write-character operation and a
write-string operation.  The common base class provides an
implementation of write-string that works by iterating over the
characters of the string and calling write-character.  Any output
stream that can write strings in a more efficient way can override
that method.

The second reason is to define a Lisp type that all of
the subclasses will be of.

In my mind, as far as the first reason goes, there should be NO
requirement that every type that implements the protocol *must*
inherit from a common abstract base class.  The abstract base class is
merely an implementation convenience.  In the case of the fhash
library, I do not have any common base class.  As a matter of fact,
it's not even implemented using CLOS at all.  (The caller doesn't
know, and should not.)  From the source file:

;;; An fhash is represented as a one-dimensional array.  It can have two
;;; representations: a linear search table, or a hash table (from the
;;; underlying Common Lisp implementation).  An fhash is represented as
;;; a vector, starting with a header.  The 0th element says whether this
;;; fhash instance is :linear or :hash.  If it's :hash, the 1th element
;;; contains the hash table, and the rest of the elements are ignored.

As far as the second reason goes, fortunately Lisp has a
powerful way to let you define types.  In fhash:

(defun fhash-table-p (fhash)

  "Return true if the argument is an FHASH.  Otherwise return
   NIL.  This is not perfect, since some array might just
   happen to have these values, but it seems unlikely."

  (and (vectorp fhash) (member (aref fhash +fhash-kind+) '(:linear :hash))))

(deftype fhash-table () '(satisfies fhash-table-p))

This is not perfect.  A programmer might just happen to create a
vector and put one of those keywords into the +fhash-kind+ slot.
(+fhash-kind+ is zero but that's a mere detail.)  But it's sort of
good enough.

OK, back to OOP.  Abstraction of the object from the caller is just
fine.  But, the commonly-heard objection to OOP as it is really used
is that there is *not* a clean modular definition isolating
*subclasses*.

Smalltalk didn't even try.  CLOS, I believe, does not try and there is
not an idiomatic way to do it.  The only language I know that makes a
good stab in this direction is C++, which has "public", "protected",
and "private".  Whatever else you say about C++, Bjarne understood
this issue and tried to fix it.  The C++ solution isn't perfect
but it's sure a lot better than anything else.

How can we provide the same thing in Common Lisp?  And without
changing CLOS?  In fact, without adding any new language feature at
all?

With packages, fhash has a defpackage that provides the protocol seen
by callers.  If you do an abstract type with CLOS, you ought to do the
same thing.  (You might have one package for many types.)  So the
callers can "use" fhash and then call putfhash and so on.

(Digression: A slightly different approach is to define a package not
intended to be "use"'ed.  I did this with the API for the logging
facility in QRes.  For example: log:add-rule.  If you did a "use", it
might be confusing to remember that add-rule is from logging, and
indeed you might "use" another package that has an "add-rule"
function.  This is a name conflict.  The *whole idea* of packages is
to avoid name conflicts!  So I think "use" is opposed to the whole
idea of packages and should be used sparingly.

What people usually do in Common Lisp, in my experience, is to name
their functions so that the ones in one module (or, more generally,
related ones) all have names starting with a common prefix.  Well,
that's what packages were invented: so that you don't have to do that!
In QRes, even though there is a "qconfig" package for the
configuration stuff, the functions are still all named "config-, and
everybody uses "use".  I think this is not as good as using packages
as they were designed.  Rather than, e.g., config-descriptor-property,
use config:descriptor-property.  Anyway.)

Now, there's a great thing about CL packages: you can have more than
one defpackage for the *same* symbols.  So we could have one package
for the code that calls into our library, which would have the usual
protocol, which we can call the "external protocol".  Then we could
have a *second package*, which presents a second protocol that
subclases would use!  This gives us modular separation, just as
"protected" does.

Of course, there's nothing stopping any code at all from including a
symbol from this second package in its code.  But we already let
people go into internal symbols using "::".  Common Lisp's general
attitude is that we don't make these things impossible; you're just
not supposed to do them.  It's very often useful to do this kind of
violation when doing interactive debugging, or when writing white-box
unit tests.  (It would be great to have a static analysis tool that
flag you when you "cheat", that you could run before releasing a new
version of your stuff.)

I have not actually written anything this way, but it seems to me as
if it would work.  And if experience shows that it does provide what I
claim it will provide, I would love for that to become
idiomatic/standard usage in Common Lisp.

I admit that there's a significant problem, namely the same problem
that we always have with packages.  Because resolution of packages
happens at runtime, it is difficult-to-impossible, in some cases, to
change the package declarations during interactive debugging and have
the effect that the environment behaves as if you had changed the
package declaration and recompiled from scratch.  All I can say is
that we really ought to do something about the problems with packages,
even if you don't buy anything I'm saying here.

-- Dan