[asdf-install-devel] Patches, Darcs, 'n' stuff

[Dan Muller]

Gary King writes:
> My apologies for being so slow in responding to your posts.

Thanks for the reply! Your email raises lots of thoughts. And although
I like quick responses (who doesn't?), I'm sure this is a sideline for
all of us. I just happened to have a lot of free time this weekend.

If I'm understanding correctly, you're talking about ASDF2 as a
from-the-ground-up rewrite of ASDF and ASDF-INSTALL. To answer one of
your last questions first: Personally, that sounds like a larger
project that I was looking for.  My Lisp programming usually occurs in
short but intense bursts, and I have another, much larger project
still pending. (And a day job that has, sadly, nothing to do with Lisp,
and often consumes more than its share of my time.)

OTOH, build systems have always interested me, so I wouldn't entirely
turn my back on such a project.

The rest of my reply aims to be more objective, although it's of
course possible that it's colored by my personal inclinations.

I wonder why a complete rewrite of either ASDF or ASDF2 is
indicated. Typical good reasons for rewrites are:

* A codebase that is not amenable to maintenance or extension. Usually
  large, poorly written, and not enough developers on hand familiar
  with it. (The "write two and throw the first one away" technique
  only works if the second is written immediately by the same
  programmers!)

* Some major shift in underlying technology is needed due to changing
  environment.

* The requirements have changed completely, such that most of the
  program needs to be replaced to accomodate them.

* Just for fun. (Sometimes I love a good, pointless rewrite as much as
  the next hacker.)

None of those really seems to apply here, as far as I understand your
idea at this point, except maybe the last one. ASDF is relatively
small (~1300 lines, straight wc count), ASDF-INSTALL is not much
larger (~1900 lines). Lots of people have hacked both of them, and
most of those folks are still accessible, I think. Common Lisp isn't
changing. And most of the requirements you listed had to do with
backwards compatibility, not new requirements.

The biggest downside of a rewrite is the usual: It's difficult to know
all the knowledge that's already embodied in the existing code. You
often end up rediscovering all the same problems that the previous
authors did, rediscovering workarounds as you go. (Heck, that goes for
my patches, too. You should reject them if they make you nervous; I
can rewrite the fixes with fewer changes, now that I understand
ASDF-INSTALL better.)

However, you didn't go into details as to what *new* things are wanted
from ASDF, so maybe there's more justification than I'm thinking. The
default choice should be to build on what's there, though.

My other concern is that, to a large extent, ASDF and ASDF-INSTALL
really handle different concerns. Glomming them together doesn't seem
like a big win to me, except for sharing a little bit of code, and
simplifying initial setup of a development environment. Now, that
latter point is nothing to sneeze at, but I think it might be better
to get there in stages. For instance:

* Fix ASDF-INSTALL's portability and configuration difficulties.

* Integrate ASDF-INSTALL with ASDF using the latter's existing
  extensibility features.  This should be relatively easy, and would
  allow people to find out if it really makes sense to combine them.

* Make ASDF-INSTALL able to understand the concept of
  "updating". Apply it to itself. Then it will be ready to try
  updating ASDF, or any other library for that matter.

* Make ASDF-INSTALL itself even easier to install. Ideally a
  single-file download, like ASDF!

* At that point, consider the benefits of combining the projects. (But
  note that some people will never really want that integration,
  because there *are* alternatives to ASDF-INSTALL that make more
  sense in some environments.)

At the very least, keeping a clean separation between build system
issues and installation issues, even if that separation were internal
to an ASDF2, would be good policy, since many of the issues are
distinct, don't you think?

ASDF already has moderately good mechanisms for this: system
definition search functions, and extensibility that is often used for
things like stowing FASLs in separate directories, and rebuilding
stale fasls. The latter extensions seem so common that they ought to
incorporated into ASDF; that's a good example of incremental
improvement.


OK, I worked pretty hard on my reply up to this point. Now let me
throw out some more stuff quickly to answer other questions you
raised, both explicitly and implicitly, so I can finally send this.

Regarding MK-DEFSYSTEM, it is my understanding that it has features
which ASDF still hasn't replicated, it has some continuing adherents,
and its author does not consider it a dead project. (I could be wrong
on some of these.) In any case, accomodating it is not hard, and doing
so encourages that clean separation that I discussed earlier.

The changes to ASDF-INSTALL that I have (the ones you haven't seen
yet) are not minor, but neither are they drastic. They separate
defsystem-specifics more cleanly from the rest of what ASDF-INSTALL
does.

> In the past, there has been concern about putting too many bells and
> whistles into ASDF (and ASDF-Install) where "too many" seems to mean
> "anything that is not there already".

IMO, making things easier for newcomers should take precedence over
disgruntling current users; there are a lot of new people coming to
Lisp, and my impression is that few of them end up using ASDF-INSTALL
because of its current limitations on Windows. If an incompatible
change is necessary, document it well and provide whatever is needed
to make it easy to adapt to it. If someone hates it, they don't have
to upgrade, or they can even fork. Someone already decided that
ASDF-INSTALL is supposed to support Windows, and it doesn't do that
well. We either undo that decision, or do what it takes to make it
really support Windows. (Similar argument for MK-DEFSYSTEM; in both
cases I'm advocating preserving the decisions.)

> ... The only concern I have is
> making any migrations smooth and of keeping all the "stakeholders" of
> ASDF and ASDF-Install in the loop.

Avoiding unintended breakage *is* important, though. I think your test
setup is a great step in the right direction, and wish I'd checked it
out earlier. But can we set things up so that candidate changes can be
easily tested on a wide variety of platforms? How many environments
(OSes, CL implementations) do you currently have the ability to test
on yourself? (BTW, you didn't comment on the unit test that's broken
for Windows.)

As to bootstrap-friendliness: One thing that ASDF has going for it,
and ASDF-INSTALL does not, is that the former is a single file.
Incorporating ASDF-INSTALL directly into ASDF would more than double
the latter's size, even if you assumed some economy of shared code
arising from the merger. Instead of going this route, it would seem
preferable to (if possible) simplify ASDF-INSTALL's internals,
gradually getting it down to where it can also be presented as a
single file. The eventual goal would be that, instead of people being
told to download and load ASDF as one of the first steps in setting up
an environment, they'd be told to do the same thing with ASDF-INSTALL
first, and then use it to install ASDF! ASDF-INSTALL doesn't actually
have to depend on *any* defsystem to do non-recursive *installation*
of a library into site-dependent locations. OK, that's kind of out
there, and would *really* beg a rename for ASDF-INSTALL. :)

> ... I think that
> there is room for better system definition facilities and am willing
> to volunteer some time towards the effort. I know that several others
> on this list have expressed interest in the past. Is this the time to
> take another crack at specifying and creating a new beast?

I'm totally into talking about what would be desired from an ASDF2,
BTW, although I'll always look at the ideas and consider whether they
could be realized in the current systems. Can you point me to earlier
discussions on this topic? Haven't found too much on my own so far.

 -- Dan Muller