[Asdf-devel] converting asdf build & test to Lisp

[Faré]

> I fear that this will sound crabby, but....  I am not very happy with
> this new makefile as a maintainer user interface.
>
I agree. The current minimakefile branch is basically working,
but its interface needs to be souped up a lot.
I just added a minimal help interface, based on extracted docstrings.
There need be a better way to register functions and document
the way to pass them arguments
(probably as part of deftestcmd, that needs be souped up).

> With the old makefile, I could look at the makefile and guess what the
> targets were that I wanted.
>
make help or ./tools/asdf-tools help will help a bit.
Most docstrings need enhancement, but that's a start.

> Better yet, as I got a relatively ok memory for what the targets were,
> bash completion would help me out.
>
I'm not proficient to help with bash completion, except that somehow
phony Makefile targets could be added semi-automatically.
I'll leave that to you.

> The new Makefile, on the other hand, has essentially no affordances. At
> least not beyond "make".  E.g., is "make test-all-no-upgrade" still
> provided?  There's no way to tell.
>
> Since there's not an API that will be called, the asdf-tools
> package.lisp file is empty.  So there's no obvious place to begin to
> look for what commands to issue.  Must we just read over the twelve lisp
> source files to figure out how to use the lisp functions?  And how to
> invoke them through make?
>
Yes, I agree that's important. It's just that it was (as usual) more work
to do the conversion that I expected, and I didn't get to improving the
interface (yet).

> And what happens to the old make variable processing?
>
>         ./tools/asdf-tools make-target $@ l="$l" L="$L" u="$u" U="$u" v="$v"
> s="$s" t="$t"
>
> this really isn't at all meaningful (I can guess -- but only based on
> past experience -- that l is the lisp implementation and t is a test
> name; other than that, I'm lost).
>
> This also gives us the union of all possible arguments, instead of
> cueing the user about which arguments are wanted for which commands.
>
That's the only way I found to pass arguments to asdf-tools from make.
Hopefully, some documentation can help explain what's going on,
and Makefile hints, bash and/or zsh completion scripts
can be automatically generated.

> IMO we are better off, as the saying goes, "in for a penny, in for a
> pound." If one wants to replace the original makefile with lisp
> scripting, that lisp scripting (a) should be invoked from the shell with
> no make scrim in between; (b) should offer entry points, probably
> modeled on multiple entry point programs like git and svn ("asdf-tools
> help", "asdf-tools help test", etc.); (c) should have a bash completion
> script with it ("asdf-tools t<TAB>" ==> test, test-all, test-no-upgrade,
> etc.).
>
asdf-tools CAN be invoked without make, as in, e.g.
    ./tools/asdf-tools install-asdf lispworks

But that make offers an easy portable bootstrap
that relies on an ubiquitous tool (make),
without having to have any particular lisp or worse to detect one.
If we could rely on either sbcl or cl-launch, we could do without make,
but I believe that's not a reasonable assumption at this time.
Also, I wanted to offer backward compatibility, at least for now.

> That said, while I'm sure that a halfway solution won't work, I'm not
> convinced that it's worth rewriting this into CL (although it's an
> interesting exercise).
>
I don't think it was worth it, either, except as indeed a learning exercise.
That said, now that it's done, I believe there's value in keeping it.

> I'm still skeptical: before lisp is ready for prime time scripting, it
> needs a *much more terse* way of specifying pathnames, at least as good
> as globbing; interaction with regexps in a less bloated way than PPCRE;
> and much more terse and transparent subprocess interaction (it's hard to
> beat "foo | bar").  Oh, and a backtrace facility where we don't have
> more than half the backtrace cluttered with pointless meta-trivia about
> the backtracing process, rather than with information about the error in
> question.
>
Looking at the code in asdf-tools,
I'm not sure how pathnames can usefully get terser.

The globbing has regressed indeed, but can be worked on:
some existing globbing library could be used, or implemented.
* still works as a wildcard, as in e.g.
   make test l=ccl t='test-*.script '

A better interface to cl-ppcre is optima.ppcre.
I just didn't want to add yet another dependency without consulting you.
But I just love it.

> You have done stellar work giving CL the tools it needs to be a
> scripting language.  But it still doesn't have syntax optimized for
> scripting, and the interaction between the shell and CL is still a lot
> more fussy than I'd like.
>
I agree — but it's only by writing, using and refining the tools
that they can be made great.
If we omit the very real but addressable UI issues you mention,
I believe that the current CL script is much much better than
the shell scripts it replaces:
* better error-handling
* much more general invoke-lisp.lisp extension to lisp-invocation.asd;
  the compose-eval-string utility is very useful.
* the run-test-lisp abstraction that we were really missing with bash.
* better, more uniform, defaulting.
* better consistency checking, thanks to richer variable manipulation
 and less unwieldy computation than in the shell

> Getting that syntax done and right is a very interesting process, but
> not one I'm willing to interleave with my ordinary workaday tasks.
>
Acknowledged. The tool can be used in quite close a syntax to the
previous one, though
— all it needs is enough documentation to describe parameter passing.

Also, it can be used from the SLIME REPL, which might be nicer than
the shell command-line:
(asdf:load-system :asdf-tools)

> Well, let's try this again.  I have looked more deeply, and it's amazing
> what you have achieved.
>
Thanks for your appreciation!

> I am left with my concerns about the interface that this scripting
> offers, but maybe by adding cl-launch (to remove the makefile layer),
> and providing better interaction for the user (completion +
> documentation), we can get this to the point where it would be a good
> solution.
>
My only concern about removing the Makefile is that I'd like a way
to bootstrap asdf.lisp from its many components without a working asdf
(that cl-launch requires, though it could be from the implementation).
Now that most implementations provide asdf, that's not as big an issue
as that used to be, but still.

> A remaining problem is that now getting an ASDF development environment
> starts to be more difficult.  How are we to manage installing the
> dependencies if the ASDF development head is going to be *ahead* of the
> state of Quicklisp (and, I believe now, *incompatibly* ahead)?
>
Yeah, well, life on the bleeding edge can be painful.
Hopefully, things will stabilize soon,
and then all dependencies will be in Quicklisp.

> It was already difficult for me using the bump script, but most of the
> makefile required only ASDF and a standard Linux or Mac +
> MacPorts|Homebrew install. I'm just worried about circular dependencies
> if we need to get a bunch of CL libraries to make this turn over.
>
That was my main reason for keeping a Makefile, albeit minimal.

In any case, please take another look and
tell me if anything more is needed to merge into master.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Politicians are like diapers: they must be changed often.
And for the same reasons. [Also, adults don't need either of them. — Faré]