Thoughts on CL-scripting

[Faré]

On Mon, Feb 29, 2016 at 3:21 PM, Robert Goldman <rpgoldman at sift.net> wrote:
> This isn't really a standard bootstrapping issue, and I think it's
> better to break the vicious cycle by simply booting the scripting engine
> out into a separate repository.
>
Indeed, one big issue is that the asdf under test shouldn't be used to
compile asdf-tools. I believe what's currently provides a solution by
building asdf-tools the first time with a hopefully stable asdf.
Certainly, it all needs to be better documented / made more usable
when upgrade is needed, etc.

> Here's the (I believe simple) approach:
>
> 1. Set up a repository that provides a *general* CL-scripting capability.
>
This is not going to happen for a while. Also, part of the appeal of
CL scripting is the ability to use standard ASDF to load systems to
extend the scripting infrastructure. If the scripting systems are
preloaded and can't be upgraded, that's a problem. If they CAN be
upgraded, using ASDF, we're back to the same problem. If they CANNOT
be upgraded, that sucks. If they CAN be upgraded, but using something
else than ASDF... then either that something else is better, and then
we can drop ASDF but we have the same issue with that ASDF
replacement; or it's not better and then it's yet another bad tool
that makes CL scripting bad, too.

> 2. Build that capability using only stable tools.
>
Nope. There is an essential bootstrapping issue, here. We can move it
around, but it will remain.

> 3. Provide in the ASDF repository code that simply uses the CL-scripting
> capability.  I.e., just provide CL scripts there, and not the scripting
> engine.
>
Well, first, a lot of the dependencies are used in the tests themselves.
Also, a lot of the code is ASDF-specific, and yet is written in a modular way,
which, in CL, means using ASDF. Even if somehow dependencies are
magically included
in the generic tool, we'll still want to use ASDF to load the
asdf-specific code in asdf-tools.

> At this point we would have broken the vicious cycle, and the CL
> scripting engine would play a role akin to perl, make, etc., in the ASDF
> development process.
>
No, we haven't broken the vicious circle. The only way to break the
vicious circle for ASDF is if there's a non-ASDF tool being used to
build the code, in which case THAT tool has the vicious circle. That
said, if we adopt, e.g. Bazel as the standard way to build and test
ASDF, then yes, we pushed back the bootstrapping issues onto Bazel.

> Instead, right now we have things like ASDF requiring to test that it
> successfully be able to build a tool suite using a new ASDF capability.
>
Well, that's why I introduced an extra step, i.e. build_asdf_tools,
which builds a ./build/asdf-tools binary, and that you should run with
a stable asdf, so you can later use it to test an unstable asdf.

> Note that this is what I'm suggesting *in the abstract*.  Right now, the
> CL scripting framework is not, IMO, ready for prime time.  I'm not
> willing to wrestle with undocumented, untested code as a precondition
> for simply fixing ASDF bugs.
>
> I'm not saying that it's bad that CL scripting is at a primitive state
> now -- that's a state all software has to go through.  All I'm saying is
> that CL scripting has a lot of promise, but it's not the kind of tested
> tool I require for ASDF development, and it's not going to get there
> (IMO) for some years yet.
>
asdf-tools depends on many systems, most of which are relatively old
and stable and widely used in the community and/or well-tested, and
not scripting-specific.

Three systems that I wrote may count as new, unstable, and
scripting-related: inferior-shell, cl-launch, and cl-scripting.

inferior-shell, which provides a user-friendlier front-end to
uiop:run-program, including string interpolation, rich fd redirections
and piping, is moderately documented, and has a small test suite. I
believe it has several users besides me, and the trivial parts used by
asdf-tools are stable.

The cl-scripting system that handles don't-stop behavior akin to make
-k on the other hand is relatively new and largely undocumented
indeed. But it now has a test suite (that I added after you reported
some bad bugs indeed), and the code is quite small and
straightforward. ASDF is the only user so far (beside other personal
scripts of mine) and I apologize for having things break on you, but I
believe the bugs have now been fixed, and with tests added, and the
parts used by ASDF should now be stable.

Finally, I believe cl-launch has several users besides me, and it has
had an extensive test suite for many years. The test suite doesn't
cover the relatively new cl-launch/dispatch code that asdf-tools
relies upon, but that code is very small and simple, stable, and
heavily used by myself (and by anyone else who might use cl-launch to
produce multi-call binaries). There haven't been any reported problems
with that code, so I presume it's stable for good.

> So I am backing slowly away from it....
>
The cl-scripting functionality (now that it's debugged) is miles ahead
of what make -k provided in terms of usability: it gives you a
readable summary of all the failures that happened during a run of
test-all-no-stop, with enough context to replay. Detailed instructions
to replay the failures are also included in the log file — but the log
file isn't vomited at the console as it is with the shell-based
testing framework, so your testing status isn't lost in horrible
testing output.

Yes, there have been issues in the past, and you've been bitten by a
lot of them, especially as you've been testing on Windows and Mac
which I don't have regular access to (but I could steal my wife's mac
once in a while and install and configure a Windows VM at some point).

There are issues, notably bootstrapping issues, with using CL to test
ASDF. I believe these issues are fixable, that most have been fixed,
and that the result is better than depending on an unreadable
hodge-podge of Make, shell, sed and perl, that also have to be
installed and curated.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
The probability of the people in power being individuals who would dislike the
possession and exercise of power is on a level with the probability that an
extremely tender-hearted person would get the job of whipping-master on a
slave plantation. — Frank H. Knight