[asdf-devel] ASDF traverse changed behavior?

[Robert Goldman]

On 3/17/10 Mar 17 -4:57 PM, Juan Jose Garcia-Ripoll wrote:
> On Wed, Mar 17, 2010 at 9:46 PM, Robert Goldman <rpgoldman at sift.info
> <mailto:rpgoldman at sift.info>> wrote:
> 

> 
> Second, ECL's extensions are part of ASDF. It is not just a game we play
> to be on this list or to ship ASDF with our system. It also sucks my
> development time and prevents me from working on other useful things
> like ECL's compiler, performance, etc, but I believe on the utility of
> the project: being able to build and ship arbitrary ASDF systems as
> libraries, executables and the like, and improving integration of ASDF
> with systems that work better with monolithic binary files instead of
> thousands of small compiled files. This is a path other implementations
> may follow, but no one seems to care right now.

OK, we need to be much more clear about what is under discussion at any
point.

If you are talking about ECL's extensions which are part of ASDF,
instead of outside ASDF, then the stricture I proposed does not apply to
you.

The restrictions I proposed are for the kinds of ASDF extensions that
are put in individual .asd files and that are put into asdf add-on
libraries.  For those it is clear we cannot allow people to simply go
off and rewrite core bits of ASDF.  For example, if I'm writing the ASDF
definition for my FLOYD-WARSHALL transitive closure library, I can't be
putting into it

(defmethod PERFORM ((op load-op) (c cl-source-file))
 ....)

because that means that anyone who loads my library will be loading it
into a radically different environment.

The ECL extensions are obviously not this sort of thing.

It seems to me that you have two choices:

1.  really fuse the ECL extensions into ASDF proper, to the extent of
putting the source in there.  We will still rely on some periodic
testing to make sure that any incompatibilities are caught, but James
Anderson's work seems like it will fix this problem.

2.  keep ECL separate and provide some statement, ideally one that can
be programmatically checked, about the kind of API you expect.  For
example, you could add a test to the test suite that would check for the
absence of a method for PERFORM on the signature (T COMPONENT).  That
would verify that you have the method free for your own use.


> I agree that a common ground should be found, but be careful with your
> sentences.
> 
> This is not just 'people' and this is not just 'extenders'. Either you
> accept this is part of ASDF and accept its growing user base, or not. If
> asdf-ecl.lisp is part of ASDF then any maintainer should also care for
> what those other components need and do, and what they are based on.

Again, if this is part of ASDF, then you are correct, you are not just
"people," and these strictures wouldn't apply to you.

However, I think you have contributed to this confusion yourself, by
objecting to strictures meant only for ASDF system developers, on behalf
of something that you say yourself does not apply to you.
>  
> 
>     Please focus your attention on helping us base define a clear set of
>     guidelines rather than attacking the notion of limitations on the
>     extenders.
> 
> 
> What we do need is clear and self explained in the asdf-ecl.lisp
> extension and I discussed this here long long ago. If this has to be
> done again, then so it be.
> 
> asdf-ecl.lisp provides six new operations that have to integrate with
> ASDF and that can be used instead of LOAD-OP or COMPILE-OP. These
> operations are DLL-OP, LIB-OP, FASL-OP, monolithic versions (that is
> everything in one file) and a PROGRAM-OP.
> 
> These operations proceed in a simple way: they use TRAVERSE with a
> LOAD-OP to know what a system uses, and then they use PERFORM with a
> COMPILE-OP to actually compile it and from the list that TRAVERSE
> creates they learn the list of compiled files and build things with them
> (library, program, whatever)
> 
> Integration with TRAVERSE is crucial and depending on how _you_ _all_
> decide that TRAVERSE should behave then we will have to adapt the code
> one way or another. This integration is fragile. If at some point the
> algorithm is randomly changed and you decide that systems should be
> listed before components, then we are f*ck*d.

Can you say what exactly has goofed this up for you?  Note that TRAVERSE
is not deterministic --- it's possible to specify multiple different
orderings that satisfy the dependency specification in the ASDF file.

It's possible that what is going on is simply a bug, but I can't tell
without knowing what has changed.  Can you give me a concrete example?
The behavior of TRAVERSE should not have changed for your application
(except up to the nondeterminism to which I refer above); the changes
should only appear on REcompilation.

> 
> Until now our own operations listed no dependencies other than the
> LOAD-OP and COMPILE-OP mentioned before. TRAVERSE was just happy with
> that. Right now ASDF is adding additional dependencies such as
> performing the same operations (DLL-OP, LIB-OP...) on the components
> itself. This was my source of surprise when I started this thread.

As far as I can tell, if ASDF was not doing that before, it was an
unintended behavior.

If individual files are not to be subjected to these operations, would
it be possible to define these operations as SYSTEM-OPERATIONS and
simply define a method on PERFORM that would ensure that
SYSTEM-OPERATIONS on COMPONENTS are no-ops?
> 
> asdf-ecl.lisp only defined methods on COMPONENT-DEPENDS-ON, INPUT-FILES,
> OUTPUT-FILES, PERFORM and OPERATION-DONE-P. All the signatures we used
> contain at least either OUR components our OUR operations, so in that
> sense they should be allowed to exist and ASDF should be careful on
> deciding what to do with these methods -- for instance, rewriting paths
> as we discussed on another thread.

I thought you were specifically referring to core methods.  I see that I
was not fully understanding.
> 
> Note that the protocol for writing these new operations was never clear
> and it has grown up with experience and by inspecting ASDF's codebase.
> At some point the extensions were accepted in the source tree, point at
> which I asked whether the integration that was done the right way or
> not. The lack of answer was understood as an ok.

This exhibits two problems ASDF development.  The underlying problem is
that people are doing this work on very thin time slices.

1.  Lack of answer can never effectively be treated as "ok" --- lack of
an answer is more likely to simply mean "I haven't had time to look at
this."  My changes to TRAVERSE went in a very long time ago, and only
after a very long discussion in the mailing list.  It's clear that you
didn't have time to review them then, and so an issue that we thought
was closed will have to be reopened.

2.  The documentation is badly bit-rotted.  The section on extending the
ASDF object model is in particularly bad shape.  It was never complete,
as far as I can tell, and it has not been updated.  We badly need a
protocol specification stating exactly what has to be defined to make
new components and operations.  I can't say that I fully understand how
to do this --- my experiments have often seemed to go just fine, then
led to me stumbling much later on a problem that came up because some
critical method was missing.

Any suggestion about how to document this protocol would be very helpful.