[asdf-devel] ANU, or changes which should be backported to ASDF 1.0 [ Re: Enforcing pure *.asd files (1)

[james anderson]

good morning;

On 2010-03-18, at 21:41 , Juan Jose Garcia-Ripoll wrote:

> This is an idea that has been long floating in the back of my mind,  
> and was brought back to life by these comments
>    http://groups.google.com/group/comp.lang.lisp/msg/f99a69797eda1caf
>
> The problem is that many people use *.asd files to do things like  
> building up packages, creating operations, defining methods, etc.  
> That makes it impossible to ship those libraries because the side  
> effects are not recorded in the system itself. So if ECL traverses  
> the system to build a standalone program it will know which sources  
> are required, but those sources may for instance rely on a package  
> that was created in the *.asd definition.
>
> My suggestion is a gradual move towards enforcing that *.asd files  
> become mere descriptions of the system and that side effects needed  
> for building and loading are listed in the system itself.


JJ.Garcia-Ripoll posed a question central to asdf: should it enforce  
'pure' system definitions?
At first, the suggestion seemed unrealistic. Upon reflection, it  
remains at odds with the prevailing reality, but it reveals that this  
reality is not the one which asdf should choose. A bit of reflection  
and a small experiment compel the realization, that asdf is not unix  
and should not persist as if it were. It should concentrate on  
describing lisp programs.

An .asd form is an abstract description of a lisp program, of the  
program's components, their construction ordering constraints, and  
their locations. The asdf facility should be able to use any  
valid .asd forms to infer that information about a system, use it to  
construct the system, and report the information to an application.  
Nothing else matters. If a form cannot be used for that purpose, it  
is invalid. asdf implementation has two essential aspects. First, to  
interpret the definition forms to construct an annotated graph which  
models the ordering, location, and operation information. Second, to  
use that graph to construct the program.  Under present  
circumstances, for between ten and twenty percent of open-source lisp  
projects, asdf does not satisfy even the first requirement. This is  
perhaps not disgraceful, but well more than lamentable. asdf in its  
present form and in the present .asd ecosystem is by no means capable  
to perform the task which distinguishes its purpose. We need to fix  
that. Fix it before one progresses to version 3, or version 2, or  
even version 1.7.

One may object, that the suggestion is unrealistic. That it is  
impossible and would sacrifice backwards compatibility. I expect the  
first assertion to turn out to be false. The second demonstrates the  
central problem, that lisp developers have had to learn to live with  
asdf inadequacies. Somehow. In the process, .asd forms have evolved  
to compensate for essential deficiencies. In order to work around  
deficiencies, developers use .asd files to extend the build facility  
itself simultaneous with defining the program to be built. To the  
extent, that the actual system definition is incidental. In some cases 
[0] to the point that the thought occurs, "maybe it would be easier  
to map some load equivalent over a file list."

Several asdf deficiencies require these practices. All should be  
corrected and the practices deprecated. To the effect that they are  
not supported in future asdf versions. One might even consider to  
back-port the corrections, but that might prove unpopular.

The primary problem is that there is no way to declare extensions. As  
bemoaned by T.Rittweiler with deference to S.Ionescu, and  
demonstrated by projects which use extensions provided by cffi- 
grovel, asdf-system-connections, as well as projects like cl-perec  
and steffil, which define extension classes in-line, there is neither  
a form which declares the system definition's dependency on an  
extension, nor one which incorporates an extension. In the case of  
the last two projects, they choose to resort to loading the .asd  
files into project-specific packages in order to  to distinguishing  
homonyms. Yes, it is possible. The effect of such license is that the  
asdf facility cannot interpret a significant percentage of the system  
descriptions in the wild without running arbitrary computations.[1]  
Yes, it is possible, but there is no reason to tolerate it. In  
particular, because this is not difficult to fix. use-package would  
suffice.

Contributing to this problem is that the system and component forms  
are not symmetrical. The intended syntax is[2], but the  
implementation is not:

? (defclass specialized-system (asdf:system) ())
#<STANDARD-CLASS SPECIALIZED-SYSTEM>
? (defclass specialized-source-file (asdf:cl-source-file) ())
#<STANDARD-CLASS SPECIALIZED-SOURCE-FILE>
? (eval '(asdf:defsystem :a-system :class specialized- 
system :components ((:file "a-file"))))
#<SPECIALIZED-SYSTEM "a-system" #xEF6724E>
? (eval '(asdf:defsystem :a-system :class specialized- 
system :components ((:file "a-file" :class specialized-source-file))))
 > Error: :CLASS is an invalid initarg to REINITIALIZE-INSTANCE for  
#<STANDARD-CLASS ASDF:CL-SOURCE-FILE>.
 >        Valid initargs: #(:CONTINGENT-ON :LONG- 
DESCRIPTION :DESCRIPTION :NAME :VERSION :IN-ORDER-TO :DO- 
FIRST :PARENT :PATHNAME :PROPERTIES).
 > While executing: CCL::CHECK-INITARGS
 > Type Command-. to abort.
See the Restarts… menu item for further choices.
1 >

This should be corrected, in order that - even in the absence of an  
intended extension, asdf can interpret the standard information in a  
system definition.

A related issue, is that, without the extension, a specialization's  
initialization arguments would not be permitted. This should be  
corrected as well.

An a-bit-more-than thought experiment issued from the following use  
case. Would it be possible to perform some kind of introspective  
analysis of the lisp open-source ecosytem without opening a server to  
arbitrary computation? Something in the same realm as clbuild, but  
without the script maintenance. The short answer: to a large percent  
yes, but there are some projects which reside well beyond the reach  
of such a service.

It was a simple experiment. Define a restricted package and  
interpret .asd files with a restricted read-eval loop.

[ continued ... ]



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20100323/542e7507/attachment.html>