[asdf-devel] ASDF 2.0 requirements : simplify the output locations mechanism

Faré fahree at gmail.com
Wed Apr 21 02:00:29 UTC 2010 

Dear James,

> before one can declare asdf 2.0 to be feature complete, one should
> have a specification of its intended function and then gauge the
> implementation in those terms.
>
I humbly disagree. The standards that I have set to myself for ASDF 2.0
are as follows:

  1- improve the ASDF interface
     * provide portable ways to do things that could not be done portably
     * have no regressions on what was previously portable
     * have a clear better way for anything that used to work widely
       in practice but was non-portable

  2- improve the ASDF implementation
     * fix bugs, clean up internals
     * performance: use hash-tables instead of linear search assoc lists
     * make all existing tests pass everywhere, add a few tests.

  3- make ASDF versioned and upgradable
     * make it easy to detect if you're using a recent-enough ASDF
     * provide a clear version 2.0 for CL implementations to provide
     * decouple _future_ release cycles of ASDF and CL implementations

  4- make ASDF user-configurable
     * allow for configuration through user-controlled files
       separate from developer-controlled build scripts
     * provide early input validation for user-controlled files
     * still allow developers to easily override when needed,
       and support previous ways of configuring ASDF.

  5- improve the documentation
     * update the documentation without introducing any regression
     * document enough so a newbie can survive
     * cleanup the manual, update the website


EXPLICITLY NOT INCLUDED amongst my goals for ASDF 2.0, are the following
goals that a future maintainer may or may not include as his goal for a
future release:

  * make the ASDF interface fully declarative.
  * make useful but backwards-incompatible changes.
  * support the distribution of common extensions (contrib, etc.)
  * have the documentation cover 100% of the interface.
  * have tests cover 100% of code.
  * keep ASDF under any specific code size.


> while the users' manual does explain
> aspects of the user and/or programmer interface, in addition to the
> descriptions of the interface, there should be something which
> describes the requirements which are being beta-tested - both to
> assess completeness and to inform future maintenance.
>
This is too abstract for me. If there are specific elements
you think should be documented, please list them concretely.

One way you could do it is to send a patch of a first draft
of a modification of the manual that you require,
with holes where you want me to fill information.


> the additions
> to the code base over the past few months were devoted to two areas:
>
> - configuration
> - output translations
>
> i neglect, for the moment, the configuration component of the asdf
> evolution and concentrate on questions related to output
> translations. even though the configuration mechanism contributes
> settings which affected the latter, they are in sufficiently
> independent, that an isolated analysis is reasonable.
>
Well, most of the translation code *IS* the configuration system.
The output-files :around method itself is small, and its helpers are
relatively simple, except for the ugly directorize-pathname-host-device
required for portability (that none of the previous solutions provided).


> in lieu of a specification, several asdf-devel messages on the topic
> of output translations[1--5] discuss requirements. the asdf-binary-
> location users' guide includes a "what it does" passage[6]. the asdf-
> binary-locations-devel archives[7] include no additional discussions.
> to follow those discussions, an output translation mechanism should
> make it possible to
>
> - place (output) files under a single root
> - place (output) files in a tree specific to (runtime x version x
> project x file-type)
> - include process information (eg, pid or user id) in the location
> - specify elements of the mapping programmatically
> - specify elements of the mapping by configuration
> - combine independent specifications
> - express location maps both globally and in connection with specific
> sub-systems (cl-controller, cl-launch, etc)
> - support simultaneous, independent mapping for binary output and
> ancillary data files
> - disable translations for a given path
> - disable translations completely
>
> in response to these requirements, "asdf binary locations" was
> developed, first as an extension to, then integrated with asdf, and
> further developed into a general output translation facility. while
> the evolution brings definite benefits, in the process, between
> version 1.362, when a-b-l was incorporated into asdf core, and the
> current +/- 1.7, the source size has increased from 56K to 132K, the
> function count has grown from less than one hundred to more than two
> hundred, and the call count has almost tripled.
>
Yes. Configuration support has doubled the size of ASDF, and added
features, cleanups and robustification have also increased its size.
I don't think any of this stuff can be done without.


> the call graphs[8] depict the situation. each node in the graph
> corresponds to a function in the asdf implementation, each edge to a
> call. the green nodes were present in 1.362, the yellow nodes in
> 1.502, and red nodes in 1.7. filled nodes have remained, while empty
> nodes indicate functions which are no longer present. the increase in
> code mass and complexity is evident. since, as expressed, the
> requirements could be met by logical pathname translations, the
> situation causes concern. there is nothing in the functional
> requirements, as such, which dictates an implementation of this
> complexity.
>
I don't think the implementation is particularly complex.
It is larger because it does more things.

Make things as simple as possible BUT NO SIMPLER.

Show me which function you'd like to see gotten rid of, and
I'll explain you why it is necessary - or get rid of it.

> there are, however, non-functional concerns which appear
> to have informed the decision: it has been held, that
> - logical pathnames are not portable;
> -  the facility must operate in terms of physical pathnames as real-
> world lisp applications require file names which do not conform to
> logical pathname syntax constraints.
>
Yes.

> in the course of testing the output location mapping itself, it was
> demonstrated[9] that the non-portability claims were unfounded. an
> experiment based on 1.502 demonstrated[10] that a mechanism which
> separates location translation from truename processing can provide
> the required functionality with much less code in terms of logical
> pathnames despite non-conformant physical pathnames. the logical
> pathname based location maps were then re-implemented in 1.702 and
> further experiments demonstrated that - with changes to the scripts
> to account for functions which would be removed, all regression tests
> pass, and that with appropriate initialization, a multi-runtime build
> is supported for the normal abcl/acl/ccl/clisp/cmucl/ecl/lw/sbcl
> complement. the following host definition, for example, produces a
> centralized per-runtime translation tree for output files.
>
> (let ((root (make-pathname :directory (butlast (pathname-directory
> *build-init-pathname*) 2)
>                            :name nil :type nil :defaults *build-init-
> pathname*))
>       (host "SOURCES"))
>   (setf (logical-pathname-translations host) '(("**;*.*"  "**;*.*")))
>   (let ((lbin (pathname-type (compile-file-pathname (make-
> pathname :host host :name "TEST" :type "LISP"))))
>         (bin (pathname-type (compile-file-pathname "test.lisp"))))
>     (setf (logical-pathname-translations host)
>         `((,(make-pathname :host host :directory '(:absolute :wild-
> inferiors) :name :wild :type lbin :version :wild)
>             ,(make-pathname :directory `(,@(pathname-directory root)
> "bin" ,(asdf::implementation-identifier) :wild-inferiors)
>                            :name :wild :type bin :version :wild))
>           (,(make-pathname :host host :directory '(:absolute :wild-
> inferiors) :name :wild :type :wild :version :wild)
>             ,(make-pathname :directory `(,@(pathname-directory
> root) :wild-inferiors)
>                             :name :wild :type :wild :version :wild)))))
>
>   (setq asdf::*translations-root*  (make-pathname :host
> host :directory '(:absolute) :name nil :type nil)))
>
> given which, the question occurs, whether the output locations
> implementation deserves further thought before it advances to beta
> status?
>
Less code that what? How is the end-user meant to deal with configuration?
The horribly complex logical pathname translation above is not what I
would like to have to explain to users in the manual.

If you want people to adopt logical pathnames -- get the spec fixed,
and convince implementers (notably SBCL implementers)
to adopt your fixed spec.

And even then you'll have a hard time fixing things that users
can now trivially do with the following translation:

(:root ("C:/my/cl/cache" :implementation))

This will work and make his file in
  \\remote-host\myshare\cl\cl-foo\foo-V2.000\bar_baz.lisp
be compiled as obvious from the spec into
  C:\my\cl\cache\sbcl-1.0.38-windows-x86\remote-host\myshare\cl\cl-foo\foo-V2.000\bar_baz.fasl
He won't have to understand the innards of any magic logical pathname
host mapping, and he won't have to write or decode such mapping
tha may map it to wherever some weirdly named file
   TEST:CL-FOO;FOO-V2-000-GENERATED-12345;BAR-BAZ-GENERATED-45678.FASL
may point.


> [1] : http://common-lisp.net/pipermail/asdf-devel/2010-March/000943.html
> [2] : http://tinyurl.com/yyv2o4w
> [3] : http://common-lisp.net/pipermail/asdf-devel/2010-February/
> 000691.html
> [4] : http://common-lisp.net/pipermail/asdf-devel/2010-February/
> 000699.html
> [5] : http://common-lisp.net/pipermail/asdf-devel/2010-February/
> 000702.html
> [6] : http://common-lisp.net/project/asdf-binary-locations/user-
> guide.html
> [7] : http://common-lisp.net/pipermail/asdf-binary-locations-devel/
> [8] : http://reports.setf.de/asdf-versions-twopi.svg,
>       http://reports.setf.de/asdf-versions-twopi.pdf,
>       http://reports.setf.de/asdf-versions-dot.svg,
>       http://reports.setf.de/asdf-versions-dot.pdf
> [9] : http://tinyurl.com/y7jmzee : the test cases are present in dual
> form, one for a physical pathname and once for a logical pathname
> [10] : http://common-lisp.net/pipermail/asdf-devel/2010-April/
> 001351.html

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
License Agreement: By reading this message, you agree to run around the room
which you are currently in, flapping your arms, and squawking like a chicken.

More information about the asdf-devel mailing list