[asdf-devel] ASDF 2.0 requirements : simplify the output locations mechanism

[james anderson]

On 2010-04-21, at 04:00 , Faré wrote:

> Dear James,
>
>> before one can declare asdf 2.0 to be feature complete, one should
>> have a specification of its intended function and then gauge the
>> implementation in those terms.
>>
> I humbly disagree. The standards that I have set to myself for ASDF  
> 2.0
> are as follows:

  [ ... which list of goals should well appear in release notes, but  
does
    not address the concern raised in this thread, except in the  
exclusion,
    that ... ]
>
> EXPLICITLY NOT INCLUDED amongst my goals for ASDF 2.0, are the  
> following
> goals that a future maintainer may or may not include as his goal  
> for a
> future release:
>
>   * [ ... ]
>   * keep ASDF under any specific code size.

were one to be in a position to consider responsibility for the  
future maintenance of the code, that item would appear quite as far  
down on the list.

>
>
>> while the users' manual does explain
>> aspects of the user and/or programmer interface, in addition to the
>> descriptions of the interface, there should be something which
>> describes the requirements which are being beta-tested - both to
>> assess completeness and to inform future maintenance.
>>
> This is too abstract for me. If there are specific elements
> you think should be documented, please list them concretely.

i will help you.

there is a section in the asdf manual called "Controlling where ASDF  
saves compiled files". that section describes how a user can  
configure asdf to translate output files.

there should be two additional sections in a manual, perhaps a  
developers' manual, called "output translation requirements" and  
"output translation implementation" which describe - independent of  
how to use the facility, what it does and how it is implemented. a  
"requirements" section should describe at least
- what kinds of output translations are required. it should do this  
in terms of abstract characteristics, specific use cases, and  
enumerative examples;
- the operating systems and the project forms to be supported by asdf  
with attention to the required performance;
- what trade-offs are apparent in supporting which functional  
features, which options were elected, and why;
- an abstract model for the entities which the implementation is  
expected to represent, support, manipulate.

an "implementation" section should provide an overview of the one  
hundred twenty functions and twenty parameters which have been  
introduced in since version 1.362, with attention to how their  
relation to the functional requirements. section "8.6" in the asdf  
manual reads as if it might have been intended as notes for a section  
of this sort.

if this is still too abstract, any one of the readily available  
descriptions of "functional requirements", eg, here[1], might may  
help you to understand their purpose and construction.


>
> One way you could do it is to send a patch of a first draft
> of a modification of the manual that you require,
> with holes where you want me to fill information.
>
>
>> the additions
>> to the code base over the past few months were devoted to two areas:
>>
>> - configuration
>> - output translations
>>
>> i neglect, for the moment, the configuration component of the asdf
>> evolution and concentrate on questions related to output
>> translations. even though the configuration mechanism contributes
>> settings which affected the latter, they are in sufficiently
>> independent, that an isolated analysis is reasonable.
>>
> Well, most of the translation code *IS* the configuration system.
> The output-files :around method itself is small, and its helpers are
> relatively simple, except for the ugly directorize-pathname-host- 
> device
> required for portability (that none of the previous solutions  
> provided).
>
>
>> in lieu of a specification, several asdf-devel messages on the topic
>> of output translations[1--5] discuss requirements. the asdf-binary-
>> location users' guide includes a "what it does" passage[6]. the asdf-
>> binary-locations-devel archives[7] include no additional discussions.
>> to follow those discussions, an output translation mechanism should
>> make it possible to
>>
>> - place (output) files under a single root
>> - place (output) files in a tree specific to (runtime x version x
>> project x file-type)
>> - include process information (eg, pid or user id) in the location
>> - specify elements of the mapping programmatically
>> - specify elements of the mapping by configuration
>> - combine independent specifications
>> - express location maps both globally and in connection with specific
>> sub-systems (cl-controller, cl-launch, etc)
>> - support simultaneous, independent mapping for binary output and
>> ancillary data files
>> - disable translations for a given path
>> - disable translations completely

is this list complete? if not, it would help if you would complete  
it. it or something equivalent should appear in a "requirements"  
section.

>>
>> in response to these requirements, "asdf binary locations" was
>> developed, first as an extension to, then integrated with asdf, and
>> further developed into a general output translation facility. while
>> the evolution brings definite benefits, in the process, between
>> version 1.362, when a-b-l was incorporated into asdf core, and the
>> current +/- 1.7, the source size has increased from 56K to 132K, the
>> function count has grown from less than one hundred to more than two
>> hundred, and the call count has almost tripled.
>>
> Yes. Configuration support has doubled the size of ASDF, and added
> features, cleanups and robustification have also increased its size.
> I don't think any of this stuff can be done without.
>
>
>> the call graphs[8] depict the situation. each node in the graph
>> corresponds to a function in the asdf implementation, each edge to a
>> call. the green nodes were present in 1.362, the yellow nodes in
>> 1.502, and red nodes in 1.7. filled nodes have remained, while empty
>> nodes indicate functions which are no longer present. the increase in
>> code mass and complexity is evident. since, as expressed, the
>> requirements could be met by logical pathname translations, the
>> situation causes concern. there is nothing in the functional
>> requirements, as such, which dictates an implementation of this
>> complexity.
>>
> I don't think the implementation is particularly complex.
> It is larger because it does more things.
>
> Make things as simple as possible BUT NO SIMPLER.
>
> Show me which function you'd like to see gotten rid of, and
> I'll explain you why it is necessary - or get rid of it.

the patch which was enclosed in the earlier message[2] demonstrated  
an implementation which modifies the three functions - component- 
pathname, input-files, and output-files, extends the function which  
constructs the component absolute pathname, and adds a function to  
map between logical and physical pathname components. with respective  
calls from the other five functions. without having deconstructed the  
asdf 1.7* implementation to distinguish which functions relate to  
which functional components, on first order, five is five percent one  
hundred and twenty and an even smaller percentage of two hundred.

>
>> there are, however, non-functional concerns which appear
>> to have informed the decision: it has been held, that
>> - logical pathnames are not portable;
>> -  the facility must operate in terms of physical pathnames as real-
>> world lisp applications require file names which do not conform to
>> logical pathname syntax constraints.
>>
> Yes.
>> in the course of testing the output location mapping itself, it was
>> demonstrated[9] that the non-portability claims were unfounded. an
>> experiment based on 1.502 demonstrated[10] that a mechanism which
>> separates location translation from truename processing can provide
>> the required functionality with much less code in terms of logical
>> pathnames despite non-conformant physical pathnames. the logical
>> pathname based location maps were then re-implemented in 1.702 and
>> further experiments demonstrated that - with changes to the scripts
>> to account for functions which would be removed, all regression tests
>> pass, and that with appropriate initialization, a multi-runtime build
>> is supported for the normal abcl/acl/ccl/clisp/cmucl/ecl/lw/sbcl
>> complement. the following host definition, for example, produces a
>> centralized per-runtime translation tree for output files.
>>
>> (let ((root (make-pathname :directory (butlast (pathname-directory
>> *build-init-pathname*) 2)
>>                            :name nil :type nil :defaults *build-init-
>> pathname*))
>>       (host "SOURCES"))
>>   (setf (logical-pathname-translations host) '(("**;*.*"  "**;*.*")))
>>   (let ((lbin (pathname-type (compile-file-pathname (make-
>> pathname :host host :name "TEST" :type "LISP"))))
>>         (bin (pathname-type (compile-file-pathname "test.lisp"))))
>>     (setf (logical-pathname-translations host)
>>         `((,(make-pathname :host host :directory '(:absolute :wild-
>> inferiors) :name :wild :type lbin :version :wild)
>>             ,(make-pathname :directory `(,@(pathname-directory root)
>> "bin" ,(asdf::implementation-identifier) :wild-inferiors)
>>                            :name :wild :type bin :version :wild))
>>           (,(make-pathname :host host :directory '(:absolute :wild-
>> inferiors) :name :wild :type :wild :version :wild)
>>             ,(make-pathname :directory `(,@(pathname-directory
>> root) :wild-inferiors)
>>                             :name :wild :type :wild :version :wild))) 
>> ))
>>
>>   (setq asdf::*translations-root*  (make-pathname :host
>> host :directory '(:absolute) :name nil :type nil)))
>>
>> given which, the question occurs, whether the output locations
>> implementation deserves further thought before it advances to beta
>> status?
>>
> Less code that what? How is the end-user meant to deal with  
> configuration?
> The horribly complex logical pathname translation above is not what I
> would like to have to explain to users in the manual.

the irony in a situation, where a doubled code size and tripled call  
count are insignificant while a two-clause logical host definition is  
incomprehensible, is appreciable, but not significant. the matter at  
issue is not whether you can read logical host specifications, but  
whether the attendant translation mechanism would simplify asdf  
implementation.

>
> If you want people to adopt logical pathnames -- get the spec fixed,
> and convince implementers (notably SBCL implementers)
> to adopt your fixed spec.

we can discuss the general utility of logical pathnames at length, at  
your leisure. elsewhere. my enquiry concerns whether logical pathname  
translations would serve as a more compact implementation mechanism  
for asdf output translations than the present method.

>
> And even then you'll have a hard time fixing things that users
> can now trivially do with the following translation:
>
> (:root ("C:/my/cl/cache" :implementation))
>
> This will work and make his file in
>   \\remote-host\myshare\cl\cl-foo\foo-V2.000\bar_baz.lisp
> be compiled as obvious from the spec into
>   C:\my\cl\cache\sbcl-1.0.38-windows-x86\remote-host\myshare\cl\cl- 
> foo\foo-V2.000\bar_baz.fasl

this would appear to be the equivalent of a translation on the order of

  `("**;*.*.* ,(concatenate 'string "C:/my/cl/cache/"
                                    (asdf::implementation-identifier)
                                    "/**/*.*.*"))

but i may have misunderstood the requirement.

---
[1] : http://en.wikipedia.org/wiki/Functional_requirement
[2] : http://common-lisp.net/pipermail/asdf-devel/2010-April/001351.html