[asdf-devel] ASDF 2.016 breaks ABCL translations for jar files

[Mark Evenson]

On 6/9/11 16:37 , Faré wrote:
[…]

> Can you write a simple test to be added to test/asdf-pathnames.script,

I'll work on that.  Can I just create a new top-level form or do you 
prefer I wedge it into TEST-COMPONENT-PATHNAMES with an #+abcl/#-abcl 
conditional?

> and/or send me an example such that I can reproduce the bug at home?
> Maybe just a trace output of the relevant functions:
> (trace translate-pathname* and compile-file-pathname* apply-output-translations)
> (add any function there necessary to show where it breaks).

A trace for the JSS system packaged in abcl-contrib.jar:

CL-USER> (asdf:compile-system :jss :force t)
   0: (ASDF:APPLY-OUTPUT-TRANSLATIONS 
#P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.abcl")
     1: (ASDF::TRANSLATE-PATHNAME* 
#P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.abcl" 
#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION ASDF::TRANSLATE-JAR-PATHNAME 
{66EF7D74}> NIL #P"jar:file:/**/*.jar!/**/*.*")
       2: (ASDF:APPLY-OUTPUT-TRANSLATIONS 
#P"/___jar___file___root___/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl")
         3: (ASDF::TRANSLATE-PATHNAME* 
#P"/___jar___file___root___/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 
#P"/___jar___file___root___/**/*.*" 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" 
NIL #P"/___jar___file___root___/**/*.*")
         3: TRANSLATE-PATHNAME* returned 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
       2: APPLY-OUTPUT-TRANSLATIONS returned 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
     1: TRANSLATE-PATHNAME* returned 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
   0: APPLY-OUTPUT-TRANSLATIONS returned 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
   0: (ASDF:APPLY-OUTPUT-TRANSLATIONS 
#P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.abcl")
     1: (ASDF::TRANSLATE-PATHNAME* 
#P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.abcl" 
#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION ASDF::TRANSLATE-JAR-PATHNAME 
{66EF7D74}> NIL #P"jar:file:/**/*.jar!/**/*.*")
       2: (ASDF:APPLY-OUTPUT-TRANSLATIONS 
#P"/___jar___file___root___/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl")
         3: (ASDF::TRANSLATE-PATHNAME* 
#P"/___jar___file___root___/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 
#P"/___jar___file___root___/**/*.*" 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" 
NIL #P"/___jar___file___root___/**/*.*")
         3: TRANSLATE-PATHNAME* returned 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
       2: APPLY-OUTPUT-TRANSLATIONS returned 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
     1: TRANSLATE-PATHNAME* returned 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
   0: APPLY-OUTPUT-TRANSLATIONS returned 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
   0: (ASDF:COMPILE-FILE-PATHNAME* 
#P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.lisp" 
:OUTPUT-FILE 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 
:OUTPUT-FILE 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl")
   0: COMPILE-FILE-PATHNAME* returned 
#P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"

And the subsequent COMPILE-FILE fails as it can't write to the jar file 
returned by COMPILE-FILE-PATHNAME*.

> Note that when I (describe #p"jar:file:///foo/bar.jar!/baz/quux.lisp")
> I get:
> 	#P"jar:file:/foo/bar.jar!/baz/quux.lisp" is an object of type
> EXTENSIONS:JAR-PATHNAME:
>    HOST         NIL
>    DEVICE       (#P"/foo/bar.jar")
>    DIRECTORY    (:ABSOLUTE "baz")
>    NAME         "quux"
>    TYPE         "lisp"
>    VERSION      NIL
>
> I believe that a pathname instead of a namestring
> makes that non-conformant. Sigh.

You mean the list with pathname in the device field?  Yes, ABCL is 
non-conformant in this manner in order to support addressing entries in 
jar files.

> Note that if requires you could keep a cache of the pathname
> in addition (or underlying store?) to any namestring in the device.

I don't think I understand this suggestion.

[…]
>
>> Recall that ABCL uses the presence of a cons in the DEVICE component to
>> indicate that a Pathname represents an entry in a jar archive.  A pathname
>> located within a jar archive is not a writable location, so we introduced
>> ABCL specific ASDF output translation rules which translate such a source
>> location to an appropriately differentiated output location in the ASDF
>> cache hierarchy.  This results in ASDF making the following kind of call in
>> preparation of the output file location it will pass to COMPILE-FILE:
>>
>>   (compile-file-pathname* <path with a DEVICE>  :output-file<path without a
>> DEVICE>)
>>
> What output-file do you get and where is it from?

0: (ASDF:COMPILE-FILE-PATHNAME* 
#P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.lisp" 
:OUTPUT-FILE 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 
:OUTPUT-FILE 
#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl")
  0: COMPILE-FILE-PATHNAME* returned 
#P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"

The arg to :output-file has come from the OUTPUT-FILES :around method 
passing in "jar:file:…" and then hitting the TRANSLATE-JAR-PATHNAME rule.

> Is any path there not absolute, and if so why?

No, it is always absolute as TRANSLATE-JAR-PATHNAME returns the fully 
translated pathname.

>> If there were some mechanism to indicate that the invocation of
>> APPLY-OUTPUT-TRANSLATIONS had really "finished" so that no further
>> processing was necessary, we could possibly plausibly short-circuit this
>> call to COMPILE-FILE-PATHAME*.
> Well, output-files accepts a secondary value of T
> to specify that no more translations are needed.

I missed that:  I'll look into this further when I get a chance.

> I don't understand yet what to change into ASDF, but
> either we can somehow reuse that mechanism, or
> we can provide something similar inside apply-output-translations,
> or somehow be more clever in translate-jar-pathname.
>
>>   I suppose my recursive invocation of
>> APPLY-OUTPUT-TRANSLATIONS in the ABCL-specific TRANSLATE-JAR-PATHNAME adds
>> the missing TYPE to the pathname which other code paths don't have at this
>> point, which is why the call to COMPILE-FILE-PATHNAME* is necessary.
>>
> My head hurts. What type is added to what where?

My understanding is that ASDF uses the COMPILE-FILE-PATHNAME* to supply 
the TYPE of fasl, which is implementation dependent.  Otherwise I didn't 
have an explanation as to why we run COMPILE-FILE-PATHNAME* before 
COMPILE-FILE which is going to return the same results, as in my testing 
the value that :output-file has at this point is already the final value 
anyways.

> Can you give examples of functions giving the "wrong" results?

No.  As I stated, it is correct that COMPILE-FILE-PATHNAME* merges in 
the device from input-file when the output-file arg has NIL for device. 
  This is the expected behavior from the ANSI perspective, but what we 
are trying to get away from in ABCL's case.

Thanks for thinking with me on this.

-- 
"A screaming comes across the sky.  It has happened before, but there
is nothing to compare to it now."