[armedbear-devel] TIcket #151 addressed (was Re: Spaces in pathnames (jar:file:/x/y/z/a space/...))
Theam Yong Chew
senatorzergling at gmail.com
Sat Jul 9 10:57:45 UTC 2011
On 6/21/11, Mark Evenson <evenson at panix.com> wrote:
> On 6/6/11 22:46 , Theam Yong Chew wrote:
> […]
>> Hope this is a good starting point? I believe this can later be folded
>> into the unit tests?
>
> With [r13353](), I believe that I finally have addressed your concerns.
> I merged what I interpreted to be the meaning of your tests in
> [abcl/test/lisp/jar-pathnames.lisp][2] avoiding the use of Alexandria to
> not have dependencies in our test suite. You may want to verify again
> with your "local conditions" (i.e. I haven't tested under windows") that
> ticket #151 is truly fixed for you.
>
> [r13353]: http://trac.common-lisp.net/armedbear/changeset/13353
> [2]:
> http://trac.common-lisp.net/armedbear/browser/trunk/abcl/test/lisp/abcl/jar-pathname.lisp
>
> The rules adopted for URI encoding are
>
> 1. All pathname components are represented "as is" without escaping.
>
> 2. Namestrings are suitably escaped if the Pathname is a URL-PATHNAME
>
> or a JAR-PATHNAME.
>
> 3. Namestrings should all "round-trip":
>
> (when (typep p 'pathname)
>
> (equal (namestring p)
>
> (namestring (pathname p))))
>
> The basics are that you should always encode #\Space as "%20" when you
> use any of the "jar:", "jar:file:", "file:", or anything starting with a
> URL schema as a namestring to construct a pathname.
>
> The functions EXT:URI-ENCODE and EXT:URI-DECODE give the user a chance
> to manipulate strings independently of the PATHNAME constrictors.
> Strictly speaking these functions are not really RFC compliant routines
> because what is escaped and what isn't is dependent on what section of
> the URI one is dealing with, but they "seem to do the right thing" for
> the basic URI schemas that I have tested. Someday this should be done
> differently (unsure of how).
>
> Hopefully this establishes a reasonable behavior for users wrt.
> whitespace in pathnames. If this is not the case, please let us know.
>
> --
Hi everyone,
Sorry for the slow reply, I've just got around to testing the pathname
stuff. Here're my discoveries, using "0.26.0-dev-svn-13379M". This is
a long email.
First, I needed to patch jar-pathname, like so (truename only works on
a pre-existing file/folder). Feel free to do something that makes more
sense.
tyc20 at ackbar:~/lisp/imp/abcl$ svn diff
Index: test/lisp/abcl/jar-pathname.lisp
===================================================================
--- test/lisp/abcl/jar-pathname.lisp (revision 13379)
+++ test/lisp/abcl/jar-pathname.lisp (working copy)
@@ -9,10 +9,11 @@
(let ((temp-file (java:jcall "getAbsolutePath"
(java:jstatic "createTempFile"
"java.io.File" "jar" "tmp"))))
(setf *tmp-directory*
- (truename (make-pathname :directory
- (append
- (pathname-directory (pathname temp-file))
- '("jar-pathname-tests"))))
+ (truename (ensure-directories-exist
+ (make-pathname :directory
+ (append
+ (pathname-directory (pathname temp-file))
+ '("jar-pathname-tests")))))
*tmp-directory-whitespace*
(merge-pathnames "a/directory with/s p a/" *tmp-directory*))))
After running the tests, I could see "25 out of 548 total tests
failed", and "23 unexpected failures".
[java] 23 unexpected failures: ABCL.TEST.LISP::UNUSED.1,
[java] ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2,
[java] ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3,
[java] ABCL.TEST.LISP::MAKE-CLASS-FILE.1,
[java] ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1,
[java] ABCL.TEST.LISP::GENERATE-METHOD.1,
ABCL.TEST.LISP::GENERATE-METHOD.2,
[java] ABCL.TEST.LISP::GENERATE-METHOD.3,
ABCL.TEST.LISP::GENERATE-METHOD.4,
[java] ABCL.TEST.LISP::GENERATE-METHOD.5,
[java] ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1,
[java] ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2,
ABCL.TEST.LISP::DMC-RETURN.1,
[java] ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3,
[java] ABCL.TEST.LISP::PHYSICAL.21, ABCL.TEST.LISP::PHYSICAL.27,
[java] ABCL.TEST.LISP::SILLY.5, ABCL.TEST.LISP::TRANSLATE-PATHNAME.5,
[java] ABCL.TEST.LISP::PATHNAME.URI-ENCODING.2,
[java] ABCL.TEST.LISP::JAR-PATHNAME.LOAD.1,
[java] ABCL.TEST.LISP::MATH.READ-FROM-STRING.1.
PROBE-FILE.4 and PROBE-FILE.5 are supposed to fail (expected), but is
LOAD.1 supposed to fail?
Another user facing issue, not reflected in the tests (yet) is
differentiating between a normal PATHNAME and URL-PATHNAME
for .abcl files.
CL-USER> (load "/tmp/nospace.lisp")
HI
T
CL-USER> (load "/tmp/nospace.abcl")
HI
T
CL-USER> (load "/tmp/a space.lisp")
HI
T
CL-USER> (load "/tmp/a space.abcl")
Failed to create URI from '/tmp/a space.abcl': Illegal character in
path at index 11: file:/tmp/a space.abcl
[Condition of type SIMPLE-ERROR]
I assume this now requires escaping. But as I write this email, I
found that I don't know how exactly to do this:
CL-USER> (load "file:/tmp/a%20space.abcl")
Failed to create URI from '/tmp/a space.abcl': Illegal character in
path at index 11: file:/tmp/a space.abcl
[Condition of type SIMPLE-ERROR]
CL-USER> (load "jar:file:/tmp/a%20space.abcl")
File not found.
[Condition of type FILE-ERROR]
Is it possible to load .abcl files? The only working example I could
come up with seems to break abstraction barriers, by having to know
about the internal format of .abcl files (name._ etc)
CL-USER> (load #P"jar:file:/tmp/a%20space.abcl!/a%20space._")
HI
T
Now that I look again, the test cases haven't covered any .abcl path
loading.
I also discovered a corner case (which had been around, but that I
wasn't as aware of before) with renamed abcl files:
If we compile a file with space in its name, but then rename the
resultant abcl file to have no spaces.
(load "/tmp/a space.lisp") => ok
(compile-file "/tmp/a space.lisp") => "/tmp/a space.abcl"
(abcl.test.lisp::cl-fad-copy-file
"/tmp/a space.abcl"
"/tmp/renamed-to-no-space.abcl")
While we have a different name, the contents "a space._" remain the
same. Now
(load "/tmp/renamed-to-no-space.abcl") => gives error
Improper URI syntax for 'file:/a space._':
java.net.URISyntaxException: Illegal character in path at index 7:
file:/a space._
[Condition of type SIMPLE-ERROR]
This is inconsistent with an example with no spaces (or other special
characters). Again:
(load "/tmp/nospace.lisp")
(compile-file "/tmp/nospace.lisp")
(abcl.test.lisp::cl-fad-copy-file
"/tmp/nospace.abcl"
"/tmp/no-space-renamed.abcl")
CL-USER> (load "/tmp/no-space-renamed.abcl")
HI
T
Perhaps it's easier to just ban the renaming of .abcl files? (Or
deprecate, or declare to be undefined behaviour...). If someone needs a
differently named .abcl file, it's simple to use
(compile-file
"/tmp/a space.lisp"
:output-file "/tmp/renamed-to-no-space.abcl")
That's all I've tested so far on Ubuntu.
;; ========================================
Unfortunately, on Windows, the .abcl file loading (pathname)
issue is much worse, since we see lots of spaces in pathnames.
Armed Bear Common Lisp 0.26.0-dev-svn-13379M
Java 1.6.0_23 Sun Microsystems Inc.
Java HotSpot(TM) Client VM
Low-level initialization completed in 1.191 seconds.
Startup completed in 3.725 seconds.
Loading C:\Documents and Settings\tyc\.abclrc completed in 0.0 seconds.
Type ":help" for a list of available commands.
CL-USER(1): (require :asdf)
("MOP" "CLOS" "PRINT-OBJECT" "FORMAT" "ASDF")
CL-USER(2): (asdf:oos 'asdf:test-op :abcl)
; Compiling C:/me/progs/abcl-0.26-test/test/lisp/abcl/rt-package.lisp ...
; (LET* (# #) ...)
; Wrote C:/Documents and Settings/tyc/.cache/common-lisp/abcl-0.26.0-dev-svn-133
79m-fasl37-win-x86/C/me/progs/abcl-0.26-test/test/lisp/abcl/ASDF-TMP-rt-package.
abcl (0.771 seconds)
#<THREAD "interpreter" {9D3F8B}>: Debugger invoked on condition of type SIMPLE-E
RROR
Failed to create URI from 'C:/Documents and Settings/tyc/.cache/common-lisp/ab
cl-0.26.0-dev-svn-13379m-fasl37-win-x86/C/me/progs/abcl-0.26-test/test/lisp/abcl
/rt-package.abcl': Illegal character in opaque part at index 17: file:C:/Documen
ts and Settings/tyc/.cache/common-lisp/abcl-0.26.0-dev-svn-13379m-fasl37-win-x86
/C/me/progs/abcl-0.26-test/test/lisp/abcl/rt-package.abcl
Restarts:
0: TRY-RECOMPILING Recompile rt-package and try loading it again
1: RETRY Retry loading FASL for #<ASDF:CL-SOURCE-FILE "abcl-test-lis
p" "abcl-rt" "rt-package">.
2: ACCEPT Continue, treating loading FASL for #<ASDF:CL-SOURCE-FILE "
abcl-test-lisp" "abcl-rt" "rt-package"> as having been successful.
3: RETRY Retry #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYS
TEM "abcl">.
4: ACCEPT Continue, treating #<STANDARD-CLASS ASDF:TEST-OP {4BD173}>
on #<ASDF:SYSTEM "abcl"> as having been successful.
5: TOP-LEVEL Return to top level.
[1] CL-USER(3):
I temporarily worked around the above problem by setting asdf output
translation to a folder path without spaces. I bumped into some
compilation errors when running the tests. Here's one:
...
; (DEFTEST PATHNAME.WINDOWS.6 ...)
#<THREAD "interpreter" {9D3F8B}>: Debugger invoked on condition of type ERROR
The URI has no path: file:z:/foo/bar
Restarts:
0: TRY-RECOMPILING Try recompiling pathname-tests
1: RETRY Retry compiling #<ASDF:CL-SOURCE-FILE "abcl-test-lisp" "tes
t" "pathname-tests">.
2: ACCEPT Continue, treating compiling #<ASDF:CL-SOURCE-FILE "abcl-te
st-lisp" "test" "pathname-tests"> as having been successful.
3: RETRY Retry #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYS
TEM "abcl">.
4: ACCEPT Continue, treating #<STANDARD-CLASS ASDF:TEST-OP {4BD173}>
on #<ASDF:SYSTEM "abcl"> as having been successful.
5: TOP-LEVEL Return to top level.
I don't have a z drive. I can type some of the forms in the REPL
without any issues, but not others:
CL-USER(24): (pathname-device #P"z:foo/bar")
"z"
CL-USER(25): (pathname-device #P"z:/foo/bar")
"z"
Trying #P"file:z:foo/bar" leads to the "URI has no path" error.
Finally, because I used the asdf
pretend-file-compiled-successfully-continue restart a few times, lots
of tests were not run in the end, but whatever ran gave the following
errors:
20 out of 290 total tests failed: ABCL.TEST.LISP::UNUSED.1,
ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2,
ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3,
ABCL.TEST.LISP::MAKE-CLASS-FILE.1,
ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1,
ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2,
ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4,
ABCL.TEST.LISP::GENERATE-METHOD.5,
ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1,
ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1,
ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3,
ABCL.TEST.LISP::URL-PATHNAME.1, ABCL.TEST.LISP::URL-PATHNAME.2,
ABCL.TEST.LISP::MATH.READ-FROM-STRING.1, ABCL.TEST.LISP::ZIP.1.
19 unexpected failures: ABCL.TEST.LISP::UNUSED.1,
ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2,
ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3,
ABCL.TEST.LISP::MAKE-CLASS-FILE.1,
ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1,
ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2,
ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4,
ABCL.TEST.LISP::GENERATE-METHOD.5,
ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1,
ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1,
ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3,
ABCL.TEST.LISP::URL-PATHNAME.1, ABCL.TEST.LISP::URL-PATHNAME.2,
ABCL.TEST.LISP::MATH.READ-FROM-STRING.1.
It sounds like the .abcl loading mechanism needs to support spaces too
(especially on Windows). That's all for now (I hope). I have seen a
few other puzzling things I'll try and investigate later.
I guess all the above problems relate to .abcl paths, so do they count
as jar-pathnames or normal pathnames, or what?
Thanks for looking into this.
Yong.
More information about the armedbear-devel
mailing list