[asdf-devel] Logical pathnames vs ASDF & SBCL

[Robert Goldman]

On 6/12/11 Jun 12 -4:34 AM, james anderson wrote:
> good morning, faré, gentlemen;
> 
> On 2011-06-12, at 02:25 , Faré wrote:
> 
>> TL;DR: ASDF currently doesn't play so well with logical pathnames.
>> I'm seeking advice on how to make things better —
>> or on whether it's worth the trouble.
>>
>>
>> Dear all,
>>
>> [...]
>>
>> POTENTIAL SOLUTIONS
>>
>> [...]
>>
>> Finally, I could do nothing, and assume that
>> people who go through the pain of setting up logical pathnames
>> before they run ASDF can just as well setup the *central-registry*
>> and have LPNs be recognized in the good old way.
>> If you use the source-registry, your LPNs will be squashed,
>> and nothing will break in the code since code works fine with PPNs,
>> just like most everyone use it, but your debug information
>> will be stored according to the PPN, not the LPN. Meh.
> 
> 
> less might well be better.
> 
> a bit over twenty years ago, when i had sent a library to john mallory
> to be included among cl-http contributions, he pointed out to me, that,
> as my apparent intent was to implement portable systems for others' use,
> it was not a good idea to have used mixed case pathnames. i trusted him,
> renamed several dozen files and directories, and have adhered to that
> restriction ever since. over the years, i have seen nothing to contract
> his advice. furthermore, over these years, as my requirements for lisp
> system management tools have remained limited to those related to
> building those aspects of larger systems which comprise just lisp source
> code, i have observed that portability and maintainability are improved
> by adopting the logical pathname naming restrictions for system
> description components. that is, in distinction to experience reported
> elsewhere, it has been possible to use asdf system declarations which
> contain just logical pathname components with all lisp implementation
> which purported to support logical pathnames. the pathname regression
> test which i implemented during the "asdf 2" testing process
> demonstrated this to have been true for all of the variations of runtime
> and pathname expression which were known at that time.

This is an interesting bit of explanation and somewhat dovetails with my
experience.  One of the problems I have had with logical pathnames is
working on projects where lisp code is /not/ in systems "compris[ing]
just lisp source code."  Larger systems on which I have worked often
contain large hunks of Java, C++ or scripting language code.  Sometimes
this software has been written across multiple companies.  It has been
difficult to limit those systems' use of pathnames; in practice it's not
worth the trouble.  It's not helpful when trying to get people to
tolerate your use of CL to have to tell them that you want them to give
up TheirCamelCase and_underscores.

Another thing that has led me to give up on logical pathnames is that it
has forced me to debug multiple colleagues' lisp configurations.

I have found it less work to simply put in our source code repository
load files that use *load-truename* to populate asdf:*central-registry*.
 This even hides ASDF configuration, and minimizes time lost in
debugging others' configurations.

These experiences may differ from much of this audience's experience,
which may be more related to individual hackers working on their own
software, or working on open source systems in a distributed way.

> 
> my particular build process - even prior to "asdf 2", involves
> variations on the two alternatives discussed in faré's original message
> [cf.
> https://github.com/lisp/de.setf.utility/blob/master/asdf/hierarchical-names.lisp]:
> 
> - it searches for system definition files recursively from the set of
> root directories specified in *central-registry*. for my own systems, a
> method to interpret system names as hierarchic paths speeds the process.
> this means that the *central-registry* is sufficient.

I use a variant of this, a portable "asd-finder" system that does a
special, portable CL version of 'find' that is tailored to find .asd
files.  The asd-finder is used to populate asdf:*central-registry*.
Using the asd-finder is similar to using Faré's :tree directive, except
it does not rely on knowing the user's absolute install pathname: you
can invoke the asd-finder using *load-truename*.

My desire to replicate this kind of configuration (the asd-finder used
in a load file) is what led me to introduce the :here directive into
Faré's DSL for configuring input file search.

> - it takes the "hard way out", as that serves maintainability. where
> possible, an additional step in the system initialization protocol
> derives and asserts a logical pathname root for the system, which (at
> least at last check) yields logical source pathnames for all system
> components.

I used to do this myself, but gave up on it.  If you are working
relative to some root directory found through *load-truename* (or a
similar mechanism), introducing the logical pathname didn't seem like a
big win to me.  If my systems need to find some files, it's just as easy
to store a copy of the directory portion of *load-truename* in a global
variable, and use merge-pathnames on the global variable.  In practice,
I have found this more reliable than wrestling with logical pathnames
(e.g., there's no need to worry that someone has installed the working
directory in some location that will break logical pathnames).

(merge-pathnames "data-file.csv" *my-system-root*)

doesn't seem a lot harder than

"my-system-root:data-file.csv"

YMMV.
> 
> this approach does limit the mechanisms available to include external
> libraries in my own builds. that is ok. for example, while i may use
> something like quicklisp to obtain source code, i have seen the
> difficulties which even advanced dependency-based build systems, such as
> ruby's, introduce into production environments. it is a nightmare to
> which i do not intend to submit. i prefer, that production systems
> always include self-curated source code. in this case, it is a minor
> issue to normalize system names.

My experience lines up with James's here.  If a project I work on needs
an external library, we blow that into our "lisp-utils" repository, and
incorporate that.  No more problems with API-breaking mods to libraries.
 Also, in my experience, CL libraries very often have corner cases which
break them.  After a bunch of times downloading a library using
ASDF-install, finding I had to patch it, and /then/ having to blow it
into a source code repository, I find that it's better to just cut to
the chase and push all library dependencies into the repository.  Again,
YMMV.  If you want to set up a web server with hunchentoot, for example,
you may find that library has been used enough that it can be collected
with quicklisp or asdf-install and simply used.  OTOH, if you are using,
say, and XML-wrangling library, a DAG-manipulation library, etc., the
chances are good that you will end up using a piece of its functionality
that has not been adequately tested and debugged.  The CL community
doesn't have enough eyeballs in it yet for its utility libraries to be
bulletproof, in my experience.

Best regards,
R