[asdf-devel] Including ASDF 2 in the next SBCL release

[Faré]

Dear SBCL developers,

could you test for inclusion with your implementation
the current release candidate of ASDF 2 (version 1.715 as of this writing)?
	http://common-lisp.net/project/asdf/

ASDF 2 is an evolution of ASDF that tries to be backwards compatible
with previous releases and their many extensions,
while at the same time improving the user experience in many important ways.
I have worked hard towards reaching the current state,
which I believe is now suitable for release.

Could you please send us fixes for any glaring bugs, and otherwise
include ASDF 2 in any alpha-testing release channel for users of your
implementation? I think it's mostly ready, and
I'd like to have it widely tested before the actual release,
which I hope will happen this month of May 2010,
hopefully without any code change (except maybe for some tweaks
to enhance support for some Lisp implementations on some operating systems),
and with slightly enhanced documentation (notably based on your feedback).

I'm using ASDF 2 at ITA. Recent versions are already included in ECL and ABCL.
Several hackers have tried building plenty of packages with it.
While it's far from my ideal of perfection, and
admittedly only a relatively small limited of previous code,
I believe it's enough of an advance over the previous generation of ASDF
that I'd like to get it out to the world.
The code has been extensively cleaned up
while preserving previous semantics everywhere that it was well-defined.
Our testing has revealed no regressions from ASDF 1.

What has changed since ASDF 1 is documented at the following page:
http://common-lisp.net/project/asdf/asdf/FAQ.html

Improvements include:

* portably handle naming of files wrt directory, type, host/device, etc.

* a builtin user-configurable output translation mechanism,
 superseding asdf-binary-locations and previous hacks in
 common-lisp-controller and cl-launch.

* a user-configurable source registry system for finding systems
 (the old *central-registry* is still available).

* improved portability to many implementations.

* improved support for Windows (though we still lack implementation-dependent
  configuration support except on LispWorks).  [NOTE:  I couldn't parse this
  item.  I juggled the sentence structure, but I still don't get it.  What kind
  of configuration support is needed for Windows?  What aspects of configuration
  that we have on other OSes is missing from Windows?]

* many bug fixes

* better documentation

* the test suite, while still largely incomplete, covers more cases
 and is more robust across implementations.

* substantial performance improvements when processing large systems with
  thousands of components.

* API improvements: load-system, system-relative-pathname, etc.

* a sensible versioning system for ASDF 2 itself.

* ASDF can be now be upgraded in a running lisp image. Using this in-place
 upgrade, ASDF releases can be decoupled from implementation releases.

* better support for various extensions used by ECL and ABCL,
 and also by POIU, ASDF-DEPENDENCY-GROVEL.


Pitfalls include:

* Output translations is enabled by default. This may surprise some users,
 most of them in pleasant way (we hope), a few of them in an unpleasant way.
 It is trivial to disable output translations.  Simply put
     (ASDF:DISABLE-OUTPUT-TRANSLATIONS)
 in a lisp startup file.

* Some systems in the large have been known not to play well with
 asdf output translations.  They were easy to fix.
 The most likely issue is to have chaining of operations
 (e.g., a preprocessing before a load) where the INPUT-FILES of one operation
 are related to the OUTPUT-FILES of another.
 In this case many issues can be avoided by simply making
 the OUTPUT-FILES primary, and having later operations' INPUT-FILES
 be defined in terms of the OUTPUT-FILES of other operations
 instead of directly defining the INPUT-FILES.
 It is also easy to disable output translations or override its configuration.

* ASDF 2 output translations do not work with ASDF-Binary-Locations.
 They replace A-B-L, and include a compatibility mode to emulate your previous
 A-B-L configuration. See ENABLE-ASDF-BINARY-LOCATIONS-COMPATIBILITY.
 But you shouldn't load ABL on top of ASDF 2.

* A notable performance bug on SBCL (and possibly other implementations, too):
 the recursive search of configured paths in your source-registry with
    (DIRECTORY "/configured/path/**/*.asd")
 can waste a few seconds when initially searching for asd files
 (depending on your configuration).
 If it is essential for you to avoid this pause, you should override
 the default source-registry and only use non-recursive :DIRECTORY
 options (or shallow trees).

* On Windows, when not using cygwin, only LispWorks has proper defaults
 for configuration pathnames. Other implementations make do.
 Patches welcome.

* Windows support has not been sufficiently well tested.

As to how to best include ASDF in your implementation,
here is an excerpt from the FAQ section of our manual:

12.3.2 "I'm a Common Lisp implementation vendor.
       When and how should I upgrade ASDF?"

Starting with current candidate releases of ASDF 2,
it should always be a good time to upgrade to a recent version of ASDF.
You may consult with the maintainer
for which specific version they recommend,
but the latest RELEASE should be correct.
We trust you to thoroughly test it with your implementation
before you release it.
If there are any issues with the current release,
it's a bug that you should report upstream and that we will fix ASAP.
In order to report bugs, please use the launchpad bug tracker.
Information about launchpad is contained in the manual.

As to how to include ASDF, we recommend the following:

* If ASDF isn't installed yet, then (require :asdf) should load the
version of ASDF that is bundled with your system.
You may have it load some other version configured by the user,
if you allow such configuration.

* If your system provides a mechanism to hook into CL:REQUIRE,
then it would be nice to add ASDF to this hook the same way
that SBCL, CCL and ABCL do.

* You may, like SBCL, have ASDF be implicitly used to require systems
that are bundled with your Lisp distribution.
If you do have a few magic systems that come with your implementation
in a precompiled way such that one should only use the binary version
that goes with your distribution, like SBCL does,
then you should add them in the beginning of wrapping-source-registry.
Please consult with the ASDF maintainers about that.

* If you have magic systems as above, then we explicitly ask you
to NOT distribute asdf.asd as part of those magic systems.
You should still include the file asdf.lisp in your source distribution
and precompile it in your binary distribution, but
asdf.asd if included at all, should be secluded from the magic systems,
in a separate file hierarchy, or you may otherwise rename the system
and its file to e.g. asdf-ecl and asdf-ecl.asd, or sb-asdf and sb-asdf.asd.

If you make asdf.asd a magic system, then users will no longer be able to
upgrade ASDF using ASDF itself. It is helpful to be able to do this
in order for users to be able to use an ASDF version of their own choosing
that they maintain independently from your Lisp distribution.

* If you do not have any such magic systems, or have other non-magic
systems that you want to bundle with your implementation, then you may
add them to the default-source-registry, and you are welcome to
include asdf.asd amongst them.

* Please send us upstream any patches you make to ASDF itself, so we
can merge them back in for the benefit of your users when they upgrade
to the upstream version.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
There are just two rules of governance in a free society:
Mind your own business. Keep your hands to yourself. — P. J. O'Rourke