[asdf-devel] The future of ASDF (was logical pathnames)

[Juan Jose Garcia-Ripoll]

This email arises from a discussion at the ECL mailing list about how ASDF
needs an urgent replacement and that is should be just discontinued and
replaced with something else. Autoconf, extended REQUIREs and other stuff
was suggested.

I do not agree. I think ASDF can be fixed and made easier to use.

Before explaining my view, please let me disgress a bit. I use autoconf a
lot. There is no way around it if one works in Unix environments with
moderately complicated software [1,2]. The need of autoconf is motivated and
justified because of certain reasons

1. All build environments in Unix work the same: all C/C++ compilers are
similar in the way they process their input, arguments they take and what
they produce: object files.
2. Components are found in rather standard locations (/usr/include,
/usr/lib, ...) and Autoconf provides information about that.
3. Components may or may not be present. Autoconf helps us determining that.
4. We normally have a simple build process with a list of files to be
compiled and linked together. Dependencies are also simple forming an open
graph, without loops. Automake takes this graph and produces makefiles which
build the system
5. Autoconf installs your software via the generated files. We know where to
place them. If the user chooses a nonstandard location he will be punished
but it has a simple solution (set an environment variable).

What about lisp?

1. Each lisp behaves differently. ECL is not at all like SBCL. Currently
this is solved by imposing a rather outmodish way of building the software
based on compile-file and load. No notion of standalone binary files, no way
of producing executables, no way of shipping things.

2. Each lisp has a different notion of how to locate "required" components.
Some have search paths (SBCL), some do not (ECL). There is not a standard
for installation.

3. There is an additional problem in the lisp world: we do not use the
shared library loader! In order to locate components we may either rely on
crappy things like REQUIRE or rely on an external agent that seeks those
components prior to loading our software.  ASDF behaves in that sense as a
shared library loader that takes into account version information!

4. System definitions provide the description of our system and how pieces
bind together. This has always been like that, even before ASDF, with
mk:defsystem. It is a simple, mostly declarative description. What we do
with it is up to the software that manipulates it, just like automake may
evolve at some point and behave differently.

5. Installation is again a problem because there is not a well defined way
to look for software. Furthermore, since current system descriptions do not
provide information about "resources" that the software may need, such as
bitmaps, icons, databases, etc, we have no information about installing them
and instead things are left in place.

On comparing both lists one must reach the conclusion that the situation is
bad. Indeed. All of the problem revolves around two things

* We need some way to build our system taking into account all dependencies
* There is no standard way to produce standalone libraries and install them
in a way that just works.

ASDF tries to solve BOTH problems, that is BUILDING and LOADING/EXECUTING.
The problem is that development has over the time forgotten the distinction
between both and one finds amazing things such as users that place some of
the execution logic (creating packages, classes, etc) IN THE SYSTEM
DEFINITION!

However, while we are in a position of say, disadvantage, w.r.t the Unix
community at large, I believe we have a good starting point to get things
right.

My vision is to *progressively* decouple the building and loading components
in ASDF. I do not want to push anything down anybody's throat, but even less
take all the existing database of knowledge about all the libraries that
build and work with ASDF and presenting myself as an illuminated that has
the solution to everything and that this solution has to be something
orthogonal to ASDF.

How can we achieve these things and make everybody's life simpler? I am not
going to produce the roadmap, but what I present below can be done
progressively, and examples for that are the bits I have been contributing
to the asdf-devel list.

* First of all understand that we have two different targets here. One is
people building standalone software (programs) and the other one is people
building libraries. In both cases the system definitions we have are good
enough to describe them and their dependencies. Let's call the component in
ASDF that handles these systems ASDF-system. This includes classes and the
parsing logic. This is very tiny and should be easy to bootstrap.

* Understand how different lisps locate software. Realize that not all lisps
can do that cleverly and that this is one of the reasons for ASDF to exist.
Produce a minimalistic ASDF-search that reuses existing logic or implements
new one for lisps that miss that part.

* Realize that we need a better way of loading software. I do not care if
that guy is improving REQUIRE, he is building a new component that people
will have to install or ship with their lisps and autoconfed scripts are not
going to solve it.

* As a solution to the previous point realize that system definitions
already have enough information for an ASDF-loader facility that builds on
ASDF-search and ASDF-system. Even for standalone systems it is possible to
create stripped down *.asd files with version information and dependencies
in them. This is already present in the patch I sent to the list and ECL can
also make it: it is the binary-op operation I introduced. The result is a
file
    (defsystem :my-system :depends-on (:cl-ppcre) :components
((:compiled-file "my-component")))
which contains just the minimal information for our "shared library" loader
to work. I personally find this better (and potentially more extensible)
than Unix version numbers in file names (libecl.so.10.2.1 ?!?)

* ASDF-loader can be a very transparent component and it may even be
possible to ship it together with _every_ library you have around, just like
libtool is shipped with every library. As a compatibility layer over REQUIRE
it should be possible to produce three files for each system
  - the system definition my-system.asd
  - a simple-minded loader my-system.fas that sets up paths, requires
asdf-loader and loads the last file which is...
  - the component of the library itself, my-component.fas

* We still need a building facility, say ASDF-builder, that uses a system
definition to compile everything. This would be the COMPILE-OP we have right
now, glorified and improved, but it would be augmented with facilities to
pack a library in a single file, building standalone programs,  etc.

* The interesting part about this decoupling is that standalone programs do
not need any of it. I can just use ASDF-builder as a glorified makefile and
produce a program that just works by concatenating or linking pieces of
precompiled software and inserting a simple logic at the beginning of the
program that sets up the environment that the components expect. No need for
ASDF-loader nor ASDF-search nor ASDF-system. Since I already removed all
dependency on ASDF from the executable code itself by having logical
pathnames, things just work. Compare this with a hand-built software where I
abuse REQUIRE here and there.

Say you are a developer of a library, MY-LIBRARY, and want the user to just
build it and install it. There are various alternatives to achieve this.

One is to continue as now. The user loads ASDF and from the prompt she is
able to compile, build and install the software with existing operations.

But from the point of view of the user things can be improved a lot by means
of scripting.

* Assume that ASDF is preinstalled by the user. This needs not be done in
complicated fashion. ASDF itself can be "autoconf'ed" and taught to find out
the existing lisps, install itself, and create a set of scripts that the
application is going to use: "asdf-build" and "asdf-install".

* Alternatively you may ship a version of ASDF with your library, hidden in
an asdf directory just like libtool hides itself in the m4 or config
directory of a Unix library. The scripts asdf-build and asdf-install are
then placed on the top of your library source tree, where the *.asd
definitions live.

Now the user only has to do something like
 ./asdf-build --lisps=sbcl,ecl,acl
 ./asdf-install
These two scripts will loop over lisps loading them with ASDF, compiling and
installing stuff, reusing the knowledge this library has about the systems
themselves. If the lisp environment does not have ASDF installed, a copy may
be installed together with the library or we may use the logic explained
above by which each library carries a copy of ASDF-loader just in case there
is none.

All this can be coded on top of the existing ASDF, better factoring the
code, and with a bits of magic here and there.

In short

* Except for standalone software, we a lisp loader that
  -- Takes into account dependencies
  -- Searches for the software at standard installation paths
* ASDF may coexist with software built using other tools.
* ASDF may build software for various distribution purposes.
* ASDF can hide itself from the user as much as we want.
* ASDF can produce software that does not rely on a configured ASDF to load
* We can write libraries that do not care much whether ASDF is there or not
* There is always going to be some amount of configuration on the side of
the user
  -- Either using scripts and assuming default installation paths
  -- Either specifying search paths SBCL_SEARCH_PATH=...
* System definitions are useful, dependencies scattered over your sources
are not.

Juanjo

[1] ECL itself
[2] Libraries I have to ship to clusters all around the world (AIX, Linux,
SGI...) See signature below.

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20100331/7db589ab/attachment.html>