[clbuild-devel] [PATCH] Broken dependencies, new WNPPs

[David Lichteblau]

Quoting Victor Kryukov (victor.kryukov at gmail.com):
> $ ./clbuild build hunchentoot
> 
> fails during master.core compilation, because hunchentoot depends on
> babel, and babel depends on rt, but rt is not in the hunchentoot
> explicit dependencies list.

Hmm, I cannot reproduce that failure.  Can you send me a complete log of
that compilation attempt?

> It looks like clbuild doesn't resolve all the dependencies - in that
> case a simple fix would be adding rt to the hunchentoot dependencies
> line.

Make sure to distinguish between project dependencies and system
dependencies.

The following process computes the "project dependencies" of a project
FOO:

  - look for all files named clbuild/source/FOO/*.asd and load them

    (Except for some special projects that force us to scan recursively
    for asd files.)

  - for each system BAR that we found a file FOO/bar.asd for, compute
    the systems that BAR depends on.

    ASDF provides no function that I am aware of to find those
    dependencies, so we scan the in-order-to slot to make an educated
    guess what those dependencies are.

    (As an extra exception, we sometimes add dependencies at this point,
    for example to force mcclim to always depend on cffi, no matter
    whether the caller currently has gtkairo enabled or not.)

  - for each such system that is a dependency, map the system back to
    its asd file, and if that asd file is found somewhere in
    clbuild/source/BAZ/, consider it to be part of project BAZ

  - The projects found in the last step are the dependencies of FOO.

This process implies that dependencies between projects are not
transitive.

For example, in the versions of hunchentoot and babel that I see here,
the project `babel' depends on the project `rt' because the system
BABEL-TEST depends on the system RT.

But the system HUNCHENTOOT depends only on BABEL, not BABEL-TEST, so the
project `hunchentoot' does not depend on the project `rt'.


(Of course, I might be overlooking a dependency, or perhaps I'm using
the wrong version of hunchentoot.  That is why it would be good to see
the exact failure.  Perhaps the stack trace gives a clue as to where the
problem lies.)

> On the other hand, it could be a better decision to resolve all the
> dependencies automatically, or even get rid of clbuild/dependencies
> completely and extract all relevant information from .asd files in a
> asdf-install fashion.

We already extract dependency information automatically.

The idea is that maintainers run
  clbuild record-dependencies
to regenerate the `dependencies' file before checking in changes to the
project file.

The exact mechanism we use is different from what asdf-install does.
asdf-install just tries loading a system, and when that errors out due
to a missing dependency, it downloads that other project and tries
again.  I don't think that this approach would be helpful in the context
of clbuild.

> Anyway, I've attached a patch that fixes that and adds some more WNPP
> projects, mainly to support cl-s3 project that provides interface to
> Amazon S3.

I'll add them once I find a darcs repository for ironclad.  (We could
instead use the darcs mirror method explained below, but I would guess
that Nathan Froyd already has a version-controlled ironclad somewhere.) 

> Two questions:
> 
> 1/ Why is get_tarball deprecated? cl-s3 and ironclad (a cryptography
> library) are only available as tarballs; I haven't found any other way
> to get them[2].

In those cases we use the automatic mirroring script on common-lisp.net,
which would check for a new tarball daily and import it into darcs.

The main advantage of this approach is that local changes will not get
lost when updating through that darcs mirror.

> 2/ What are the reasons for having clbuild as a shell script, and not
> as a lisp program? It may be just my inexperience/unfamiliarity with
> bash, but clbuild with 1000+ LOC is hard to grasp/modify/support. And
> having a lisp is our core dependency anyway :). Two potential reasons
> come in mind, though I may miss something:
> 
> - it is just a legacy of clbuild development, which was started as a
> short shell script, or

Luke's original decision to write it as a shell script is certainly the
main reason it is (still) implemented that way.

Personally I do not see clbuild as a mere shell script, at least since
the recompile and record-dependencies steps are now written in Lisp.
But it is true that the use of a shell script is a distinguishing
feature of clbuild.

Note that other projects have tried to explore other parts of the design
space for lisp software installers.  There is cl-librarian written in
Lisp, clget written in Perl, and so on.  At this point, I see no reason
why clbuild should try to take over, say, cl-librarian.  It probably is
good at what it does, and we are good at what we do, and while the
overall goal may be similar, there are considerable differences between
our approaches.

> - clbuild does a lot of OS-level tasks, and there's no portable way to
> do it in Common Lisp.

That is one reason which people often mention.  We basically just call
out to darcs, git, cvs, and svn anyway.

I also prefer being able to install lisp projects before having built a
clean SBCL (and without having to worry whether the Lisp in my PATH is
good enough).


d.