[asdf-devel] :ASDF-DEPENDENCIES implemented

Tue Apr 20 20:55:14 UTC 2010

On Tue, Apr 20, 2010 at 9:51 PM, james anderson <james.anderson at setf.de>wrote:

> if the concepts are "difficult to fit in one or two words", then the
> expression form should disambiguate rather than obscure. just as
>  (defstruct (person (:print-function print-person)) name age sex)
>

It is good to make more elaborate explanations. When you said "change the
name structure" I understood something like (defsystem
some-dependency-my-system ...) or something like that.

> ia pattern such as
>  (defsystem (a-system :class extended-system :require some-implementation)
>   :components ((:file x)))
> is better than
>  (defsystem a-system
>   :class extended-system
>   :require some-implementation
>   :components ((:file x)))
>

I would have said that the latter is less different than current practice
and thus it would have allowed for a simpler evolution of people's code.
Your notation above is not at all disgusting, but it would change the way
the :class argument is parsed right now.

> if it might be possible to confirm that the shortlist, above, is a complete
> statement of the problem, that would help.
>

Your list seems mostly complete.

1 it should be possible to interpret a system definition without performing
arbitrary computation
1a the interpretation includes constructing and/or interrogating the intra-
and inter-system dependency graph
1b it includes identifying designated files
1c it should be possible to do this using ASDF's classes and parsers
1d it will be allowed to have additional "properties" or annotations for
extensions to use
1e the system definition itself will be read in a "null" package, destroyed
after reading the definition
1f* the system definition will not contain symbols in packages other than
  - CL
  - KEYWORD
  - The null package
  - ASDF
  - CL-USER (for setting configuration variables)
1g* If a symbol is needed from some other package it will be done with a
special designator helped by a reader macro (~FUTURE-PACKAGE:SYMBOL-NAME or
something like that).
1h* ASDF functions will recognize those designators and "intern" them in the
appropriate packages only after dependencies are satisfied.

2 it should be possible to extend asdf operations on system, module, and
file model components declaratively, in a manner such that the declaration
still yield a valid definition in the absence of any
extension
2a  this includes to specify additional initargs in the specification
2b  this includes to specify classes which are not defined in the asdf core.

3 Guarantees about extensions
3a The extensions will be available when the first operation is PERFORMed on
the system or on any of its components. By then the system and component
instances will have been processed and finished by the extensions.
3b In particular ASDF does not promise that the extensions will be available
when reading
3c Those extensions will be loaded using ASDF's mechanisms and may be
reloaded if their own dependencies change.
3d Reloading extensions may result in an update of the system classes and
components and extensions should be coded to allow instance
reinitialization.
3e** Extensions should not change the list of dependencies of the components
as previously computed by ASDF. However, they are allowed to insert harmless
dependencies in the edges or as hanging leaves, such as intermediate actions
to be performed, as far as they do not change the dependencies between
components and systems.

(*) Some of these things would require further work, the rest is mostly
there.
(**) This could be formulated in a more rigorous way.

> [... listing the dependencies outside DEFSYSTEM ...]
> That would cause less pain because it could be macroexpanded, but it would
> fail to annotate the system form itself with a dependency,
>
> is that one of the requirements? one of the earlier discussions suggested
> that the load protocol be implemented as a restricted read-eval loop. which
> would itself be in a position to collect any annotations.
>

If you consider the system definition as the whole file, then I agree with
you that it really does not matter where things are placed. I just find it
more appealing to have just one thing to read: the DEFSYSTEM form, where all
annotations and dependencies are placed together -- just like you find it
more natural to have the meta-information in the name.

> it depends on whether the requirement is that it behave like use/in-package
> or like :metaclass.
> given the present formulation, what is the consequence of a dependency
> which appears in more than one system definition. is it reloaded or not?
> what happens if the features have changed since it was first loaded? is the
> behavior by intent, or a side-effect of the implementation?
>

Even metaclasses can change themselves. What I do not want to emphasize is
the view that :asdf-dependencies or :defsystem-depends-on or whatever
formula we arrive at is viewed as an imperative USE-PACKAGE that is used
_before_ the defsystem form itself and before it is read. It is just part of
it, listing where system classes, component classes and perhaps additional
parsing routines live, and informing us that they will be used to create the
system itself, somehow similar to how a metaclass handles creation,
instantiation, access to slots, etc.

> 1- system definitions continue to be read in the appropriate package.
>
> as of last reading, they were loaded by default into a temporary package
> which was then deleted.
> is this the appropriate package? if so, why?
>

I find it appropriate because it is a temporary package and we can detect
symbols that are pulled from other packages, issuing a warning if needed.

>
> 2- at some point we may introduce tokens with late bindings to packages --
> this is essential for rules that depend on functions that depend on packages
> that are not created.
>
> why tokens? what function do they support and why are tokens the best way
> to accomplish that? why is it better to specify the future home package that
> it would be to search for it? why is that better than it would be to always
> use the same package? why is it better than to use a package named after the
> system?
>

The defsystem form has to have some way of referring to future packages
because we are going to need it. We have a file "generated-lisp.lsp" which
is listed in the defsystem

(defsystem :example
  :components
   ((:file "dsl-parser")
    (:file "generated"
     :create (~dsl-parser:transform "generated-source.dsl"
"generated.lisp"))))

Please do not take this literally: I do not mean this to be the right
syntax. However you appreciate the need for a rule which is not general, not
reusable, just to create this file. This rule needs a function that does not
exist when we create the system and which has a name that lives in a package
that also does not exist. If we allow this kind of "tokens" (is this the
right word?) and teach ASDF to generate the right substitution when the
dependencies are satisfied (in this case after dsl-parser is loaded and when
generated is created), then we can be happy.

This is something we will face with this example or with other ones that
have dependencies in non-existent packages. For instance components such as
the ones defined by CFFI-GROVEL. This time taken from a real example:

(asdf:defsystem :fsbv
  :description "Foreign Structures By Value."
  :maintainer "Liam Healy <lhealy at common-lisp.net>"
  :licence "See readme.html"

  :depends-on (:cffi :cffi-grovel :trivial-features)
  ;;:pathname (merge-pathnames "syscalls/" *load-truename*)
  :serial t
  :components
  ((:file "init")
   (cffi-grovel:grovel-file "libffi" :pathname #+unix "libffi-unix")

We could list CFFI-GROVEL as a meta dependency, but even then we would not
be able to read this form without loading the extension. If we instead allow
for some kind of "future package" designator, then we could.

 3- we annotate the system definition with meta-dependencies.
>
> what are meta-dependencies?
> - packages?
> - classes in packages?
> - systems?
> - features?
> - ?
>

So far I have only talked about two: classes (:class) and libraries that
implement extensions (:asdf-dependency)

> 4- We make _no_promise_, absolutely no promise about when the
> meta-dependencies will be use. In this sense this is exactly like the MOP
> class finalization protocol.
>
> before perform?
> before operate-on-system?
> before shared-initialize starts? completes?
> before initialize-instance starts, completes?
>

This is debatable and it will depend how much liberty we want to give to
extension writers. I would argue that just before PERFORM. We should be able
to traverse the dependencies, create the appropriate graphs, etc, using just
ASDF's classes, without having the extensions around.

The problem I see is that people may find this too limiting. For instance I
will take a real life example: a user that creates his own CSS-FILE class
because STATIC-FILE does not provide extensions
(defclass css-file (doc-file) ())
(defmethod source-file-type ((c css-file) (s module)) "css")
used as follows
(:module "doc" :components
  ((:html-file "ironclad")
   ;; XXX ASDF bogosity
   (:css-file "style")))))

If we enforce the policy that the extension is not enforced by TRAVERSE,
this class will not be loaded and we will fail to recognize that the system
depends on "style.css" and not on "style".

Now one may argue precisely that this particular example is artificial,
because if he used the keyword :type "css" he could have used the :doc-file
component instead. But I believe that indeed this shows that if we provide a
rich, expressive and concise defsystem grammar by default, then extensions
may leave the handling of dependencies to ASDF and focus on other things:
how actual operations are performed.

I am sorry I was too verbose, as usual, and you will by now totally bored,
but I hope you will not find this writing void of logic, as you usually do
:-)

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20100420/956c53eb/attachment.html>