Handling of readme and license files

[Russell Sim]

Robert Goldman <rpgoldman at sift.info> writes:

> I have been out of town and away from my email for a while. Sorry for 
> the late response. Checked ASDF code and I think you are right: it would 
> make sense to make `:read-file-form` more generally usable than just in 
> the `:version` property.
>
> Unfortunately, if we were to do this, it would cause new `defsystem` 
> forms to be backwards incompatible, which would be undesirable.

I was thinking about this and I understand the need for backwards
compatibility, I know that read time conditionals could be used to
enable backwards compatibility `#+asdf-3.4` or something of that
nature?

> We could probably instead add a `:license-file` `defsystem` option. We 
> should think carefully about how this is to be done, though. Probably 
> this should *not* point to a file directly, but to a component 
> (generally a `:static-file`): that would be desirable because using a 
> component already provides a mechanism for specifying the location of a 
> source file.
>
> At this point, though, we are starting to get a structure that is so 
> fussy that it is likely to cause more problems than it fixes.

I agree this sounds overly complicated for the user, introducing
dependent fields i think is too much. Would it make sense to add a
special :license-file component type like
https://gitlab.com/daewok/asdf-release-ops introduces? or implicitly add
a `:static-file` when a `:license-file` is declared as a `defsystem`
option?

Supporting multiple license files is better for dual licensed projects,
so in some ways adding a new component type is the simplest.

> I wonder if this is a technical problem at all. Maybe the problem is 
> just that developers are not putting the right static files into their 
> systems, and the issue is that our social practices need to be fixed, 
> not our code. If programmers should be putting license files in as 
> static files, maybe we should just be teaching them to do this.

I think it's a combination of the 2, if the tool doesn't provide an
easy/obvious way to do fill the `:long-description` field.  We are
relying on the users to copy and paste good examples, or to try and
write portable code.  I do agree it's a balance, and in a library like
this it's important consider all options.  Some of this could be fixed
by providing tooling and better examples.


This discussion has given me some good ideas.  Like you say, it's partly
a social practices issue, which makes me think that tooling like a
linter would help.  But read time evaluation makes that hard to
implement.  If the need for that could be removed somehow, that would
improve things significantly.  One thing that can be seen from the
`:version`, `:read-file-form` option usage below is that if there is a
better way to read a file, people will use it.

Cheers,
Russell


--------------------
Below are some numbers to gauge usage of existing fields and features of
ASDF.  I have gone and pulled all the systems from Quicklisp so we can
look at the current usage.

So looking at 4921 asd files found in quicklisp (NB, this total isn't
perfect there a few minor duplicates, ~10 due to locally having multiple
versions of some packages) But the rest of the results I manually
verified.

70 asd files currently declare a license via a `:static-file` component.
1961 asd files currently declare a `:license` option

76 asd files that declare the readme as either a `:static-file` or a
`:doc-file` (6 use the `:doc-file` declaration) There is some overlap
with the following libraries, but not all, so some people are correctly
declaring a `:static-file` and using read time evaluation.  Sorry I was
too lazy to calculate the set difference.

130 systems using read-file-string
-----------
  :long-description #.(uiop:read-file-string
                       (uiop:subpathname *load-pathname* "README.org"))
-----------

95 asd files using the `with-open-file` similar to this
-----------
  :long-description
  #.(with-open-file (stream (merge-pathnames
                             #p"README.markdown"
                             (or *load-pathname* *compile-file-pathname*))
                            :if-does-not-exist nil
                            :direction :input)
      (when stream
        (let ((seq (make-array (file-length stream)
                               :element-type 'character
                               :fill-pointer t)))
          (setf (fill-pointer seq) (read-sequence seq stream))
          seq)))
-----------

Regarding general use of read time evaluation, there are 7 total
instances where users are incorrectly using `with-open-file` to fill the
`:version` defsystem option from a file instead of using
`:read-file-form` or equivalent line option.

There are ~258 correct usages of the `:version (:read-file-form`
expression.