[asdf-devel] source file encoding

[Douglas Crosher]

Attached is my suggestion for adding external-format support.

* A table of translations is included, based on asdf-encodings, but if not found then the external-format is passed through.  The
intention is to increase the range of aliases supported to make it easier to write portable system definitions.  It is not the
intention that the list be exhaustive or that any attempt be made to verify the encoding.

* It uses :external-format.  Users will be working with external-formats, perhaps for a foreign CL implementation but still
external-formats.  Introducing new terminology of 'encoding' seems a mistake.

* No attempt is made to verify the external format.  This is does seem necessary and not even possible.

* A declarative system definition can be used for both portable :utf-8 and implementation dependant (non-portable) external-formats.
There is no need to add code methods or extend asdf-encodings to use user defined or implementation dependant external formats.
Supporting declarative definitions has many advantages over the alternative of requiring asdf-encoding code or asdf methods to
support user defined or implementation dependant external-formats.

* The default is :default.   The external-format support in ASDF would seem to be needed to write 'portable' libraries with UTF-8
source files so it will not be possible until users have upgraded anyway.  Portability is not gained now by making :utf-8 the
default, so I just don't see the advantage of making :utf-8 the default when this would break backward compatibility and make
migration problematic and run contra to the ANSI CL standard.

* At less than 200 lines of code it is just included in asdf.lisp.

Regards
Douglas Crosher


On 04/09/2012 12:36 AM, Faré wrote:
> Abstract:
> I think requiring a few marginal hackers doing weird things
> to specifiy :encoding :default
> is a small price to pay for everyone to be able to specify
> their encoding in a portable way, with a sane default
> that is already almost universally accepted (i.e. :utf-8).
> 
> On Sun, Apr 8, 2012 at 07:31, Douglas Crosher <dtc-asdf at scieneer.com> wrote:
>> The portable-hemlock is still maintained and was updated a few months ago to avoid the use of non-ascii characters in the source so
>> it builds cleanly with UTF-8 as the input external-format.  The code is not in great shape, but is being improved.  See:
>> http://gitorious.org/hemlock/pages/Home
>>
> Oh, I hadn't noticed this new page for hemlock.
> Is CMUCL using the portable hemlock these days, or still including its own?
> 
>> Even if you get all the quick lisp projects converted to be UTF-8 clean,
>> this still represents a subset of ASDF users.  I wish you
>> would reconsider these changes to ASDF because I fear it is divisive.
>>
> Well, I recognize that not all code is in Quicklisp and that
> there is a need for a backward compatibility mode.
> Putting :encoding :default in your defsystem will achieve just that.
> 
> At the same time, if :encoding :default
> rather than :encoding :utf-8 were the default,
> then we'd gain nothing, and it would still be a horrible mess
> to ascertain which system has been compiled with which encoding.
> 
>> It is not reasonable to expect users of ASDF to hack on
>> external support code just to use non-UTF-8 external-formats,
>> and the external library you plan for can never be complete because
>> the external-format is user extensible.
>>
> Well, on the one hand, for portability's sake,
> one should probably one's lisp file to a universally supported external format.
> On the other hand, where portability is not a problem,
> one can either use :encoding :default and be back to the current semantics,
> or extend asdf-encodings as one extends external formats.
> 
>> ASDF could easily be flexible regarding the external-format
>> and not a limited bastion of portable open source code.
>>
> Agreed. Currently, ASDF is not flexible at all -- rather it is uncontrolled.
> 
>>  It would be very easy and workable to just name this :external-format,
>> and to pass through encodings not recognised - all the quicklisp projects
>> would work just fine using :utf-8
>> and other CL users could use encodings as needed.
> Unhappily, passing through external formats is not portable,
> if only for CLISP.
> But if you're doing non-portable things,
> you can keep doing whatever you were previously doing
> with :encoding :default,
> or you can now define methods on asdf::component-external-format
> to do whatever you want, to override the default behavior of
> checking *encoding-external-format-hook*.
> Or then again, you can extend asdf-encodings to make it smarter.
> 
> In practice, how many people do you know who use a non-UTF-8 encoding,
> and how many of them will be majorly annoyed by having to either
> recode their source, explicitly specify their encoding,
> or add :encoding :default to preserve backwards compatibility?
> 
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
> "I've finally learned what `upward compatible' means.  It means we
>  get to keep all our old mistakes."
>  — Dennie van Tassel
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: asdf-external-format.patch
Type: text/x-patch
Size: 13816 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20120410/5023980a/attachment.bin>