[Ecls-list] Help needed, really

[Gabriel Dos Reis]

On Sun, Jan 2, 2011 at 12:42 PM, Juan Jose Garcia-Ripoll
<juanjose.garciaripoll at googlemail.com> wrote:
> On Sun, Jan 2, 2011 at 10:54 AM, Gabriel Dos Reis
> <gdr at integrable-solutions.net> wrote:
>>
>> On Sun, Jan 2, 2011 at 3:21 AM, Juan Jose Garcia-Ripoll
>> <juanjose.garciaripoll at googlemail.com> wrote:
>>
>> > This seems to be the current attitude in the Autoconf'ed world, where
>> > uniform build environments are assumed and deviations from them are a
>> > responsibility of the packager or builder.
>>
>> I don't know what you mean by that.  I don't recall that the GMP library
>> is supposed to be linked against only in a uniform build environments, nor
>> do I understand why that has to be assumed (in this specific case.)
>
> What I mean is that right now I am using prepackaged software built with
> standard compiler flags.

None of what I suggest should change that.

> Developers provide the basic software and port
> maintainers keep track of how the build/compile/link flags of their platform
> evolve and pass those to their libraries, compilers, editors, etc, using
> standard Autoconf techniques. That provides a "uniform" environment in which
> the developer of a piece of software is isolated from the complexities of
> the platforms in which it is delivered, as far as standard features (POSIX,
> C, math, etc) are concerned, of course.

Of course, that is one of the reasons why I believe it is ECL's job
to isolate the binary personally complexity from its users -- like other
Lisp systems do -- so that N users don't have to invent N buggy ways
of getting information already contained in ECL and that ECL should provide.

> It also may safely expect that all
> software is properly installed and binary compatible.

A further complexity here is that ECL is a Lisp compiler that
targets the C language -- variously implememted by various
C compilers with several ABI and whatnots.  The implication is that
if ECL finds it great to profit from the  isolation provided
by the various packages it is using, it should also isolate its users from
the low-level complexities brought up by its design choices.

> That is how I expected ECL to be used and so far I have not received any
> feedback that this mode of use makes it totally useless and impossible to
> integrate in reasonable software.

You can take what I've reported so far as a feedback, or you can also
 dismiss it and maintain the invariant that  you have not received any feedback.

> Where particular needs had to be met, the
> configuration mechanism is flexible enough that developers may provide
> nonstandard build and link options *in their private copies*, and be
> consistent then with their own software.

So, is it your official recommendation to maintain private own copy of ECL?

I'm not claiming it is a wrong suggestion, I just wanted to make sure that
I am not reading too much into your suggestion and I'm not putting words
in your mouth.  Please confirm.

> We should put also things in perspective. What do other lisps really
> provide? Lisps around are not that smart and might fool you.

That may very well be the case.  But, they have not done that yet, and
if I concretely ran into that issue and can reproduce it, be sure I would
report it, just like I reported to ECL.


> If I take my
> copy of SBCL the *features* it supplies are just names of assembler
> backends. From base-target-features.lisp-expr
>  ;; machine architecture features:
>  ;;   :x86      any Intel 386 or better, or compatibles like the AMD K6 or
> K7
>  ;;   :x86-64      any x86-64 CPU running in 64-bit mode
>  ;;   :alpha;      DEC/Compaq Alpha CPU
>  ;;   :sparc      any Sun UltraSPARC (possibly also non-Ultras -- currently
> untested)
>  ;;   :ppc      any PowerPC CPU
>  ;;   :hppa      any PA-RISC CPU
>  ;;   :mips      any MIPS CPU (in little-endian mode with :little-endian)
> Note how this is incomplete.

It may be incomplete, but so far it suffices for what the kind of information
I have been asking from ECL.

> You can not deduce from here how C software has
> to be built and linked to be properly loaded and used by SBCL.

Please note that I have never asked ECL to give me an information so that
I "can deduce how C software has to be built and linked to be properly
loaded and used by SBCL".

> This is
> ultimately confirmed by looking at the features of the Solaris/Sparc copy
> that is installed in Sage's buildnet (v 1.0.17). It does not offer really
> useful hints about what ABI it uses
> (:ANSI-CL :COMMON-LISP :SBCL :SB-DOC :SB-TEST :SB-LDB :SB-PACKAGE-LOCKS
>  :SB-UNICODE :SB-EVAL :SB-SOURCE-LOCATIONS :IEEE-FLOATING-POINT :SPARC :UNIX
>  :ELF :SUNOS :LINKAGE-TABLE :STACK-ALLOCATABLE-CLOSURES :OS-PROVIDES-DLOPEN
>  :OS-PROVIDES-DLADDR :OS-PROVIDES-PUTWC :OS-PROVIDES-BLKSIZE-T
>  :OS-PROVIDES-SUSECONDS-T)
> On looking at other platforms, the number of combinations are always
> limited.

Of course, the number of combinations will be always limited, in practice.
I never asked for an unlimited combinations.  I'm really surprised
that your messages
seem to indicate a systematic reduction to absurd (with slight modification
of the original idea if needed) and yet claim you are not being resistive.
There is nothing wrong for ECL to decide that it is not going to provide that
information, based purely on personal preference.  However, I am disturbed that
pretend technical reasons are being advanced against things not suggested.

> The implementation that excels at providing information is
> Lispworks, which
> has http://www.lispworks.com/documentation/lw60/LW/html/lw-600.htm#marker-1039551
> with :lispworks-32bit and :lispworks-64bit and a total of 14 combinations
> perfectly documented -- what those combinations imply for the C world,
> though, is left to the user.

yes, that is already something!  If only ECL can do 1/10 of that, this
discussion would
certainly be different in nature.

> Other platforms, such as CCL have no documented features but seem to offer a
> name for the backend -- which I can not guess from their web.

Again, if you offer a name for your backend that is sufficient enough
for a knowledge person to work with it, it would be a progress.  ECL
pretends :I686 when that is completely bogus and I have read one of
your messages saying it is intentionally done so!

> OpenAxiom also takes two different approaches when it comes to GCL.

Your are missing history.  See below.

> From /open-axiom.trunk/config/open-axiom.m4
> ## For all values of L, except gcl, the assumption is that the Lisp
> ## image L is available in the build environment.  For  gcl,
> ## we make an exception: if no GCL image is available, or if
> ## the option --enable-gcl is specified then OpenAxiom builds its
> ## own version from the source tree.

A bit of history here.  The original open source version of AXIOM
could build only with GCL (for historical reasons, work on GCL, and AKCL,
"grand father" in ECL's pedigree, were funded by IBM in connection
with the AXIOM/SCRATCPAD system.)
For reasons I never fully understood, but that Tim Daly must have
documented somewhere, he decided to keep "known working copies" of
GCL in the AXIOM distribution. That was against common software
packaging practice and was causing lot of problems with many distributions.
The first thing I did when I embarked on modernizing AXIOM's build system
was to take out those "known working copies" out of the repository.
That in itself caused lot of angst, but eventually that decision was also
adopted by FriCAS.  However, since GCL is the "traditional" Lisp
platform of OpenAxiom, AND Lisp system are were not
common places in standard OS distributions, I kept the flags necessary to
build GCL and also instructions for people who prefer to build the historically
old combination of AXIOM + CL.

When I ported AXIOM to SBCL, CLISP, and eventually ECL and Clozure CL,
it did not strike me as a good thing to duplicate the nightmare with GCL.
So, I assumed that a working Lisp system is available.  Once reliable
versions of GCL are officially released, that is going to be less of an issue.

> So OpenAxiom contemplates and does build GCL from sources providing it with
> the right compiler flags (the default ones plus optimizations)

Since you have been asking to put things in perspective, you should follow that
advice and consider the history.  I did not start with a clean plate and
Ifavored GCL (if that is what is causing you trouble.)  It is because
AXIOM has historically been built only with GCL from sources and when one
has the sources, one can set the parameters.

> and guessing  the "openaxiom_precision" from the C compiler itself.

That is done ONLY in build situation where the users wants to
build OpenAxiom with GCL's source.  That is never done when it is
the system-installed GCL -- and GCL is treated just like SBCL, ECL, CLISP,
Clozure CL.  And we look on *FEATURES* to determine the binary
personality.

Said differently, in my normal operation and my recommendation to people,
the binary personality is never guessed from the C compiler.  That would
be obviously the wrong thing to do, since we are linking against the Lisp
compiler which is pre-existing.

Now, if your suggestion is that OpenAxiom users should build their own
ECL system from sources as opposed to using existing pre-packages,
then that is fine.  We would just draw our conclusions and move on.


> When preinstalled, GCL itself will only supply OpenAxiom with the host_cpu
> value given by Autoconf, which is what ECL is currently doing, so it would
> suffer the same problem when trying to identify the appropriate binary
> format in OS X -- I say "would" because I have not been able to build GCL
> myself.

I do not know to what extent you are willing to admit concrete facts to
get into the discussion, but here it is:  GCL gives me:

ring:~ gdr$ gcl
GCL (GNU Common Lisp)  2.6.8 CLtL1    Nov  3 2010 17:45:50
Source License: LGPL(gcl,gmp), GPL(unexec,bfd,xgcl)
Binary License:  GPL due to GPL'ed components: (READLINE UNEXEC)
Modifications of this banner must retain notice of a compatible license
Dedicated to the memory of W. Schelter

Use (help) to get some basic information on how to use GCL.
Temporary directory for compiler files set to /tmp/

>*features*

(:COMPILER :NUMLIB :SDEBUG :DEFPACKAGE :UNEXEC :NATIVE-RELOC :READLINE
    :TRUNCATE_USE_C :CLX-LITTLE-ENDIAN :BSD :DARWIN10.4.0 :X86_64
    :IEEE-FLOATING-POINT :UNIX :GMP :GCL :AKCL :COMMON :KCL)

>


where you see :X86_64 on the same machine where you were "predicting"
"suffer same problem".  And yes, GCL uses Autoconf on that same machine.
And yes, OpenAxiom uses Autoconf on that same machine.
And yes I have built OpenAxiom with GCL on that same, and reported
successful builds using the system installed GCL.

Now, can we put this "Autoconf's fault" nonsense to rest?  It is a distraction.

> So put in perspective, some of the lisps out there provide information for
> the limited number of platforms they support, some of them do not provide
> enough.

I'll take any day useful information on limited supported platforms, over
useless information on a wide variety of supported platforms.

> Could ECL do better?

I thought ECL can do significantly better otherwise I would not have
brought up the issue.  From where I sit, the real question is: would ECL
maintainers want to do better?

> But what?
> To sort out this discussion I may just add a single keyword called
> :ecl-32bit-pointers and :ecl-64bit-pointers which will denote the size of
> pointers.

Please, please, please, the issue is not about the size of pointers.
So, please, whatever you do, don't name it after pointers.  It is a
property of the ECL executable or the ECL library.  For example,
on macintel, you can have 32-bit, 64-bit, or universal (which means
the program or binary essentially has duplicates to run in both
32-bit and 64-bit.)


> This will be equivalent to doing the following in OA's
> configuration
> if test "$axiom_lisp_flavor" = ecl; then
>    openaxiom_host_lisp_precision=`ecl -eval '(print (* 8
> (ffi:size-of-foreign-type :pointer-void)))' -eval '(quit)'`
> fi

As I pointed out in private conversion, and again on this list,
the size of the pointer does not tell me whether it is 32-bit or
64-bit or universal application.

> But will it match the logic after that that OpenAxiom uses to determine
> compiler flags?
> ## Augment C and C++ compiler flags with ABI directives as appropriate
> ## before we proceed to infer other host datatype properties.
> if test -n "$openaxiom_host_lisp_precision"; then
>    case $GCC in
>      yes)
>        CPPFLAGS="$CPPFLAGS -m$openaxiom_host_lisp_precision"
>        LDFLAGS="$LDFLAGS -m$openaxiom_host_lisp_precision"
>        ;;
>      no)
>        # cross fingers and pray.
>        ;;
>    esac
> fi
> Well, this I can not assure.

No, you cannot assure, and this undoubtely will require some iterations,
but at least we need an initial point to iterate from.  Also, please don't
concentrate on the size of `void*'.

> But as I said privately, even the -m32/-m64 option that OpenAxiom is using
> might not be enough to make the software compatible with a preinstalled ECL.

That may be correct, but the -m32 and -m64 in fact already abstracts
away lot of low-level things we should not really have to know.

> What if other compilers are used? Take a Linux/PPC with IBM's toolchain.

That part of the ball is in my camp, you don't have to worry about how
I have to constructs the compiler flags to build a compatible library.

> I
> may choose to build ECL using xlc and 128-bit long double, or 64-bit ones,
> or ones that are library compatible or ones that are not. This will affect
> how your software interacts with ECL. I would expect that other options,
> such as the thread-local-storage might as well, but I do not know enough
> about this and I would rather remain isolated from that -- is this really a
> fool's choice?

Because of ECL's design decision to target the C programming language,
and the C programming language has lots ot parameters left to the
implementation, it is certainly ECL's responsability to document choices
that affect inter-operation with other C library -- at least if ECL claims
to be embeddable.  It has to document enough of the characteristics
so that a knowledgeable person can construct a compatible library.

-- Gaby