[Ecls-list] Project status and changes (directions? help?)

[Matthew Mondor]

On Mon, 7 Oct 2013 10:54:01 +0200
Juan Jose Garcia-Ripoll <juanjose.garciaripoll at gmail.com> wrote:

> * *I am resigning and opening the position of ECL maintainer for anyone to
> take*. I will grant him or her with full administrative rights and full
> responsibility over the project's future. No need to fork the project: if
> you really feel you can make it better, step ahead and take it. I will
> remain as a support developer and help you as much as I can.
> 
> * If no one takes over maintenance, I will *continue working as I can*,
> when I can. This may be *unsatisfactory* for many of you -- if this is the
> case, I am sorry, but that's all there is.
>
> * I am also *abandoning the effort to do extensive testing of ECL on
> various platforms*. If you are interested on a particular platform, you
> will have to build and test yourself and report problems through SF's bug
> report.

I was told recently that the SAGE project could possibly help to provide
development/testing resources (I have had no confirmation of this
yet though, it was an informal IRC discussion with a NetBSD developer
who would have liked me to offer to maintain ECL).  It would be nice to
build a list of people or organizations who care enough about ECL to
join in occasional such OS/compiler related tests and fixes for the
platform(s) they use the most.

I'm unsure if it's because we are too few on this list, if ECL users
are alltogether too few (I know the CL community itself is not large),
or if it's because most ECL users don't consider themselves qualified,
but I do remember Juan asking a few times on this list for help, and
recently he reminded us that noone actually joined the project yet.
It's also possible that at such requests we had no idea in which
direction help was needed.

I'm reticent to ask to officially maintain ECL for the following
reasons:

- Time constraints, as always
- I consider myself at an intermediate (non-expert) Common Lisp level,
  and have no official CS credentials
- I only develop and test on specific environments (NetBSD/amd64,
  NetBSD/i686, Linux x86_64)
- Despite having used ECL a few years on and off and having read
  carefully a good part of its source, I'm still not very familiar with
  a number of aspects, most notably the compiler internals
- Some aspects of the signalling are still mysterious to me, I just
  recently learned what the SIMPLE prefix actually meant technically
  IRT to ECL's implementation
- I also see some old and new redundant code sections which might
  indicate that some cleanup should eventually take place, but I'm still
  confused as to what needs keeping or dropping/moving.  Examples that
  come to mind are two symbol lists, two allocators, several garbage
  collectors, there might be other redundancies
- This can be remedied with some time, but my experience with GIT is
  limited versus with CVS.  I currently use the SF GIT repository for
  tracking changes and commiting my custom changes locally, but have
  had to extract my diffs and reapply them to a new freshly extracted
  tree a few times since (the last time when the repository moved and
  the git-documentation-suggested migration method failed and I had
  trouble recovering).
- I might likely let ASDF rot, I would not notice when it breaks
  (excepts when it alters the behaviour of aspects of ECL iself).
  Someone else would have to maintain it.
- I have few experience with other implementations, I'm in no
  position to make decisions when the hyperspec is obscure, or about
  details of "defacto standards" relative to their precedents, as I
  simply lack the experience.


That said, Juan also wrote that future ECL developers would have access
to his technical support.  Having occasionally suggested patches for
ECL in the past, I would gladly register an SF account and apply for
access, but as another developer rather than as project leader.  I hope
that some other developers also join in (and that Juan is able to also
keep doing what he can, when he can, even if only that)...

With some directions there are probably tasks which I can participate
on over time.  It would be nice to build a TODO list for people (and
myself) to know what needs to be done and work on it if/when possible.


Ideally the mailing list should not be an official place for such a
projects list in the long term, but here are a few starting ideas I can
come up with easily.  Comments are welcome, and it would be most helpful
if others are interested on working on some of these, or have other
suggestions.  If some of these points are considered counterproductive,
it would also be nice to know.

- Testing on various OSs and toolchains (used to exist)
  The ideal for this would be access to a system with various VMs and
  OSs relatively up to date, but it can also be split among users of
  various platforms, as long as they can report errors, produce and/or
  test patches...

- Benchmark comparisons between a few ECL versions, HEAD and a known
  performant CL implementation (i.e. SBCL), on at least one common
  OS+architecture (also used to exist)

- There is an existing TODO file as part of the ECL tree:
  src/new-cmp/TODO

- I think that I've noticed some non-public-API C macros which were
  lowercase.  It would not be very difficult to make those uppercase as
  per usual C convention to enhance code clarity (especially that it's
  easy to overlook the bad use of a C macro causing unintended side
  effects.

- Some macros might be written as C99 inline functions to help
  debugging and avoid side effects without affecting performance
  dramatically

  (copied from above as it also fits here)
- I also see some old and new redundant code sections which might
  indicate that some cleanup should eventually take place, but I'm still
  confused as to what needs keeping or dropping/moving.  Examples that
  come to mind are two symbol lists, two allocators, several garbage
  collectors, there might be other redundancies

- The configure script options, or their descriptions seem inconsistent,
  I have the impression that where two methods are provided to
  specify if a library should be built statically from the
  distribution, or dynamically from the OS, both are marked as
  obsolete.  There ideally should only be one consistent method.

- I have only really surveyed once the ECL interpreter, but if it does
  not have these features already, it would be possible to explore
  various areas, such as a "hardened interpreter" option, perhaps
  eventually using a JIT, LLVM, etc.  I am not necessarily qualified
  for all of this, though.

- Enhancing the documentation: some aspects are not yet documented at
  all.  Others might need some examples or minor additions.  Eventually
  even review, adapt or (re)write some documentation on the internals
  of the implementation, intended for developers.

- I have noticed that when heavily testing threads some difficult to
  explain pauses and lags can suddently begin to occur after some
  sustained high load.  This is something that would be nice to
  understand better and fix.  When this occurs, reducing the load for
  some time before trying again helps.  I could reproduce it on the two
  OSs I use.  I should try to come up with the smallest test case which
  reproduces the problem.

- There are bug/support tickets on SF, some of which might be obsolete,
  others which should be closed as resolved, unreasonable request or
  fixed.

- I have some ideas to extend or complement the native C FFI.  I don't
  have distribution-ready code, but have worked on extensions and have
  tested them using SDL and GD.  Unfortunately it's low-priority and
  long-term work-in-progress.  Following from that I could distribute
  an optional general POSIX module as contrib supporting a number of
  syscalls and libc functions.  I have an internal edition of the latter
  which was built over less powerful extensions to the FFI and which
  are NetBSD-specific only.  The second implementation based on the new
  FFI extensions would be easier to port to various POSIX systems.
  Possibly that libevent would be used instead of exposing kqueue
  directly as well.  Another project on TODO would be full support for
  PostgreSQL's libpq (including asynchronous features, I don't envision
  this as part of ECL, but as OS-friendly package, i.e. initially
  pkgsrc+debsrc).

- I think that eventual transparent support for "UTF8-B" with streams
  would be a good idea.  In this mode invalid UTF-8 sequences would be
  read and mapped automatically to characters within the UTF-16
  surrogate range, to preserve their original octets.  In output mode,
  those would be output as octets.  This would provide lower overhead
  than having to use signals to detect invalid sequences and restarts
  to convert them, and would allow to preserve the original "mixed
  encoding" data.  Examples where mixed UTF-8 and LATIN-* data are
  common are filenames in some file systems, IRC lines, etc.  Currently
  one can gracefully deal with these when treated as bytes or 8-bit
  characters, unless one also needs to have an extended character set
  interface, in which case some conversion must be done anyway.
  If I remember there exists a limitation in the current system to
  easily implement this without touching the stream or
  encoding/decoding system much, but I forgot the details (I have notes
  somewhere on this though).

- There are various situations where although not necessary by the
  standard, printing, describing or inspecting objects with the default
  generic functions could provide more information.  This also helps
  debugging.  Some work has already been done like printing the
  condition type, although the results may sometimes be surprising or
  not totally clear.  Of note are SYS/SYSTEM/SI, SIMPLE, etc.  On
  occasion it was difficult for me to discover exactly which symbol to
  use in HANDLER-CASE/HANDLER-BIND to handle a condition, which
  function to use to invoke a restart, for instance.  Perhaps that
  instead of only printing the type printing its hierarchy would also
  be useful.  Also compare ECL and SBCL when using DESCRIBE on the
  symbol of a condition type (on SBCL the hierarchy is shown).

- It might be nice to have documented (obviously
  implementation-specific) environment objects, as well as features to
  evaluate at macro-expansion and compile-time the state of custom (and
  system) definition/declaration (DECLARATIONS) variables without
  needing custom define macros.  Apparently cltl2 VARIABLE-INFORMATION?

- Inconsistent package name reporting was previously mentioned,
  possibly that if some names are deprecated since a while, they should
  be phased out, and every reporting system should also use consistent
  names.

- Might be slightly redundant with the last point, but deserves its own
  point: review deprecated interfaces, evaluate if they have been
  deprecated long enough to have spawned at least a few official
  releases, and if so, consider dropping those legacy interfaces, after
  making sure that the new interface provides the equivalent
  functionality and are documented.  Unfortunately doing this always
  incurs some annoyance for some users, but it helps to keep a clean
  source tree, which is especially useful when there are few
  developers/maintainers.


Thanks,
-- 
Matt