[Ecls-list] Benchmarking (and versioning)

[Matthew Mondor]

On Tue, 6 Dec 2011 00:27:26 +0100
Juan Jose Garcia-Ripoll <juanjose.garciaripoll at googlemail.com> wrote:

> Out of curiosity, I have produced some plots comparing the performance of
> ECL with a recent copy of SBCL (the one that came with Ubuntu). The goal is
> not really to compare implementations, but to see the evolution of ECL and
> how it works in different modes: interpreted, multithreaded, compiled, etc.
> 
> http://ecls.sourceforge.net/pictures/index.html
> 
> There is still room for improvement :)

The few tests to look closer at seem to be: BIGNUM/PARI-200-5,
MANDELBROT/DFLOAT, FACTORIAL, FFT, WALK-LIST/SEQ, SUM-PERMUTATIONS.

I can also see a very slight preformance regression shown by the PI-*
benchmarks, nothing major.

Is there an easy way for users to reproduce these benchmarks and
generate plots?  I remember that when trying some benchmark suite I had
some trouble to set it up, but it was a while ago.  It'd be nice to
have a comparision between official release versions too once the next
release it out.


This reminds me that the releases are rare, and that the CVS/GIT
snapshots continue to pretend to be the same version over long periods
of time, which is confusing when users submit bug reports.  Since CVS
has a different revision per file, and that GIT has only mostly
meaningless hashes, it's hard to use these for automatic global
versioning of current sources.

There are various conventions which I've seen in use to deal with this:

NetBSD's latest stable release is 5.1 (cut from the netbsd-5 branch),
so its -current snapshots use a 5.99.x until netbsd-6 is branched, with
x being bumped everytime an API or ABI compatibility issue is
introduced by a commit.

Another frequently used system is a datestamp being used for
development code (either an update/checkout or build timestamp).
In this case, a build time stamp would probably not be very useful, but
perhaps that the the date of the latest CVS or Git commit would; an
exemple ECL development snapshot's version could be: ECL 20111205, or
ECL 11.11.1-20111205, etc.

Another possibility would be to only release x.x[.0] releases and cause
the subrevision to be bumped at every commit on the official repository,
i.e. release 11.1 and development snapshot 11.1.3364...

Would any such solution be desirable and easy to implement?


Thanks,
-- 
Matt