[Ecls-list] Benchmarking (and versioning)

[Matthew Mondor]

On Tue, 6 Dec 2011 10:02:09 +0100
Juan Jose Garcia-Ripoll <juanjose.garciaripoll at googlemail.com> wrote:

> On Tue, Dec 6, 2011 at 1:44 AM, Matthew Mondor <mm_lists at pulsar-zone.net>wrote:

> > Is there an easy way for users to reproduce these benchmarks and
> > generate plots?  I remember that when trying some benchmark suite I had
> > some trouble to set it up, but it was a while ago.  It'd be nice to
> > have a comparision between official release versions too once the next
> > release it out.
> 
> It is a collection of hacks, plus fixes of Eric's benchmarks, so that they
> also run on ECL, catch errors, etc. I uploaded a tarball with the current
> version of those files here
>    http://ecls.sourceforge.net/ecl-bench.zip

Thanks a lot for sharing, I might try them for local tests here

> This reminds me that the releases are rare, and that the CVS/GIT
> > snapshots continue to pretend to be the same version over long periods
> > of time, which is confusing when users submit bug reports
> 
> This only partially true. Right now ECL uses also the VCS (git) commit id
> to identify itself. This distinguishes the different versions and allows me
> to better find out what changes the user has.

This is nice to know.  Is the commit ID only a hash though?  If so it's
often difficult to immediately assess what is earlier or later, or the
time of the snapshot, without access to the GIT logs.

> OTOH, the release engineering problem continues to be that: a problem. The
> only reasons why there are no releases is because testing on all platforms
> takes time and the current infraestructure I have does not work well
> enough. See for instance how many of the platforms here have outdated
> tests: http://ecls.sourceforge.net/logs.html

I am not complaining about this and I realize that release engineering
is a lot of work.  Fortunately the -current snapshots usually work well
for my uses, and I'm already grateful for that :)

Thanks,
-- 
Matt