[pro] [LibCL-devel] A brave new world

[Daniel Herring]

[Most discussion should take place on the libcl-devel list.]

Hi all,

It has been several months since LibCL saw significant progress.  This was 
a period of small experiments, reflection, and numerous "real-world"
distractions.

This past week, I had a bit of free time; and recent rumors of quicklisp 
focused the need to sieze the moment and put thoughts into action. 
[Thanks Zach.  Honestly, with everything in my head, I couldn't have 
handled volunteers anyway.]

Here's the result: a revised plan for LibCL, and a skeleton code base 
ready for contributors.

http://libcl.com/ [main website, as yet unchanged]
http://gitorious.org/libcl/world [new git repo]


* User experience

The user experience will see relatively few changes; just more stuff will 
be available.  ASDF 1 to ASDF 2 is the biggest configuration change.


* Developer experience

This is much improved.  No more tediously maintaining two git repos for 
each package.  No more harrowing forced conversions of upstream sources 
into git.  No more massive packages.lisp with a metadata defstruct.

Now, each package is defined by a simple package.lisp file.  This contains 
a single plist of information.  Many of the keys are inherited from the 
old packages.lisp file; others are defined in the new metadata.txt.  Key 
new metadata types are :maintainers, :tags, :todo, and :source.

Now a single file describes everything needed to obtain and describe a 
(nearly) arbitrary upstream package.  Helper scripts standardize the 
process.

A simple framework structures the steps needed for building a release. 
It also applies local patches until they can be merged upstream.  Names 
aren't finalized, but see workflow.txt.  The html index will have a mark 
denoting local patches, except for patches following a naming convention 
indicating "trivial" operations like renaming directories and building 
documentation.


* Expanded vision

History:  LibCL began after yet-another-computer/OS-upgrade cycle left me 
spending months reconstructing my libraries.  The current systems just 
didn't work for me.  I started with a bunch of things; but realized that 
many packages needed to be dropped because they were hard to support 
(most notably, anything using FFI).  Then I moved towards the idea of 
creating a "core standard library", with a team of developers, etc.  That 
never gelled.  People were asking for more; I even found myself needing 
one-off, non-portable documentation tools to build the docs for LibCL...

LibCL now aims to be a 1-stop shop.  If a library isn't in LibCL, please 
send a metadata file to this mailing list, with a subject line like 
"[metadata] awesome-package.lisp".  Or use a merge request on the 
gitorious repository.  Same thing goes for applications.  If/when the 
downloads start bloating, we can offer variants as needed.

How are non-portable tools handled?  Through tags.  Only portable, pure CL 
libraries may be tagged with :pure.  Only :pure libraries seeing 
widespread use may be tagged with :core.

The default invocation of compile-libcl will only compile :core libraries. 
Calling compile-libcl with other tags allow people to compile things 
specific to their interests.  Selecting a package by name will compile it 
and all its dependencies.  Selecting T will compile everything.


* Upcoming feature

PJB and others had a great idea.  After "(compile-libcl t)", walk the 
packages and generate a master index of all available symbols.  Link this 
index to docstrings, and eventually add links to the full documentation. 
Good for searching.

This will be included in the next LibCL release.


-------------
Call for help

The current libcl-world is a rough skeleton; but key pieces are already in 
place.  Any help would be appreciated; please coordinate on the 
libcl-devel list; here are some suggested places to start.

Easy tasks:
- add metadata for new packages
- create/add tags to packages
- add maintainer information to packages


Harder tasks:
- coordinate metadata and/or the tarball cache with quicklisp?
- fix package :todos, and add more as needed
- update configure-libcl for ASDF 2
- website stuff
   - create a cache system for package tarballs and log files
   - create a comment/voting system for packages
- sort through the old upstream/local repos for patches which never made
   it upstream
- design a "fetch on demand" layer a la quicklisp or MiKTeX
- hook "fetch on demand" into ASDF
- design a system for caching .so and .dll files for FFI bindings


Most changes may be handled through git merge requests.  Metadata changes 
may be sent to libcl-devel with a subject like "[metadata] package.lisp". 
Contact me directly if you need a shell account for web changes (the 
current website should be available on gitorious).


Thanks for reading this far.  This will be a busy week personally, and I 
should be offline during September 12-14; but feel free to make progress 
without me.  Let's see where this thing goes.

Later,
Daniel