[Antik-devel] First post: common-lisp -> numerical-lisp->CL-statistics

[Liam Healy]

Hi Mirko, Tamas, Tony,

I am happy to read of these developments; they align very closely with what
I have in mind in building grid and Antik.

A little bit of background: I do numerical computation as part of my "day
jobs," and have used Lisp for  25 years now, starting with the Symbolics.
Over the decades, I developed for personal use only a rather (ahem) hackish
set of numerical libraries and utilities.  When GSL came out in the late
90s, I saw it as an opportunity to gain some numerical utilities without
having to write them myself, but it took until CFFI's release in 2005 to
spur me to start GSLL.  I am of the mind that any correct library is
potentially useful and should be available in CL; while I prefer to code in
CL myself, if the library is already written, then I will try to use it.  I
realize others may differ.  My goal in writing GSLL was to make the
non-lispness of it as invisible as possible, so that it can be used with
all the power of lisp and none of the hassles of C.

One of the hacks I had in my old personal library was that of generic
arrays.  With some GSL functions needing arrays, I was forced to reinvent
that.  I did a lot of research of existing CL libraries, and settled on
Tamas's affine library (which predates xarray, I believe) as a basis, and
built on that.   Eventually, this was split out from GSLL into the grid
package in Antik, which also includes other numerical utilities.  I
envision that grids could address any kind of array-like structure
(including, say SQL tables, structured files like FITS, HDF, etc.) because
these are the kinds of things I use in my work.  They should be as
transparent as possible, so that if for example there is a function in GSL
that does the same thing as another library (written in C, or CL, or
whatever), I can send the exact same inputs and compare the outputs.

There is a lot of work to be done to get Antik and grids to that point, and
I welcome ideas, contributions, and patches.  Unfortunately, my time on
this has been very limited recently, but I intend to move this forward as
best as I can.  Mirko has sent me emails and I've been slow to respond.  I
think the antik-devel<http://lists.common-lisp.net/cgi-bin/mailman/listinfo/antik-devel>list
is a good place to discuss any ideas, and its email archive may be
useful in the future.  It is my intent to be inclusive of different
approaches as possible, so that even working entirely within CL using
arrays and displaced arrays should be supported (currently, displaced
arrays are not there).  I look at languages like Fortran 90, Python, and
Matlab, and I think there is opportunity to borrow idioms and features, and
maybe have interfaces to some of those libraries.  I don't know R but
suspect it too has something to offer.  As Mirko noted, I use (shadowed) CL
symbols where possible.

One special note about the current status: all development for the last
several months or so (and there hasn't been much)  is on the "cffi-libffi"
branches.  Over last fall and winter, working with Luís Oliveira, I have
replaced FSBV with the system cffi-libffi, currently distributed on the
master branch of CFFI.  This is a significant improvement on FSBV because
calling structures by value require nothing more than CFFI calls and having
libffi (also, things like recursive structures should work).  While this is
in the master branch of CFFI, it is not yet in a released version of CFFI,
so I have held off merging my cffi-libffi branches into master because
Quicklisp only pulls from released versions of software, where such a
concept is defined (it isn't for Antik and GSLL).  However, I don't
maintain the FSBV-based master branches any more.

(I am copying antik-devel so that other Antik users will see.)

Liam


On Fri, Oct 12, 2012 at 7:45 AM, A.J. Rossini <blindglobe at gmail.com> wrote:

> Hi Mirko -
>
> The other possibility is that depending on the complexity of Antik (sounds
> like it does a good deal, whether it should all be in the same place is
> another matter, which is just a question that I shouldn't ask until I read
> the code later this weekend, Sorry for posing it prematureley!), we just
> replace xarray with antik.  Some of this might be pretty simple.
>
>
>
> On Friday, October 12, 2012 2:02:06 AM UTC+2, Mirko Vukovic wrote:
>>
>> This is a bit long-winded, but it may have some relevance to the
>> architecture of CL-Statistics
>>
>> The other day, I was writing this as part of a sequence generating
>> library documentation that I hope to release soon:
>>
>> The Intoruction to R<http://cran.r-project.org/doc/manuals/R-intro.html#Top> describes
>> vectors<http://cran.r-project.org/doc/manuals/R-intro.html#Simple-manipulations-numbers-and-vectors>.
>> Of interest are also sections on factors<http://cran.r-project.org/doc/manuals/R-intro.html#Factors>,
>> arrays and matrices<http://cran.r-project.org/doc/manuals/R-intro.html#Arrays-and-matrices>,
>> and lists and data frames<http://cran.r-project.org/doc/manuals/R-intro.html#Lists-and-data-frames>.
>> Finally, reading data<http://cran.r-project.org/doc/manuals/R-intro.html#Reading-data-from-files>is of interest. At this point, this is starting to sound like Rossini's
>> project common-lisp-stat. However note that this project uses Tamas
>> Papp's xarray.
>>
>> At that point, I went to google what Rossini was up to, and came upon
>> this list.  It was up for only a day or so when I found it.
>>
>> The reason for this post is that I am using Liam Healy's gsll and antik
>> libraries.  These use his grid library for representing vectors, matrices,
>> etc.  It is unfortunate that at the time Liam was releasing grid, Tamas
>> released his xarray.
>>
>> Personally, I dislike fragmentation (grid vs xarray), but now nothing can
>> be done about it.  And some competition is good.
>>
>> The interesting part of Liam's antik is that antik shadows several of
>> CL's symobls: many math functions, and also aref and aref*.  He
>> reimplements these as generic functions, allowing extension for other data
>> types (such as grids).  In a way, Liam is on his way to creating what I
>> would call Numerical Lisp: Extension of CL for numerical computing that may
>> be incompatible with CL.  But the extension is a relatively thin layer on
>> top of CL, leaving all of CL accessible.
>>
>> Futhermore, Numerical-Lisp (NL) can unify grid and xarray using the
>> following three-layered structure:
>>  - Application layer, such as CL-Statistics
>>  - Numerical-Lisp layer
>>  - Raw libraries (grid/xarray)
>>
>> The numerical lisp layer would consist of Liam's redefinitions of CL
>> mathematics related functions via generic functions.  The raw libraries
>> would hook into this via methods.  In addition to CL mathematics functions,
>> NL can define interface to higher mathematics functions also using generic
>> functions.  Then the raw libraries can plug in using packages such as GSL,
>> LAPACK, etc.
>>
>> What needs to be done is
>> - creation of the Numerical-Lisp library
>> - hooks in grid and xarray
>> - write the application layer to use Numerical-Lisp interface.
>>
>> It seems to me that much of this should be relatively straightforward.  I
>> would be glad to help in testing (sbcl, clisp) and documenting.
>>
>> Finally, what does this have to do with CL-statistics?  Well, I am
>> suggesting that the code be organized in a way to make it easy to
>> transition from interfacing to CL & xarray into NL.
>>
>> Best,
>>
>> Mirko
>>
>> PS - and in the end, when all is said and done, someone writes a great
>> book: Practical Numerical Lisp
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/antik-devel/attachments/20121014/1598e7f8/attachment.html>