From fahree at gmail.com  Fri Dec 19 17:06:32 2008
From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=)
Date: Fri, 19 Dec 2008 12:06:32 -0500
Subject: [xcvb-devel] First release of XCVB
Message-ID: <653bea160812190906l7c6ebe9bo8c10eba5bcbeef09@mail.gmail.com>

I'm glad to announce the first public release of XCVB.

After a week or so of hacking, I could add a rough user interface to
sbrody's working prototype, and am including an example of how I could
successfully migrate exscribe from ASDF and compile it with XCVB
(after I wrapped an eval-when around exscribe's macros.lisp).

It's all in the git repo. Very rough. Don't expect great error
messages. No support for anything but the basics. But at least, it's
there and it works (in simple cases).

Only tested with SBCL and CFASLs so far.

[ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
The naturalistic fallacy: "if it's natural, it's good."
The anti-naturalistic fallacy: "if it's natural, it's bad."
The a-naturalistic fallacy: "nature has no relationship to good and bad."



From fahree at gmail.com  Wed Dec 24 05:00:04 2008
From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=)
Date: Wed, 24 Dec 2008 00:00:04 -0500
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs, search paths,
	etc.
Message-ID: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>

Since the initial prototype release, I've been cleaning up a bit and
adding to the documentation, focusing notably on what's TODO next.

One topic that's at the same time relatively easy to code and urgent
is the ability to handle multiple projects that must be compiled
together.

ASDF has its infamous *central-registry*. gcc has /usr/include and
various -I flags. then there is PERL5PATH, PYTHONPATH, etc.

What should xcvb do?

I was thinking of identifying every BUILD with a URN, possibly using
xcvb: as the prefix. But how shall we map some top-level directories
to some URNs?

Any ideas on what xcvb should do? Remember that currently, xcvb takes
a xcvb-specialized lisp configuration / program as a parameter -- but
we may eventually want to also/instead have a full command-line
interface to drive xcvb from Make.

My current thoughts: XCVB at startup initializes a path from each of
the following, in order, that may either modify or replace the former:
1- builtin default, say /usr/share/common-lisp/source/
2- environment variable, say XCVB_PATH, : delimited, some magic entry
! means splice the former value
3- command-line argument, say --path, same behavior as above
4- lisp configuration, say xcvb:*search-path*, a list of pathnames.

How each pathname is interpreted is that whenever a new module is
requested, it is looked up:
1- in the current build
2- in each of the parents of the current build
3- in each of the paths specified above, from first to last (left to right).

Any comments?

[ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
"I've finally learned what `upward compatible' means.  It means we
 get to keep all our old mistakes."
 -- Dennie van Tassel



From krewinkel at gmx.net  Wed Dec 24 07:45:39 2008
From: krewinkel at gmx.net (Albert Krewinkel)
Date: Tue, 23 Dec 2008 23:45:39 -0800
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
	(=?utf-8?Q?=22Far=C3=A9=22's?= message of "Wed, 24 Dec 2008 00:00:04
	-0500")
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
Message-ID: <m23ageqan0.fsf@gmx.net>

"Far?" <fahree at gmail.com> writes:

> Since the initial prototype release, I've been cleaning up a bit and
> adding to the documentation, focusing notably on what's TODO next.

That is some impressiv TODO list.  Reading my way through the docs and
the todo, I believe you are on to something really cool here.  I
especially like the good support for a more hierarchical code
organisation.

> One topic that's at the same time relatively easy to code and urgent
> is the ability to handle multiple projects that must be compiled
> together.
>
> ASDF has its infamous *central-registry*. gcc has /usr/include and
> various -I flags. then there is PERL5PATH, PYTHONPATH, etc.
>
> What should xcvb do?
>
> I was thinking of identifying every BUILD with a URN, possibly using
> xcvb: as the prefix. But how shall we map some top-level directories
> to some URNs?
>
> Any ideas on what xcvb should do? Remember that currently, xcvb takes
> a xcvb-specialized lisp configuration / program as a parameter -- but
> we may eventually want to also/instead have a full command-line
> interface to drive xcvb from Make.
>
> My current thoughts: XCVB at startup initializes a path from each of
> the following, in order, that may either modify or replace the former:
> 1- builtin default, say /usr/share/common-lisp/source/
> 2- environment variable, say XCVB_PATH, : delimited, some magic entry
> ! means splice the former value
> 3- command-line argument, say --path, same behavior as above
> 4- lisp configuration, say xcvb:*search-path*, a list of pathnames.
>
> How each pathname is interpreted is that whenever a new module is
> requested, it is looked up:
> 1- in the current build
> 2- in each of the parents of the current build
> 3- in each of the paths specified above, from first to last (left to right).
>
> Any comments?

Just a random thought which sprang into my mind: I always kinda wished
ASDF was offering the option to put a couple of definitions into a
single file.  Although slow, this is convenient and windows friendly
(new project: just append to file).

Albert



From geocar at gmail.com  Wed Dec 24 12:37:15 2008
From: geocar at gmail.com (Geo Carncross)
Date: Wed, 24 Dec 2008 07:37:15 -0500
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
Message-ID: <527511140812240437w1c31e99em9303b0c8eb1c0a85@mail.gmail.com>

On Wed, Dec 24, 2008 at 12:00 AM, Far? <fahree at gmail.com> wrote:

> ASDF has its infamous *central-registry*. gcc has /usr/include and
> various -I flags. then there is PERL5PATH, PYTHONPATH, etc.
>

Hi Fare,

My current thoughts: XCVB at startup initializes a path from each of
> the following, in order, that may either modify or replace the former:
> 1- builtin default, say /usr/share/common-lisp/source/
> 2- environment variable, say XCVB_PATH, : delimited, some magic entry
> ! means splice the former value
> 3- command-line argument, say --path, same behavior as above
> 4- lisp configuration, say xcvb:*search-path*, a list of pathnames.
>

This seems like it covers everything, but I find I lump more loading stuff
in
deployed systems into the setup/startup script - and setting XCVB_PATH there
just seems messy.

May I suggest having a xcvb:*find-modules* var (or similar) that contains a
list of these functions (or
symbols to these functions perhaps) so that instead of just adding new
pathnames, we can add
functions to find lists of these pathnames. They could optionally accept a
module name in case the searching module can use the information to narrow
the search better.

An extension of this could also open the possibility of network paths for
finding new modules.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/xcvb-devel/attachments/20081224/c833578c/attachment.html>

From fahree at gmail.com  Fri Dec 26 00:43:22 2008
From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=)
Date: Thu, 25 Dec 2008 19:43:22 -0500
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <m23ageqan0.fsf@gmx.net>
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
	<m23ageqan0.fsf@gmx.net>
Message-ID: <653bea160812251643g517bec58sf8e45554661cb446@mail.gmail.com>

2008/12/24 Albert Krewinkel <krewinkel at gmx.net>:
> "Far?" <fahree at gmail.com> writes:
>
>> Since the initial prototype release, I've been cleaning up a bit and
>> adding to the documentation, focusing notably on what's TODO next.
>
> That is some impressiv TODO list.  Reading my way through the docs and
> the todo, I believe you are on to something really cool here.  I
> especially like the good support for a more hierarchical code
> organisation.
>
The TODO list just grew even bigger with today's commit
as I dumped my informal notes and integrated items from the presentation.
Moreover, I'm still working on this module naming issue,
which I think is crucial. I'm heading towards adopting the same
path-based naming convention that
Perl, PLT Scheme, Python, OCaml and about everyone seems to be using.


> Just a random thought which sprang into my mind: I always kinda wished
> ASDF was offering the option to put a couple of definitions into a
> single file.  Although slow, this is convenient and windows friendly
> (new project: just append to file).
>
I'm not sure what you mean here. Can you explain?

Merry Gravmas!

[ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Well, the fact is, God does exist and I'm His prophet. And my prophecy is
that you shouldn't believe in God, for it would be a blasphemy against His
gift of Reason to you. So don't you believe in Him, lest you go to Hell!
		-- Far?



From attila.lendvai at gmail.com  Fri Dec 26 02:22:36 2008
From: attila.lendvai at gmail.com (Attila Lendvai)
Date: Fri, 26 Dec 2008 03:22:36 +0100
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <527511140812240437w1c31e99em9303b0c8eb1c0a85@mail.gmail.com>
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
	<527511140812240437w1c31e99em9303b0c8eb1c0a85@mail.gmail.com>
Message-ID: <b54b34af0812251822o4d1d5105n952e0c91af8df420@mail.gmail.com>

> May I suggest having a xcvb:*find-modules* var (or similar) that contains a
> list of these functions (or
> symbols to these functions perhaps) so that instead of just adding new
> pathnames, we can add
> functions to find lists of these pathnames. They could optionally accept a
> module name in case the searching module can use the information to narrow
> the search better.


+1 from here, modulo details, naming, etc.

i've never liked the static *system-registry* with the manual
symlinking on the filesystem. using a directory scanner yields a much
pleasing user experience for me. so, please make the functional way
the default or the first in the docs to advertise it for the
newcomers...

-- 
 attila



From fahree at gmail.com  Fri Dec 26 03:28:52 2008
From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=)
Date: Thu, 25 Dec 2008 22:28:52 -0500
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <b54b34af0812251822o4d1d5105n952e0c91af8df420@mail.gmail.com>
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
	<527511140812240437w1c31e99em9303b0c8eb1c0a85@mail.gmail.com>
	<b54b34af0812251822o4d1d5105n952e0c91af8df420@mail.gmail.com>
Message-ID: <653bea160812251928j6522ba4cn50c63831299aa907@mail.gmail.com>

2008/12/25 Attila Lendvai <attila.lendvai at gmail.com>:
>> May I suggest having a xcvb:*find-modules* var (or similar) that contains a
>> list of these functions (or
>> symbols to these functions perhaps) so that instead of just adding new
>> pathnames, we can add
>> functions to find lists of these pathnames. They could optionally accept a
>> module name in case the searching module can use the information to narrow
>> the search better.
>
> +1 from here, modulo details, naming, etc.
>
> i've never liked the static *system-registry* with the manual
> symlinking on the filesystem. using a directory scanner yields a much
> pleasing user experience for me. so, please make the functional way
> the default or the first in the docs to advertise it for the
> newcomers...
>
OK for directory scanning, though I'm not sure whether to make it
eager scanning (might take a longer time, but allows arbitrary mapping
from module name to filesystem path) or lazy scanning (faster, but
assumes a trivial mapping from module to filesystem path).

I'm not sure what you mean by "the functional way". Can you elaborate?
Even produce a mock API?

(I'm currently reading AITR-874 and trying to complete the redesign of
the internals.)

[ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Documentation is worth it just to be able to answer all your mail with 'RTFM'.
        -- Alan Cox



From attila.lendvai at gmail.com  Fri Dec 26 12:36:08 2008
From: attila.lendvai at gmail.com (Attila Lendvai)
Date: Fri, 26 Dec 2008 13:36:08 +0100
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <653bea160812251928j6522ba4cn50c63831299aa907@mail.gmail.com>
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
	<527511140812240437w1c31e99em9303b0c8eb1c0a85@mail.gmail.com>
	<b54b34af0812251822o4d1d5105n952e0c91af8df420@mail.gmail.com>
	<653bea160812251928j6522ba4cn50c63831299aa907@mail.gmail.com>
Message-ID: <b54b34af0812260436i4d667041vac750907ed847aa8@mail.gmail.com>

> I'm not sure what you mean by "the functional way". Can you elaborate?
> Even produce a mock API?

nothing fancy, just something that does not rely on symlinks on the
filesystem or otherwise a strict directory structure, but instead runs
something on the lisp side.

sometimes i experiment with two branches of a dependency, which may
mean a different branch of 5-10 transitive dependencies... and
switching between them should be simpler than a bunch of ln -s.

for example it would be good if there was a way to customize xcvb to
prefer 'foo.some-branch' to 'foo' while scanning for the module
definition for 'foo'.

as of eager/lazy scanning: in our experience, it works fine to use an
eager scanner that builds up the *cenral-registry* at build-image
time. this is mostly ok, but a bit inconvenient in the rare occasion
when a new system is added, because i either have to
(reread-asdf-registry) every time i start the old image or rebuild the
image. i mostly do the latter because i can start and forget the build
process in the background while doing something else.

-- 
 attila

ps: sorry for the vague ideas in this mail. unfortunately it's more a
wishlist than something useful...



From geocar at gmail.com  Fri Dec 26 16:23:44 2008
From: geocar at gmail.com (Geo Carncross)
Date: Fri, 26 Dec 2008 11:23:44 -0500
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <b54b34af0812251822o4d1d5105n952e0c91af8df420@mail.gmail.com>
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
	<527511140812240437w1c31e99em9303b0c8eb1c0a85@mail.gmail.com>
	<b54b34af0812251822o4d1d5105n952e0c91af8df420@mail.gmail.com>
Message-ID: <527511140812260823h565e9c5avb23f38c62ade13c2@mail.gmail.com>

On Thu, Dec 25, 2008 at 9:22 PM, Attila Lendvai <attila.lendvai at gmail.com>wrote:

> > May I suggest having a xcvb:*find-modules* var (or similar) that contains
> a
> > list of these functions (or
> > symbols to these functions perhaps) so that instead of just adding new
> > pathnames, we can add
> > functions to find lists of these pathnames. They could optionally accept
> a
> > module name in case the searching module can use the information to
> narrow
> > the search better.
>
>
> +1 from here, modulo details, naming, etc.


The naming isn't important to me either. Neither are a lot of the details :)


>
> i've never liked the static *system-registry* with the manual
> symlinking on the filesystem. using a directory scanner yields a much
> pleasing user experience for me. so, please make the functional way
> the default or the first in the docs to advertise it for the
> newcomers...
>

Well, I wasn't thinking about it as something incompatible; I'm talking
about implementing the suggested search system in an extensible way.
Something like this:

(defvar *module-finders* '(find-in-home find-in-environ find-in-sys
find-on-http))
(defun find-in-home (id) "Searches ~/.xcvb/systems for ID" ...)
(defun find-in-environ (id) "Searches $XCVB_SYSTEMS env" ...)
(defun find-in-sys (id) "Searches /usr/lib/xcvb/systems for ID" ...)
(defun find-on-http (id) "Searches various websites for xcvb systems" ...)

Such that it would implement the suggested default path, but specifically
export *module-finders*
so that instead of merely setting *system-registry* (or similar), we can
push something interesting
onto module-finders to do a local (or packaged) search; For example, my
application which ships with its own dependencies might push on a function
which prefers the included files, or when porting an app from sbcl to ecl
(or whatever), I might find i want to change the "home" of some modules on
an ad-hoc basis so I can use an older version of a particular module,
without having to move files around on the disk.

Originally I was thinking the finders would return a list of pathname
objects, and that network-methods would use a cache. This would also make
implementing (modules-apropos) or something, easier.

I've thought more since then:

The finders could return some object that we could interact with using:

(defgeneric finder-get-more-findees (what))
(defgeneric findee-get-definition (found))

Where "what" was returned by one of the functions listed in *module-finders*
(to implement lazy lists- perhaps just a list and a generating function
would be enough...), and "found" is one of the members of "what".

"found" would usually be a pathname (in which case, findee-get-definition
would open, read it, and return the definition), but it could also be other
things- perhaps a network socket or some rows from an sql database.

Once a defintion had been read, xcvb would have to check to make sure it was
actually the desired module; The finders would be able to assume ID was just
a substring to give searching and retrieving a common interface, but by
allowing ID to be T (all systems) we could mirror/cache all systems easily,
or debug the search tree easier.

Of course, this still supports the search path you outlined previously.

Also, if the found object is dynamically dispatched, different module file
formats could be implemented by a special loader, and (more interestingly)
:around methods could be used to dynamically process the defined systems
(instead of having to edit or fork the definition files). I don't know how
valuable this would be to others though.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/xcvb-devel/attachments/20081226/8c57fd94/attachment.html>

From attila.lendvai at gmail.com  Fri Dec 26 17:05:48 2008
From: attila.lendvai at gmail.com (Attila Lendvai)
Date: Fri, 26 Dec 2008 18:05:48 +0100
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <527511140812260823h565e9c5avb23f38c62ade13c2@mail.gmail.com>
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
	<527511140812240437w1c31e99em9303b0c8eb1c0a85@mail.gmail.com>
	<b54b34af0812251822o4d1d5105n952e0c91af8df420@mail.gmail.com>
	<527511140812260823h565e9c5avb23f38c62ade13c2@mail.gmail.com>
Message-ID: <b54b34af0812260905i446ebcf3wd388e6a03b8cf56f@mail.gmail.com>

>> +1 from here, modulo details, naming, etc.
>
> The naming isn't important to me either. Neither are a lot of the details :)

naming *is* important for me, i just don't feel informed enough to
chose the right ones in this context.

imho, naming is of about 60-80% importance in clear and readable code...

-- 
 attila



From marcoxa at cs.nyu.edu  Sat Dec 27 09:58:15 2008
From: marcoxa at cs.nyu.edu (Marco Antoniotti)
Date: Sat, 27 Dec 2008 11:58:15 +0200
Subject: [xcvb-devel] Next on the TODO list: multiple BUILDs,
	search paths, etc.
In-Reply-To: <b54b34af0812260436i4d667041vac750907ed847aa8@mail.gmail.com>
References: <653bea160812232100g40fe3f65g830704a68c33c699@mail.gmail.com>
	<527511140812240437w1c31e99em9303b0c8eb1c0a85@mail.gmail.com>
	<b54b34af0812251822o4d1d5105n952e0c91af8df420@mail.gmail.com>
	<653bea160812251928j6522ba4cn50c63831299aa907@mail.gmail.com>
	<b54b34af0812260436i4d667041vac750907ed847aa8@mail.gmail.com>
Message-ID: <19EA6776-3F34-4945-B20C-1BFEE045E194@cs.nyu.edu>


On Dec 26, 2008, at 14:36 , Attila Lendvai wrote:

>> I'm not sure what you mean by "the functional way". Can you  
>> elaborate?
>> Even produce a mock API?
>
> nothing fancy, just something that does not rely on symlinks on the
> filesystem or otherwise a strict directory structure, but instead runs
> something on the lisp side.

You want something like

MK:ADD-REGISTRY-LOCATION

A scanned can be built on top of this functionality.   And I agree  
that symlinks are not to be trusted.  I got hit very very badly  
recently when I moved a code stash from a SMB accessible Linux file  
system to a Windows file system.

--
Marco Antoniotti