[cltl3-devel] RFC: CLtL3 Charter

[Drew Crampsie]

[snipped some context]

>> Yes, avoiding portability layers is a goal.. so when you state " it is
>> just a matter of creating a small layer on top of them (like
>> usocket)", we're directly against that goal. usocket and the like
>> exist already, so there's not much value in working to standardize the
>> interface... just use usocket! :)
>
> Well, maybe I didn't expressed myself clearly. And I didn't quite get your
> point. If you go down that road, we could say that we don't need to
> standardize FFI or threads as well ("just use CFFI or bordaux-threads" -
> usocket is just as much of a portability layer as CFFI or bordeaux-threads
> is). In the part of "just create a small layer on top of them", I meant
> internally in each implementation. Each implementation will take the
> functions to manipulate sockets that it already has and create the standard
> functions on top of them or make small modifications appropriately to make
> those functions conform with the standard.

Right, i getcha, and i don't necessarily disagree. But, why require
that an implementation have sockets when you don't need them to be
implementation specific?

Sure, given sockets you can create a crude FFI, but it will not be
fast or pretty. However, the socket interface i can create via FFI and
extensible streams is just as good, if not better, than those provided
by individual implementations.

What if i want more functionality for our sockets then those provided
by an implementation? do i ask the implementation to implement my
extension? Again, if we try to force implementors to do anything,
we've lost. We need to gain traction and momentum first.

The point i'm trying to make is that including sockets and networking
in the CLtL3 specification (it's not a standard, lets not call it one)
will require work, and potentially a lot of work, on both our parts
and the parts of implementors. I see a good argument for excluding
them (a perfectly good implementation can be built on the features we
provide), but i've yet to hear a good argument for providing them (in
the description rather than the library) beyond 'because some
implementations have sockets already'.

> This is what I believe that will
> be done by implementations to conform FFI, thread and any other standard we
> create here, isn't it?

That remains to be seen.. this is not ANSI, is not funded by ARPA, and
has little chance of getting anywhere if we start making demands of
implementors. Keep that in mind... nobody has to listen to anything we
come up with. It's our job to make it acceptable.

>> I'm not sure what you mean by 'changing the functions that deal with
>> them' in this context, but a goal of cltl3 (i will add this to the
>> charter) is to require as little effort on the part of implementors as
>> possible. In the case of sockets and networking, we can provide
>> library code from a single source... so why duplicate that effort
>> across every distribution or implementation of lisp? Seems like a
>> wasted effort and a barrier to adoption from my POV.
>
> My point is that every implementation already support sockets, it should be
> much better if there was a standard interface so they could implement socket
> interface uniformly across implementations.

what do you mean by 'standard' and who is the 'they' that will
implement this interface? If the they is us (which is the only they it
could be), then FFI and Extensible Streams are all we require from our
implementations. If the they is implementors, who will pay, who will
convince them it's worthwhile, etc? This is where the main thrust of
my argument lies. We don't need sockets supported by an implementation
if we're given the proper tools to write them ourselves.

What difference does it make if they are in CLtL3 or in the 'standard library'?


>> It is possible, given FFI and gray streams, to implement sockets
>> without using _any_ implementation defined socket code. Rather than a
>> compatibility layer that attempts to paper over the differences in
>> implementations (trouble with such layers being a large motivation for
>> the creation of CLtL3), i'd prefer a single canonical implementation
>> based on constructs provided by CLtL3.
>
> I believe it would also be possible to implement multiple threading that way
> (using FFI, if it support callbacks (as CFFI does)). But I'm not going
> further on this topic.

You'd better not, because that is demonstrably false. Threading
requires implementation support.. think of the simple case of two
threads mutzing about with a hash table. Either you lock the world
(then why bother with threads!), or the implementation must have
primitives that support the notion of multiple strands of execution in
order to avoid race conditions and data corruption.

>>
>> The prior art here (as for a lot of things we will be discussing) is
>> IOlib. IOlib uses its own implementation of sockets and networking
>> based on the OS's underlying implementation (through FFI). It is
>> mostly portable and uniform across implementations, and does not
>> require any support for networking in the implementation itself
>> (fe[nl]ix had better correct me if i'm wrong).
>
> Using IOlib has its drawbacks. It is a big library, one more dependency and
> it will only add functionality that is already present in all Lisp
> implementations.

All lisp implementations? are you sure that all lisp implementations
support all that's in IOlib? Hint: they don't, and that's part of the
reason we're doing CLtL3 :).


> It also uses grays streams, which are not optimized streams
> (so implementation specific's sockets should be faster for good
> implementations). And it uses FFI, which is a barrier when you need to save
> and load lisp images.

Hence a good reason for describing a FFI and an extensible stream
implementation (with a specific stated goal of supporting sockets and
networking, on those platforms where they are available). FFI is
hardly a barrier to saving images... just save the .so in the image as
an array and write it out at run time if you can't find it.

If you think that implementors are going to go ahead and code to
anything we can come up with, you are barking up the wrong tree.
However, if we define a minimal set of features as 'CLtL3', and
provide a reference implementation, there is a chance that it might
gain hold, and users might demand CLtL3 compliance from vendors.


>>
>> I'm still not dismissing inclusion of networking and sockets, but i've
>> not heard a convincing argument for them, and i think our time would
>> be better spent elsewhere.
>
> I still can't really see why standardizing sockets is so different from
> other topics that were included.

You cannot extend sequences without breaking ANSI conformance. You
cannot have FFI and extensible streams without implementation support.
Threads require runtime support. Lisps should have some access to the
OS and filesystem in order to support things like ASDF, file
compilation, etc.

Sockets and networking, OTOH, can be implemented without implementation support.

If we implement it as a library, an implementation is free to clone
the API using fast native streams etc. If we try to force something on
implementors, we will get nowhere.

> I agree with you when you say our time will
> be better spent in other subjects for now, but I really think we should
> discuss this at some point. Lack of standard sockets is one of the topics
> that are most oftenly mentioned when the subject is "what are CL's negative
> points".

Let me reiterate the point that CLtL3 is not a 'standard', and we will
not be standardising anything. CLtL3 is merely a description of
features that modern lisps should provide in order to support our
eventual standard library.

>
> Well, I guess I am the one playing the devil's advocate now :)

It goes back and forth :). Discussion is good!

> By the way, a few more topics that can be discussed:
>
> * Features (as in trivial-features - e.g. when should x86-64 or x86 or
> darwin features be provided)

Excellent.. it's a simple one that needs little discussion but has a
lot of benefit,

> * (Per-package ?) Readtables (I believe we really should discuss about
> readtables, it would avoid many of the hacks that are needed for using read
> macros - clsql is the canonical example)

 I can't believe i forgot readtables! It'll be on the next draft for sure.

> * Declarations and other optimizations - For instance, ECL has a declaration
> to freeze some generic function to avoid redefinition or including new
> methods, which is used for optimization. Freezing functions and types may be
> useful as well. Something like a standard sb-ext:defglobal, including
> sb-ext:global and sb-ext:always-bound proclamations, are welcome. I don't
> know if these are good ideas, but it would be very simple and easy to both
> include and implement (since actually using the declarations is optional,
> and defglobal can be implemented as an usual defvar).

I'll add this to the list as well, in some form or another.

Thanks for all the feedback! Your input has been quite valuable so far.

Cheers,

drewc


>
> _______________________________________________
> cltl3-devel mailing list
> cltl3-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/cltl3-devel
>
>