Would love some feedback on wip feature

[Chris Bagley]

Hi Faré, thanks for all the answers.

I'm starting to see where the mismatch lies between my thinking and
what is needed. However like with Stelian's, this email is excellent
and I'd like to go through it fully. I'll do it in an odd order
though.

> If it's anyone but Integrators, then mucking with implementation identifiers is wrong-headed
> …
>  you're not using it the intended way, either:
> 1) There's a bug in ASDF, so go fix ASDF, or
> 2) There's a bug in your code, so go fix your code.
> …

True, and I'm fine with the idea that my approach was poor, I would
love a better one. The static link capabilities did look really
interested but SBCL is on an older ASDF (I guess for the reasons you
mentioned). Maybe that is part of ASDF I can start reading with the
hope I can contribute at some point.

> But please don't introduce towers of complexity that defeat the design of ASDF.

I can only ask for your trust that this isnt what I don't want more
complexity or to defeat the design on ASDF (even if I achieved both in
the current PR). I'm just looking to ship useful things for lispers to
use, publicly making these kind of mistakes isn't a problem to me if
it gets me closer to making something that is useful and works well
with the existing tooling (ASDF being a huge part of that for me).


> may I suggest you give a look at bazelisp

Thanks, I'll have a peek. But I really like ASDF and want to be useful
on as many lisp implementations as is feasible. So I'd rather stay in
this pond (and help if I can). I just need to understand more deeply
where certain things should be done.


> 1- Author, who writes a given system.
> 2- Referrer, who uses the system from another system.
> 3- Integrator, who assembles a bunch of systems into an application.
> 4- (End) User, who uses the application and shouldn't even have to know it's in Lisp.
>
> What problem are you trying to solve for people in which role? How is it not already solved by the ASDF cache?

I'm trying to allow 2 & 3 to consume the lisp library without having
to worry about the implementation decisions 1 made (unless they want
to).

Let's say I am 1 in this case for example, and that I'm making a GL
based UI for people to use. I decide to wrap an existing GL UI system
written in C and make a nice lisp interface on top.
I would argue that the fact my implementation uses a C library is
irrelevant to the consumer of my library. They wanted a GL UI.

Of course this shouldn't stop them from digging into this if they want
to, but the standard experience should be as standard a lisp
experience as possible. load-system and start being productive.


So my closing question is: If I want to be able to use shared-object
files with my lisp libraries what is the correct asdf way for this to
be done? Given that:
- The shared-object file is not available through any package manager
- people should be able to just quickload the project and start working
- I need to support osx/linux/windows so I will be including a
selection of .so/dll/dylib files.
- Requiring every user to have a C compiler set up on their machine is
not acceptable.

Note that I don't mind some people having to build it themselves as
it's not feasible include precompiled objects for all platforms. But I
want to cover a basic subset

Thanks again for the time you are taking to go through this.

Regards
Chris

On 6 October 2016 at 01:34, Faré <fahree at gmail.com> wrote:
> On Wed, Oct 5, 2016 at 3:26 AM, Chris Bagley <chris.bagley at gmail.com> wrote:
>> I started this feature after finding it a pain to use osicat as
>> neither Windows or OSX ship with a C compiler installed. When the
>> groveler failed the output was very unhelpful, and as I wasn't a
>> developer on the groveler I had no idea what it was or how I should be
>> interacting with it. I was just a consumer who saw value in osicat.
>>
> I like to distinguish 4 roles:
> 1- Author, who writes a given system.
> 2- Referrer, who uses the system from another system.
> 3- Integrator, who assembles a bunch of systems into an application.
> 4- (End) User, who uses the application and shouldn't even have to
> know it's in Lisp.
>
> Only the Integrator should have to worry about matching together Lisp
> systems, Lisp implementations, C compilers, etc. For others, it should
> "just work". By default.
>
> What problem are you trying to solve for people in which role? How is
> it not already solved by the ASDF cache?
>
> If it's anyone but Integrators, then mucking with implementation
> identifiers is wrong-headed. If you're trying to write something for
> Integrators, then look at the ASDF bundle operations as a good start.
>
> And if you're not satisfied with ASDF, may I suggest you give a look
> at bazelisp? Sure, it only works on Linux amd64 using SBCL at this
> point, but it might be easier to port for much better results than
> would be trying to abuse ASDF and try to go against the design of
> ASDF.
>
>> Where can I find details on these 'internal invariants'? This sounds
>> like a problem with the currently implementation and not the concept
>> of avoiding duplication of work (caching).
>>
> The invariants of ASDF are mainly that you should be using the ASDF
> cache and it should just work. If you're not using it the intended
> way, either:
> 1) There's a bug in ASDF, so go fix ASDF, or
> 2) There's a bug in your code, so go fix your code.
>
>> I really would like to understand this as I attempt to ship games in
>> lisp and dealing with C dependencies from projects using CFFI is by
>> far the single biggest issue of shipping.
>>
> That's an application delivery issue, for Integrators.
> For that, please use ASDF (monolithic) bundle operations,
> GDL famously does just that.
>
> See also the cffi-toolchain static-link capabilities, that could use
> some love. (SBCL notably reverted my patch to generate self-contained
> static linked binaries, because it was broken on MacOS and Windows.)
>
> Or use Bazel (which patches SBCL).
>
> But please don't introduce towers of complexity that defeat the design of ASDF.
>
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
> If you're a quiet, law-abiding citizen most of the time but occasionally cut
> someone up and bury them in your backyard, you're a bad guy.
>        — Paul Graham, "The Power of the Marginal"