Would love some feedback on wip feature

[Chris Bagley]

> My apologies for the delay in answering, but I was a bit busy last
> quarter.

No problem at all, thanks for such a thorough reply

> This patch goes against both principles, so as far as I am concerned my
> vote is a NACK. I'm sorry if you've worked so long on this, but in my
> opinion this feature should not be a part of CFFI.

Again no problem, the concept has value to me (even if the current
implementation could be improved) and this will exist in some form as
a separate library if it doesn't belong here.

> You haven't stated what those issues are, not why you think this
> approach is the best way to solve them nor why do you think it's
> reasonable for a development machine to have a CL compiler but not a C
> compiler.

The answers to these will be longer than the questions I'm afraid. But
you are right that I didn't provide enough in the PR.
As I have done no rigorous survey of new lisp programmers as large so
I can only speak to my experiences an those I has watched get started.

> why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler

TLDR: Because I don't believe library consumers should have to change
their environment to accommodate an implementation detail of the
library's developer.

I make libraries for lisp programmers to use, and naturally I use C
libraries to avoid duplicating good work. The people that use my
libraries shouldn't have to know the innards of the library they use,
they should only *have to* be concerned in how they interact with it
and the benefits it gives them.

To me being a developer on some libraries doesn't preclude you from
being a consumer of others.

I have seen newcomers who were both excited by lisp and willing to
make the emacs switch, drop it because they kept running into C when
they were trying to pull a projects they were interested in.

So the reason to provide the option for compiling on  with C compilers
and not on those without is, I expect, similar to why we don't require
people to install c2ffi on every machine that uses a library that was
made using it: Namely to let the consumers of the library just be
consumers and let the developers of the library handle that.


> not why you think this approach is the best way to solve them

I certainly don't want to make that claim. It's one potential solution
to an issue and has been provided as a PR for review as I expect it
would be improved with more brains on the problem.


> You haven't stated what those issues are

The issue is that a library's consumer should only have to be
interested in the features on the library, and not have to go making
changes to their environment just to use it (caveat: where this can be
avoided etc).

I started this feature after finding it a pain to use osicat as
neither Windows or OSX ship with a C compiler installed. When the
groveler failed the output was very unhelpful, and as I wasn't a
developer on the groveler I had no idea what it was or how I should be
interacting with it. I was just a consumer who saw value in osicat.

It seemed a shame to me that work, which could be valid on many
people's machines (who share the same OS, Architecture, etc), was
being duplicated on all of them. Maybe I'm missing the benefit's, but
the potential benefits so me sounds like the argument for using gentoo
over debian, and whilst I appreciate the benefits gentoo users have
given. I still prefer my dependencies shipped compiled where possible.



> 1) Duplicates ASDF functionality in incompatible ways - like the choice
> of a mapping from source file name to build artifact name

Cool this sounds fixable, I have read the asdf manual but didn't see
any guide on mappings. I copied the style already in place where I
could.

If there are duplication of functionality then would you mind listing
them? I would MUCH prefer to rely on ASDF features where they exist.


2) Undermines ASDF's internal invariants by copying files into the ASDF
cache from a secondary cache and creates empty files in the ASDF cache


Where can I find details on these 'internal invariants'? This sounds
like a problem with the currently implementation and not the concept
of avoiding duplication of work (caching).


3) Creates cache keys using one of the most brittle mechanism in Common
Lisp (*features*).
Try thinking what happens during interactive development when a
developer changes *features* between compilations.

The C library loaded is no longer valid. This is an issue without my
change though as the wrapper specifications use reader conditionals
already. If you change *features* and have already loaded a grovelled
library whos contents were dependent on one of the features
changed..then the assumption for that library are now invalid.

If this is an issue then I can raise a ticket for cffi-grovel, though
I am not sure how to fix it.

> 4) Makes things difficult for the creation of a binary package because
> now that different developers can set different :cache-dir directories
> in the .asd files they control, a user must review and possibly
> manually edit all .asd files of its dependencies (think pgloader for
> example) in order to enforce the use of the same cache directory and to
> be able to find the build artifacts.
>
> If at all, this code should be part of ASDF as a cache available for
> all extension types. There are 2 important rules in doing cross-OS
> development and deployment:
>
> 1) Follow the conventions of the host OS, don't fight them
> 2) Work with your build system

This one is very interesting to me but also has me confused.

I haven't seen anything in ASDF that can package the shared-objects
with the generated binary, so (in my case) they would have to be
shipped anyway (As OS package managers are, as a rule, out of date).
Whether I'm pulling those from some system folder or from some cffi
cache folder doesn't seem to make any difference. I'm assuming you
aren't suggesting trawling through the asdf cache was the correct was
of finding one's compiled dependencies?

:) I expect at this point you are shaking your head and that I have
totally missed your point, I'm also have had no need for pgloader so
I'm unaware of it's details. I'll ask a question instead.

Why would a user ever be changing the cache directory of a library
they don't maintain? If the library is using this feature it's that
the developer of that library saw value in shipping the artifacts of
the grovel/wrapping.

I really would like to understand this as I attempt to ship games in
lisp and dealing with C dependencies from projects using CFFI is by
far the single biggest issue of shipping.



Once again, I make no claim that this version of this feature is the
final best one, just that such a feature has value. This view was
shared by Luis when I approached him online before starting this and
is driven in no small part by the difficulties seen around shipping C
libs in the lisp-games community.


Thanks go going over this and I really hope I can find some nugget in
here that can be of use CFFI.

Warm Regards,

Chris

On 4 October 2016 at 21:35, Stelian Ionescu <sionescu at cddr.org> wrote:
> My apologies for the delay in answering, but I was a bit busy last
> quarter.
>
> On Sun, 2016-08-21 at 23:41 +0200, Chris Bagley wrote:
>> Hi folks,
>>
>> I'm currently working on adding caching of .so and wrapper files to
>> cffi. This is to handle issues arising from trying to use various
>> libraries on machines without a C compiler. I'm hoping you can take a
>> peek at my branch [0] and see if I'm going in the right direction.
>
> You haven't stated what those issues are, not why you think this
> approach is the best way to solve them nor why do you think it's
> reasonable for a development machine to have a CL compiler but not a C
> compiler.
>
>> The changes are very simple, we allow users to specify a cache
>> directory by using an optional :cache-dir argument in :grovel-file &
>> :wrapper-file forms. CFFI will then look there first for the required
>> files before trying to compile them itself.
>>
>> The tricky bit is that we recommend that people use the #+ & #-
>> reader
>> conditionals in the grovel and wrapper specifications, which means
>> the
>> cached results are only applicable if all the required feature-forms
>> match.
>>
>> To handle this I am currently using my with-cached-reader
>> -conditionals
>> library [1] which, whilst small and standalone, could be replicated
>> inside CFFI if required. The library modifies the readtable so that
>> feature expression still work as before, but the also record the
>> feature-forms used.
>>
>> With the captured feature information we can make a subdirectory
>> inside the :cache-dir directory, whose name is based on the feature
>> information. In my current experiment it does something simple & easy
>> to read, however it would likely make directory names too long for
>> windows users.
>>
>> For example I took osicat and added a cache directory to the
>> following lines:
>>
>>     (:grovel-file "basic-unixint" :cache-dir "foo")
>>     (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo")
>>
>> The "foo" directory has the following contents
>>
>>     bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil
>>         unixint.processed-grovel-file
>>     darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil
>>         basic-unixint.processed-grovel-file
>>     linux_t_windows_nil
>>         libosicat.so
>>         wrappers.processed-wrapper-file
>>
>> All the system does is copy these files to the same directory the
>> system would have put the compiled results in the standard system.
>
> So this patch
>
> 1) Duplicates ASDF functionality in incompatible ways - like the choice
> of a mapping from source file name to build artifact name
> 2) Undermines ASDF's internal invariants by copying files into the ASDF
> cache from a secondary cache and creates empty files in the ASDF cache
> 3) Creates cache keys using one of the most brittle mechanism in Common
> Lisp (*features*).
> Try thinking what happens during interactive development when a
> developer changes *features* between compilations.
> 4) Makes things difficult for the creation of a binary package because
> now that different developers can set different :cache-dir directories
> in the .asd files they control, a user must review and possibly
> manually edit all .asd files of its dependencies (think pgloader for
> example) in order to enforce the use of the same cache directory and to
> be able to find the build artifacts.
>
> If at all, this code should be part of ASDF as a cache available for
> all extension types. There are 2 important rules in doing cross-OS
> development and deployment:
>
> 1) Follow the conventions of the host OS, don't fight them
> 2) Work with your build system
>
> This patch goes against both principles, so as far as I am concerned my
> vote is a NACK. I'm sorry if you've worked so long on this, but in my
> opinion this feature should not be a part of CFFI.
>
> --
> Stelian Ionescu a.k.a. fe[nl]ix
> Quidquid latine dictum sit, altum videtur.
>
>