Would love some feedback on wip feature

[Luís Oliveira]

Hello Chris,

On Sun, Aug 21, 2016 at 10:41 PM, Chris Bagley <chris.bagley at gmail.com> wrote:
> I'm currently working on adding caching of .so and wrapper files to
> cffi. This is to handle issues arising from trying to use various
> libraries on machines without a C compiler. I'm hoping you can take a
> peek at my branch [0] and see if I'm going in the right direction.

I didn't review your commits thoroughly but it's looking good!


> The tricky bit is that we recommend that people use the #+ & #- reader
> conditionals in the grovel and wrapper specifications, which means the
> cached results are only applicable if all the required feature-forms
> match.

Interesting, that's definitely a difficulty I hadn't anticipated.

The identifier I had in mind was the usual cpu-vendor-os triplet
(e.g., i686-pc-linux-gnu). Even if the grovel file contains no reader
conditionals whatsoever, it'll depend on this triplet due to varying
type sizes across platforms at the very least.

I was thinking there might be an extra dependency here, maybe, which
is the foreign library version. But usually bindings will depend on
binary compatibility with a specific major version anyway, so perhaps
that is a non-issue. If something like that did turn out to be an
issue, perhaps we could add some extensible mechanism for users to add
extra info to the cache key.


> To handle this I am currently using my with-cached-reader-conditionals
> library [1] which, whilst small and standalone, could be replicated
> inside CFFI if required. The library modifies the readtable so that
> feature expression still work as before, but the also record the
> feature-forms used.

Looks neat. dirty-featurep is not super pretty but it seems like the
way to go. (Or we could just warn people that #+ #- will have a
possibly different behaviour than expected within grovel files and
just use alexandria:featurep.) I'm leaning towards just including it
directly into cffi-grovel at this point, but feel free to try and
change my mind. :-)


> With the captured feature information we can make a subdirectory
> inside the :cache-dir directory, whose name is based on the feature
> information. In my current experiment it does something simple & easy
> to read, however it would likely make directory names too long for
> windows users.
>
> For example I took osicat and added a cache directory to the following lines:
>
>     (:grovel-file "basic-unixint" :cache-dir "foo")
>     (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo")
>
> The "foo" directory has the following contents
>
>     bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil
>         unixint.processed-grovel-file
>     darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil
>         basic-unixint.processed-grovel-file
>     linux_t_windows_nil
>         libosicat.so
>         wrappers.processed-wrapper-file

In Osicat's case, the cpu-vendor-os triplet would do the job on its
own, but double-checking which features were used seems sensible
nevertheless.

Indeed, my experience with Windows pathnames makes me very cautios of
using the filename like that for arbitrary feature names because (a)
Windows is picky about what a valid pathname is and (b) it's got a 260
character limit for pathnames.

Perhaps we just need to record the result of each reader conditional,
store those boolean results as increasingly significant bits in an
integer, then base64-encode that and append it to the cpu-vendor-os
triplet? (Then maybe just append that to the grovel file name without
a need for a subdirectory.)


> All the system does is copy these files to the same directory the
> system would have put the compiled results in the standard system.
>
> My next addition is going to be a function the user can call that will
> do the following:
>
> - Work out if cffi-grovel has cached files for the grovel and wrapper results
> - If it didnt have those files it will copy the cachable files to a
> user specified location (or maybe just the cache directory)

Can we avoid the copying by just writing to and loading from the cache
directory unconditionally?

Anyway, it's looking good. I think this has the potential to improve
usability for Windows (and perhaps OS X) users quite a bit. :-)

HTH,

-- 
Luís Oliveira
http://kerno.org/~luis/