Shared/static libraries in static-program-op context

[Florian Margaine]

Le lun. 25 mai 2020 à 11:57, Luís Oliveira <luismbo at gmail.com> a écrit :

> Hello Florian,
>
> On Sun, 24 May 2020 at 20:58, Florian Margaine <florian at margaine.com>
> wrote:
> > - CFFI loads a static foreign library, either as :grovel-wrapper or
> :system
> > - The image is dumped with the loaded library
> > - When restored, the runtime will try to reload those "shared" libraries
> and fails because the path doesn't exist on the target system
>
> Let me point out, that this is an issue for regular foreign libraries.
> Take cl-sqlite as an example:
>
> (define-foreign-library sqlite3-lib
>   (:darwin (:default "libsqlite3"))
>   (:unix (:or "libsqlite3.so.0" "libsqlite3.so"))
>   (t (:or (:default "libsqlite3") (:default "sqlite3"))))
>
> On UNIX, (load-foreign-library 'sqlite3-lib) will attempt to load
> "lbsqlite3.so.0" first. If that fails, "libsqlite3.so" will be used
> instead. Whichever happens to succeed when building an image, is what
> will be retried when starting up the image. Things get worse if things
> like *foreign-library-directories* come into play. When that's used,
> the library will be loaded using an absolute path, and that's what
> will be retried at startup.
>
> The proper way to handle this is to close all foreign libraries before
> dumping an image, and reload them using load-foreign-library at
> startup.
>
> I think we should either (a) have load-foreign-library register these
> hooks or (b) somehow hook into image/runtime-op (not the static
> versions) to perform this unload/reload dance.
>
> We might have to be mindful that people shipping applications most
> likely do this dance themselves, so we might clash. For instance, we
> could offer this functionality as a convenient hook users can use in
> their dump and startup routines.
>

Maybe a new option in the ASDF integration? I'm a big fan of backwards
compatibility.


>
> > Specifically for the 2nd one, CFFI provides a :c-file ASDF integration,
> which, when used with :static-program-op, will compile the C file into an
> object file (among others), load it as a foreign library, and statically
> link the object file with the program.
>
> For :c-file, when shipped dynamically, I think the best we can do is
> make sure it's reloaded by name instead of absolute path. That'd work
> on Windows, by placing the DLL next to the executable. Not sure what
> the proper way would be on UNIX.
>
> For a statically linked :c-file (or wrapper, or whatever), then we of
> course need to avoid reopening.
>
> I conclude that we /always/ want to close on dump. Should we just do
> that unconditionally?
>

This actually plays really well with the recent canary addition to cffi, as
the load will be skipped for static libraries.


> As for reloading, we'd still need to distinguish between what should
> and shouldn't be reloaded, so I'm leaning towards your CLOSE-ON-DUMP
> idea but call it RELOAD-ON-RESTART (or something along those lines).
>

We don't actually need to, if we just use the canary mechanism, right?

Why do you say it implies exposing REGISTER-FOREIGN-LIBRARY? Seems
> like a LOAD-FOREIGN-LIBRARY option, no?
>

That was mostly for plumbing, but I'm not quite sure myself now.


> Cheers,
>
> --
> Luís Oliveira
> http://kerno.org/~luis/


Regards,
Florian

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/cffi-devel/attachments/20200525/d12cfb5b/attachment.htm>