Shared/static libraries in static-program-op context

[Florian Margaine]

Le lun. 25 mai 2020 à 13:57, Florian Margaine <florian at margaine.com> a
écrit :

>
>
> Le lun. 25 mai 2020 à 11:57, Luís Oliveira <luismbo at gmail.com> a écrit :
>
>> Hello Florian,
>>
>> On Sun, 24 May 2020 at 20:58, Florian Margaine <florian at margaine.com>
>> wrote:
>> > - CFFI loads a static foreign library, either as :grovel-wrapper or
>> :system
>> > - The image is dumped with the loaded library
>> > - When restored, the runtime will try to reload those "shared"
>> libraries and fails because the path doesn't exist on the target system
>>
>> Let me point out, that this is an issue for regular foreign libraries.
>> Take cl-sqlite as an example:
>>
>> (define-foreign-library sqlite3-lib
>>   (:darwin (:default "libsqlite3"))
>>   (:unix (:or "libsqlite3.so.0" "libsqlite3.so"))
>>   (t (:or (:default "libsqlite3") (:default "sqlite3"))))
>>
>> On UNIX, (load-foreign-library 'sqlite3-lib) will attempt to load
>> "lbsqlite3.so.0" first. If that fails, "libsqlite3.so" will be used
>> instead. Whichever happens to succeed when building an image, is what
>> will be retried when starting up the image. Things get worse if things
>> like *foreign-library-directories* come into play. When that's used,
>> the library will be loaded using an absolute path, and that's what
>> will be retried at startup.
>>
>> The proper way to handle this is to close all foreign libraries before
>> dumping an image, and reload them using load-foreign-library at
>> startup.
>>
>> I think we should either (a) have load-foreign-library register these
>> hooks or (b) somehow hook into image/runtime-op (not the static
>> versions) to perform this unload/reload dance.
>>
>> We might have to be mindful that people shipping applications most
>> likely do this dance themselves, so we might clash. For instance, we
>> could offer this functionality as a convenient hook users can use in
>> their dump and startup routines.
>>
>
> Maybe a new option in the ASDF integration? I'm a big fan of backwards
> compatibility.
>
>
>>
>> > Specifically for the 2nd one, CFFI provides a :c-file ASDF integration,
>> which, when used with :static-program-op, will compile the C file into an
>> object file (among others), load it as a foreign library, and statically
>> link the object file with the program.
>>
>> For :c-file, when shipped dynamically, I think the best we can do is
>> make sure it's reloaded by name instead of absolute path. That'd work
>> on Windows, by placing the DLL next to the executable. Not sure what
>> the proper way would be on UNIX.
>>
>> For a statically linked :c-file (or wrapper, or whatever), then we of
>> course need to avoid reopening.
>>
>> I conclude that we /always/ want to close on dump. Should we just do
>> that unconditionally?
>>
>
> This actually plays really well with the recent canary addition to cffi,
> as the load will be skipped for static libraries.
>

Actually, the canary option implies that we know an existing ELF symbol in
the foreign library, which is not possible in our case.


>
>> As for reloading, we'd still need to distinguish between what should
>> and shouldn't be reloaded, so I'm leaning towards your CLOSE-ON-DUMP
>> idea but call it RELOAD-ON-RESTART (or something along those lines).
>>
>
> We don't actually need to, if we just use the canary mechanism, right?
>

Still trying to think of a good way to do that. I'll likely come back with
a PR.


> Why do you say it implies exposing REGISTER-FOREIGN-LIBRARY? Seems
>> like a LOAD-FOREIGN-LIBRARY option, no?
>>
>
> That was mostly for plumbing, but I'm not quite sure myself now.
>
>
>> Cheers,
>>
>> --
>> Luís Oliveira
>> http://kerno.org/~luis/
>
>
> Regards,
> Florian
>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/cffi-devel/attachments/20200525/d39de291/attachment.htm>