Fwd: Re: [monolith builds]

[Daniel Kochmański]

Current ML is ecl-devel at common-lisp.net, I'm forwarding this mail there.

Daniel Kochmański writes:

> Faré writes:
>
>> [Adding ECL-list to this conversation, that I believe should be
>> publicly archived]
>
> Good idea.
>>
>> On Fri, Oct 2, 2015 at 7:24 AM, Daniel Kochmański <daniel at turtleware.eu> wrote:
>>> Hey,
>>>
>>> when fixing problem with randomized names etc I have taken a closer look
>>> at the build process. You asked earlier how ECL manages to do monolithic
>>> builds.
>>>
>>> Each module is initialized with a special function which registers the
>>> whole module and creates lisp objects. Before calling this function ECL
>>> isn't aware of the existance of it neither it has environment.
>>>
>>> Module may be composed on other modules. Builder, when constructing
>>> initialization function for "top module" gathers all submodules (object
>>> files or libraries) and parses the binary file to find special tag which
>>> contains name of functions which initialize submodules. Top module in
>>> it's initialization functions calls (beside the other things) all
>>> submodule initialization functions. Then it concatanates all of them in
>>> .a, .fasl, .so, ... files
>>>
>>> Top-module
>>> Sub-module-1
>>> Sub-module-2
>>> …
>>>
>>> So it is guaranteed, that if this top-module will be embedded in the
>>> other one it's special tag will be found first and the correct
>>> initialization function will be called (which will initialize the futher
>>> submodules).
>>>
>>> This description might be not 100% accurate but I believe it is how
>>> things work there. Reverse engineering might be misleading :-)
>>>
>>> I've fixed bundles once again and ASDF passes tests. It's your call if
>>> you want to enable it by default.
>>>
>>> Best regards,
>>> Daniel
>>>
>>> --
>>> Daniel Kochmański | Poznań, Poland
>>> ;; aka jackdaniel
>>>
>>> "Be the change that you wish to see in the world." - Mahatma Gandhi
>>
>> Thanks for these explanations.
>>
>> Questions:
>>
>> 1- what is a "tag" in this context?
>
> Tag is a string embedded in the C code with deterministic prefix
> followed by a name of the object function. For instance assuming
> initialization function has name lib_init_xyz, then it will look
> something like:
>
> void lib_init_xyz (cl_object block) {
>      ...
>      Cblock->cblock.data_text = (const cl_object *)"@EcLtAg_lib:lib_init_xyz@";
>      ...
> }
>
> When emiting code for top module we scan the binary object files for
> appropriate tag (in this example @EcLAg_lib) to find the inititalization
> function name which we embed in a source code as a normall call:
>
> void lib_init_top_module (cl_object block) {
>      ...
>      Cblock->cblock.data_text = (const cl_object *)"@EcLtAg_lib:lib_init_top_module@";
>      ...
>      lib_init_xyz(block);
>      ...
> }
>
> Note that now function names are randomized and this is mere a
> psudocode. For further details consult src/cmp/cmpmain.lsp.
>
>>
>> 2- once again, why not use the linker support for initialization?
>>   If behavior has to depend on whether ECL was initialized yet,
>>   the initialization could consider in calling ecl_register_init_function,
>>   which depending on whether ECL was booted, would either call the function
>>   and/or add it to a hook.
>>   See once again attached files on how to use linker support for
>>   initialization.
>
> Fact that we might bundle many objects in one archive doesn't mean that
> they don't have dependencies on one another. *I think* that we have to
> strictly control initialization order. It's possible that I'm wrong here
> though.
>
> Also do we want to *always* initialize *everything* what is linked? We
> may have compiled-in support for number of lisp libraries in one module
> just for conveniance and require them on demand depending on the
> application using our code.
>
> I'm guessing here since it's a design decision made by someone
> else. Your proposition is surely worth investigating - it is like it is
> because it was that way when I approached the codebase.
>>
>> 3- How does MKCL do it?
>
> MKCL creates these initialization functions but when it loads files it
> "slurps" some vector with objects - it doesn't call initialization
> functions. Don't know if function creation is
> rendundant/obsolete/inconsistent/correct - you have to ask MKCL
> maintainer for details. It looks to me that it works for files built
> with MKCL but breaks linking from the outer code with the libraries
> created by it (and by breakage I mean that compilation goes fine but
> such modules doesn't work). Once again I'm not 100% about that. MKCL
> code diverged a lot with this regards.
>
> Regards,
> Daniel
>
>>
>> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
>> Suppose 2/3 Congress were incinerated, what would we lose to offset our gain
>> of their salaries and the salaries of their parasites?  — H. L. Mencken

-- 
Daniel Kochmański | Poznań, Poland
;; aka jackdaniel

"Be the change that you wish to see in the world." - Mahatma Gandhi