Issues with new testing scripts

Sat Jan 2 08:37:48 UTC 2016

On 2016-01-02 00:33 CET, Robert Goldman <rpgoldman at sift.net> wrote:

> On 1/1/16 Jan 1 -6:18 AM, Kambiz Darabi wrote:
>> On 2016-01-01 09:59 CET, Kambiz Darabi <darabi at m-creations.com> wrote:
>> 
>>> Happy New Year,
>>>
>>> On 2015-12-31 16:12 CET, Robert Goldman <rpgoldman at sift.net> wrote:
>>>
>>>> On 12/31/15 Dec 31 -5:46 AM, Kambiz Darabi wrote:
>>>>> On 2015-12-29 01:58 CET, Robert Goldman <rpgoldman at sift.net> wrote:
>>>>>
>>>>>> The problem of having clones not get auto-populated seems not worse than
>>>>>> needing to do a magical incantation when you merge stuff back and forth
>>>>>> at the cost of garbling your repo.
>>>>>
>>>>> The complete magical incantation reads:
>>>>>
>>>>> git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master
>>>>>
>>>>> and to update it to the lastest master, replace 'add' with 'pull'.
>>>>>
>>>>>> The submodules cause annoying quiet failures, but even when the annoying
>>>>>> quiet failures happen, you don't end up mangling your repo.
>>>>>
>>>>> The problem with submodules is that every single person who checks out
>>>>> the repo is concerned with it, whereas git subtree has to be performed
>>>>> only by one single person and for all others it just works without them
>>>>> even knowing that there is a 'subtree' being used somewhere. 
>>>>
>>>> Can you explain that stuff about needing to squash? I read the subtree
>>>> tutorial and that discussion just seemed like gibberish to me.  Very far
>>>> from clear.  Maybe it's easier than the discussion made it seem.  As I
>>>> read it, it seemed to be saying that if you didn't remember to do
>>>> --squash at the right times you would end up with a repo clogged with
>>>> duplicate commits.
>>>>
>>>> If that's the case...ugh.  But perhaps it was just a bad explanation?
>>>> Or does your subtree add expression above ensure that the squashing
>>>> becomes the default?
>>>
>>> My command above does contain the '--squash' option. The git-subtree
>>> author just chose the wrong default: usually, you are not interested in
>>> seeing each and every commit in the external repo.
>>>
>>> But again, if you do the wrong thing, it's just a matter of:
>>>
>>> git reset --hard HEAD~2
>>> git subtree add/pull --squash ...
>>>
>>>> If we could figure this out maybe subtrees would be easier.  But if you
>>>> have to remember to squash every time, forget it....
>>>
>>> IMO such commands should be automated with a makefile entry, so you
>>> wouldn't have to remember the --squash option.
>
> Unfortunately, right now the makefile is in the worst condition of the
> whole project, because of the recent merge of minimakefile, which by its
> nature was largely untestable (since exercising its functionality would
> change the repository, and even the upstream repository).

AFAICS there are only two places where a remote repo ist changed:

https://gitlab.common-lisp.net/asdf/asdf/blob/master/tools/git.lisp#L18
https://gitlab.common-lisp.net/asdf/asdf/blob/master/tools/git.lisp#L44

So testing makefile changes is only a matter of renaming the remotes
'origin', 'cl.net', and 'github'.

One can even git clone locally, add the local 'remotes', name them
appropriately and test the changes.

> Indeed, just recently, I wasn't able to use the makefile to fix my
> problems with the addition of new ext/ submodules because..... the
> makefile didn't work until the submodules were added.

This is also a point in favor of subtree, as the dependencies are then
part of the repo.

These 9 lines were necessary to implement the new behaviour:

https://gitlab.common-lisp.net/darabi/asdf/commit/fc24dae0a7ba8ccbe7633584da2b26846f37a48e#e603f9495ad27b4e1779e05c500c25f6cf682de8_77_60

I found it quite easy to find the correct place to modify the code
although I'm not at all familiar with the code base.

>> I pushed a branch to my repo which contains the necessary changes:
>> 
>> https://gitlab.common-lisp.net/darabi/asdf/tree/ext-subtree
>> 
>> The specification of what to get where and which version is in
>> ext/dependencies.lisp-expr
>> 
>> and
>> 
>> git checkout ext-subtree
>> make ext
>> 
>> does the job.
>
> I'm happy to consider switching to using git-subtree. It may indeed be
> better.
>
> But not until it gets some better documentation.  The discussion in the
> Pro-Git book, to be honest, just makes my head hurt.  See
> https://git-scm.com/book/en/v1/Git-Tools-Subtree-Merging

This is the documentation for the concept of subtree merging, not the
'git subtree' command. You don't have to deal with the subtree merge
yourself, the command 'git subtree' does it for you.

I apologise for not having provided the pointer to the docs in the
first place (and will give a short example below):

man git subtree

should give you this:

http://manpages.ubuntu.com/manpages/trusty/man1/git-subtree.1.html

> So, the remote repository is added as a remote for the *containing*
> repository?  ASDF would have the cl-scripting as an upstream remote for
> ASDF, not for a contained thing?

No, the commits in cl-scripting (leading to HEAD) are imported into a
local branch (but you don't have to deal with that branch in any way).

Let's go through the process, after you pull my branch ext-subtree:

after Faré's last commit 'tools: Simplify a ,@ into a ,', there is
'Remove git submodules in ext' which deletes .gitmodules and all ext
subdirs:

https://gitlab.common-lisp.net/darabi/asdf/commit/d028e6d2b9d729a47a7380b05e99e17c17080492

then, I performed

git subtree add --squash --prefix=ext/alexandria <alexandria-repo-url> master

which creates two commits, look at the first one with 'git log --name-status 915257':

    Squashed 'ext/alexandria/' content from commit 5a17c07

    git-subtree-dir: ext/alexandria
    git-subtree-split: 5a17c072970cf50213f7f896c40e6e640638391f

A       .boring
A       .gitignore
A       AUTHORS
A       LICENCE
A       README
A       alexandria-tests.asd
A       alexandria.asd
...

As one can see, this is the content of the alexandria repo. Hidden
behind the git-subtree-split property, is the branch which contains the
history of alexandria which led to master:

git log --oneline 5a17c07

5a17c07 fix hygiene and multiple evaluation in ENSURE-GETHASH
b1c6ee0 Fix default ARRAY-INDEX and ARRAY-LENGTH.
a019eda Extended DELETE-FROM-PLIST.1 test to test the removal of duplicate keys.
4c4361c Use the original DELETE-FROM-PLIST replacement.

Now back to the second commit which was performed by 'git subtree add':

commit 62116639895d8c2f537b0cb8f26314716edf3900
Merge: d028e6d 9152572
Author: Kambiz Darabi <darabi at m-creations.net>
Date:   Fri Jan 1 12:53:03 2016 +0100

    Add ext/alexandria as git subtree

    git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master

This commit is a merge and doesn't add any specific files, but if you
look at ext/alexandria, you can see that is was populated with the
content of "Squashed 'ext/alexandria/'".

That's it. From now on, Alexandria's commit 5a17c07 will live in our
repo at ext/alexandria.

If one day you decide that it's time to update to the latest version of
alexandria, you perform:

git subtree pull --squash --prefix=ext/alexandria <alexandria-repo-url> master

and the same thing happens again.

Now to the makefile automation part:

The file ext/dependencies.lisp-expr

https://gitlab.common-lisp.net/darabi/asdf/commit/fc24dae0a7ba8ccbe7633584da2b26846f37a48e#diff-2

currently contains:

(setf asdf-tools::*ext-dependencies*
      '(("alexandria" "master" "https://gitlab.common-lisp.net/alexandria/alexandria.git")
        ...
        ("optima" "master" "https://github.com/m2ym/optima.git")))

performing 'make ext' leads to a 'git subtree pull --squash' as
described above. So, as a developer, you can decide at which point in
time you would like to upgrade to newer versions of the dependencies.

If I was the maintainer, I would edit dependencies.lisp-expr and replace
'master' with the latest release tag of the corresponding repos or a
specific commit if the repo doesn't have release tags. This would lead
to clearly defined dependencies which would make build and testing
reproducible.

But this is of course your decision.

HTH

Kambiz