From:  Frank-Rainer Grahl <frg@gmx.com>
Date:  10 Oct 2018 03:17:46 Hong Kong Time
Newsgroup:  news.mozilla.org/mozilla.dev.planning
Subject:  

Re: comm-central future as a branch

NNTP-Posting-Host:  31.18.187.72

Well not knowing if SeaMonkey will see another working version from 
comm-central but it is still compiling so lets assume the best.

Cons:
- I would still need separate comm-repos for at least development and esr 
release. Working with branches and mercurial queues is cumbersome. Switching 
branches on the fly and compiling a different release will usually just result 
in a clobber build. Same for working with a stable and unstable central. 
Doesn't matter much.

If unified:
- Working with a current mozilla repo is dead slow under Windows and somewhat 
slow under Linux. These are too big already. Adding the comm history on top 
and you probably end up with super dead slow :)

- Needs synchronisation with the mozilla repos. So might need many more merges 
which will probably won't make it faster.

Pros:
- Easier for new contributors to just check out one unified repo.
- Merges between releases might be easier (or not).
- Makes a possible future m-c fork easier.
- Tool integration might be easier but most of them are a complete overkill 
for smaller projects such as TB and SM anyway.

Overall indifferent. Current system works for me but there is always room for 
improvement.

FRG


Ben Bucksch wrote:
> I like this idea. I'm in favor.
> 
> If I understood correctly, you're proposing to take a mozilla-central (or a 
> mozilla-unified) repo, put comm-central checkout into the comm/ subdirectory, 
> and push that to the branch "comm/default". I think that's the right way to do 
> things, yes.
> 
> We do not even need buy-in from Firefox people, because hg is a distributed 
> system, so we can simply have our own repository that contains a copy of 
> mozilla-central plus comm/ (from former comm-central).
> 
> I'm also in favor of re-adding CVS history. It's important for us, more 
> important than for Firefox, due to the longevity of our project.
> 
> As for "which combination of mozilla and thunderbird is building and 
> working?", I would suggest to make a branch comm/bleeding where we merge 
> mozilla-central immediately and completely without tests. So, if there are API 
> changes in Mozilla and Thunderbird hasn't been updated yet, it will be broken. 
> (Code-wise, this is what we currently get, by checking out mozilla-central and 
> comm-central independently.) So, you have the option of easily continuing to 
> check out the latest and bleeding edge.
> Then, once we tested the combination and we confirmed that it's working, we 
> merge comm/bleeding into comm/default, and it becomes the new official 
> Thunderbird development branch. This way, fixing an API breakage becomes a hg 
> commit that fixes the API and hg merge operation.
> 
> The main advantage of this setup is that it's clear which version of Mozilla 
> works with which version of the Thunderbid code, down to the commit level. And 
> we capture this information using the version control system itself, but with 
> some build scripts.
> 
> That should dramatically simplify development and build scripts.
> 
> Ben
> 
> 
> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
> 
>> I'd like feedback on the below plan for having Thunderbird in a single 
>> repository.
>> Let's have follow-ups go to dev-planning
>>
>> TL;DR version: planning to have current comm-central live as a branch 
>> instead of a separate repository.
>>
>> ---
>>
>> Background
>> ----------------
>>
>> When the Mozilla move from CVS to hg took place in 2008, the code related to 
>> Thunderbird was not included in mozilla-central and instead placed in the 
>> comm-central repository. Under the hood Thunderbird uses the mozilla-central 
>> code, so to build it currently has to check out both repositories. This is 
>> not ideal, for many many reasons.
>>
>> I wrote [an email to 
>> maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html) 
>> last year listing some of the drawbacks of this setup - but essentially what 
>> it comes down to is that version control was not meant to be used like this 
>> and it makes workflows error prone, in addition to having to do various 
>> hacks in the build automation. Enabling autoland, phabricator and other 
>> tools used by Mozilla core could also be done much more easily for 
>> comm-central with a single repository.
>>
>> The mail spurred some related discussion on git vs hg and importing of 
>> history, but basically there was agreement among the comm-central developers 
>> to go ahead with this change.
>>
>> Thunderbird has now migrated its build automation to Taskcluster, and also 
>> moved to building with mozilla-central as topdir, with a comm/ sub-folder 
>> containing comm-central code. (We used to instead have a mozilla subdir 
>> checkout of mozilla-central as child the comm-central checkout.)
>>
>> In 2014 there was [a proposal to merge comm-central code into 
>> mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ) 
>> but this was rejected since the powers that be did not want 1) the confusion 
>> about what code belongs to what project, 2) pull/tree size, and 3) it was 
>> seen as irrelevant to Firefox. The proposal below has none of these 
>> obstacles, and depending on the physical repository location, may not change 
>> anything at all for a Firefox developer.
>>
>> I think it's now time to revisit the single repository idea and pull that 
>> through in one way or the other. So, on to the proposal. For reference, on 
>> disk comm/ is on currently 122 MB (+ comm/.hg 192 MB), so size-wise it is 
>> comparatively small, with a mozilla-central checkout weighing in at 2.3 GB. 
>> Including the relevant CVS history, the comm/.hg would be 317 MB in total.
>>
>> How
>> ------
>>
>> To capture the essence of what Thunderbird is, we would make comm-central a 
>> named branch of the mozilla-central. Let's call this branch "comm/default".
>>
>> Initially, we'd pull in a hg converted version of the current comm-central 
>> default branch, putting all the files into a comm sub-folder in the base 
>> directory of the comm/default branch. This is the way that the Thunderbird 
>> build system already wants the code laid out. With the comm sub-folder it 
>> will be clear to everyone involved which code is on mozilla-central code and 
>> which is not. It is possible to import the CVS history for the related 
>> directories too while we're at it, but it adds some size. Importing related 
>> CVS history only adds around 125 MB extra so I'm inclined to include it. 
>> With the new comm/default branch pulled in, we merge the curent 
>> mozilla-central default branch to comm/default. The resulting code is now 
>> the same as status quo (of the both repos pulled in), just that it's all 
>> version controlled in the same repository.
>>
>> Once this is all set up, Thunderbird development work would move on on the 
>> comm/default branch, and there would be periodical merges with the 
>> (mozilla-central) default branch. These merges could be automated if wanted.
>>
>> Quite often it's the case that Thunderbird code needs to be adjusted to 
>> build, due to changes in mozilla-central code. For cases of known incoming 
>> bustage Thunderbird now would have the possibility of waiting to do the 
>> merge until a fix is available. I would suggest never to back out bustage 
>> causing mozilla-central changesets from the comm/default branch, but to 
>> detect other incoming bustage by doing builds from a comm/band-aid branch 
>> (branced from comm/default). The details on this can be discussed later 
>> though, not to derail discussion about the main issue here. There are many 
>> alternatives, and it's also completely possible to do what we currently do.
>>
>> Where
>> ---------
>>
>> In the earlier discussions i's become clear there is a bunch of confusion 
>> around what is a branch and what is a repository, so please note the 
>> difference. Aa repository is the physical location where the history of a 
>> project is stored. Every repository does not have to carry every branch. 
>> I.e., the mozilla-central repository could carry the comm/default branch, or 
>> the comm/default branch could exist only in a comm-central2 repository 
>> somewhere else. In all cases it's trivial for developers who have 
>> mozilla-central checked out, to add the comm/default branch to their local 
>> checkout.
>>
>> So for the location there are at least three options:
>>
>> A: branch in the mozilla-central repository
>> B: branch in the mozilla-unified repository
>> C: branch in a repository elsewhere
>>
>> Not to disrupt operations too much I think it would be preferable to use 
>> option B - create the branch in the mozilla-unified repository. It does 
>> depend a bit on what plans Mozilla has for these repositories.
>>
>> Creating the branch in mozilla-unified of course needs buy-in from the 
>> Mozilla hg people, so please let me hear your opinions.
>>
>> What about history?
>> --------------------------
>>
>> For the proposed approach (going through `hg convert`) commits are 
>> preserved, but AFAIK it's not possible to preserve actual commit hashes. `hg 
>> convert` will graft the original commits, and this adds the original hash as 
>> an extra field to the new commit (use `hg log --debug` to see it). I don't 
>> think this is such a big problem. The original commits would be linked in 
>> the pushlog, the same way there is "converted from" for mozilla-central-cvs, 
>> like 
>> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959. I 
>> don't see them e.g. in 
>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611 
>> so perhaps there is some server side feature that needs turning on?
>>
>> Can I try it?
>> ---------------
>>
>> You can check 
>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which 
>> carries the comm/default branch.
>>
>> In your mozilla tree, pull in the branch like this:
>>
>>      hg pull -u -b comm/default 
>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
>>
>> To completely remove it again, use `hg strip "branch(comm/default)`
>>
>> How did you do it?
>> ------------------------
>>
>>     CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
>>     CC_HG_REPO=~/Code/tb/mozilla/comm/
>>
>>     git clone https://github.com/ehsan/mozilla-cvs-history.git $CVS_GIT_REPO
>>
>>     echo default comm/default | tee branchmap.txt
>>     echo rename . comm | tee filemap.txt cc0.txt
>>
>>     hg --cwd=$CC_HG_REPO up 0
>>     (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /') >> 
>> cc0.txt
>>
>>     hg convert --filemap=cc0.txt --branchmap=branchmap.txt $CVS_GIT_REPO 
>> single-repo/
>>
>>     CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
>>     CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
>>     echo $CC_0 $CVS_TIP | tee splicemap.txt
>>     # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa 
>> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
>>
>>     hg convert --config convert.hg.saverev=True --splicemap=splicemap.txt 
>> --filemap=filemap.txt --branchmap=branchmap.txt $CC_HG_REPO single-repo/
>>
>> For fun, see all the history down to 1999 is there, like
>>
>>     hg log comm/mailnews/mime/src/mimei.cpp
>>
>> Then go on and add the default branch from mozilla-central and do the merge
>>
>>     cd single-repo
>>     hg pull -f -b default $CC_HG_REPO/..
>>
>>     hg up comm/default
>>     hg merge default && hg commit -m "Merge default to comm/default branch"
>>
>>     hg push -f -b comm/default --new-branch thunderbird-central-push
>>
>> ---
>>
>> Thoughts, comments, feedback appreciated!
>>
>>  -Magnus
>>
>> _______________________________________________
>> tb-planning mailing list
>> tb-planning@mozilla.org
>> https://mail.mozilla.org/listinfo/tb-planning
>