From:  Axel Hecht <l10n@mozilla.com>
Date:  07 Sep 2017 20:05:31 Hong Kong Time
Newsgroup:  news.mozilla.org/mozilla.dev.builds
Subject:  

Re: localized builds, status and questions

NNTP-Posting-Host:  79.210.97.238

Am 07.09.17 um 00:45 schrieb Mike Hommey:
> On Thu, Sep 07, 2017 at 12:19:14AM +0200, Axel Hecht wrote:
>> Am 29.08.17 um 06:16 schrieb Mike Hommey:
>>> On Mon, Aug 28, 2017 at 06:10:26PM -0700, Gregory Szorc wrote:
>>>> On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht  wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> as some of you probably noticed, I've been hacking around our localized
>>>>> builds a bit over the past few weeks.
>>>>>
>>>>> Context:
>>>>>
>>>>> For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where
>>>>> to put the generation of res/multilocale.json in a regular build.
>>>>> And I hope that someone volunteers to tackle l10n repacks for local
>>>>> artifact builds.
>>>>>
>>>>> Status:
>>>>>
>>>>> For desktop, we're now at the point where all that we do outside of
>>>>> browser/locales is jar packaging, and installer packaging. For android, we
>>>>> just need to move the search stuff from mobile/locales to
>>>>> mobile/android/locales.
>>>>>
>>>>> We're also only running the jar packaging once, and we're only running
>>>>> repack-l10n.py once.
>>>>>
>>>>> I also think we can drop the -j1 from the entry points.
>>>>>
>>>>> My hope here is that now that we're only using recursive make in a defined
>>>>> scope, and that parallelism isn't a problem anymore, we should have a
>>>>> better chance to get other backends to support repacks? Notably the
>>>>> artifact builds one?
>>>>>
>>>>> The flow of an l10n repack is also constructive now. It's preparing the
>>>>> localization, possibly getting a repo, and running compare-locales for
>>>>> l10n-merge. Then it's doing all the jar stuff, and then the app-specific
>>>>> funkyness like search.
>>>>> Then it packages the stuff as applicable for the project. One desktop,
>>>>> that's langpack, package, and possibly full mar. Then it creates a windows
>>>>> installer if on windows. On mobile, it's just the apk (and full mar?).
>>>>>
>>>>> All of this is done by explicit steps in the recipe, and I got rid of
>>>>> dependencies as much as possible. The dependencies turned out to be primary
>>>>> pain point, as they weren't dependencies to begin with.
>>>>>
>>>>> Questions:
>>>>>
>>>>> Where to put multilocale.json, in en-US builds, and in multi-locale
>>>>> builds, and in single-locale repacks?
>>>>>
>>>>> Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
>>>>> recipes way more complex than I think they should be. Some conditional
>>>>> hacks that we have depending on this could be replaced with
>>>>> IS_LANGUAGE_REPACK ?
>>>>>
>>>>
>>>> XPI_ROOT_APPID feels like something that should be defined once in the
>>>> entrypoint and shouldn't need to be passed around everywhere. But I have no
>>>> clue what all is keyed off of it.
>>>>
>>>>
>>>>>
>>>>> What do we really need to do to support l10n repacks on artifact builds?
>>>>> The build backend code is above me, sadly.
>>>>>
>>>>
>>>> Conceptually, an artifact build is a glorified cache. At the top of the
>>>> build we extract files from a previous build into their normal locations in
>>>> the objdir. And we disable all the build rules that would produce those
>>>> files.
>>>>
>>>> If l10n repacks were implemented as a post-build action that didn't
>>>> interfere with existing files (e.g. didn't delete binaries or other files
>>>> from the objdir it didn't itself produce), I would expect l10n repacks to
>>>> be compatible with artifact builds. The work to actually make them
>>>> compatible is likely a bunch of configure muckery and ensuring the artifact
>>>> build backend has enough context to let l10n repacks work. I wouldn't be
>>>> surprised if we were disabling things required by l10n repacks.
>>>
>>> I, for one, think l10n repacks should be deprecated in favor of artifact
>>> builds. Because, really, when you take a step back, l10n repacks are
>>> just a special kind of artifact builds where the strings data is taken
>>> from a different place.
>>>
>>
>> In the light of https://bugzilla.mozilla.org/show_bug.cgi?id=1352595#c38
>> (brotli and artifact builds), I actually strongly disagree with that
>> sentiment.
>>
>> Our localized builds should have all the optimizations we have for en-US
>> release builds, and I read your comment in the bug that that's not the case
>> for artifact builds.
> 
> That's not the case for artifact builds *today*. Because the brotli
> support was made partial on purpose because, what's the point of
> implementing the whole thing if we're not going to do it in the end?

Yet, the purpose of artifact builds and l10n repacks are rather 
different. Artifact builds (I guess) should make building Firefox 
locally easy for developers. L10n repacks minimize the risk of shipping 
builds in 100 languages on 5 operating systems, and minimizing the time 
it takes.

As much as I like synergies, trying to shoehorn one into the other 
doesn't sound right to me.

All of that is said with minimal experience on artifact builds how they 
work today, and no idea on how they're expected to work in the long run.

Axel