From:  Chris AtLee <catlee@mozilla.com>
Date:  29 May 2019 22:52:53 Hong Kong Time
Newsgroup:  news.mozilla.org/mozilla.tools
Subject:  

Re: Standardizing on archive formats in CI

NNTP-Posting-Host:  63.245.210.105

I wasn't sure what was best for Android. For testing, do we need to have
the builds in apk format? If so, then I don't think there's any point in
generating anything except APKs for Android.

I think we should heavily favour time over space. Storage is relatively
cheap compared to human or machine time waiting for archives to be produced
or extracted. Perhaps one caveat to this is anything that we end up
shipping to users. If we can spend more cpu time producing a smaller
archive for users, then I think we should.

In automation, perhaps our evaluation metric could be something like "m =
compression_time + N*(transfer_time + decompression_time)", where N is the
number of downstream tasks from a build. Further to this, maybe we should
assume a constant network speed, and then the only thing to compare would
be compression and decompression time on set of reference archives.

On Mon, 27 May 2019 at 15:51, Geoffrey Brown  wrote:

> Chris,
>
> That sounds good to me.
>
> You didn't mention Android explicitly; will Android be treated just like
> Linux?
>
> If you encounter a trade-off between time and space, how will you decide
> which is "best"?
>
>  - Geoff
>
> On Mon, May 27, 2019 at 12:19 PM Chris AtLee  wrote:
>
>> We currently use a mix of various archive formats in automation. For
>> example, we use:
>>
>> * .tar.bz2 for linux builds
>> * .tar.gz for test archives
>> * .zip for jshell, crashreporter symbols, and mozharness.zip
>> * .dmg for macos builds
>> * .exe AND .zip for windows builds
>>
>> Supporting this variety of different formats has several drawbacks:
>> * Complicates automation scripts that need to know what format to expect,
>> depending on the platform
>> * .tar.bz2 is slow to compress and decompress, and has poor compression
>> compared with modern compression methods
>> * .tar.gz is pretty fast, but we can also improve compression ratios with
>> modern methods
>> * .zip is similar to .tar.gz in terms of speed and compression ratio
>> * producing both .zips and .exes on Windows is a waste of time and space
>> * .dmgs are hard to work with on non-mac platforms
>>
>> I'd like to propose a few changes for what archive formats we produce and
>> consume in CI:
>> 1) Standardize on .tar.xz or .tar.zst for all build and test archives used
>> in CI. If this proposal has general support, I'd like to see a good
>> analysis of each of xz and zst so we can ideally pick one that works best
>> for our use cases.
>>
>> 2) Change our canonical linux packaging format to .tar.xz or .tar.zst.
>>
>> 3) Change Windows builds to produce only .tar.xz or .tar.zst in CI. We can
>> produce the installers from the tarballs in separate "repackage" tasks
>>
>> 4) Change the macOS builds to produce only .tar.xz or .tar.zst in CI. We
>> already are producing the .dmg files from tarballs in separate "repackage"
>> tasks
>>
>> With these changes I hope we can simplify the code used to support
>> builds/tests in CI, and also achieve some cpu time and space improvements.
>>
>> Thoughts?
>>
>> Cheers,
>> Chris
>> _______________________________________________
>> tools mailing list
>> tools@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/tools
>>
>