From:  Axel Hecht <axel@mozilla.com>
Date:  28 May 2019 02:43:32 Hong Kong Time
Newsgroup:  news.mozilla.org/mozilla.tools
Subject:  

Re: Standardizing on archive formats in CI

NNTP-Posting-Host:  63.245.210.105

Hi catlee,

responding here with the perspective of doing local (and automated) l10n 
repacks.

How would this affect on what `./mach package` does, and how would it 
affect l10n repacks? Both local and in automation?

Axel

Am 27.05.19 um 20:19 schrieb Chris AtLee:
> We currently use a mix of various archive formats in automation. For
> example, we use:
> 
> * .tar.bz2 for linux builds
> * .tar.gz for test archives
> * .zip for jshell, crashreporter symbols, and mozharness.zip
> * .dmg for macos builds
> * .exe AND .zip for windows builds
> 
> Supporting this variety of different formats has several drawbacks:
> * Complicates automation scripts that need to know what format to expect,
> depending on the platform
> * .tar.bz2 is slow to compress and decompress, and has poor compression
> compared with modern compression methods
> * .tar.gz is pretty fast, but we can also improve compression ratios with
> modern methods
> * .zip is similar to .tar.gz in terms of speed and compression ratio
> * producing both .zips and .exes on Windows is a waste of time and space
> * .dmgs are hard to work with on non-mac platforms
> 
> I'd like to propose a few changes for what archive formats we produce and
> consume in CI:
> 1) Standardize on .tar.xz or .tar.zst for all build and test archives used
> in CI. If this proposal has general support, I'd like to see a good
> analysis of each of xz and zst so we can ideally pick one that works best
> for our use cases.
> 
> 2) Change our canonical linux packaging format to .tar.xz or .tar.zst.
> 
> 3) Change Windows builds to produce only .tar.xz or .tar.zst in CI. We can
> produce the installers from the tarballs in separate "repackage" tasks
> 
> 4) Change the macOS builds to produce only .tar.xz or .tar.zst in CI. We
> already are producing the .dmg files from tarballs in separate "repackage"
> tasks
> 
> With these changes I hope we can simplify the code used to support
> builds/tests in CI, and also achieve some cpu time and space improvements.
> 
> Thoughts?
> 
> Cheers,
> Chris
>