>From a CI perspective, I chatted with :gps in person in SF about this.
While I personally am of the opinion of CI pulling in the external repo,
:gps and release folks want to vendor the tests in. This is because they do
not want our tree's open-ness to be dependent on an external service
(GitHub). But, like, I'm pretty sure GH's SLA is better our own...
Given that, here are the concrete next steps:
- Import test262 into m-c, keeping it up-to-date by periodic imports.
- Get a tier-2 test262 task (i.e. test262 failing does not close the tree)
and gather feedback about the update process/repo perf with the 20k extra
- If everything looks well, remove the tier-2 task and run test262 as part
of regular jstests.
On Wed, Jan 25, 2017 at 11:26 AM, Tom Schuster wrote:
> Thank you (and anbda of course)! I am so glad we are finally going to
> track test262 progress \o/.
> This should also make it really easy for everyone to figure out which
> tests are still failing.
> I second Shu's opinion on defaults and excluding directories.
> What is the process of keeping test262 up-to-date going to be?
> On Wed, Jan 25, 2017 at 7:34 PM, Shu-yu Guo wrote:
> > Woohoo, thanks for the CI work!
> > Outside of CI, I am strongly in favor jstests.py running test262 by
> > default. Concretely, I am in favor of:
> > - Default set to be everything, including test262, and
> > - Ability to exclude entire subdirectories
> > They supersede many of our tests and is the general source of compliance
> > truth for many corner cases. The increased running time is worth it for
> > correctness when implementing new features. JIT hackers can most likely
> > by most of the time without running the full suite.
> > On Wed, Jan 25, 2017 at 9:55 AM, Steve Fink wrote:
> >> anba has done some major work to get test262 runnable as a CI test, and
> >> I've been looking into creating taskcluster jobs for it. One thing that
> >> could use input from a wider audience is how we want it to work with
> >> running jstests manually. The tests replace the js/src/tests/test262
> >> directory, which means that if we do nothing, the runtime of jstests.py
> >> an opt build goes from about 50 seconds to about 260 seconds on my
> >> I haven't tried a debug build.
> >> One option is to suck it up. If everyone (but me) tends to use jit-tests
> >> for their smoke tests already, then it doesn't matter. And you can
> >> explicitly choose subdirectories to run.
> >> Another option, which I've implemented but wanted some feedback before
> >> landing, is to say that running jstests.py with no arguments gives you
> >> default set", rather than its current "everything". And that default set
> >> would be "all but test262". If you want everything, you can say
> >> ./jstests.py $JS *
> >> But in order to implement that, I had to implement directory exclusions.
> >> Which means it wouldn't be much extra effort to implement
> >> --exclude=test262, in which case the first option is a little more
> >> tolerable: |./jstests.py $JS| would give you everything, and if you
> >> want the huge pile of test262 tests, you could do |./jstests.py $JS
> >> --exclude=test262|.
> >> A third possibility that comes to mind is to have presets. |./jstests.py
> >> $JS --smoke| or something.
> >> Opinions?
> >> _______________________________________________
> >> dev-tech-js-engine-internals mailing list
> >> email@example.com
> >> https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
> > --
> > shu
> > _______________________________________________
> > dev-tech-js-engine-internals mailing list
> > firstname.lastname@example.org
> > https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals