From:  Shu-yu Guo <shu@mozilla.com>
Date:  26 Jan 2017 03:32:08 Hong Kong Time
Newsgroup:  news.mozilla.org/mozilla.dev.tech.js-engine.internals
Subject:  

Re: RFC: test262 integration in jstests

NNTP-Posting-Host:  63.245.214.181

>From a CI perspective, I chatted with :gps in person in SF about this.
While I personally am of the opinion of CI pulling in the external repo,
:gps and release folks want to vendor the tests in. This is because they do
not want our tree's open-ness to be dependent on an external service
(GitHub). But, like, I'm pretty sure GH's SLA is better our own...

Given that, here are the concrete next steps:

 - Import test262 into m-c, keeping it up-to-date by periodic imports.
 - Get a tier-2 test262 task (i.e. test262 failing does not close the tree)
and gather feedback about the update process/repo perf with the 20k extra
tests.
 - If everything looks well, remove the tier-2 task and run test262 as part
of regular jstests.

On Wed, Jan 25, 2017 at 11:26 AM, Tom Schuster  wrote:

> Thank you (and anbda of course)! I am so glad we are finally going to
> track test262 progress \o/.
> This should also make it really easy for everyone to figure out which
> tests are still failing.
>
> I second Shu's opinion on defaults and excluding directories.
>
> What is the process of keeping test262 up-to-date going to be?
>
> -Tom
>
>
> On Wed, Jan 25, 2017 at 7:34 PM, Shu-yu Guo  wrote:
> > Woohoo, thanks for the CI work!
> >
> > Outside of CI, I am strongly in favor jstests.py running test262 by
> > default. Concretely, I am in favor of:
> >  - Default set to be everything, including test262, and
> >  - Ability to exclude entire subdirectories
> >
> > They supersede many of our tests and is the general source of compliance
> > truth for many corner cases. The increased running time is worth it for
> > correctness when implementing new features. JIT hackers can most likely
> get
> > by most of the time without running the full suite.
> >
> > On Wed, Jan 25, 2017 at 9:55 AM, Steve Fink  wrote:
> >
> >> anba has done some major work to get test262 runnable as a CI test, and
> >> I've been looking into creating taskcluster jobs for it. One thing that
> >> could use input from a wider audience is how we want it to work with
> >> running jstests manually. The tests replace the js/src/tests/test262
> >> directory, which means that if we do nothing, the runtime of jstests.py
> on
> >> an opt build goes from about 50 seconds to about 260 seconds on my
> machine.
> >> I haven't tried a debug build.
> >>
> >> One option is to suck it up. If everyone (but me) tends to use jit-tests
> >> for their smoke tests already, then it doesn't matter. And you can
> always
> >> explicitly choose subdirectories to run.
> >>
> >> Another option, which I've implemented but wanted some feedback before
> >> landing, is to say that running jstests.py with no arguments gives you
> "the
> >> default set", rather than its current "everything". And that default set
> >> would be "all but test262". If you want everything, you can say
> >>
> >>   ./jstests.py $JS *
> >>
> >> But in order to implement that, I had to implement directory exclusions.
> >> Which means it wouldn't be much extra effort to implement
> >> --exclude=test262, in which case the first option is a little more
> >> tolerable: |./jstests.py $JS| would give you everything, and if you
> didn't
> >> want the huge pile of test262 tests, you could do |./jstests.py $JS
> >> --exclude=test262|.
> >>
> >> A third possibility that comes to mind is to have presets. |./jstests.py
> >> $JS --smoke| or something.
> >>
> >> Opinions?
> >>
> >>
> >>
> >> _______________________________________________
> >> dev-tech-js-engine-internals mailing list
> >> dev-tech-js-engine-internals@lists.mozilla.org
> >> https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
> >>
> >
> >
> >
> > --
> >        shu
> > _______________________________________________
> > dev-tech-js-engine-internals mailing list
> > dev-tech-js-engine-internals@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
>



-- 
       shu