From:  Steve Fink <sfink@mozilla.com>
Date:  26 Jan 2017 08:18:27 Hong Kong Time
Newsgroup:  news.mozilla.org/mozilla.dev.tech.js-engine.internals
Subject:  

Re: RFC: test262 integration in jstests

NNTP-Posting-Host:  63.245.214.181

Hm. On a debug build, jstests for all but test262 takes 115sec; test262 
takes 317sec. So we're talking about quadrupling the time -- of 
something that really doesn't take all that much total time at all.

So if we're willing to vendor, I would suggest that we don't need any 
new patches at all -- what is already in the bug is sufficient. Just 
land it as-is, and let the additional tests run as part of the existing 
jobs. Unless the compacting run starts timing out or something?

As a separate thing, I can land my changes to implement whole-directory 
exclusions. My normal smoke test is to run a debug build on jstests, and 
I'm not going to bother doing that as often if it's going to go from 2 
minutes to 5 minutes. But I'm ok with having to pass --exclude=test262. 
I also don't work on the frontend, so I'm a lot more interested in eg 
js1_8_5/extensions than most of the stuff under test262/. That gets me 
typed arrays, structured cloning, and a fair amount of GC, which are all 
relevant to what I work on.

That's not a great way of dividing things up, though. I would rather 
have a more rational basis for choosing smoke tests to run. Like 
"interesting tests like js1_8_5/extensions" plus "really fast tests"? 
Not that js1_8_5/extensions is a very good collection, either. If only 
we had some magic to list out the most "useful" tests, as in ones that 
have caught problems in the past -- including those that caught stuff 
before it was pushed to try. Hm.

On 01/25/2017 10:34 AM, Shu-yu Guo wrote:
> Woohoo, thanks for the CI work!
>
> Outside of CI, I am strongly in favor jstests.py running test262 by
> default. Concretely, I am in favor of:
>  - Default set to be everything, including test262, and
>  - Ability to exclude entire subdirectories
>
> They supersede many of our tests and is the general source of compliance
> truth for many corner cases. The increased running time is worth it for
> correctness when implementing new features. JIT hackers can most likely get
> by most of the time without running the full suite.
>
> On Wed, Jan 25, 2017 at 9:55 AM, Steve Fink  wrote:
>
>> anba has done some major work to get test262 runnable as a CI test, and
>> I've been looking into creating taskcluster jobs for it. One thing that
>> could use input from a wider audience is how we want it to work with
>> running jstests manually. The tests replace the js/src/tests/test262
>> directory, which means that if we do nothing, the runtime of jstests.py on
>> an opt build goes from about 50 seconds to about 260 seconds on my machine.
>> I haven't tried a debug build.
>>
>> One option is to suck it up. If everyone (but me) tends to use jit-tests
>> for their smoke tests already, then it doesn't matter. And you can always
>> explicitly choose subdirectories to run.
>>
>> Another option, which I've implemented but wanted some feedback before
>> landing, is to say that running jstests.py with no arguments gives you "the
>> default set", rather than its current "everything". And that default set
>> would be "all but test262". If you want everything, you can say
>>
>>   ./jstests.py $JS *
>>
>> But in order to implement that, I had to implement directory exclusions.
>> Which means it wouldn't be much extra effort to implement
>> --exclude=test262, in which case the first option is a little more
>> tolerable: |./jstests.py $JS| would give you everything, and if you didn't
>> want the huge pile of test262 tests, you could do |./jstests.py $JS
>> --exclude=test262|.
>>
>> A third possibility that comes to mind is to have presets. |./jstests.py
>> $JS --smoke| or something.
>>
>> Opinions?
>>
>>
>>
>> _______________________________________________
>> dev-tech-js-engine-internals mailing list
>> dev-tech-js-engine-internals@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
>>
>
>
>