Hm. On a debug build, jstests for all but test262 takes 115sec; test262
takes 317sec. So we're talking about quadrupling the time -- of
something that really doesn't take all that much total time at all.
So if we're willing to vendor, I would suggest that we don't need any
new patches at all -- what is already in the bug is sufficient. Just
land it as-is, and let the additional tests run as part of the existing
jobs. Unless the compacting run starts timing out or something?
As a separate thing, I can land my changes to implement whole-directory
exclusions. My normal smoke test is to run a debug build on jstests, and
I'm not going to bother doing that as often if it's going to go from 2
minutes to 5 minutes. But I'm ok with having to pass --exclude=test262.
I also don't work on the frontend, so I'm a lot more interested in eg
js1_8_5/extensions than most of the stuff under test262/. That gets me
typed arrays, structured cloning, and a fair amount of GC, which are all
relevant to what I work on.
That's not a great way of dividing things up, though. I would rather
have a more rational basis for choosing smoke tests to run. Like
"interesting tests like js1_8_5/extensions" plus "really fast tests"?
Not that js1_8_5/extensions is a very good collection, either. If only
we had some magic to list out the most "useful" tests, as in ones that
have caught problems in the past -- including those that caught stuff
before it was pushed to try. Hm.
On 01/25/2017 10:34 AM, Shu-yu Guo wrote:
> Woohoo, thanks for the CI work!
> Outside of CI, I am strongly in favor jstests.py running test262 by
> default. Concretely, I am in favor of:
> - Default set to be everything, including test262, and
> - Ability to exclude entire subdirectories
> They supersede many of our tests and is the general source of compliance
> truth for many corner cases. The increased running time is worth it for
> correctness when implementing new features. JIT hackers can most likely get
> by most of the time without running the full suite.
> On Wed, Jan 25, 2017 at 9:55 AM, Steve Fink wrote:
>> anba has done some major work to get test262 runnable as a CI test, and
>> I've been looking into creating taskcluster jobs for it. One thing that
>> could use input from a wider audience is how we want it to work with
>> running jstests manually. The tests replace the js/src/tests/test262
>> directory, which means that if we do nothing, the runtime of jstests.py on
>> an opt build goes from about 50 seconds to about 260 seconds on my machine.
>> I haven't tried a debug build.
>> One option is to suck it up. If everyone (but me) tends to use jit-tests
>> for their smoke tests already, then it doesn't matter. And you can always
>> explicitly choose subdirectories to run.
>> Another option, which I've implemented but wanted some feedback before
>> landing, is to say that running jstests.py with no arguments gives you "the
>> default set", rather than its current "everything". And that default set
>> would be "all but test262". If you want everything, you can say
>> ./jstests.py $JS *
>> But in order to implement that, I had to implement directory exclusions.
>> Which means it wouldn't be much extra effort to implement
>> --exclude=test262, in which case the first option is a little more
>> tolerable: |./jstests.py $JS| would give you everything, and if you didn't
>> want the huge pile of test262 tests, you could do |./jstests.py $JS
>> A third possibility that comes to mind is to have presets. |./jstests.py
>> $JS --smoke| or something.
>> dev-tech-js-engine-internals mailing list