From:  Jim Blandy <jblandy@mozilla.com>
Date:  11 May 2016 14:26:16 Hong Kong Time
Newsgroup:  news.mozilla.org/mozilla.dev.tech.js-engine.internals
Subject:  

Re: Rust JSAPI bindings in js/src?

NNTP-Posting-Host:  63.245.214.181

I fat-fingered the list address at first, so I'm forwarding this here, and
updating the CC list.

---------- Forwarded message ----------
From: Anthony Ramine 
Date: Tue, May 10, 2016 at 4:59 AM
Subject: Re: Rust JSAPI bindings in js/src?
To: Jim Blandy 
Cc: dev-tech-js-engine-internals ,
Eddy Bruel , Josh Matthews , Jack
Moffitt 


Replied inline. Please reply to all because I'm not in
dev-tech-js-engine-internals btw.

> Le 9 mai 2016 à 22:37, Jim Blandy  a écrit :
>
> Hi, folks. How does the SpiderMonkey team feel about moving the Servo
project's Rust JSAPI bindings from the separate github repo they're in now
into js/src in Mozilla Central?
>
> Two weeks ago I posted a message, "Rust bindings for Debugger API",
pointing folks at the discussion in bug 1263317 about adding a Rust API for
SpiderMonkey's debugging facilities. Since then we've learned two
interesting things:
>
> First: Because of the way Rust crates are compiled and linked, it's
difficult to have one Rust API (rust-mozjs) located in a separate crate,
and another Rust API (for the debugger) located in js/src, as originally
discussed. Everything would be much easier if there could be a single crate
providing Rust bindings for the SpiderMonkey library. (Anthony, could you
follow up with more details here?)

So we have two things:

mozjs [1], which packages a SM tarball as a Rust crate and instructs Cargo
on how to link against libjs_static.

rust-mozjs [2], which depends on the former and includes safe wrappers for
some JS APIs and the bindings, which we generate from parsing SM's C++ code
with a fork of rust-bindgen [3][4][5].

> Second: Servo is finding their periodic updates to the latest
SpiderMonkey to require exorbitant amounts of work. At present, an engineer
needs to dedicate an entire quarter to it, which is completely untenable.
Having the Rust API be treated as just another build and test in
SpiderMonkey's ordinary CI processes would be a huge improvement: problems
could be addressed as they arise.

The problem is that we put into the Git repository the bindings for all the
platforms we handle, so that would mean that committers would somehow need
to update each version of the bindings (currently there are 8 of them)
every time they change something. That's not scalable.

I have two solutions in mind:

- We stop putting in Git the bindings generated before-hand, at
upgrade-time, and we start depending on rust-bindgen (and thus libclang) at
compile-time (that would mean that SM developers too would need libclang to
work on SM if the bindings are put there).

- We don't upstream the bindings at all, but we make a complete, scalable
bindgen tool that can generate for any target we need, by behaving like a
cross compiler, where we would "only" need to provide system/libc headers
for the targets we are interested in. That should probably be a quarter
goal, and I think for that to work we would need to make bindgen a proper
Research project. After all it's not like it would hurt Rust to be able to
talk with C, C++ and Objective-C. That could even supersede our own Cocoa
bindings and maybe the winapi stuff too.


I see on Google Groups that Jason Orendorff had some questions too,
replying here.

> What *should* happen if we add a parameter to a JSAPI function? Should we
> add it to the rust-mozjs equivalent at the same time, immediately breaking
> downstream users?

The bindings are generated, so the parameter would be added when updating
mozjs and rust-mozjs. That's really not the blocking part of making a
SpiderMonkey update.

> Who's doing the "release engineering", and what's involved? Just bumping
> version numbers and running `cargo publish` from time to time?


I did the last SM upgrade.

What was involved:

- Generate an SM tarball from mozilla-central's tip with
make-source-package.js
- Excitement.
- Try to compile that.
- Install an Ubuntu VM to setup libclang and Rust and mozjs to generate
bindings for Linux 32 for our Android target.
- Ask a third party to generate bindings for Linux 64 while I'm busy with
Linux 32.
- Despair.
- Install Windows to setup libclang and Rust and mozjs to generate bindings
for Windows.
- Fix a few linking issues on Linux 32.
- Fix some issues where the tarball is missing some things on Windows
because m-c has no CI for that.
- Fix a few linking issues on Windows related to some msys/mingw insanity.
- Suffering.
- Patch some C++ code so that our bindgen tool can cope with it.
- Patch the code in Servo to use the new SM.
- Land.

Oh and did I mention that there are 8 versions of the bindings? We support
4 platforms, but enabling JS_DEBUG means we need another set of bindings
because some structs change layout, and some functions are defined only
with JS_DEBUG.

We don't publish rust-mozjs because we are its only user for now and we
don't claim anything about the usefulness of our current setup.

I hope I'm making sense and that replies all of your questions.



On Tue, May 10, 2016 at 5:40 AM, Till Schneidereit <
till@tillschneidereit.net> wrote:

> On Mon, May 9, 2016 at 11:31 PM, Jack Moffitt  wrote:
>
>> > What *should* happen if we add a parameter to a JSAPI function? Should
>> we
>> > add it to the rust-mozjs equivalent at the same time, immediately
>> breaking
>> > downstream users?
>>
>> Generally you'd make a corresponding change to the Rust bindings
>> immediately and we'd have unit tests for them that ensured they
>> worked. This will only break downstream users if they are pointed
>> directly to the github repo as their dependency *and* they are not
>> pointed at a specific working revision.
>>
>
> In the ideal - and common - case, this shouldn't be a manual process: most
> of the bindings are auto-generated. Ideally, the script doing this would
> run in automation and, e.g., upload the generated bindings to
> archive.mozilla.org, similar to how jsshell bundles are uploaded there.
> (E.g.
> https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/)
>
> The main reason why having this in-tree would be nice is that much of the
> breakage in the bindings is caused by using C++ features that're too clever
> for the bindings generator and hence need annotations or dumbing-down. If
> during automation the bindings were generated and perhaps a very simple
> Rust embedding compiled (such as the tests in [1] or the stupendously
> bare-bones shell in [2]) that should alleviate 90% of the pain that's
> currently associated with updating Servo's SpiderMonkey.
>
> The bindings generator is a Rust application itself:
> https://github.com/crabtw/rust-bindgen
> It should be pretty straight-forward to install this on our build servers:
> they already have all the preconditions available.
>
> I think most of the work involved in getting this working is in getting
> the bindings working with current mozilla-central without patches, and in
> releng. The former entails (cleaning up and) landing a bunch of patches we
> currently have in the mozjs github repo[3].
>
> > Currently rust-mozjs is not published to crates.io, so keeping it
> > compiling and writing some unit tests and ensuring those are part of
> > normal js automation is probably all that is needed right now. If we
> > wanted to publish to crates.io we'd need to occasionally update the
> > version info and do a cargo publish at some regular cadence.
>
> If we had everything needed to create the bindings in mozilla-central and
> published to crates.io, we wouldn't even need a repository for mozjs at
> all: creating a new release would entail testing that it works, fixing any
> potential issues (with patches landed in mozilla-central), bumb the version
> (again with a patch landed in mozilla-central), and publish to crates.io.
> mozilla-central would remain the one source of truth for SpiderMonkey
> instead of the current situation where we effectively have a fork in the
> mozjs repo.
>
> To be clear, under this proposal the Servo team would be on the hook for
> very quickly helping out with regressions to the bindings generation
> process itself, and for fixing (potentially with help by SpiderMonkey
> peers) issues with Servo's usage of said bindings when it comes to
> deploying a new version to crates.io.
>
>
> Till
>
> [1] https://github.com/servo/rust-mozjs/tree/master/tests
> [2] https://github.com/tschneidereit/mozjs-shell
> [3]
> https://github.com/servo/mozjs/compare/85a4f5fcaae2ccd08233d4208d387243483fed40...master
>
>
>