Add Nutrient benchmark

[pweiskircher]

As discussed on our call, a Nutrient benchmark!

This is a pretty real-world workload. We import some annotations (same as if a user adds them, but less messy in the code). We render pages. We extract the text.

The benchmark doesn't require any license keys and just uses a trial version. This has no limitations except a runtime limit (1 hour) and a watermark in the rendered pages.

[netlify]

✅ Deploy Preview for webkit-jetstream-preview ready!

Name	Link
🔨 Latest commit	99ee890
🔍 Latest deploy log	https://app.netlify.com/projects/webkit-jetstream-preview/deploys/68a62751e09a2100084cb733
😎 Deploy Preview	https://deploy-preview-92--webkit-jetstream-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[danleh]

Thanks a lot, that was super quick! Overall it looks great to me.

I had a quick look at the CPU profile/flamegraph with 40 iterations (see comment), and in d8 we spend ~40% of the cycles compiling. That's quite compilation heavy, so we could consider increasing the runtime of the workload further. We also spend ~20% of the cycles in the top 3 hottest functions, but that LGTM.

CC @kmiller68 @eqrion for input from the other engines.

[pweiskircher]

Thanks a lot, that was super quick! Overall it looks great to me.

I had a quick look at the CPU profile/flamegraph with 40 iterations (see comment), and in d8 we spend ~40% of the cycles compiling. That's quite compilation heavy, so we could consider increasing the runtime of the workload further. We also spend ~20% of the cycles in the top 3 hottest functions, but that LGTM.

I'm glad to hear that! We have a lot more we can benchmark. Just wanted to make sure we're going in the right direction. Will work on that and ping you again soon!

[pweiskircher]

Marking it ready for review. This is a pretty real-world workload. We import some annotations (same as if a user adds them, but less messy in the code). We render pages. We extract the text. What do you think?

[pweiskircher]

@danleh @kmiller68 Anything I can do to get this reviewed? :)

[danleh]

(Sorry for the late reply.) Thanks for addressing the comments and adding a license for the binary!

From our (Google's) side, this is a nice real-world workload to have, but there have been some concerns in the last meeting regarding using a "non-standard"/proprietary license and not having source code available.

You mentioned in an earlier email that you might be able to use the original, unmodified Wasm binary from your publicly released NPM package in this benchmark. Would that be possible? If we just 1:1 copy the Wasm binary from an upstream package and have open sourced JavaScript code for the rest (setup, polyfills if required), that might make things easier wrt. licensing and also makes it clear that this is the exact Wasm code that real users are running in the wild.

@kmiller68 @eqrion Happy to discuss this here or in the next meeting.

[pweiskircher]

(Sorry for the late reply.) Thanks for addressing the comments and adding a license for the binary!

From our (Google's) side, this is a nice real-world workload to have, but there have been some concerns in the last meeting regarding using a "non-standard"/proprietary license and not having source code available.

You mentioned in an earlier email that you might be able to use the original, unmodified Wasm binary from your publicly released NPM package in this benchmark. Would that be possible? If we just 1:1 copy the Wasm binary from an upstream package and have open sourced JavaScript code for the rest (setup, polyfills if required), that might make things easier wrt. licensing and also makes it clear that this is the exact Wasm code that real users are running in the wild.

I'm happy to do that! Our latest release - 1.5.0 - unfortunately doesn't have the changes in it to work in the shell, but our next one will (freezes next Tuesday). I can use a nightly build for now and adjust when our next release is out or I can wait. Either one is fine with me.

Happy to help in any way to get this into JetStream. The only thing I can't do it open source the code unfortunately.

[eqrion]

@danleh I opened up that NPM package and the license points here.

I am not a lawyer, but I do not think the terms will be acceptable to be included in this repo.

Specifically:

LICENSEE MAY ONLY USE LICENSED TECHNOLOGY FOR EVALUATION PURPOSES, WITH
A VIEW TO PURCHASING A FULL COMMERCIAL DEVELOPMENT LICENSE. IN ALL CASES,
LICENSED TECHNOLOGY'S OBJECT CODE MAY NOT BE SUBMITTED TO APPLE'S APP
STORE, GOOGLE’S PLAY STORE, ANY THIRD PARTY WHATSOEVER, OR MADE
AVAILABLE ON ANY WEBSITE, PRIVATE OR PUBLIC, OR USED IN PRODUCTION. ANY
REDISTRIBUTION OF LICENSED TECHNOLOGY IN EITHER SOURCE OR BINARY FORM IS
STRICTLY PROHIBITED.
3.4. ANY DISTRIBUTION OR MAKING AVAILABLE OF DERIVED WORKS OR LICENSEE
APPLICATIONS TO LICENSEE'S END USERS IS STRICTLY PROHIBITED.

3.2. Licensee shall not (and is not licensed to):
3.2.1. reverse engineer, decompile, disassemble, bypass any code obfuscation, or otherwise
attempt to derive the source code of any part of the Licensed Technology. (To the
extent that applicable law expressly permits you to decompile the Licensed
Technology in order to obtain information necessary to render the object code libraries
interoperable with other software, you must first obtain written permission from us to
provide the necessary information)

I followed up internally at Mozilla and we agreed that we do not want to include any benchmark where we cannot view the source code. It could maybe be okay for a disabled workload that's not part of the main score, but not the main score.

[pweiskircher]

@danleh I opened up that NPM package and the license points here.

I am not a lawyer, but I do not think the terms will be acceptable to be included in this repo.

I talked with our legal team and everyone is very excited at the possibility to get this included here. I'm sure we could adjust licensing to allow this usage.

I followed up internally at Mozilla and we agreed that we do not want to include any benchmark where we cannot view the source code. It could maybe be okay for a disabled workload that's not part of the main score, but not the main score.

This is more problematic. There's no way we can open source our solution.

[eqrion]

This is more problematic. There's no way we can open source our solution.

That's very understandable, I'm definitely not realistically asking that you do that.

[danleh]

Recording offline discussions with Nutrient and in the last JetStream meeting, next steps: Patrik will reach out to their legal team to (potentially) change the license to allow:

• Licensees to use the binary "for benchmarking, performance comparisons of different execution environments, and performance analysis" or something along those lines (not just for evaluating a purchase).
• Hosting a website with the JetStream suite, including this workload, e.g., on https://browserbench.org/JetStream/ (where it could be run manually, e.g., by adding a URL parameter such as ?test=Nutrient-sdk-wasm)
• Putting the binary into the JetStream repository.
• Putting the binary into Chromium, V8, Firefox, SpiderMonkey etc. repositories as part of a vendored copy of the JetStream repo.
• Allow disassembling and analyzing Wasm bytecode for performance investigations.

[pweiskircher]

@danleh @eqrion I received a new license from legal that should hopefully cover this usage.

[pweiskircher]

@kmiller68 Hi there! Is there anything actionable from my side to help this along? Daniel mentioned I should ping you.

[kmiller68]

Hi @pweiskircher, are you sure you want to put this in the codebase? You might want to run this past your legal again anyway since other line items in the benchmark are GPL licensed. Apologies that this was missed earlier but I don't want you to be put on a limb.

I'd also have to run it past our legal too since I'm almost certain the provided license is not GPL compatible. I'm unsure if, that would be sufficient anyway. I think this test would have be removed from the codebase before serving the content.

[pweiskircher]

Hi @pweiskircher, are you sure you want to put this in the codebase? You might want to run this past your legal again anyway since other line items in the benchmark are GPL licensed. Apologies that this was missed earlier but I don't want you to be put on a limb.

I'd also have to run it past our legal too since I'm almost certain the provided license is not GPL compatible. I'm unsure if, that would be sufficient anyway. I think this test would have be removed from the codebase before serving the content.

Thanks for your reply! Yeah, I didn't realize there is GPL benchmarks in this project. Going to check with legal, but you're right, the most likely answer is no.

[matej]

Hi everyone. Following an internal conversation, we (Nutrient) don’t believe that this would technically be an issue for our side. Other licenses in this project would not change the license applying to our code. However, I’m reasonably confident that this poses a problem for the JetStream project itself, given that there’s GPL code in the project which then limits compatibility of licenses for other code in the project, as already pointed out by @kmiller68. We didn’t do a full legal assessment, and I’m not a lawyer, and this is no legal advice; so please feel free to check with your own legal team to make sure.

Given the overall amount of legal uncertainty in this PR and that at best there’s going to be the need for special treatment of this test, the unfortunate conclusion seems to be that we have to scrap this one. Unless you see a different option?

In any case, https://www.nutrient.io/webassembly-benchmark/ would remain available for testing. We’re happy to listen for feedback, if there’s anything we can do to make that more useful to you or if there are other ways we could help.

[pweiskircher]

Considering we don't see a way forward, I will close this PR for now. Please contact us if this is something we want to try to get in again!