Migrate to TypeScript 6.0, set TypeScript target to ES6 (ES2015 / ECMAScript 2015)

[generalmimon]

Related issue: kaitai-io/kaitai_struct#539

See also kaitai-io/kaitai_struct#542 (comment)

---

The ECMAScript 5 (ES5) target was deprecated in TypeScript 6.0, see https://www.typescriptlang.org/docs/handbook/release-notes/typescript-6-0.html#deprecated-target-es5. ES6 (ES2015) is now the TypeScript's lowest target.

ES6 is widely supported across JavaScript engines, see https://compat-table.github.io/compat-table/es6/. As can be seen from this table and also from https://zoo.js.org/, there are several non-mainstream JS engines that don't support ES6 classes (which is the main feature we need), namely:

• Rhino 1.9.0 - see ES2015 Classes and inheritance mozilla/rhino#835 and https://github.com/ivankra/javascript-zoo-data/blob/8d76f329c686f006081842b5c5091880e49efd03/compat-table/rhino.json#L400
• (Hermes 0.12.0) - see ES Classes support facebook/hermes#685; Hermes 0.13.0 likely already supports them, Hermes V1 (the default static_h branch of the https://github.com/facebook/hermes repo; see https://zoo.js.org/#hermes-v1) definitely does, see https://github.com/ivankra/javascript-zoo-data/blob/8d76f329c686f006081842b5c5091880e49efd03/compat-table/hermes_full.json#L381-L404
• Nashorn 15.7 (labeled "JJS 1.8" at https://compat-table.github.io/compat-table/es6/) - see https://github.com/ivankra/javascript-zoo-data/blob/8d76f329c686f006081842b5c5091880e49efd03/compat-table/nashorn.json#L401 and https://bugs.openjdk.org/browse/JDK-8066046
• Duktape 2.7 - apparently abandoned, see https://github.com/svaarala/duktape (last release in February 2022 and no commits since January 2024)

I believe it makes sense to leave these rather outdated JavaScript runtimes behind and no longer support them out of the box in future versions of Kaitai Struct. If any of our users use one of the runtimes listed above or another ES5-only runtime, they can continue using existing versions of Kaitai Struct, or use Babel to transpile ES6 code to ES5.

[generalmimon]

Cc @Azq2, @cherue, @EdJoPaTo, @mfalkvidd

[mfalkvidd]

Thanks for including me @generalmimon . I am not 100% sure why I was tagged, but I have probably asked for / discussed es5 support since I use Thingsboard, which is using Nashorn.

Nowadays, Thingsboard recommends remote js executor (which is based on nodejs and therefore supports es6) or TBEL which is a fork of MVEL.

It would be nice if Kaitai could support generating MVEL code, but since we can use the remote js executor in Thingsboard it should still be possible to use es6 Kaitai anyway.

And as you mentioned, we could always stay with an older version of Kaitai or use Babel.

The only thing I would add is that it would be nice if this was considered a breaking change and trigger a new version, especially since switching to semver is already being discussed. That way it would be clear which version to stay on.

[generalmimon]

@mfalkvidd Thank you for your feedback!

I am not 100% sure why I was tagged, but I have probably asked for / discussed es5 support since I use Thingsboard, which is using Nashorn.

Yes, your past pull requests were the reason, because they showed that you care(d) about ES5 compatibility :)

The only thing I would add is that it would be nice if this was considered a breaking change and trigger a new version, especially since switching to semver is already being discussed. That way it would be clear which version to stay on.

I guess that makes sense. The transition to SemVer is discussed in kaitai-io/kaitai_struct#1293. For now, we plan for 1.x versions to be compatible with 0.x, while 2.x versions will have breaking changes.

If we consider this a breaking change, then this PR should go into 2.x, so let me mark this PR as a draft so that we don't accidentally merge it prematurely (as I've done with other PRs containing breaking changes: kaitai-io/kaitai_struct#1288 (comment)).

[Azq2]

I'm still working on the ES6 & TypeScript. During some periods, I just don't have enough time to work on it. But yesterday I got back to it again.

I can send all my work as a draft PR's.

Current progress (very short description):

Runtime

https://github.com/yet-another-org-with-forks/kaitai_struct_javascript_runtime/tree/esm-and-write-support

Done

Package & Build System

• Migrated to TS6
• Migrated to ES6
• Switched from Rollup to tsc

KaitaiStruct

• Added KaitaiStruct as a base class for all KaitaiStructs (synchronized with the Java implementation)
• Implemented ES6 Proxy-based setters (synchronized with the Python implementation)
• Disabled enumeration for "protected" fields

KaitaiStream

• Correct handling of UInt8Array (it's actually a view and is not compatible with ArrayBuffer)
• Fixed miscellaneous issues caused by newer TypeScript versions
• All exceptions synchronized with the Java implementation
• New way to configure zlibHandler (KaitaiStream.zlibHandler = ...)
• Native zlibHandler for NodeJS (implemented via different entry points in package.json)
• Removed IconvLite because TextDecoder is now standard
• Zero-copy substreams

Planned

1. Pass all tests
2. Add write-mode support

Compiler

https://github.com/yet-another-org-with-forks/kaitai_struct_compiler_typescript/tree/typescript-support

Done

• Implemented ECMAScriptCompiler / ECMAScriptTranslator, used by both TypeScript and JavaScript
• Implemented a fully featured TypeScript TS6 compiler with write-mode support (synchronized with Java/Python)
• Implemented a fully featured JavaScript ES6 compiler with write-mode support (synchronized with Java/Python)
• Added opaque paths for processors imports

Planned

1. Pass all tests

CI System

https://github.com/yet-another-org-with-forks/kaitai_struct_tests/tree/typescript_and_esm

Done

1. Implemented JavaScript ES6 test generation
2. Implemented JavaScript ES6 test runtime
3. All tests are passing (all kst + 21 manual):

• 59 tests added (all passed)
• 2 tests fixed
• 14 tests changed (all failed, but only backtrace changed, no regression)

SUMMARY: {"kst":0,"passed":312,"failed":14}

LOG: https://gist.github.com/Azq2/9203b1b0718693e790fc456f5f874a76

Planned

1. Pass all tests
2. Add write-mode tests
3. Add tests for TypeScript (or reuse the same tests for both JavaScript & TypeScript)

[Azq2]

And about targets... my opinion:

• Any target that is out of scope for CI is unsupported by definition.
• In the JS world, it is safe to use bleeding-edge features:
  • I'd be surprised if anyone used a broken and outdated NodeJS from Ubuntu repo in real projects. The NodeJS (LTS version) from nvm or https://nodesource.com/ is the standard.
  • The IE/Android WebKit era is over. Every browser is self-updating, and we can now use modern features without waiting 10+ years.
  • Any browser app is built using bundlers - using Babel or other tools is not a problem, if truly needed.

I think it would be good to align the minimum NodeJS version with TypeScript. For example, the minimum version for TS6 is Node 14:
https://github.com/microsoft/TypeScript/blob/v6.0.3/package.json#L28
They have already given thought to reasonable backward compatibility.

For the ES target, I think ES2022 is a good choice: https://caniuse.com/?search=es2022

Example Gif.js: https://gist.github.com/Azq2/fb3aebd0c0b96c3f73136171c14f048f

The current version of the JS compiler uses these features:

• https://caniuse.com/?search=Public+class+fields
  Usage: explicitly declaring struct members + docs
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Classes/Private_elements
  Usage: preventing KaitaiStruct object pollution with many private data members
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Classes/static
  Usage: nesting KaitaiStruct as a namespaces alternative

All of these features are supported even in legacy NodeJS 12.

However, if ES2015 is mandatory, the following can be polyfilled:

• Public class fields - just remove or comment out
• Private class fields - can be replaced with a WeakMap polyfill (this.#XXX → _privateMembers.get(this).XXX)
• Static class fields - can be replaced with a variable assignment (static BBB = class { → AAA.BBB = class {)

The current JavaScriptCompiler can support this without big changes.

// Not applicable to the TypeScriptCompiler, since TSC will gracefully downgrade all of these features.

[EdJoPaTo]

• Every browser is self-updating, and we can now use modern features without waiting 10+ years.

Keep in mind that a bunch of people are using old tablets in a bunch of places (like wall displays or signs) which don't receive updates anymore. (I have an iOS 12 tablet in use and I know more hobbyists using similar older tech for stuff like that.)

Projects that should run there could not use Kaitai Struct output when they require newer JS features. Not sure how many people would need that. Supporting these 10+ years old mobile browsers would be nice.

[Azq2]

Supporting these 10+ years old mobile browsers would be nice.

This case can be handled with babel & polyfills

Fancy fact - 10+ years ago it was 2016, not 2000's.... Most 2016 browsers ES6 capable: https://caniuse.com/?search=ES6
Also, this means there's no particular problem with ES2022 → ES2015 with just only transplitter.

[Azq2]

Another argument for a higher ES version - BigInt was added in ES2020: https://caniuse.com/?search=es2020
This is important for correct 64-bit number handling (kaitai-io/kaitai_struct#183). Possibly more important than legacy support, which is theoretical anyway (this not tested in CI).

// Also, BigInt is one feature that cannot be transpiled to lower targets. It seems sooner or later a choice will have to be made: the future or legacy.
// Or another way - adding an ES level target to the compiler.