Add types/tokenizers.d.ts for Angular/Vite/Rollup compatibility

[dpfens]

Closes #31

TypeScript doesn't rewrite path aliases in .d.ts output. tsc-alias handles this as a post-processing step, but it doesn't work under "moduleResolution": "bundler" (Angular 17+). Also, /types/static/tokenizers.d.ts is not being generated.

* Replaces internal path aliases (@utils, @core/*, @static/*) with relative imports so the emitted declarations are correct without any post-processing. This makes tsc-alias longer needed and is removed.

• Renames src/static/tokenizer.d.ts to src/static/tokenizer.ts to ensure tsc emits types/static/tokenizer.d.ts
• Updates tsconfig.json from src/utils/index.ts to src/utils/index to ensure tsc-alias rewrites @utils correctly.
• Adds a "bundler" exports condition to package.json

Without these changes, Angular 17+ (and other bundlers that resolve the "bundler" exports condition) fail to build with errors like:

TS2305: Module '"@static/tokenizer"' has no exported member 'TokenizerConfigDecoder'
TS2693: 'Callable' only refers to a type, but is being used as a value here
TS2305: Module '"@static/tokenizer"' has no exported member 'TokenizerConfigNormalizerBert'.
TS2305: Module '"@static/tokenizer"' has no exported member 'TokenizerConfigNormalizerLowercase'.
TS2305: Module '"@static/tokenizer"' has no exported member 'TokenizerConfigNormalizerPrecompiled'.
etc.

[nico-martin]

Hi @dpfens, thank you so much for looking into this!
My main concern is that you replace all alias imports with relative imports. That not necessarily bad, but I'd rather have the aliases in the source code. Also I don't think it solves the underlying problem.

I've looked into the problem and it seems like in some places types/static/tokenizer.d.ts is referenced, but this file does not exist. Whether imported via an alias or as a relative path (tsc-alias has also worked correctly for me with bundler).

The reason is that tsc does not copy src/static/tokenizer.d.ts to the types folder. Therefore other code refrencing types/static/tokenizer.d.ts. I think the solution would be to rename src/static/tokenizer.d.ts to src/static/tokenizer.ts so its treated as a ts file in tsc.

Could you try that and see if it solves your problem?

• Keep the source aliases
• Rename src/static/tokenizer.d.ts to src/static/tokenizer.ts so tsc emits types/static/tokenizer.d.ts.
• Export the first SpecialToken interface in that file so the duplicate SpecialToken declarations can merge cleanly in a .ts file.
• Keep tsc-alias, but change the @utils path from src/utils/index.ts to src/utils/index. With the .ts extension removed, tsc-alias rewrites @utils correctly.

[nico-martin]

looks good to me now 🤗

[xenova]

Thanks for the PR! It's in a very clean state, and I am happy to merge the implementation.

One question though: I haven't seen other libraries use "bundler" exports before

      "bundler": {
        "import": "./dist/tokenizers.mjs"
      },

@dpfens Could you point me to other libraries which do this and/or the docs/part of angular which accepts this?

Thanks!

[dpfens]

I haven't seen other libraries use "bundler" exports before
...
Could you point me to other libraries which do this and/or the docs/part of angular which accepts this?

@xenova I mixed this up with a module condition I use in another project, my mistake. For the record, bundler isn't a real/recognized condition in the exports spec or in Angular's package format (or anywhere else I can find), so it wouldn't actually resolve anywhere.

That change has been dropped.

[xenova]

That change has been dropped.

Thanks for updating 👍 and can you confirm that with these current changes, it fixes the issues you had for angular? If so, I'm happy to merge.

[dpfens]

can you confirm that with these current changes, it fixes the issues you had for angular?

@xenova Yes, I just confirmed this fixes the tokenizers.js compilation issues in Angular. Tested by doing the following:

# in this tokenizers.js branch
npm pack 

# in my project
npm install ~/path/to//bundler-support-branch/huggingface-tokenizers-0.1.3.tgz
npm run build

and the compilation/missing type issues were fixed.

Thank you again for your help with this change.

[xenova]

Thanks for iterating!