LCORE-2493: benchmarks for tokenizing 1000 lines sources

[tisnik]

Description

LCORE-2493: benchmarks for tokenizing 1000 lines sources

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up service version
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Konflux configuration change
• Unit tests improvement
• Integration tests improvement
• End to end tests improvement
• Benchmarks improvement

Tools used to create PR

• Assisted-by: N/A
• Generated by: N/A

Related Tickets & Documents

• Related Issue #LCORE-2493

[coderabbitai]

Warning

Review limit reached

@tisnik, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 52 minutes and 8 seconds. Learn how PR review limits work.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: b7e0668d-f26e-4219-b1a1-e60bf0b95e45

📥 Commits

Reviewing files that changed from the base of the PR and between ad85ff7 and 9d44495.

📒 Files selected for processing (6)

• tests/benchmarks/data/js_1000_lines.js
• tests/benchmarks/data/json_1000_lines.json
• tests/benchmarks/data/python_1000_lines.py
• tests/benchmarks/data/xml_1000_lines.xml
• tests/benchmarks/data/yaml_1000_lines.yml
• tests/benchmarks/test_token_estimator.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

✨ Simplify code

• Create PR with simplified code

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}