feat: add outputFormat: 'tokens' support

[AVGVSTVS96]

hook only and experimental - no renderer yet

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
react-shiki	Ready	Preview, Comment	May 19, 2026 10:59am
react-shiki-docs	Error		May 19, 2026 10:59am

[changeset-bot]

🦋 Changeset detected

Latest commit: 7681b5b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
react-shiki	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[claude]

Claude finished @AVGVSTVS96's task —— View job

---

PR Review: outputFormat: 'tokens' Support

Reviewing the implementation of the new outputFormat: 'tokens' feature...

Review Checklist:

• Read repository guidelines (CLAUDE.md)
• Review core implementation changes
• Check type definitions and API design
• Verify changeset inclusion
• Verify documentation updates
• Review test coverage
• Assess code quality and architecture
• Check for potential bugs/performance/security issues
• Provide final feedback

---

✅ Overall Assessment: Excellent Implementation

This PR successfully adds outputFormat: 'tokens' support with a clean, well-architected approach. The implementation is thoughtful, well-tested, and properly documented.

🌟 Strengths

1. Clean Type System Design

• package/src/lib/types.ts:171-193: Excellent type safety with OutputFormat, HighlightResultMap, and conditional return types
• Smart restriction of 'tokens' format to hook-only usage via ComponentOutputFormat = Exclude<OutputFormat, 'tokens'>
• Proper generic inference allowing HighlightResult<F> to be inferred from the outputFormat parameter

2. Robust Implementation

• package/src/lib/hook.ts:61-72: Clean output mapping with proper type safety
• All three bundles (core, web, full) support the new format consistently
• Backwards compatible with existing API

3. Comprehensive Testing

• package/tests/hook.test.tsx:774-804: Well-structured test verifying tokens structure and content
• Tests cover the complete token object shape including content, offset, and arrays
• Good integration with existing test suite

4. Documentation & DX

• README properly documents the new feature with clear examples
• Changeset included with appropriate semver level (patch)
• JSDoc comments clearly indicate hook-only limitation

🔍 Technical Review

Component Safety ✅
package/src/lib/component.tsx:151: Component correctly excludes 'tokens' format via ComponentOutputFormat, preventing runtime errors.

Error Handling ✅
Inherits robust error handling from existing infrastructure with proper request deduplication.

Performance ✅
tokens output uses Shiki's optimized codeToTokens() method directly - no additional overhead.

Security ✅
No security concerns - tokens format doesn't involve HTML generation or DOM manipulation.

🚀 Code Quality

• Architecture: Follows existing patterns perfectly, maintains consistency across all bundles
• Maintainability: Clear separation of concerns, easy to understand and extend
• Readability: Well-structured code with appropriate comments and type annotations

✨ Minor Suggestions (Non-blocking)

1. Consider adding a small example in the README showing how to render tokens for custom syntax highlighting
2. The JSDoc for outputFormat could mention performance characteristics of each format option

📋 Verification Summary

• ✅ Changeset: Properly included with patch-level change
• ✅ Documentation: README updated with clear examples and table entry
• ✅ Tests: Comprehensive coverage of new functionality
• ✅ Type Safety: Excellent TypeScript implementation
• ✅ API Design: Clean, consistent, and backwards compatible

🎯 Recommendation: APPROVE

This is a high-quality implementation that adds valuable functionality while maintaining the library's excellent standards. The hook-only restriction is appropriate for this experimental feature, and the implementation provides a solid foundation for future enhancements.

---

[AVGVSTVS96]

• feat: add outputFormat: 'tokens' support #146 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/bassims-projects-2e899514?upgradeToPro=build-rate-limit