Skip to content

feat: add outputFormat: 'tokens' support#146

Draft
AVGVSTVS96 wants to merge 3 commits into
mainfrom
tokens
Draft

feat: add outputFormat: 'tokens' support#146
AVGVSTVS96 wants to merge 3 commits into
mainfrom
tokens

Conversation

@AVGVSTVS96

Copy link
Copy Markdown
Owner

hook only and experimental - no renderer yet

hook only and experimental - no renderer yet
@vercel

vercel Bot commented May 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
react-shiki Ready Ready Preview, Comment May 19, 2026 10:59am
react-shiki-docs Error Error May 19, 2026 10:59am

@changeset-bot

changeset-bot Bot commented May 3, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 7681b5b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
react-shiki Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@claude

claude Bot commented May 3, 2026

Copy link
Copy Markdown

Claude finished @AVGVSTVS96's task —— View job


PR Review: outputFormat: 'tokens' Support

Reviewing the implementation of the new outputFormat: 'tokens' feature...

Review Checklist:

  • Read repository guidelines (CLAUDE.md)
  • Review core implementation changes
  • Check type definitions and API design
  • Verify changeset inclusion
  • Verify documentation updates
  • Review test coverage
  • Assess code quality and architecture
  • Check for potential bugs/performance/security issues
  • Provide final feedback

Overall Assessment: Excellent Implementation

This PR successfully adds outputFormat: 'tokens' support with a clean, well-architected approach. The implementation is thoughtful, well-tested, and properly documented.

🌟 Strengths

1. Clean Type System Design

  • package/src/lib/types.ts:171-193: Excellent type safety with OutputFormat, HighlightResultMap, and conditional return types
  • Smart restriction of 'tokens' format to hook-only usage via ComponentOutputFormat = Exclude<OutputFormat, 'tokens'>
  • Proper generic inference allowing HighlightResult<F> to be inferred from the outputFormat parameter

2. Robust Implementation

  • package/src/lib/hook.ts:61-72: Clean output mapping with proper type safety
  • All three bundles (core, web, full) support the new format consistently
  • Backwards compatible with existing API

3. Comprehensive Testing

  • package/tests/hook.test.tsx:774-804: Well-structured test verifying tokens structure and content
  • Tests cover the complete token object shape including content, offset, and arrays
  • Good integration with existing test suite

4. Documentation & DX

  • README properly documents the new feature with clear examples
  • Changeset included with appropriate semver level (patch)
  • JSDoc comments clearly indicate hook-only limitation

🔍 Technical Review

Component Safety
package/src/lib/component.tsx:151: Component correctly excludes 'tokens' format via ComponentOutputFormat, preventing runtime errors.

Error Handling
Inherits robust error handling from existing infrastructure with proper request deduplication.

Performance
tokens output uses Shiki's optimized codeToTokens() method directly - no additional overhead.

Security
No security concerns - tokens format doesn't involve HTML generation or DOM manipulation.

🚀 Code Quality

  • Architecture: Follows existing patterns perfectly, maintains consistency across all bundles
  • Maintainability: Clear separation of concerns, easy to understand and extend
  • Readability: Well-structured code with appropriate comments and type annotations

Minor Suggestions (Non-blocking)

  1. Consider adding a small example in the README showing how to render tokens for custom syntax highlighting
  2. The JSDoc for outputFormat could mention performance characteristics of each format option

📋 Verification Summary

  • Changeset: Properly included with patch-level change
  • Documentation: README updated with clear examples and table entry
  • Tests: Comprehensive coverage of new functionality
  • Type Safety: Excellent TypeScript implementation
  • API Design: Clean, consistent, and backwards compatible

🎯 Recommendation: APPROVE

This is a high-quality implementation that adds valuable functionality while maintaining the library's excellent standards. The hook-only restriction is appropriate for this experimental feature, and the implementation provides a solid foundation for future enhancements.


Copy link
Copy Markdown
Owner Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

minor when renderer is added and supported in component
@vercel

vercel Bot commented May 7, 2026

Copy link
Copy Markdown

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/bassims-projects-2e899514?upgradeToPro=build-rate-limit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant