feat: add experimental outputFormat: 'tokens' support

.changeset/token-output-format.md

@@ -0,0 +1,7 @@
+---
+"react-shiki": minor
+---
+
+Add experimental `outputFormat: 'tokens'` support to `useShikiHighlighter`, returning Shiki's raw `TokensResult` for custom rendering. The hook's return type narrows based on the literal `outputFormat` passed: `'react'` returns `ReactElement`, `'html'` returns `string`, and `'tokens'` returns `TokensResult`.
+
+Token output is hook-only. `'tokens'` is accepted through the hook's generic signature but excluded from `HighlighterOptions` and the component's props, so existing option objects and wrappers keep their current types. The `ShikiHighlighter` component warns and falls back to `'react'` if `'tokens'` is passed at runtime.

package/README.md

@@ -222,7 +222,7 @@ See [Shiki - RegExp Engines](https://shiki.style/guide/regex-engines) for more i
 | `transformers`      | `array`            | `[]`            | Custom Shiki transformers for modifying the highlighting output               |
 | `cssVariablePrefix` | `string`           | `'--shiki'`     | Prefix for CSS variables storing theme colors                                 |
 | `defaultColor`      | `string \| false`  | `'light'`       | Default theme mode when using multiple themes, can also disable default theme |
-| `outputFormat`      | `string`           | `'react'`       | Output format: 'react' for React nodes, 'html' for HTML string                 |
+| `outputFormat`      | `string`           | `'react'`       | Output format: 'react' for React nodes, 'html' for HTML string, or 'tokens' for Shiki tokens (hook only, experimental) |
 | `tabindex`          | `number`           | `0`             | Tab index for the code block                                                  |
 | `decorations`       | `array`            | `[]`            | Custom decorations to wrap the highlighted tokens with                        |
 | `structure`        | `string`           | `'classic'`  | The structure of the generated HAST and HTML - `classic` or `inline`               |
@@ -630,7 +630,7 @@ const highlightedCode = useShikiHighlighter(code, "tsx", "github-dark", {
 
 ### Output Formats
 
-`react-shiki` can return highlighted code in two formats:
+`react-shiki` can return highlighted code in three formats:
 
 **React Nodes (Default)** - Rendered as React elements, no `dangerouslySetInnerHTML` required
 ```tsx
@@ -656,10 +656,29 @@ const highlightedCode = useShikiHighlighter(code, "tsx", "github-dark", {
 </ShikiHighlighter>
 ```
 
-The two formats spend their rendering work in different phases. HTML output skips per-token React element creation and reconciliation, reducing render-phase overhead for large, one-shot highlights, but each update replaces the code block's entire DOM subtree via `innerHTML`. React output pays for element creation and diffing on every update, but commits only incremental DOM mutations, minimizing DOM churn for frequently re-highlighted code such as streaming LLM output.
+**Shiki Tokens (Experimental)** - Hook-only output for custom renderers
+```tsx
+const highlighted = useShikiHighlighter(code, "tsx", "github-dark", {
+  outputFormat: "tokens",
+});
+
+const rendered = highlighted?.tokens.map((line, i) => (
+  <div key={i}>
+    {line.map((token, j) => (
+      <span key={j} style={{ color: token.color }}>
+        {token.content}
+      </span>
+    ))}
+  </div>
+));
+```
+
+The React and HTML formats spend their rendering work in different phases. HTML output skips per-token React element creation and reconciliation, reducing render-phase overhead for large, one-shot highlights, but each update replaces the code block's entire DOM subtree via `innerHTML`. React output pays for element creation and diffing on every update, but commits only incremental DOM mutations, minimizing DOM churn for frequently re-highlighted code such as streaming LLM output.
 
 HTML output hands the highlighted markup to the DOM via `dangerouslySetInnerHTML`, so only use it with trusted code sources. The default React output is the safe choice for untrusted content.
 
+Token output is for when you need to own rendering yourself; it is intentionally available on the hook, not the `ShikiHighlighter` component, since react-shiki's markup-based features no longer apply once you control the markup. Note that Shiki's `codeToTokens` does not run transformers or decorations, so markup-producing options, including transformers, decorations, line numbers, and `structure`, have no effect on token output.
+
 ---
 
 Made with ❤️ by [Bassim (AVGVSTVS96)](https://github.com/AVGVSTVS96)

package/src/core.ts

@@ -18,6 +18,9 @@ export type {
   Themes,
   Element,
   HighlighterOptions,
+  HighlighterOptionsFor,
+  HighlightResult,
+  OutputFormat,
 } from './lib/types';
 
 export { createHighlighterCore } from 'shiki/core';
@@ -27,7 +30,7 @@ export {
   createJavaScriptRawEngine,
 } from 'shiki/engine/javascript';
 
-export type { LanguageRegistration } from 'shiki/core';
+export type { LanguageRegistration, TokensResult } from 'shiki/core';
 
 /**
  * Highlight code with shiki (core bundle)
@@ -36,7 +39,7 @@ export type { LanguageRegistration } from 'shiki/core';
  * @param lang - Language (bundled or custom)
  * @param theme - Theme (bundled, multi-theme, or custom)
  * @param options - react-shiki options + shiki options
- * @returns Highlighted code as React elements or HTML string
+ * @returns Highlighted code as React elements (default), HTML string, or Shiki tokens, based on `outputFormat`
  *
  * @example
  * ```tsx

package/src/index.ts

@@ -18,14 +18,17 @@ export type {
   Themes,
   Element,
   HighlighterOptions,
+  HighlighterOptionsFor,
+  HighlightResult,
+  OutputFormat,
 } from './lib/types';
 
 export {
   createJavaScriptRegexEngine,
   createJavaScriptRawEngine,
 } from 'shiki/engine/javascript';
 
-export type { LanguageRegistration } from 'shiki';
+export type { LanguageRegistration, TokensResult } from 'shiki';
 
 /**
  * Highlight code with shiki (full bundle)
@@ -34,7 +37,7 @@ export type { LanguageRegistration } from 'shiki';
  * @param lang - Language (bundled or custom)
  * @param theme - Theme (bundled, multi-theme, or custom)
  * @param options - react-shiki options + shiki options
- * @returns Highlighted code as React elements or HTML string
+ * @returns Highlighted code as React elements (default), HTML string, or Shiki tokens, based on `outputFormat`
  *
  * @example
  * ```tsx
@@ -55,15 +58,7 @@ export const useShikiHighlighter: UseShikiHighlighter = (
   lang,
   themeInput,
   options = {}
-) => {
-  return useHighlight(
-    code,
-    lang,
-    themeInput,
-    options,
-    createFullHighlighter
-  );
-};
+) => useHighlight(code, lang, themeInput, options, createFullHighlighter);
 
 /**
  * ShikiHighlighter component using the full bundle.

package/src/lib/component.tsx

@@ -5,12 +5,34 @@ import { clsx } from 'clsx';
 import type {
   HighlighterOptions,
   Language,
+  OutputFormat,
   Theme,
   Themes,
   UseShikiHighlighter,
 } from './types';
 import { forwardRef } from 'react';
 
+let warnedTokensOutputFormat = false;
+
+/**
+ * The component's props exclude `outputFormat: 'tokens'`, but plain-JS
+ * callers can still pass it. Tokens are not renderable React children,
+ * so fall back to 'react' instead of crashing the render.
+ */
+const resolveOutputFormat = (
+  outputFormat: HighlighterOptions['outputFormat']
+): HighlighterOptions['outputFormat'] => {
+  if ((outputFormat as OutputFormat) !== 'tokens') return outputFormat;
+
+  if (!warnedTokensOutputFormat) {
+    warnedTokensOutputFormat = true;
+    console.warn(
+      "[react-shiki] outputFormat 'tokens' is only supported by the useShikiHighlighter hook, falling back to 'react'"
+    );
+  }
+  return 'react';
+};
+
 /**
  * Props for the ShikiHighlighter component
  */
@@ -127,11 +149,11 @@ export const createShikiHighlighterComponent = (
         as: Element = 'div',
         customLanguages,
         preloadLanguages,
+        outputFormat,
         ...shikiOptions
       },
       ref
     ) => {
-      // Destructure some options for use in hook
       const options: HighlighterOptions = {
         delay,
         transformers,
@@ -142,6 +164,7 @@ export const createShikiHighlighterComponent = (
         cssVariablePrefix,
         startingLineNumber,
         highlightLineNumbers,
+        outputFormat: resolveOutputFormat(outputFormat),
         ...shikiOptions,
       };
 

package/src/lib/hook.ts

@@ -4,10 +4,13 @@ import { toJsxRuntime } from 'hast-util-to-jsx-runtime';
 import type { CodeToHastOptions } from 'shiki';
 
 import type {
-  HighlightedCode,
   HighlighterFactory,
   HighlighterOptions,
+  HighlighterOptionsFor,
+  HighlightResult,
+  HighlightResultMap,
   Language,
+  OutputFormat,
   Theme,
   Themes,
   TimeoutState,
@@ -22,22 +25,22 @@ import {
 import { resolveTheme } from './theme';
 import { buildShikiOptions } from './options';
 
-export async function highlight(
+export async function highlight<F extends OutputFormat = 'react'>(
   code: string,
   resolved: {
     languageId: string;
     langsToLoad: Language[];
     themesToLoad: Theme[];
     shikiOptions: CodeToHastOptions;
   },
-  opts: Pick<
-    HighlighterOptions,
-    'highlighter' | 'outputFormat' | 'engine'
-  >,
+  opts: Pick<HighlighterOptions, 'highlighter' | 'engine'> & {
+    outputFormat?: F;
+  },
   factory: HighlighterFactory
-) {
+): Promise<HighlightResultMap[F]> {
   const { languageId, langsToLoad, themesToLoad, shikiOptions } =
     resolved;
+  const format: OutputFormat = opts.outputFormat ?? 'react';
 
   const highlighter =
     opts.highlighter ??
@@ -55,24 +58,31 @@ export async function highlight(
   );
   const options = { ...shikiOptions, lang: language };
 
-  return opts.outputFormat === 'html'
-    ? highlighter.codeToHtml(code, options)
-    : toJsxRuntime(highlighter.codeToHast(code, options), {
+  const outputs: { [K in OutputFormat]: () => HighlightResultMap[K] } = {
+    react: () =>
+      toJsxRuntime(highlighter.codeToHast(code, options), {
         jsx,
         jsxs,
         Fragment,
-      });
+      }),
+    html: () => highlighter.codeToHtml(code, options),
+    tokens: () => highlighter.codeToTokens(code, options),
+  };
+
+  // Untyped call sites can pass formats outside OutputFormat; degrade to
+  // react rendering like the pre-tokens ternary did instead of throwing.
+  return (outputs[format] ?? outputs.react)() as HighlightResultMap[F];
 }
 
-export const useHighlight = (
+export const useHighlight = <F extends OutputFormat = 'react'>(
   code: string,
   lang: Language,
   themeInput: Theme | Themes,
-  options: HighlighterOptions = {},
+  options: HighlighterOptionsFor<F> = {},
   highlighterFactory: HighlighterFactory
-) => {
+): HighlightResult<F> => {
   const [highlightedCode, setHighlightedCode] =
-    useState<HighlightedCode>(null);
+    useState<HighlightResult<F>>(null);
 
   const stableLang = useStableValue(lang);
   const stableTheme = useStableValue(themeInput);
@@ -116,7 +126,7 @@ export const useHighlight = (
 
     const run = async () => {
       try {
-        const result = await highlight(
+        const result = await highlight<F>(
           code,
           resolved,
           stableOpts,

package/src/lib/options.ts

@@ -5,7 +5,11 @@ import type {
   CodeToHastOptions,
 } from 'shiki';
 
-import type { HighlighterOptions } from './types';
+import type {
+  HighlighterOptions,
+  HighlighterOptionsFor,
+  OutputFormat,
+} from './types';
 import type { ResolvedTheme } from './theme';
 import {
   highlightedLinesTransformer,
@@ -35,7 +39,7 @@ const buildThemeOptions = (
 export const buildShikiOptions = (
   languageId: string,
   resolvedTheme: ResolvedTheme,
-  options: HighlighterOptions
+  options: HighlighterOptionsFor<OutputFormat>
 ): CodeToHastOptions => {
   const {
     delay,

package/src/lib/types.ts

@@ -12,6 +12,7 @@ import type {
   SpecialLanguage,
   StringLiteralUnion,
   ThemeRegistrationAny,
+  TokensResult,
 } from 'shiki';
 
 import type { ReactElement } from 'react';
@@ -87,6 +88,11 @@ interface ReactShikiOptions {
    * Output format for the highlighted code.
    * - 'react': Returns React nodes (default, safer)
    * - 'html': Returns HTML string (rendered via dangerouslySetInnerHTML)
+   *
+   * The hook additionally accepts 'tokens' (experimental) to return raw
+   * Shiki tokens for custom rendering. It is deliberately excluded here
+   * and only enters through the hook's generic signature, so option
+   * objects typed with this interface keep the classic return type.
    * @default 'react'
    */
   outputFormat?: 'react' | 'html';
@@ -169,22 +175,46 @@ interface TimeoutState {
 }
 
 /**
- * Public API signature for the useShikiHighlighter hook.
+ * Supported output formats and their corresponding result shapes.
  */
-type HighlightedCode = ReactElement | string | null;
+type OutputFormat = 'react' | 'html' | 'tokens';
+
+interface HighlightResultMap {
+  react: ReactElement;
+  html: string;
+  tokens: TokensResult;
+}
+
+type HighlightResult<F extends OutputFormat = 'react'> =
+  | HighlightResultMap[F]
+  | null;
+
+/**
+ * Per-call options shape: every highlighter option except `outputFormat`,
+ * plus a narrowed `outputFormat?: F` so the return type can be inferred
+ * from the format the caller actually passes.
+ *
+ * This is the only place 'tokens' is accepted. Keeping it out of
+ * `HighlighterOptions` means values typed with that interface infer
+ * `F = 'react' | 'html'` and keep their pre-tokens return type.
+ */
+type HighlighterOptionsFor<F extends OutputFormat> = Omit<
+  HighlighterOptions,
+  'outputFormat'
+> & { outputFormat?: F };
 
 type HighlighterFactory = (
   langsToLoad: Language[],
   themesToLoad: Theme[],
   engine?: Awaitable<RegexEngine>
 ) => Promise<Highlighter | HighlighterCore>;
 
-export type UseShikiHighlighter = (
+export type UseShikiHighlighter = <F extends OutputFormat = 'react'>(
   code: string,
   lang: Language,
   themeInput: Theme | Themes,
-  options?: HighlighterOptions
-) => HighlightedCode;
+  options?: HighlighterOptionsFor<F>
+) => HighlightResult<F>;
 
 export type {
   Language,
@@ -193,6 +223,9 @@ export type {
   Element,
   TimeoutState,
   HighlighterOptions,
-  HighlightedCode,
+  HighlighterOptionsFor,
+  HighlightResult,
+  HighlightResultMap,
+  OutputFormat,
   HighlighterFactory,
 };