Commit 268d046
fix: accept output_text, reasoning, and input_image in /v1/responses mapper (mlx-node#50)
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> **Medium Risk**
> Medium risk because it changes request/history mapping and VLM
session/cache handling (image placeholders, M-RoPE offsets, and delta
continuations), which can affect multi-turn correctness and replayed
history integrity.
>
> **Overview**
> **Expands the `/v1/responses` request mapper and stored-history codec
to properly handle client replay and images.** `mapRequest` now accepts
replayed assistant `output_text`, top-level `reasoning` items, and
`refusal/summary_text` parts; it also decodes `input_image` base64 data
URLs into `ChatMessage.images` and coalesces interleaved assistant
`reasoning`/`message`/`function_call` items into a single assistant
turn.
>
> **Adds strict text/image ordering guards** in both the OpenAI and
Anthropic mappers to reject any text-after-image interleaving within a
single user turn (previously the pipeline could silently reorder
content).
>
> **Makes images survive persistence and VLM templating, and fixes
Qwen3.5 VLM session correctness.** Stored `inputJson` is now written via
`stringifyStoredInputMessages` (base64 sentinel) and revived during
`reconstructMessagesFromChain` so `Uint8Array` images round-trip;
tokenizer sanitization/serialization now preserves user images so Jinja
templates can emit inline vision markers; Qwen3.5/Qwen3.5-MoE update
image placeholder expansion (per-image counts), robust M-RoPE position
indexing for multiple image runs, and session-delta handling so
text-only deltas can continue on image-bearing sessions without clearing
`cached_image_key`/rope deltas. Extensive new unit tests cover these
behaviors.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
9c33817. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent bfdeef5 commit 268d046
10 files changed
Lines changed: 2160 additions & 230 deletions
File tree
- __test__/server
- crates/mlx-core/src
- models
- qwen3_5_moe
- qwen3_5
- packages/server/src
- endpoints
- mappers
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
263 | 263 | | |
264 | 264 | | |
265 | 265 | | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
266 | 270 | | |
267 | 271 | | |
268 | 272 | | |
| |||
278 | 282 | | |
279 | 283 | | |
280 | 284 | | |
281 | | - | |
| 285 | + | |
282 | 286 | | |
283 | 287 | | |
284 | 288 | | |
| |||
857 | 861 | | |
858 | 862 | | |
859 | 863 | | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
860 | 972 | | |
0 commit comments