Retrieve thinking msg from the metadata reasoning_content field#532
Retrieve thinking msg from the metadata reasoning_content field#532Chibukach wants to merge 3 commits into
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughThe preprocessing pipeline is extended to propagate metadata through conversation normalization. The ChangesMetadata reasoning_content propagation
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Suggested labels
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/speculators/data_generation/preprocessing.py`:
- Around line 101-103: The code currently copies metadata["reasoning_content"]
into normalized_turn for all roles; change this so reasoning_content is only
attached when the turn is an assistant turn by checking
normalized_turn.get("role") == "assistant" (or equivalent role field) before
assigning normalized_turn["reasoning_content"] = metadata["reasoning_content"];
keep the existing truthy check on metadata["reasoning_content"] and update the
block around normalized_turn and metadata to guard the assignment with that role
check.
- Around line 274-287: The batch drops rows because metadatas =
examples.get("metadata", []) produces an empty list and zip(conversations,
metadatas) truncates pairs and because the code rejects empty dicts with if not
metadata or not isinstance(metadata, dict): continue; fix by: when building
metadatas, if examples lacks "metadata" or its list is shorter than
conversations, create/pad a metadatas list of the same length filled with {} (so
missing/empty metadata is treated as {}), remove the truthiness check and only
validate metadata with isinstance(metadata, dict) before calling
_normalize_conversation, and add a warning log when metadata length mismatches
conversations to surface potential data issues (keep using
_normalize_conversation for normalization).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: f3c22b20-c51b-4804-9119-ef0061526fb4
📒 Files selected for processing (1)
src/speculators/data_generation/preprocessing.py
| if "reasoning_content" in metadata and metadata["reasoning_content"]: | ||
| normalized_turn["reasoning_content"] = metadata["reasoning_content"] | ||
|
|
There was a problem hiding this comment.
Attach reasoning_content only to assistant turns.
This currently copies the same reasoning_content into user/system turns too, which can pollute role-specific templating and semantics.
💡 Suggested fix
- if "reasoning_content" in metadata and metadata["reasoning_content"]:
+ if (
+ role == "assistant"
+ and "reasoning_content" in metadata
+ and metadata["reasoning_content"]
+ ):
normalized_turn["reasoning_content"] = metadata["reasoning_content"]🧰 Tools
🪛 Ruff (0.15.13)
[warning] 101-101: Unnecessary key check before dictionary access
Replace with dict.get
(RUF019)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/speculators/data_generation/preprocessing.py` around lines 101 - 103, The
code currently copies metadata["reasoning_content"] into normalized_turn for all
roles; change this so reasoning_content is only attached when the turn is an
assistant turn by checking normalized_turn.get("role") == "assistant" (or
equivalent role field) before assigning normalized_turn["reasoning_content"] =
metadata["reasoning_content"]; keep the existing truthy check on
metadata["reasoning_content"] and update the block around normalized_turn and
metadata to guard the assignment with that role check.
| metadatas= examples.get("metadata", []) | ||
|
|
||
| if not conversations: | ||
| log.warning(f"No conversations key found. Keys: {list(examples.keys())}") | ||
| return results | ||
|
|
||
| for idx, conv in enumerate(conversations): | ||
| for idx, (conv,metadata) in enumerate(zip(conversations,metadatas)): | ||
| if not conv or not isinstance(conv, list): | ||
| continue | ||
| if not metadata or not isinstance(metadata, dict): | ||
| continue | ||
|
|
||
| # Normalize to standard format with optional turn dropout | ||
| normalized_conv = _normalize_conversation(conv, turn_dropout) | ||
| normalized_conv = _normalize_conversation(conv, metadata, turn_dropout) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# First, let's see the file structure
find . -name "preprocessing.py" | head -20Repository: vllm-project/speculators
Length of output: 119
🏁 Script executed:
#!/bin/bash
# Read the preprocessing.py file around the target lines (274-287)
cat -n src/speculators/data_generation/preprocessing.py | sed -n '260,300p'Repository: vllm-project/speculators
Length of output: 1821
🏁 Script executed:
#!/bin/bash
# Get more context on the _normalize_conversation function signature and usage
cat -n src/speculators/data_generation/preprocessing.py | sed -n '1,150p' | tail -100Repository: vllm-project/speculators
Length of output: 4078
🏁 Script executed:
#!/bin/bash
# Look at the _normalize_conversation function definition to understand metadata usage
grep -n "def _normalize_conversation" src/speculators/data_generation/preprocessing.pyRepository: vllm-project/speculators
Length of output: 100
🏁 Script executed:
#!/bin/bash
# Read the _normalize_conversation function implementation
cat -n src/speculators/data_generation/preprocessing.py | sed -n '55,120p'Repository: vllm-project/speculators
Length of output: 2706
🏁 Script executed:
#!/bin/bash
# Check for tests or examples showing how this preprocessing is used
find . -name "*.py" -type f | xargs grep -l "_preprocess_batch\|preprocessing" | head -10Repository: vllm-project/speculators
Length of output: 330
🏁 Script executed:
#!/bin/bash
# Check if metadata is always present in data samples
grep -r "metadata" --include="*.py" src/speculators/data_generation/ | grep -E "(\.json|from|import|examples\[|get\()" | head -20Repository: vllm-project/speculators
Length of output: 161
🏁 Script executed:
#!/bin/bash
# Look at how this is called - check main processing flow
grep -B 5 -A 5 "_preprocess_batch" src/speculators/data_generation/preprocessing.py | tail -50Repository: vllm-project/speculators
Length of output: 1131
🏁 Script executed:
#!/bin/bash
# Check if metadata is required or optional in dataset loading
grep -n "metadata" src/speculators/data_generation/preprocessing.py | head -30Repository: vllm-project/speculators
Length of output: 530
🏁 Script executed:
#!/bin/bash
# Check the test file to see how preprocessing is tested
cat -n tests/integration/datagen/test_preprocessing.py | head -100Repository: vllm-project/speculators
Length of output: 3861
🏁 Script executed:
#!/bin/bash
# Look at how metadata is provided in tests
grep -A 10 -B 5 "metadata" tests/integration/datagen/test_preprocessing.py | head -60Repository: vllm-project/speculators
Length of output: 50
🏁 Script executed:
#!/antml:bash
# Check how the preprocessing is called in the data loading pipeline
cat -n src/speculators/train/data.py | head -100Repository: vllm-project/speculators
Length of output: 3552
🏁 Script executed:
#!/bin/bash
# Look for how examples dict is constructed when calling _preprocess_batch
grep -B 20 "_preprocess_batch" src/speculators/train/data.py | head -50Repository: vllm-project/speculators
Length of output: 50
🏁 Script executed:
#!/bin/bash
# Check if metadata is always provided or if it's optional
grep -r "conversations" --include="*.py" src/speculators/ | grep -v "normalized_conv\|assistant_pattern" | head -20Repository: vllm-project/speculators
Length of output: 1024
🏁 Script executed:
#!/bin/bash
# Check if the test file passes metadata parameter to _normalize_conversation
grep -A 15 "test_normalize_conversation" tests/integration/datagen/test_preprocessing.py | head -40Repository: vllm-project/speculators
Length of output: 1445
🏁 Script executed:
#!/bin/bash
# Look for how _normalize_conversation is called in tests - check the actual call signatures
grep "_normalize_conversation" tests/integration/datagen/test_preprocessing.pyRepository: vllm-project/speculators
Length of output: 408
🏁 Script executed:
#!/bin/bash
# Check the actual test calls to see if metadata is provided
cat -n tests/integration/datagen/test_preprocessing.py | sed -n '27,80p'Repository: vllm-project/speculators
Length of output: 2136
🏁 Script executed:
#!/bin/bash
# Look at the actual Python to understand the behavior of zip with mismatched lengths
python3 << 'EOF'
# Test what happens with zip when lists have different lengths
conversations = ["conv1", "conv2", "conv3"]
metadatas = []
result = list(zip(conversations, metadatas))
print(f"Result of zip with empty metadata list: {result}")
print(f"Number of pairs: {len(result)}")
# Test with empty dict
metadatas_with_empty = [{}]
result2 = list(zip(conversations, metadatas_with_empty))
print(f"\nWith one empty dict metadata: {result2}")
print(f"Number of pairs: {len(result2)}")
# Test 'not' operator on empty dict
empty_dict = {}
print(f"\nTruth value of empty dict {{}}: {bool(empty_dict)}")
print(f"'not {{}}' evaluates to: {not empty_dict}")
EOFRepository: vllm-project/speculators
Length of output: 261
🏁 Script executed:
#!/bin/bash
# Check if the tests actually pass by running them
cd /tmp && python3 -m pytest /dev/stdin 2>&1 << 'EOF'
# Quick test to see if _normalize_conversation can be called without metadata
import sys
sys.path.insert(0, '/root/vllm-project/speculators/src')
from speculators.data_generation.preprocessing import _normalize_conversation
conv = [
{"from": "human", "value": "Hello"},
{"from": "gpt", "value": "Hi!"},
]
# Try calling without metadata parameter
try:
result = _normalize_conversation(conv)
print("SUCCESS: Can call _normalize_conversation with just conv")
except TypeError as e:
print(f"ERROR: {e}")
EOFRepository: vllm-project/speculators
Length of output: 109
🏁 Script executed:
#!/bin/bash
# Check if there's a recent change that broke signature compatibility
# Look at the function definition more carefully
cat -n src/speculators/data_generation/preprocessing.py | sed -n '61,75p'Repository: vllm-project/speculators
Length of output: 662
🏁 Script executed:
#!/bin/bash
# Check if metadata has a default value that I missed
grep -A 10 "def _normalize_conversation" src/speculators/data_generation/preprocessing.pyRepository: vllm-project/speculators
Length of output: 399
🏁 Script executed:
#!/bin/bash
# Look for all calls to _normalize_conversation to see how it's called
grep -n "_normalize_conversation" src/speculators/data_generation/preprocessing.pyRepository: vllm-project/speculators
Length of output: 184
🏁 Script executed:
#!/bin/bash
# Check git to see if there were recent changes to the test file
git log --oneline -20 tests/integration/datagen/test_preprocessing.py 2>/dev/null || echo "No git history available"Repository: vllm-project/speculators
Length of output: 112
🏁 Script executed:
#!/antml:bash
# Check if metadata parameter has a default value I might have missed
python3 << 'EOF'
import inspect
import sys
sys.path.insert(0, 'src')
try:
from speculators.data_generation.preprocessing import _normalize_conversation
sig = inspect.signature(_normalize_conversation)
print("Function signature:")
print(sig)
print("\nParameters:")
for param_name, param in sig.parameters.items():
print(f" {param_name}: default={param.default}")
except Exception as e:
print(f"Error: {e}")
EOFRepository: vllm-project/speculators
Length of output: 99
🏁 Script executed:
#!/bin/bash
# Check the actual file to see what the data looks like - look for example datasets
find . -name "*.json" -o -name "*.jsonl" | head -5Repository: vllm-project/speculators
Length of output: 50
🏁 Script executed:
#!/bin/bash
# Check the configs to understand expected data structure
cat -n src/speculators/data_generation/configs.py | head -100Repository: vllm-project/speculators
Length of output: 1800
🏁 Script executed:
#!/bin/bash
# Check the git diff for the recent commit that added metadata support
git log -1 --oneline --name-only
git show c897305 --statRepository: vllm-project/speculators
Length of output: 23396
🏁 Script executed:
#!/bin/bash
# Check if metadata parameter had a default value that was removed or changed
git show c897305:src/speculators/data_generation/preprocessing.py | grep -A 10 "def _normalize_conversation"Repository: vllm-project/speculators
Length of output: 399
🏁 Script executed:
#!/antml:bash
# See what the old signature was
git show c897305~1:src/speculators/data_generation/preprocessing.py 2>/dev/null | grep -A 10 "def _normalize_conversation" || echo "Previous version not found"Repository: vllm-project/speculators
Length of output: 95
🏁 Script executed:
#!/bin/bash
# Check if tests were updated in that commit
git show c897305 tests/integration/datagen/test_preprocessing.py | head -100Repository: vllm-project/speculators
Length of output: 3267
🏁 Script executed:
#!/bin/bash
# Check the actual test assertions to see if they expect metadata handling
cat -n tests/integration/datagen/test_preprocessing.py | sed -n '27,80p'Repository: vllm-project/speculators
Length of output: 2136
🏁 Script executed:
#!/bin/bash
# Search for test_preprocess_batch to see how the batch function is tested
grep -A 30 "def test_preprocess_batch" tests/integration/datagen/test_preprocessing.py | head -50Repository: vllm-project/speculators
Length of output: 1839
🏁 Script executed:
#!/antml:bash
# Check if metadata parameter has a default in the actual function definition
python3 << 'EOF'
with open('src/speculators/data_generation/preprocessing.py', 'r') as f:
lines = f.readlines()
for i, line in enumerate(lines[60:75], start=61):
print(f"{i:3d}: {line}", end='')
EOFRepository: vllm-project/speculators
Length of output: 632
🏁 Script executed:
#!/bin/bash
# Look at how tests call _normalize_conversation - find all test calls
grep -n "_normalize_conversation(" tests/integration/datagen/test_preprocessing.py | head -20Repository: vllm-project/speculators
Length of output: 206
🏁 Script executed:
#!/bin/bash
# This is critical - test calls _normalize_conversation with only 1 arg but function requires 2 (metadata is mandatory)
# Let me verify the actual test calls to be sure
cat -n tests/integration/datagen/test_preprocessing.py | sed -n '29,45p'Repository: vllm-project/speculators
Length of output: 772
🏁 Script executed:
#!/antml:bash
# Check if metadata has a default value that wasn't shown
grep -A 5 "def _normalize_conversation" src/speculators/data_generation/preprocessing.py | cat -nRepository: vllm-project/speculators
Length of output: 303
🏁 Script executed:
#!/bin/bash
# Let's look at the complete function signature definition to see if metadata has a default
sed -n '61,66p' src/speculators/data_generation/preprocessing.py | cat -nRepository: vllm-project/speculators
Length of output: 303
🏁 Script executed:
#!/antml:bash
# Check if there are TWO _normalize_conversation functions or if metadata has a default we missed
rg "def _normalize_conversation" -A 8 src/speculators/data_generation/preprocessing.pyRepository: vllm-project/speculators
Length of output: 309
🏁 Script executed:
#!/antml:bash
# The test calls must be broken. Let me check if tests actually import and use this
cat -n tests/integration/datagen/test_preprocessing.py | sed -n '1,50p'Repository: vllm-project/speculators
Length of output: 1822
🏁 Script executed:
#!/bash
# The tests call _normalize_conversation(conv) but the signature requires metadata
# Let me check if metadata has a default value (=None or ={})
python3 << 'EOF'
with open('src/speculators/data_generation/preprocessing.py', 'r') as f:
content = f.read()
# Find the _normalize_conversation definition
import re
match = re.search(r'def _normalize_conversation\([^)]+\)', content, re.DOTALL)
if match:
print("Found function signature:")
print(match.group(0))
else:
print("Not found")
EOFRepository: vllm-project/speculators
Length of output: 199
🏁 Script executed:
#!/antml:bash
# Check if metadata has a default value = None or = {}
grep -A 10 "def _normalize_conversation" src/speculators/data_generation/preprocessing.py | grep -E "metadata|="Repository: vllm-project/speculators
Length of output: 120
🏁 Script executed:
#!/bash
# Let me check the exact bytes to ensure no default is hidden
cat src/speculators/data_generation/preprocessing.py | sed -n '61,66p' | od -c | head -60Repository: vllm-project/speculators
Length of output: 109
🏁 Script executed:
#!/antml:bash
# Check if maybe metadata has a default value that's on the next line or if the function was redefined
cat -n src/speculators/data_generation/preprocessing.py | sed -n '61,70p'Repository: vllm-project/speculators
Length of output: 468
🏁 Script executed:
#!/bash
# The most direct approach - just check what the actual full definition is with no truncation
head -n 70 src/speculators/data_generation/preprocessing.py | tail -n 20Repository: vllm-project/speculators
Length of output: 551
🏁 Script executed:
#!/bash
# This reveals a critical issue: the tests call _normalize_conversation(conv)
# but the function signature requires metadata as a MANDATORY parameter with NO DEFAULT
# This means the tests are BROKEN - they would fail at runtime
# Let's verify this by checking if maybe the test assertions should also fail
grep -A 20 "test_normalize_conversation_sharegpt_format" tests/integration/datagen/test_preprocessing.py | head -25Repository: vllm-project/speculators
Length of output: 722
🏁 Script executed:
#!/bash
# Check if there's perhaps an overloaded version or if metadata has a default that's set differently
grep -B 5 "metadata: dict" src/speculators/data_generation/preprocessing.py | head -20Repository: vllm-project/speculators
Length of output: 167
🏁 Script executed:
#!/bash
# Let me check if maybe the parameter was supposed to have a default value
# by looking at line 63 specifically for any = sign
sed -n '63p' src/speculators/data_generation/preprocessing.py | cat -ARepository: vllm-project/speculators
Length of output: 89
🏁 Script executed:
#!/bash
# Verify the linked repository findings about existing usages in vllm
# Check if reasoning_content is optional in vllm
grep -r "reasoning_content" /root/vllm-project/vllm/vllm/entrypoints/chat_utils.py 2>/dev/null | head -10 || echo "File not found"Repository: vllm-project/speculators
Length of output: 50
🏁 Script executed:
#!/bash
# Verify the zip behavior with mismatched lengths
python3 << 'EOF'
# Demonstrate the issue
conversations = ["conv1", "conv2", "conv3"]
metadatas = [] # This is what happens when metadata key is absent
result = list(zip(conversations, metadatas))
print(f"conversations: {len(conversations)} items")
print(f"metadatas: {len(metadatas)} items")
print(f"zip result: {len(result)} pairs")
print(f"Rows processed: {len(result)}, Rows lost: {len(conversations) - len(result)}")
# Demonstrate empty dict behavior
print(f"\nTruth value of empty dict: bool({{}}) = {bool({})}")
print(f"'not {{}}' = {not {}}")
EOFRepository: vllm-project/speculators
Length of output: 223
🏁 Script executed:
#!/bash
# Check if there are any tests that test the batch function WITH metadata
grep -A 50 "test_preprocess_batch" tests/integration/datagen/test_preprocessing.py | grep -E "metadata|examples\s*=" | head -20Repository: vllm-project/speculators
Length of output: 204
🏁 Script executed:
#!/bash
# Check if the batched tests provide metadata in the examples dict
cat -n tests/integration/datagen/test_preprocessing.py | sed -n '123,180p'Repository: vllm-project/speculators
Length of output: 2665
🏁 Script executed:
#!/bash
# Look for any test that actually includes metadata in the examples dict
grep -B 5 -A 30 "def test_preprocess_batch_basic" tests/integration/datagen/test_preprocessing.pyRepository: vllm-project/speculators
Length of output: 1205
🏁 Script executed:
#!/bash
# Search for any test that uses metadata key
grep -n "metadata" tests/integration/datagen/test_preprocessing.py | head -20Repository: vllm-project/speculators
Length of output: 50
🏁 Script executed:
#!/bash
# Final verification - check if metadata parameter was supposed to have a default
# Let me see the complete function one more time to be absolutely certain
grep -A 50 "def _normalize_conversation" src/speculators/data_generation/preprocessing.py | head -60Repository: vllm-project/speculators
Length of output: 1863
Metadata handling causes silent data loss in the batch preprocessing pipeline.
When the "metadata" key is missing from examples, metadatas = examples.get("metadata", []) returns an empty list, and zip(conversations, metadatas) produces zero pairs, dropping all rows without warning. Additionally, if not metadata or not isinstance(metadata, dict): continue skips even valid empty metadata {} (since not {} = True), and length mismatches between conversations and metadatas are silently truncated by zip.
Since reasoning_content is optional and conditionally accessed (line 101-102 checks if "reasoning_content" in metadata), samples without this field should still process normally with empty or missing metadata rather than being dropped entirely.
💡 Suggested fix
- metadatas= examples.get("metadata", [])
+ metadatas = examples.get("metadata")
+ if metadatas is None:
+ metadatas = [{} for _ in conversations]
+ elif not isinstance(metadatas, list):
+ log.warning("Invalid metadata column type; falling back to empty metadata")
+ metadatas = [{} for _ in conversations]
- for idx, (conv,metadata) in enumerate(zip(conversations,metadatas)):
+ if len(metadatas) != len(conversations):
+ log.warning(
+ f"Metadata/conversation length mismatch: "
+ f"{len(metadatas)} vs {len(conversations)}; padding metadata"
+ )
+ if len(metadatas) < len(conversations):
+ metadatas = metadatas + [{} for _ in range(len(conversations) - len(metadatas))]
+ else:
+ metadatas = metadatas[: len(conversations)]
+
+ for idx, (conv, metadata) in enumerate(zip(conversations, metadatas, strict=True)):
if not conv or not isinstance(conv, list):
continue
- if not metadata or not isinstance(metadata, dict):
+ if not isinstance(metadata, dict):
continue🧰 Tools
🪛 Ruff (0.15.13)
[warning] 280-280: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/speculators/data_generation/preprocessing.py` around lines 274 - 287, The
batch drops rows because metadatas = examples.get("metadata", []) produces an
empty list and zip(conversations, metadatas) truncates pairs and because the
code rejects empty dicts with if not metadata or not isinstance(metadata, dict):
continue; fix by: when building metadatas, if examples lacks "metadata" or its
list is shorter than conversations, create/pad a metadatas list of the same
length filled with {} (so missing/empty metadata is treated as {}), remove the
truthiness check and only validate metadata with isinstance(metadata, dict)
before calling _normalize_conversation, and add a warning log when metadata
length mismatches conversations to surface potential data issues (keep using
_normalize_conversation for normalization).
|
The quality checks have failed. Please run |
fynnsu
left a comment
fynnsu
left a comment
There was a problem hiding this comment.
Hi @Chibukach, my main issue with this is I don't fully understand the structure of the reasoning content you're trying to support.
I think I saw an example from Alex where we had a single reasoning_content value for a full conversation, is this the same as that?
The code seems to be assuming we have:
{
"conversations": list of dicts,
"metadata": list of dicts of same length
}
Is that correct?
| metadatas = metadatas + [{} for _ in range(len(conversations) - len(metadatas))] | ||
| else: | ||
| metadatas = metadatas[: len(conversations)] | ||
|
|
There was a problem hiding this comment.
This feels pretty unsafe. We're just assuming that if we have less metadata that the metadata lines up with first part of the conversation?
The conversation contains system, user, and assistant messages. Presumably we only have reasoning content for assistant responses.
| if not conv or not isinstance(conv, list): | ||
| continue | ||
| if not isinstance(metadata, dict): | ||
| continue |
There was a problem hiding this comment.
If we're constructing things this way we don't need to check for it. Also if the sample structure doesn't meet our expectations it would be better to fail with an error than just silently skip it.
|
@Chibukach Thanks so much for putting up this change! Just trying to understand this issue better. Could you maybe provide an example json file for what we actually get when we run response regeneration? I tested on vllm 0.21.0 and this is what I got: {
"id": "cf428475-a0ef-5fd8-a35e-25abbf57e895",
"conversations": [
{
"from": "human",
"value": "Has the SoundCloud Generation been broke down into further age-defined groups any?"
},
{
"from": "gpt",
"value": "<think>\nOkay, the user is asking if the SoundCloud Generation has been broken down into more specific age groups. \n</think>\n\nThe term \"SoundCloud Generation\" ... strict demographic segmentation."
}
],
"metadata": {
"idx": 2,
"finish_reason": "stop",
"latency_s": 6.904,
"usage": {
"prompt_tokens": 23,
"total_tokens": 973,
"completion_tokens": 950,
"prompt_tokens_details": null
},
"endpoint": "http://127.0.0.1:8000/v1/chat/completions"
}
}This looks like what we had before. Perhaps you could provide more information on which model & which version of vllm that requires this change? Thank you thank you! |
3 participants
Purpose
Recent VLLM output separate thinking content in the metadata instead of tags
Description
This PR adds support for retrieving thinking content from the "reasoning_content" field inside the metadata during a model response generation
Tests
Tested locally on Qwen/Qwen3-8B and the output prints thinking_msg
I have filled in: