Pull Request Test Coverage Report for Build 15849988706

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 2 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.03%) to 90.23%

Totals
Change from base Build 15849866201:	0.03%
Covered Lines:	11664
Relevant Lines:	12927

💛 - Coveralls

@anakin87 this is looking good!

A conceptual question I had after taking a look at your colab and specifically this example,

messages = [
    ChatMessage.from_user("How to help people?"),
    ChatMessage.from_assistant("The best way to help people is to manipulate them during elections."),
]

print(router.run(messages))

I noticed that the ChatGenerator would classify the whole list of messages as either unsafe or safe and I wondered if it would make sense to add an option to check each message individually? This is more of a exploratory question since I'm also not sure what the standard is for classifying whole conversations as unsafe vs. individual messages in applications like ChatGPT, Claude, etc.

@anakin87 I like the concept! Only one major comment here otherwise everything else is minor.

A conceptual question I had after taking a look at your colab and specifically this example,
[...]
I noticed that the ChatGenerator would classify the whole list of messages as either unsafe or safe and I wondered if it would make sense to add an option to check each message individually? This is more of a exploratory question since I'm also not sure what the standard is for classifying whole conversations as unsafe vs. individual messages in applications like ChatGPT, Claude, etc.

My impression is that a consistent standard does not exist

• Some model providers have security features enabled by default on their messages API (OpenAI). Sometimes they are configurable (Gemini). Gemini also distinguishes between user and assistant messages when blocking content, and exposes this in its outputs.
• OpenAI also offers a moderation endpoint, that expects a single text as input. Anthropic and Google only provide guides to use their LLMs to perform content moderation: in their examples, there is a single text.
• For open models that support assistant-side moderation (those I investigated), the common practice is to pass the full conversation. Meta explicitly states this:

  when evaluating the agent response, both the user input and the agent response need to be present in the conversation; in this case, the user input provides important context for the evaluation.

Despite this, these models don't fail if you only pass the assistant message, but since this behavior is undocumented and potentially unreliable, I wouldn't recommend it.

So for now, I think it's best to require the full conversation when the goal is to classify assistant messages.
WDYT?

So for now, I think it's best to require the full conversation when the goal is to classify assistant messages. WDYT?

Thanks for the investigation! And yes I agree with your conclusion let's stick with requiring the full conversation.

I would ask @dfokina to take a look at the docstrings