add endpoint to return prompts

[rogthefrog]

BaaS --get prompts--> modelrunner --get prompts--> modelbench job <-- this part

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[bkorycki]

Isn't this the requests endpoint in the indirect sut? I'm actually not even sure that IndirectSUTServer.prompts gets used. I think that should be deleted.

[wpietri]

Isn't this the requests endpoint in the indirect sut? I'm actually not even sure that IndirectSUTServer.prompts gets used. I think that should be deleted.

That's my understanding too.

[rogthefrog]

I added prompts because it wasn't clear to me what requests did (it returns list(self.outstanding_requests.values())) since the word "prompt" doesn't appear anywhere in that code. So we should tidy up / standardize our domain language and clarify what the difference is between a "prompt" and an "outstanding request."

Are they the same thing? If so, then we should change some names and/or add comments explaining what's what, because it's confusing.

@bkorycki @wpietri do you have the answers?

[wpietri]

I added prompts because it wasn't clear to me what requests did (it returns list(self.outstanding_requests.values())) since the word "prompt" doesn't appear anywhere in that code. So we should tidy up / standardize our domain language and clarify what the difference is between a "prompt" and an "outstanding request."

Are they the same thing? If so, then we should change some names and/or add comments explaining what's what, because it's confusing.

@bkorycki @wpietri do you have the answers?

From the IndirectSUT's pespective, requests have a prompt and a request id (which is not our notion of a prompt id). Responses have a request id and a response. Each request comes from an IndirectSUTRequest and each response gets turned into an IndirectSUTResponse.

These in turn derive from our core notions of SUT requests and responses. So the terms align with our usual domain language. I think the modelbench code should stay consistent with that. It should be the BaaS server's job to translate between the BaaS user's domain model and our internal one.

[rogthefrog]

Are we using the request ID as the key to reconcile a prompt we sent the customer and the response to that prompt they send back to us? @wpietri

[bkorycki]

@rogthefrog that's right

[rogthefrog]

Do we want to call it "request ID" in the response BaaS sends? I think that'd be fine.

I.e. the customer would receive this:

[
{ "request_id": "12345", "prompt": "why did the chicken cross the road?" }, # (etc)
]

Is everyone good with this?

@bkorycki @wpietri

[bkorycki]

Yes that makes sense since the request should contain more than just the prompt (e.g. max_tokens, temperature..).

[rogthefrog]

Great, thank you.

[wpietri]

Yes, that's also good by me.