Skip to content

Commit a2f109a

Browse files
waylaidwandererwaylaidwanderer
authored
feat: per-message client options (#173)
* feat(client): allow setting per-message client options * feat(browser-client): allow setting per-message client options * feat(bing): allow setting per-message client options * feat: configurable whitelist for per-message client option properties * fix: override options properly * feat: implement ability to swap clients per message * docs: explain perMessageClientOptionsWhitelist * fix: make it more clear that openaiApiKey can be set per-message * fix: create new client for every message to avoid option pollution
1 parent 8e0f26e commit a2f109a

7 files changed

Lines changed: 272 additions & 74 deletions

File tree

‎README.md‎

Lines changed: 54 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -180,9 +180,9 @@ module.exports = {
180180
// If set, `ChatGPTClient` will use `keyv-file` to store conversations to this JSON file instead of in memory.
181181
// However, `cacheOptions.store` will override this if set
182182
storageFilePath: process.env.STORAGE_FILE_PATH || './cache.json',
183-
// Your OpenAI API key (for `ChatGPTClient`)
184-
openaiApiKey: process.env.OPENAI_API_KEY || '',
185183
chatGptClient: {
184+
// Your OpenAI API key (for `ChatGPTClient`)
185+
openaiApiKey: process.env.OPENAI_API_KEY || '',
186186
// (Optional) Support for a reverse proxy for the completions endpoint (private API server).
187187
// Warning: This will expose your `openaiApiKey` to a third party. Consider the risks before using this.
188188
// reverseProxyUrl: 'https://chatgpt.hato.ai/completions',
@@ -242,8 +242,29 @@ module.exports = {
242242
host: process.env.API_HOST || 'localhost',
243243
// (Optional) Set to true to enable `console.debug()` logging
244244
debug: false,
245-
// (Optional) Possible options: "chatgpt", "chatgpt-browser", "bing".
246-
// clientToUse: 'bing',
245+
// (Optional) Possible options: "chatgpt", "chatgpt-browser", "bing". (Default: "chatgpt")
246+
clientToUse: 'chatgpt',
247+
// (Optional) Set this to allow changing the client or client options in POST /conversation.
248+
// To disable, set to `null`.
249+
perMessageClientOptionsWhitelist: {
250+
// The ability to switch clients using `clientOptions.clientToUse` will be disabled if `validClientsToUse` is not set.
251+
// To allow switching clients per message, you must set `validClientsToUse` to a non-empty array.
252+
validClientsToUse: ['bing', 'chatgpt', 'chatgpt-browser'], // values from possible `clientToUse` options above
253+
// The Object key, e.g. "chatgpt", is a value from `validClientsToUse`.
254+
// If not set, ALL options will be ALLOWED to be changed. For example, `bing` is not defined in `perMessageClientOptionsWhitelist` above,
255+
// so all options for `bingAiClient` will be allowed to be changed.
256+
// If set, ONLY the options listed here will be allowed to be changed.
257+
// In this example, each array element is a string representing a property in `chatGptClient` above.
258+
chatgpt: [
259+
'promptPrefix',
260+
'userLabel',
261+
'chatGptLabel',
262+
// Setting `modelOptions.temperature` here will allow changing ONLY the temperature.
263+
// Other options like `modelOptions.model` will not be allowed to be changed.
264+
// If you want to allow changing all `modelOptions`, define `modelOptions` here instead of `modelOptions.temperature`.
265+
'modelOptions.temperature',
266+
],
267+
},
247268
},
248269
// Options for the CLI app
249270
cliOptions: {
@@ -263,12 +284,32 @@ Alternatively, you can install and run the package directly.
263284
- using `npm start` or `npm run server` (if not using Docker)
264285
- using `docker-compose up` (requires Docker)
265286

287+
#### Endpoints
288+
<details>
289+
<summary><strong>POST /conversation</strong></summary>
290+
291+
Start or continue a conversation.
292+
Optional parameters are only necessary for conversations that span multiple requests.
293+
294+
| Field | Description |
295+
|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
296+
| message | The message to be displayed to the user. |
297+
| conversationId | (Optional) An ID for the conversation. |
298+
| parentMessageId | (Optional, for `ChatGPTClient` only) The ID of the parent message. |
299+
| conversationSignature | (Optional, for `BingAIClient` only) A signature for the conversation. |
300+
| clientId | (Optional, for `BingAIClient` only) The ID of the client. |
301+
| invocationId | (Optional, for `BingAIClient` only) The ID of the invocation. |
302+
| clientOptions | (Optional) An object containing options for the client. |
303+
| clientOptions.clientToUse | (Optional) The client to use for this message. Possible values: `chatgpt`, `chatgpt-browser`, `bing`. |
304+
| clientOptions.* | (Optional) Any valid options for the client. For example, for `ChatGPTClient`, you can set `clientOptions.openaiApiKey` to set an API key for this message only, or `clientOptions.promptPrefix` to give the AI custom instructions for this message only, etc. |
305+
306+
</details>
307+
266308
#### Usage
267309
<details>
268310
<summary><strong>Method 1 (POST)</strong></summary>
269311

270-
To start a conversation with ChatGPT, send a POST request to the server's `/conversation` endpoint with a JSON body in the following format.
271-
Optional parameters are only necessary for conversations that span multiple requests:
312+
To start a conversation with ChatGPT, send a POST request to the server's `/conversation` endpoint with a JSON body with parameters per **Endpoints** > **POST /conversation** above.
272313
```JSON
273314
{
274315
"message": "Hello, how are you today?",
@@ -289,7 +330,7 @@ The server will return a JSON object containing ChatGPT's response:
289330
"conversationSignature": "your-conversation-signature (for `BingAIClient` only)",
290331
"clientId": "your-client-id (for `BingAIClient` only)",
291332
"invocationId": "your-invocation-id (for `BingAIClient` only - pass this new value back into subsequent requests as-is)",
292-
"details": "additional details about the AI's response (for `BingAIClient` only)"
333+
"details": "an object containing the raw response from the client"
293334
}
294335
```
295336

@@ -316,7 +357,7 @@ If there was an error sending the message to ChatGPT:
316357
You can set `"stream": true` in the request body to receive a stream of tokens as they are generated.
317358

318359
```js
319-
import { fetchEventSource } from '@waylaidwanderer/fetch-event-source';
360+
import { fetchEventSource } from '@waylaidwanderer/fetch-event-source'; // use `@microsoft/fetch-event-source` instead if in a browser environment
320361

321362
const opts = {
322363
method: 'POST',
@@ -327,7 +368,8 @@ const opts = {
327368
"message": "Write a poem about cats.",
328369
"conversationId": "your-conversation-id (optional)",
329370
"parentMessageId": "your-parent-message-id (optional)",
330-
"stream": true
371+
"stream": true,
372+
// Any other parameters per `Endpoints > POST /conversation` above
331373
}),
332374
};
333375
```
@@ -346,7 +388,7 @@ Successful output:
346388
{ data: ' you', event: '', id: '', retry: undefined }
347389
{ data: ' today', event: '', id: '', retry: undefined }
348390
{ data: '?', event: '', id: '', retry: undefined }
349-
{ data: '<result JSON here>', event: 'result', id: '', retry: undefined }
391+
{ data: '<result JSON here, see Method 1>', event: 'result', id: '', retry: undefined }
350392
{ data: '[DONE]', event: '', id: '', retry: undefined }
351393
// Hello! How can I help you today?
352394
```
@@ -408,7 +450,7 @@ Instructions are provided below.
408450
* **This is NOT the same thing as the _session token_.**
409451
* Automatically fetching or refreshing your ChatGPT access token is not currently supported by this library. Please handle this yourself for now.
410452
2. Set `reverseProxyUrl` to `https://chatgpt.hato.ai/completions` in `settings.js > chatGptClient` or `ChatGPTClient`'s options.
411-
3. Set the "OpenAI API key" parameter (e.g. `settings.openaiApiKey`) to the ChatGPT access token you got in step 1.
453+
3. Set the "OpenAI API key" parameter (e.g. `settings.chatGptClient.openaiApiKey`) to the ChatGPT access token you got in step 1.
412454
4. Set the `model` to `text-davinci-002-render`, `text-davinci-002-render-paid`, or `text-davinci-002-render-sha` depending on which ChatGPT models that your account has access to. Models **must** be a ChatGPT model name, not the underlying model name, and you cannot use a model that your account does not have access to.
413455
* You can check which ones you have access to by opening DevTools and going to the Network tab. Refresh the page and look at the response body for https://chat.openai.com/backend-api/models.
414456

@@ -426,7 +468,7 @@ Instructions are provided below.
426468
* **This is NOT the same thing as the _session token_.**
427469
* Automatically fetching or refreshing your ChatGPT access token is not currently supported by this library. Please handle this yourself for now.
428470
2. Set `reverseProxyUrl` to `https://chatgpt.pawan.krd/api/completions` in `settings.js > chatGptClient` or `ChatGPTClient`'s options.
429-
3. Set the "OpenAI API key" parameter (e.g. `settings.openaiApiKey`) to the ChatGPT access token you got in step 1.
471+
3. Set the "OpenAI API key" parameter (e.g. `settings.chatGptClient.openaiApiKey`) to the ChatGPT access token you got in step 1.
430472
4. Set the `model` to `text-davinci-002-render`, `text-davinci-002-render-paid`, or `text-davinci-002-render-sha` depending on which ChatGPT models that your account has access to. Models **must** be a ChatGPT model name, not the underlying model name, and you cannot use a model that your account does not have access to.
431473
* You can check which ones you have access to by opening DevTools and going to the Network tab. Refresh the page and look at the response body for https://chat.openai.com/backend-api/models.
432474

‎bin/cli.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ switch (clientToUse) {
8888
break;
8989
default:
9090
client = new ChatGPTClient(
91-
settings.openaiApiKey,
91+
settings.openaiApiKey || settings.chatGptClient.openaiApiKey,
9292
settings.chatGptClient,
9393
settings.cacheOptions,
9494
);

‎bin/server.js‎

Lines changed: 88 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -45,26 +45,7 @@ if (settings.storageFilePath && !settings.cacheOptions.store) {
4545
}
4646

4747
const clientToUse = settings.apiOptions?.clientToUse || settings.clientToUse || 'chatgpt';
48-
49-
let client;
50-
switch (clientToUse) {
51-
case 'bing':
52-
client = new BingAIClient(settings.bingAiClient);
53-
break;
54-
case 'chatgpt-browser':
55-
client = new ChatGPTBrowserClient(
56-
settings.chatGptBrowserClient,
57-
settings.cacheOptions,
58-
);
59-
break;
60-
default:
61-
client = new ChatGPTClient(
62-
settings.openaiApiKey,
63-
settings.chatGptClient,
64-
settings.cacheOptions,
65-
);
66-
break;
67-
}
48+
const perMessageClientOptionsWhitelist = settings.apiOptions?.perMessageClientOptionsWhitelist || null;
6849

6950
const server = fastify();
7051

@@ -109,14 +90,24 @@ server.post('/conversation', async (request, reply) => {
10990
// noinspection ExceptionCaughtLocallyJS
11091
throw invalidError;
11192
}
112-
const parentMessageId = body.parentMessageId ? body.parentMessageId.toString() : undefined;
113-
result = await client.sendMessage(body.message, {
93+
94+
let clientToUseForMessage = clientToUse;
95+
const clientOptions = filterClientOptions(body.clientOptions, clientToUseForMessage);
96+
if (clientOptions && clientOptions.clientToUse) {
97+
clientToUseForMessage = clientOptions.clientToUse;
98+
delete clientOptions.clientToUse;
99+
}
100+
101+
const messageClient = getClient(clientToUseForMessage);
102+
103+
result = await messageClient.sendMessage(body.message, {
114104
jailbreakConversationId: body.jailbreakConversationId ? body.jailbreakConversationId.toString() : undefined,
115105
conversationId: body.conversationId ? body.conversationId.toString() : undefined,
116-
parentMessageId,
106+
parentMessageId: body.parentMessageId ? body.parentMessageId.toString() : undefined,
117107
conversationSignature: body.conversationSignature,
118108
clientId: body.clientId,
119109
invocationId: body.invocationId,
110+
clientOptions,
120111
onProgress,
121112
abortController,
122113
});
@@ -173,3 +164,77 @@ function nextTick() {
173164
return new Promise(resolve => setTimeout(resolve, 0));
174165
}
175166

167+
function getClient(clientToUse) {
168+
switch (clientToUse) {
169+
case 'bing':
170+
return new BingAIClient(settings.bingAiClient);
171+
case 'chatgpt-browser':
172+
return new ChatGPTBrowserClient(
173+
settings.chatGptBrowserClient,
174+
settings.cacheOptions,
175+
);
176+
case 'chatgpt':
177+
return new ChatGPTClient(
178+
settings.openaiApiKey || settings.chatGptClient.openaiApiKey,
179+
settings.chatGptClient,
180+
settings.cacheOptions,
181+
);
182+
default:
183+
throw new Error(`Invalid clientToUse: ${clientToUse}`);
184+
}
185+
}
186+
187+
/**
188+
* Filter objects to only include whitelisted properties set in
189+
* `settings.js` > `apiOptions.perMessageClientOptionsWhitelist`.
190+
* Returns original object if no whitelist is set.
191+
* @param {*} inputOptions
192+
* @param clientToUse
193+
*/
194+
function filterClientOptions(inputOptions, clientToUse) {
195+
if (!inputOptions || !perMessageClientOptionsWhitelist) {
196+
return null;
197+
}
198+
199+
// If inputOptions.clientToUse is set and is in the whitelist, use it instead of the default
200+
if (
201+
perMessageClientOptionsWhitelist.validClientsToUse
202+
&& inputOptions.clientToUse
203+
&& perMessageClientOptionsWhitelist.validClientsToUse.includes(inputOptions.clientToUse)
204+
) {
205+
clientToUse = inputOptions.clientToUse;
206+
} else {
207+
inputOptions.clientToUse = clientToUse;
208+
}
209+
210+
const whitelist = perMessageClientOptionsWhitelist[clientToUse];
211+
if (!whitelist) {
212+
// No whitelist, return all options
213+
return inputOptions;
214+
}
215+
216+
const outputOptions = {};
217+
218+
for (let property in inputOptions) {
219+
const allowed = whitelist.includes(property);
220+
221+
if (!allowed && typeof inputOptions[property] === 'object') {
222+
// Check for nested properties
223+
for (let nestedProp in inputOptions[property]) {
224+
const nestedAllowed = whitelist.includes(`${property}.${nestedProp}`);
225+
if (nestedAllowed) {
226+
outputOptions[property] = outputOptions[property] || {};
227+
outputOptions[property][nestedProp] = inputOptions[property][nestedProp];
228+
}
229+
}
230+
continue;
231+
}
232+
233+
// Copy allowed properties to outputOptions
234+
if (allowed) {
235+
outputOptions[property] = inputOptions[property];
236+
}
237+
}
238+
239+
return outputOptions;
240+
}

‎settings.example.js‎

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ export default {
66
// If set, `ChatGPTClient` will use `keyv-file` to store conversations to this JSON file instead of in memory.
77
// However, `cacheOptions.store` will override this if set
88
storageFilePath: process.env.STORAGE_FILE_PATH || './cache.json',
9-
// Your OpenAI API key (for `ChatGPTClient`)
10-
openaiApiKey: process.env.OPENAI_API_KEY || '',
119
chatGptClient: {
10+
// Your OpenAI API key (for `ChatGPTClient`)
11+
openaiApiKey: process.env.OPENAI_API_KEY || '',
1212
// (Optional) Support for a reverse proxy for the completions endpoint (private API server).
1313
// Warning: This will expose your `openaiApiKey` to a third party. Consider the risks before using this.
1414
// reverseProxyUrl: 'https://chatgpt.hato.ai/completions',
@@ -68,8 +68,29 @@ export default {
6868
host: process.env.API_HOST || 'localhost',
6969
// (Optional) Set to true to enable `console.debug()` logging
7070
debug: false,
71-
// (Optional) Possible options: "chatgpt", "chatgpt-browser", "bing".
72-
// clientToUse: 'bing',
71+
// (Optional) Possible options: "chatgpt", "chatgpt-browser", "bing". (Default: "chatgpt")
72+
clientToUse: 'chatgpt',
73+
// (Optional) Set this to allow changing the client or client options in POST /conversation.
74+
// To disable, set to `null`.
75+
perMessageClientOptionsWhitelist: {
76+
// The ability to switch clients using `clientOptions.clientToUse` will be disabled if `validClientsToUse` is not set.
77+
// To allow switching clients per message, you must set `validClientsToUse` to a non-empty array.
78+
validClientsToUse: ['bing', 'chatgpt', 'chatgpt-browser'], // values from possible `clientToUse` options above
79+
// The Object key, e.g. "chatgpt", is a value from `validClientsToUse`.
80+
// If not set, ALL options will be ALLOWED to be changed. For example, `bing` is not defined in `perMessageClientOptionsWhitelist` above,
81+
// so all options for `bingAiClient` will be allowed to be changed.
82+
// If set, ONLY the options listed here will be allowed to be changed.
83+
// In this example, each array element is a string representing a property in `chatGptClient` above.
84+
chatgpt: [
85+
'promptPrefix',
86+
'userLabel',
87+
'chatGptLabel',
88+
// Setting `modelOptions.temperature` here will allow changing ONLY the temperature.
89+
// Other options like `modelOptions.model` will not be allowed to be changed.
90+
// If you want to allow changing all `modelOptions`, define `modelOptions` here instead of `modelOptions.temperature`.
91+
'modelOptions.temperature',
92+
],
93+
},
7394
},
7495
// Options for the CLI app
7596
cliOptions: {

0 commit comments

Comments
 (0)