Potential minor Improvements

[dipamsen]

• could we somehow tell elevenlabs certain .s are not periods? (maybe we replace the . by dot or something). Such .s can maybe be identified if the . does not have any whitespace on either side of it (eg. p5.js, this.position)
• collapse multiple consecutive word updates into one? currently the diffing algorithm is by WORDS, which looks good if in a line a single token is being changed, only that part will be removed and then typed. But if an entire line changes, each word gets selected and typed, one by one. Maybe multiple consecutive word diffs can be collapsed into one update (upto a newline)

[dipamsen]

also maybe:

• speak comments? mostly when instructed to not write comments, it (gpt-4o) does so as it thinks it is speaking whatever is being written as a comment.

[shiffman]

could we somehow tell elevenlabs certain .s are not periods? (maybe we replace the . by dot or something). Such .s can maybe be identified if the . does not have any whitespace on either side of it (eg. p5.js, this.position)

Yes, we should manually replace the . with "dot" I think! We could make a list of known terms, but I think the no whitespace pattern will work well!

collapse multiple consecutive word updates into one? currently the diffing algorithm is by WORDS, which looks good if in a line a single token is being changed, only that part will be removed and then typed. But if an entire line changes, each word gets selected and typed, one by one. Maybe multiple consecutive word diffs can be collapsed into one update (upto a newline)

Game for this! Maybe lower priority!

speak comments? mostly when instructed to not write comments, it (gpt-4o) does so as it thinks it is speaking whatever is being written as a comment.

I like this idea! Do we think we could have it speak and type at the same time?

[dipamsen]

My evaluation on how easy it is to implement

1. should be easy enough, though the issue is that the model may stream a chunk p5. and such chunk would be categorised as end of sentence by our parsing logic. so idk how to elegantly handle that
2. i think this is pretty doable
3. this seems hard because the parsing logic will have to be modified to go into speech mode on receiving a // token, and then going back to code on receiving a \n

Do we think we could have it speak and type at the same time?

probably have to change lots of stuff to have this, but possible