Fixed Whitespace Error in Streaming mode

[enochlev]

When a TensorRTLLM is deployed with streaming mode tokens have no white spaces in between streaming chunks.

Link to Issue

This is because calling tokenizer.decode does a whitespace strip of the output and this functionality cannot be disabled. So we must manually look for those white spaces and add.

The standard way of going about this is holding tokens in cache until a space is detected, in which everything after the space is put again into cache.

The other suggested method decodes the token_id text instead of the string text to look for a "_" symbol

This PR should be able to fix these issues

[dbuades]

I confirm that this change fixes the issue. Thank you!

[yiding]

I notice that LlamaTokenizer has an add_prefix_space option, which when active, will insert a space at the beginning of a string passed to encode, and trim a space from the first character of the output returned by decode. This behavior can be turned off by setting add_prefix_space=False in postprocessing/1/model.py

        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_dir,
                                                       legacy=False,
                                                       padding_side='left',
                                                       trust_remote_code=True,
                                                       add_prefix_space=False)

Doing so will allow spaces to be emitted in the streaming output. However I think this is an indication that Tokenizer.decode is not meant for decoding sub-sequences of emitted tokens from an output, and there might be other oddities with this usage.