Skip to content

[fix] Handle out of range values in scalar quantization#2865

Open
meiravgri wants to merge 1 commit into
huggingface:mainfrom
meiravgri:meiravg_sq_handle_out_of_range
Open

[fix] Handle out of range values in scalar quantization#2865
meiravgri wants to merge 1 commit into
huggingface:mainfrom
meiravgri:meiravg_sq_handle_out_of_range

Conversation

@meiravgri
Copy link
Copy Markdown

@meiravgri meiravgri commented Jul 30, 2024

Issue description

This PR addresses an issue with out-of-range values during scalar quantization. Previously, when quantizing data, values outside the calibration range could result in incorrect quantization.

Solution

The PR fixes this issue by introducing a clip operation in the quantization process. This operation ensures that quantized values are correctly bounded within the expected range, thus preventing out-of-range values from producing incorrect results.

Example

calibration_set = np.array([[1,20,-3], [4,5,-60]], dtype=np.float32)
dataset = np.array([[-1,15,1]], dtype=np.float32). 
sq_embeddings = quantize_embeddings(
    dataset,
    precision="int8",
    calibration_embeddings=my_np_array,
)
print("quantizing dataset = ", sq_embeddings)

Result:
incorrect quantized values for the first and third entries, respectively, which are outside the calibration range.

quantizing dataset =  [[ -42   42 -112]]

After the PR:
The quantized values are correctly bounded and thus match the expected output.

quantizing dataset =  [[-128   42  127]]

@meiravgri
Copy link
Copy Markdown
Author

Hi @tomaarsen, could you please review my PR? Thanks!

@meiravgri meiravgri changed the title Handle out of range values in scalar quantization [fix] Handle out of range values in scalar quantization Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant