feat(metrics): add Character Error Rate (CER) metric#3710
Open
Manimaran-tech wants to merge 1 commit into
Open
feat(metrics): add Character Error Rate (CER) metric#3710Manimaran-tech wants to merge 1 commit into
Manimaran-tech wants to merge 1 commit into
Conversation
Add CharacterErrorRate metric for NLP tasks that measures the edit distance at the character level between predicted and reference strings. Features: - Implements Levenshtein distance algorithm for edit distance computation - Supports batch processing and distributed computation - Follows same API pattern as other NLP metrics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
@Manimaran-tech, could you link the issue number for this PR , also the Empty reference strings aren't handled in update and Distributed integration tests are missing as the other metric bleu and rogue have this |
Contributor
|
@vfdev-5 should i update the pr or this one is good to be merged? |
Comment on lines
+108
to
+116
| for p, r in zip(y_pred, y): | ||
| p_chars = list(p) | ||
| r_chars = list(r) | ||
|
|
||
| errors += _edit_distance(r_chars, p_chars) | ||
| refs += len(r_chars) | ||
|
|
||
| self._num_errors += torch.tensor(errors, device=self._device) | ||
| self._num_refs += torch.tensor(refs, device=self._device) |
Collaborator
There was a problem hiding this comment.
I think we should add a check for p and r is string or not .
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new NLP metric,
CharacterErrorRate, to Ignite.Character Error Rate (CER) measures the normalized edit distance between predicted and reference strings at the character level.
What’s included
CharacterErrorRatemetric implementationWhy
CER is a standard evaluation metric for ASR/OCR and other sequence-to-text tasks where character-level accuracy is important.
Fixes #{issue_number}
Checklist