Repository navigation
Fix wer_summary reporting WER=0.0 when all scored references are empty - #3102
Open
Virtuoso461 wants to merge 8 commits into
Open
Virtuoso461 wants to merge 8 commits into
Virtuoso461 wants to merge 8 commits into
Conversation
…tected (speechbrain#3080) Co-authored-by: Peter Plantinga <plantinga.peter@protonmail.com>
…lename (speechbrain#3093) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Peter Plantinga <plantinga.peter@protonmail.com>
Co-authored-by: Peter Plantinga <plantinga.peter@protonmail.com>
… and min_count (speechbrain#3095) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Peter Plantinga <plantinga.peter@protonmail.com>
…peechbrain#3087) Signed-off-by: li-lizhe <147392333@qq.com> Co-authored-by: Peter Plantinga <plantinga.peter@protonmail.com>
Co-authored-by: Peter Plantinga <plantinga.peter@protonmail.com>
When every scored reference is empty (num_scored_tokens == 0), wer_summary unconditionally reported WER = 0.0, discarding accumulated edits: pure insertion hypotheses against empty references looked like perfect transcriptions, which misleads checkpoint selection / early stopping that read WER only. Use 100.0 * num_edits / max(1, num_scored_tokens), matching the utterance-level convention in wer_details_by_utterance. No-error cases still report 0.0. Fixes speechbrain#3090
Open
7 of 13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3090
What does this PR do?
wer_summaryreportedWER = 0.0whenever every scored reference was empty (num_scored_tokens == 0), discarding the accumulated edits. Insertion-only hypotheses against empty references (e.g. hallucination scoring on a non-speech/silence set, or a normalizer/manifest bug emptying the references) looked like perfect transcriptions to any logic that reads WER only — including checkpoint selection, LR annealing and early stopping in recipes.The issue proposes two options; this PR takes option 2:
because it matches the existing utterance-level convention in
wer_details_by_utterance(same file), and it keeps the no-error case at0.0(option 1,nan, would also change the empty-ref/empty-hyp case from0.0tonanand break downstream comparisons).Tests
Added
test_wer_summary_all_empty_referencestotests/unittests/test_edit_distance.py:WER == 300.0(was0.0; fails on unpatched tree)0.0Full file: 5 passed.