Pewarta Proxy

📋 Server Request & Proxy Logs

Loading active proxy settings...
Developed by Khairul Adha (adhexa.id)
Mengambil log server...

⚙️ Pengaturan Proxy Target

Skip to content

Normalize smoothed KL mean by non-padding targets - #3115

Open
kocean9-freedom wants to merge 1 commit into
speechbrain:developfrom
kocean9-freedom:fix-kldiv-mean-reduction
Open

kocean9-freedom wants to merge 1 commit into
speechbrain:developfrom
kocean9-freedom:fix-kldiv-mean-reduction

Conversation

@kocean9-freedom

Copy link
Copy Markdown

What does this PR do?

Fix the mean reduction in kldiv_loss when label smoothing is enabled. It currently runs loss.sum().mean(), so it returns a sum. Duplicating a batch increases the supposedly averaged loss and gradients. The unsmoothed path uses an actual token mean. The small-smoothing limit is checked on unpadded inputs; the existing difference between its length mask and the smoothed path's pad_idx mask is outside this change.

Divide the summed KL divergence by the number of non-pad_idx targets. Keep all-padding input at zero, and document this reduction explicitly. The smoothing distribution, sum, batchmean, batch, and existing length handling are unchanged. Behavior change: smoothed mean values and gradients are now normalized; callers relying on the old sum should request sum explicitly. The bundled LibriSpeech Transformer configuration uses batchmean and retains its behavior.

On two bundled speech samples through the existing MFCC/CRDNN/attention ASR example, 60 valid target tokens produce a loss of 153.60635 before and 2.56011 after, matching an independent torch.distributions KL reference. With the fix, one public Brain.fit_batch SGD step matches independently calculated updates for all 31 trainable parameters on CPU and MPS. This validates loss scaling, not recognition accuracy.

Validation (Python 3.12.14, torch/torchaudio 2.10.0, macOS arm64):

  • Original implementation: 7 new regression failures, 3 passing controls. Fixed implementation: 40 passed, 1 skipped across loss unit tests and module doctests; 11 new tests passed on MPS. Coverage includes classification/sequence inputs, repeated batches, padding and its gradients, all-padding input, independent loss/gradient references, and the small-smoothing NLL limit.
  • Full unit suite: 767 passed, 6 skipped, 6 failed. All six failures reproduce with the original upstream loss file: four rescorer tests need uncached models, test_fallback_when_all_fail assumes os.sched_getaffinity, and test_run_shell assumes shell echo -n behavior.
  • Existing ASR_seq2seq integration test passed. All configured pre-commit hooks passed. Built sdist/wheel with the declared setuptools backend; the extracted wheel passed the same real-audio CPU training-step check.

CUDA, large-scale training, and online model tests were not run. AI assistance: implementation, tests, validation, review, and PR preparation used OpenAI Codex; no independent human review is claimed.

Before submitting
  • Read the contributor guideline.
  • Keep the PR focused on one behavior.
  • Update the reduction documentation.
  • Add necessary regression tests.
  • Verify relevant tests; full-suite environment failures are documented above.
  • List behavior changes and unchanged reductions.
  • Follow repository code style; all pre-commit checks pass.

PR review

Reviewer checklist
  • Is this pull request ready for review?
  • Check the submission checklist.
  • Confirm reduction semantics and compatibility.
  • Add appropriate labels and milestones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant