[dipo101]

bridge.generate(input_features) now transcribes:

bridge.tokenizer.set_prefix_tokens(language="english", task="transcribe")
bridge.generate(input_features, do_sample=False, return_type="str")

• Takes the audio features tensor. The decoder starts from decoder_input (one row shared by every clip, or one per clip) or, by default, the tokenizer's prefix tokens. Returns the decoder tokens, prompt included; return_type="str" gives the transcripts.
• Encodes the audio once per call, as HF's generate does, instead of once per token. Encoder hooks fire on that pass and decoder hooks every step. The pass sets output_attentions=False so HF does not hold every layer's attention weights: two whisper-large-v3-turbo clips now peak at 5.6 GB instead of running out of memory at 15 GB.
• The encoder-decoder loop applies generation_config.suppress_tokens and begin_suppress_tokens with transformers' own processors, as HF's generate does by default. Whisper sets both.
• A prompt plus max_new_tokens beyond the decoder's positions raises, and so does text input. generate_stream raises NotImplementedError for audio seq2seq.
• hf_generate sends temperature only when sampling. Whisper's generate samples whenever it gets a temperature above 0, so greedy hf_generate calls sampled.

Greedy output matches HF's Whisper generate token for token on two LibriSpeech clips, with whisper-tiny and whisper-large-v3-turbo.

Tests: tiny random-weight parity against HF's GenerationMixin.generate (default and explicit prompts, suppressed tokens, string output, encoder hooks once and decoder hooks every step, error paths); slow whisper-tiny parity against HF's Whisper generate and through hf_generate; a unit test for the temperature change.

Part of #1938

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility

[jlarson4]

Looks good @dipo101, thank you for putting this together!