make faster_whisper.assets as a valid python package to distribute (#772 ) (#774 )

Foolproof: Disable VAD if clip_timestamps is in use (#769 )
* Foolproof: Disable VAD if clip_timestamps is in use Prevent silly things to happen.
2024-04-02 18:22:22 +02:00 · 2024-04-02 18:20:34 +02:00 · 2024-04-02 18:18:12 +02:00 · 2024-03-27 08:31:17 +01:00 · 2024-03-26 14:58:39 +01:00 · 2024-03-12 15:44:49 +01:00
17 changed files with 1026 additions and 216 deletions
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -0,0 +1,31 @@
+# Contributing to faster-whisper
+
+Contributions are welcome! Here are some pointers to help you install the library for development and validate your changes before submitting a pull request.
+
+## Install the library for development
+
+We recommend installing the module in editable mode with the `dev` extra requirements:
+
+```bash
+git clone https://github.com/SYSTRAN/faster-whisper.git
+cd faster-whisper/
+pip install -e .[dev]
+```
+
+## Validate the changes before creating a pull request
+
+1. Make sure the existing tests are still passing (and consider adding new tests as well!):
+
+```bash
+pytest tests/
+```
+
+2. Reformat and validate the code with the following tools:
+
+```bash
+black .
+isort .
+flake8 .
+```
+
+These steps are also run automatically in the CI when you open the pull request.
--- a/2
+++ b/2
@@ -1,6 +1,6 @@
 MIT License

-Copyright (c) 2023 Guillaume Klein
+Copyright (c) 2023 SYSTRAN

 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1 +1,3 @@
 include faster_whisper/assets/silero_vad.onnx
+include requirements.txt
+include requirements.conversion.txt
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-[![CI](https://github.com/guillaumekln/faster-whisper/workflows/CI/badge.svg)](https://github.com/guillaumekln/faster-whisper/actions?query=workflow%3ACI) [![PyPI version](https://badge.fury.io/py/faster-whisper.svg)](https://badge.fury.io/py/faster-whisper)
+[![CI](https://github.com/SYSTRAN/faster-whisper/workflows/CI/badge.svg)](https://github.com/SYSTRAN/faster-whisper/actions?query=workflow%3ACI) [![PyPI version](https://badge.fury.io/py/faster-whisper.svg)](https://badge.fury.io/py/faster-whisper)

 # Faster Whisper transcription with CTranslate2

@@ -8,11 +8,13 @@ This implementation is up to 4 times faster than [openai/whisper](https://github

 ## Benchmark

+### Whisper
+
 For reference, here's the time and memory usage that are required to transcribe [**13 minutes**](https://www.youtube.com/watch?v=0u7tTptBo9I) of audio using different implementations:

 * [openai/whisper](https://github.com/openai/whisper)@[6dea21fd](https://github.com/openai/whisper/commit/6dea21fd7f7253bfe450f1e2512a0fe47ee2d258)
 * [whisper.cpp](https://github.com/ggerganov/whisper.cpp)@[3b010f9](https://github.com/ggerganov/whisper.cpp/commit/3b010f9bed9a6068609e9faf52383aea792b0362)
-* [faster-whisper](https://github.com/guillaumekln/faster-whisper)@[cce6b53e](https://github.com/guillaumekln/faster-whisper/commit/cce6b53e4554f71172dad188c45f10fb100f6e3e)
+* [faster-whisper](https://github.com/SYSTRAN/faster-whisper)@[cce6b53e](https://github.com/SYSTRAN/faster-whisper/commit/cce6b53e4554f71172dad188c45f10fb100f6e3e)

 ### Large-v2 model on GPU

@@ -36,6 +38,71 @@ For reference, here's the time and memory usage that are required to transcribe

 *Executed with 8 threads on a Intel(R) Xeon(R) Gold 6226R.*

+
+### Distil-whisper
+
+| Implementation | Precision | Beam size | Time | Gigaspeech WER |
+| --- | --- | --- | --- | --- |
+| distil-whisper/distil-large-v2 | fp16 | 4 |- | 10.36 |
+| [faster-distil-large-v2](https://huggingface.co/Systran/faster-distil-whisper-large-v2) | fp16 | 5 | - | 10.28 |
+| distil-whisper/distil-medium.en | fp16 | 4 | - | 11.21 |
+| [faster-distil-medium.en](https://huggingface.co/Systran/faster-distil-whisper-medium.en) | fp16 | 5 | - | 11.21 |
+
+*Executed with CUDA 11.4 on a NVIDIA 3090.*
+
+<details>
+<summary>testing details (click to expand)</summary>
+
+For `distil-whisper/distil-large-v2`, the WER is tested with code sample from [link](https://huggingface.co/distil-whisper/distil-large-v2#evaluation). for `faster-distil-whisper`, the WER is tested with setting:
+```python
+from faster_whisper import WhisperModel
+
+model_size = "distil-large-v2"
+# model_size = "distil-medium.en"
+# Run on GPU with FP16
+model = WhisperModel(model_size, device="cuda", compute_type="float16")
+segments, info = model.transcribe("audio.mp3", beam_size=5, language="en")
+```
+</details>
+
+## Requirements
+
+* Python 3.8 or greater
+
+Unlike openai-whisper, FFmpeg does **not** need to be installed on the system. The audio is decoded with the Python library [PyAV](https://github.com/PyAV-Org/PyAV) which bundles the FFmpeg libraries in its package.
+
+### GPU
+
+GPU execution requires the following NVIDIA libraries to be installed:
+
+* [cuBLAS for CUDA 11](https://developer.nvidia.com/cublas)
+* [cuDNN 8 for CUDA 11](https://developer.nvidia.com/cudnn)
+
+There are multiple ways to install these libraries. The recommended way is described in the official NVIDIA documentation, but we also suggest other installation methods below.
+
+<details>
+<summary>Other installation methods (click to expand)</summary>
+
+#### Use Docker
+
+The libraries are installed in this official NVIDIA Docker image: `nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04`.
+
+#### Install with `pip` (Linux only)
+
+On Linux these libraries can be installed with `pip`. Note that `LD_LIBRARY_PATH` must be set before launching Python.
+
+```bash
+pip install nvidia-cublas-cu11 nvidia-cudnn-cu11
+
+export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
+```
+
+#### Download the libraries from Purfview's repository (Windows & Linux)
+
+Purfview's [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) provides the required NVIDIA libraries for Windows & Linux in a [single archive](https://github.com/Purfview/whisper-standalone-win/releases/tag/libs). Decompress the archive and place the libraries in a directory included in the `PATH`.
+
+</details>
+
 ## Installation

 The module can be installed from [PyPI](https://pypi.org/project/faster-whisper/):
@@ -44,32 +111,31 @@ The module can be installed from [PyPI](https://pypi.org/project/faster-whisper/
 pip install faster-whisper
 ```

-**Other installation methods:**
+<details>
+<summary>Other installation methods (click to expand)</summary>
+
+### Install the master branch

 ```bash
-# Install the master branch:
-pip install --force-reinstall "faster-whisper @ https://github.com/guillaumekln/faster-whisper/archive/refs/heads/master.tar.gz"
-
-# Install a specific commit:
-pip install --force-reinstall "faster-whisper @ https://github.com/guillaumekln/faster-whisper/archive/a4f1cc8f11433e454c3934442b5e1a4ed5e865c3.tar.gz"
-
-# Install for development:
-git clone https://github.com/guillaumekln/faster-whisper.git
-pip install -e faster-whisper/
+pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/refs/heads/master.tar.gz"
 ```

-### GPU support
+### Install a specific commit

-GPU execution requires the NVIDIA libraries cuBLAS 11.x and cuDNN 8.x to be installed on the system. Please refer to the [CTranslate2 documentation](https://opennmt.net/CTranslate2/installation.html).
+```bash
+pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/a4f1cc8f11433e454c3934442b5e1a4ed5e865c3.tar.gz"
+```
+
+</details>

 ## Usage

-### Library
+### Faster-whisper

 ```python
 from faster_whisper import WhisperModel

-model_size = "large-v2"
+model_size = "large-v3"

 # Run on GPU with FP16
 model = WhisperModel(model_size, device="cuda", compute_type="float16")
@@ -93,8 +159,27 @@ for segment in segments:
 segments, _ = model.transcribe("audio.mp3")
 segments = list(segments)  # The transcription will actually run here.
 ```
+### Faster Distil-Whisper

-#### Word-level timestamps
+The Distil-Whisper checkpoints are compatible with the Faster-Whisper package. In particular, the latest [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)
+checkpoint is intrinsically designed to work with the Faster-Whisper transcription algorithm. The following code snippet 
+demonstrates how to run inference with distil-large-v3 on a specified audio file:
+
+```python
+from faster_whisper import WhisperModel
+
+model_size = "distil-large-v3"
+
+model = WhisperModel(model_size, device="cuda", compute_type="float16")
+segments, info = model.transcribe("audio.mp3", beam_size=5, language="en", condition_on_previous_text=False)
+
+for segment in segments:
+    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
+```
+
+For more information about the distil-large-v3 model, refer to the original [model card](https://huggingface.co/distil-whisper/distil-large-v3).
+
+### Word-level timestamps

 ```python
 segments, _ = model.transcribe("audio.mp3", word_timestamps=True)
@@ -104,7 +189,7 @@ for segment in segments:
        print("[%.2fs -> %.2fs] %s" % (word.start, word.end, word.word))
 ```

-#### VAD filter
+### VAD filter

 The library integrates the [Silero VAD](https://github.com/snakers4/silero-vad) model to filter out parts of the audio without speech:

@@ -112,33 +197,61 @@ The library integrates the [Silero VAD](https://github.com/snakers4/silero-vad)
 segments, _ = model.transcribe("audio.mp3", vad_filter=True)
 ```

-The default behavior is conservative and only removes silence longer than 2 seconds. See the available VAD parameters and default values in the function [`get_speech_timestamps`](https://github.com/guillaumekln/faster-whisper/blob/master/faster_whisper/vad.py). They can be customized with the dictionary argument `vad_parameters`:
+The default behavior is conservative and only removes silence longer than 2 seconds. See the available VAD parameters and default values in the [source code](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/vad.py). They can be customized with the dictionary argument `vad_parameters`:

 ```python
-segments, _ = model.transcribe("audio.mp3", vad_filter=True, vad_parameters=dict(min_silence_duration_ms=500))
+segments, _ = model.transcribe(
+    "audio.mp3",
+    vad_filter=True,
+    vad_parameters=dict(min_silence_duration_ms=500),
+)
 ```

-#### Going further
+### Logging

-See more model and transcription options in the [`WhisperModel`](https://github.com/guillaumekln/faster-whisper/blob/master/faster_whisper/transcribe.py) class implementation.
+The library logging level can be configured like this:

-### CLI
+```python
+import logging

-You can use [jordimas/whisper-ctranslate2](https://github.com/jordimas/whisper-ctranslate2) to access `faster-whisper` through a CLI interface similar to what is offered by Whisper.
+logging.basicConfig()
+logging.getLogger("faster_whisper").setLevel(logging.DEBUG)
+```
+
+### Going further
+
+See more model and transcription options in the [`WhisperModel`](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/transcribe.py) class implementation.
+
+## Community integrations
+
+Here is a non exhaustive list of open-source projects using faster-whisper. Feel free to add your project to the list!
+
+
+* [WhisperX](https://github.com/m-bain/whisperX) is an award-winning Python library that offers speaker diarization and accurate word-level timestamps using wav2vec2 alignment
+* [whisper-ctranslate2](https://github.com/Softcatala/whisper-ctranslate2) is a command line client based on faster-whisper and compatible with the original client from openai/whisper.
+* [whisper-diarize](https://github.com/MahmoudAshraf97/whisper-diarization) is a speaker diarization tool that is based on faster-whisper and NVIDIA NeMo.
+* [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) Standalone CLI executables of faster-whisper for Windows, Linux & macOS. 
+* [asr-sd-pipeline](https://github.com/hedrergudene/asr-sd-pipeline) provides a scalable, modular, end to end multi-speaker speech to text solution implemented using AzureML pipelines.
+* [Open-Lyrics](https://github.com/zh-plus/Open-Lyrics) is a Python library that transcribes voice files using faster-whisper, and translates/polishes the resulting text into `.lrc` files in the desired language using OpenAI-GPT.
+* [wscribe](https://github.com/geekodour/wscribe) is a flexible transcript generation tool supporting faster-whisper, it can export word level transcript and the exported transcript then can be edited with [wscribe-editor](https://github.com/geekodour/wscribe-editor)
+* [aTrain](https://github.com/BANDAS-Center/aTrain) is a graphical user interface implementation of faster-whisper developed at the BANDAS-Center at the University of Graz for transcription and diarization in Windows ([Windows Store App](https://apps.microsoft.com/detail/atrain/9N15Q44SZNS2)) and Linux.
+* [Whisper-Streaming](https://github.com/ufal/whisper_streaming) implements real-time mode for offline Whisper-like speech-to-text models with faster-whisper as the most recommended back-end. It implements a streaming policy with self-adaptive latency based on the actual source complexity, and demonstrates the state of the art.
+* [WhisperLive](https://github.com/collabora/WhisperLive) is a nearly-live implementation of OpenAI's Whisper which uses faster-whisper as the backend to transcribe audio in real-time.
+* [Faster-Whisper-Transcriber](https://github.com/BBC-Esq/ctranslate2-faster-whisper-transcriber) is a simple but reliable voice transcriber that provides a user-friendly interface.

 ## Model conversion

-When loading a model from its size such as `WhisperModel("large-v2")`, the correspondig CTranslate2 model is automatically downloaded from the [Hugging Face Hub](https://huggingface.co/guillaumekln).
+When loading a model from its size such as `WhisperModel("large-v3")`, the corresponding CTranslate2 model is automatically downloaded from the [Hugging Face Hub](https://huggingface.co/Systran).

 We also provide a script to convert any Whisper models compatible with the Transformers library. They could be the original OpenAI models or user fine-tuned models.

-For example the command below converts the [original "large-v2" Whisper model](https://huggingface.co/openai/whisper-large-v2) and saves the weights in FP16:
+For example the command below converts the [original "large-v3" Whisper model](https://huggingface.co/openai/whisper-large-v3) and saves the weights in FP16:

 ```bash
 pip install transformers[torch]>=4.23

-ct2-transformers-converter --model openai/whisper-large-v2 --output_dir whisper-large-v2-ct2 \
-    --copy_files tokenizer.json --quantization float16
+ct2-transformers-converter --model openai/whisper-large-v3 --output_dir whisper-large-v3-ct2
+--copy_files tokenizer.json preprocessor_config.json --quantization float16
 ```

 * The option `--model` accepts a model name on the Hub or a path to a model directory.
@@ -146,6 +259,18 @@ ct2-transformers-converter --model openai/whisper-large-v2 --output_dir whisper-

 Models can also be converted from the code. See the [conversion API](https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html).

+### Load a converted model
+
+1. Directly load the model from a local directory:
+```python
+model = faster_whisper.WhisperModel("whisper-large-v3-ct2")
+```
+
+2. [Upload your model to the Hugging Face Hub](https://huggingface.co/docs/transformers/model_sharing#upload-with-the-web-interface) and load it from its name:
+```python
+model = faster_whisper.WhisperModel("username/whisper-large-v3-ct2")
+```
+
 ## Comparing performance against other implementations

 If you are comparing the performance against other Whisper implementations, you should make sure to run the comparison with similar settings. In particular:
--- a/faster_whisper/init.py
+++ b/faster_whisper/init.py
@@ -1,10 +1,13 @@
 from faster_whisper.audio import decode_audio
 from faster_whisper.transcribe import WhisperModel
-from faster_whisper.utils import download_model, format_timestamp
+from faster_whisper.utils import available_models, download_model, format_timestamp
+from faster_whisper.version import __version__

 __all__ = [
+    "available_models",
    "decode_audio",
    "WhisperModel",
    "download_model",
    "format_timestamp",
+    "__version__",
 ]
--- a/faster_whisper/assets/init.py
+++ b/faster_whisper/assets/init.py
--- a/faster_whisper/audio.py
+++ b/faster_whisper/audio.py
@@ -6,6 +6,7 @@ system dependencies. FFmpeg does not need to be installed on the system.
 However, the API is quite low-level so we need to manipulate audio frames directly.
 """

+import gc
 import io
 import itertools

@@ -42,7 +43,7 @@ def decode_audio(
    raw_buffer = io.BytesIO()
    dtype = None

-    with av.open(input_file, metadata_errors="ignore") as container:
+    with av.open(input_file, mode="r", metadata_errors="ignore") as container:
        frames = container.decode(audio=0)
        frames = _ignore_invalid_frames(frames)
        frames = _group_frames(frames, 500000)
@@ -53,6 +54,11 @@ def decode_audio(
            dtype = array.dtype
            raw_buffer.write(array)

+    # It appears that some objects related to the resampler are not freed
+    # unless the garbage collector is manually run.
+    del resampler
+    gc.collect()
+
    audio = np.frombuffer(raw_buffer.getbuffer(), dtype=dtype)

    # Convert s16 back to f32.
@@ -96,3 +102,18 @@ def _resample_frames(frames, resampler):
    # Add None to flush the resampler.
    for frame in itertools.chain(frames, [None]):
        yield from resampler.resample(frame)
+
+
+def pad_or_trim(array, length: int, *, axis: int = -1):
+    """
+    Pad or trim the audio array to N_SAMPLES, as expected by the encoder.
+    """
+    if array.shape[axis] > length:
+        array = array.take(indices=range(length), axis=axis)
+
+    if array.shape[axis] < length:
+        pad_widths = [(0, 0)] * array.ndim
+        pad_widths[axis] = (0, length - array.shape[axis])
+        array = np.pad(array, pad_widths)
+
+    return array
--- a/faster_whisper/feature_extractor.py
+++ b/faster_whisper/feature_extractor.py
@@ -142,11 +142,15 @@ class FeatureExtractor:
            data[f] = np.fft.fft(fft_signal, axis=0)[:num_fft_bins]
        return data.T

-    def __call__(self, waveform, padding=True):
+    def __call__(self, waveform, padding=True, chunk_length=None):
        """
        Compute the log-Mel spectrogram of the provided audio, gives similar results
        whisper's original torch implementation with 1e-5 tolerance.
        """
+        if chunk_length is not None:
+            self.n_samples = chunk_length * self.sampling_rate
+            self.nb_max_frames = self.n_samples // self.hop_length
+
        if padding:
            waveform = np.pad(waveform, [(0, self.n_samples)])

--- a/faster_whisper/tokenizer.py
+++ b/faster_whisper/tokenizer.py
@@ -19,15 +19,21 @@ class Tokenizer:
        self.tokenizer = tokenizer

        if multilingual:
+            if task not in _TASKS:
+                raise ValueError(
+                    "'%s' is not a valid task (accepted tasks: %s)"
+                    % (task, ", ".join(_TASKS))
+                )
+
+            if language not in _LANGUAGE_CODES:
+                raise ValueError(
+                    "'%s' is not a valid language code (accepted language codes: %s)"
+                    % (language, ", ".join(_LANGUAGE_CODES))
+                )
+
            self.task = self.tokenizer.token_to_id("<|%s|>" % task)
-            if self.task is None:
-                raise ValueError("%s is not a valid task" % task)
-
-            self.language_code = language
            self.language = self.tokenizer.token_to_id("<|%s|>" % language)
-            if self.language is None:
-                raise ValueError("%s is not a valid language code" % language)
-
+            self.language_code = language
        else:
            self.task = None
            self.language = None
@@ -102,7 +108,7 @@ class Tokenizer:
    def split_to_word_tokens(
        self, tokens: List[int]
    ) -> Tuple[List[str], List[List[int]]]:
-        if self.language_code in {"zh", "ja", "th", "lo", "my"}:
+        if self.language_code in {"zh", "ja", "th", "lo", "my", "yue"}:
            # These languages don't typically use spaces, so it is difficult to split words
            # without morpheme analysis. Here, we instead split words at any
            # position where the tokens are decoded as valid unicode points
@@ -161,3 +167,112 @@ class Tokenizer:
                word_tokens[-1].extend(subword_tokens)

        return words, word_tokens
+
+
+_TASKS = (
+    "transcribe",
+    "translate",
+)
+
+_LANGUAGE_CODES = (
+    "af",
+    "am",
+    "ar",
+    "as",
+    "az",
+    "ba",
+    "be",
+    "bg",
+    "bn",
+    "bo",
+    "br",
+    "bs",
+    "ca",
+    "cs",
+    "cy",
+    "da",
+    "de",
+    "el",
+    "en",
+    "es",
+    "et",
+    "eu",
+    "fa",
+    "fi",
+    "fo",
+    "fr",
+    "gl",
+    "gu",
+    "ha",
+    "haw",
+    "he",
+    "hi",
+    "hr",
+    "ht",
+    "hu",
+    "hy",
+    "id",
+    "is",
+    "it",
+    "ja",
+    "jw",
+    "ka",
+    "kk",
+    "km",
+    "kn",
+    "ko",
+    "la",
+    "lb",
+    "ln",
+    "lo",
+    "lt",
+    "lv",
+    "mg",
+    "mi",
+    "mk",
+    "ml",
+    "mn",
+    "mr",
+    "ms",
+    "mt",
+    "my",
+    "ne",
+    "nl",
+    "nn",
+    "no",
+    "oc",
+    "pa",
+    "pl",
+    "ps",
+    "pt",
+    "ro",
+    "ru",
+    "sa",
+    "sd",
+    "si",
+    "sk",
+    "sl",
+    "sn",
+    "so",
+    "sq",
+    "sr",
+    "su",
+    "sv",
+    "sw",
+    "ta",
+    "te",
+    "tg",
+    "th",
+    "tk",
+    "tl",
+    "tr",
+    "tt",
+    "uk",
+    "ur",
+    "uz",
+    "vi",
+    "yi",
+    "yo",
+    "zh",
+    "yue",
+)
--- a/faster_whisper/transcribe.py
+++ b/faster_whisper/transcribe.py
@@ -1,20 +1,23 @@
 import itertools
+import json
 import logging
 import os
 import zlib

+from inspect import signature
 from typing import BinaryIO, Iterable, List, NamedTuple, Optional, Tuple, Union

 import ctranslate2
 import numpy as np
 import tokenizers

-from faster_whisper.audio import decode_audio
+from faster_whisper.audio import decode_audio, pad_or_trim
 from faster_whisper.feature_extractor import FeatureExtractor
-from faster_whisper.tokenizer import Tokenizer
-from faster_whisper.utils import download_model, format_timestamp, get_logger
+from faster_whisper.tokenizer import _LANGUAGE_CODES, Tokenizer
+from faster_whisper.utils import download_model, format_timestamp, get_end, get_logger
 from faster_whisper.vad import (
    SpeechTimestampsMap,
+    VadOptions,
    collect_chunks,
    get_speech_timestamps,
 )
@@ -28,18 +31,17 @@ class Word(NamedTuple):


 class Segment(NamedTuple):
+    id: int
+    seek: int
    start: float
    end: float
    text: str
-    words: Optional[List[Word]]
-    avg_log_prob: float
+    tokens: List[int]
+    temperature: float
+    avg_logprob: float
+    compression_ratio: float
    no_speech_prob: float
-
-
-class AudioInfo(NamedTuple):
-    language: str
-    language_probability: float
-    duration: float
+    words: Optional[List[Word]]


 class TranscriptionOptions(NamedTuple):
@@ -47,12 +49,15 @@ class TranscriptionOptions(NamedTuple):
    best_of: int
    patience: float
    length_penalty: float
+    repetition_penalty: float
+    no_repeat_ngram_size: int
    log_prob_threshold: Optional[float]
    no_speech_threshold: Optional[float]
    compression_ratio_threshold: Optional[float]
    condition_on_previous_text: bool
+    prompt_reset_on_temperature: float
    temperatures: List[float]
-    initial_prompt: Optional[str]
+    initial_prompt: Optional[Union[str, Iterable[int]]]
    prefix: Optional[str]
    suppress_blank: bool
    suppress_tokens: Optional[List[int]]
@@ -61,6 +66,19 @@ class TranscriptionOptions(NamedTuple):
    word_timestamps: bool
    prepend_punctuations: str
    append_punctuations: str
+    max_new_tokens: Optional[int]
+    clip_timestamps: Union[str, List[float]]
+    hallucination_silence_threshold: Optional[float]
+
+
+class TranscriptionInfo(NamedTuple):
+    language: str
+    language_probability: float
+    duration: float
+    duration_after_vad: float
+    all_language_probs: Optional[List[Tuple[str, float]]]
+    transcription_options: TranscriptionOptions
+    vad_options: VadOptions


 class WhisperModel:
@@ -73,13 +91,15 @@ class WhisperModel:
        cpu_threads: int = 0,
        num_workers: int = 1,
        download_root: Optional[str] = None,
+        local_files_only: bool = False,
    ):
        """Initializes the Whisper model.

        Args:
          model_size_or_path: Size of the model to use (tiny, tiny.en, base, base.en,
-            small, small.en, medium, medium.en, large-v1, or large-v2) or a path to a converted
-            model directory. When a size is configured, the converted model is downloaded
+            small, small.en, medium, medium.en, large-v1, large-v2, large-v3, or large), a path to a
+            converted model directory, or a CTranslate2-converted Whisper model ID from the HF Hub.
+            When a size or a model ID is configured, the converted model is downloaded
            from the Hugging Face Hub.
          device: Device to use for computation ("cpu", "cuda", "auto").
          device_index: Device ID to use.
@@ -94,15 +114,21 @@ class WhisperModel:
            having multiple workers enables true parallelism when running the model
            (concurrent calls to self.model.generate() will run in parallel).
            This can improve the global throughput at the cost of increased memory usage.
-          download_root: Directory where the model should be saved. If not set, the model
-            is saved in the standard Hugging Face cache directory.
+          download_root: Directory where the models should be saved. If not set, the models
+            are saved in the standard Hugging Face cache directory.
+          local_files_only:  If True, avoid downloading the file and return the path to the
+            local cached file if it exists.
        """
        self.logger = get_logger()

        if os.path.isdir(model_size_or_path):
            model_path = model_size_or_path
        else:
-            model_path = download_model(model_size_or_path, download_root)
+            model_path = download_model(
+                model_size_or_path,
+                local_files_only=local_files_only,
+                cache_dir=download_root,
+            )

        self.model = ctranslate2.models.Whisper(
            model_path,
@@ -121,7 +147,8 @@ class WhisperModel:
                "openai/whisper-tiny" + ("" if self.model.is_multilingual else ".en")
            )

-        self.feature_extractor = FeatureExtractor()
+        self.feat_kwargs = self._get_feature_kwargs(model_path)
+        self.feature_extractor = FeatureExtractor(**self.feat_kwargs)
        self.num_samples_per_token = self.feature_extractor.hop_length * 2
        self.frames_per_second = (
            self.feature_extractor.sampling_rate // self.feature_extractor.hop_length
@@ -133,6 +160,27 @@ class WhisperModel:
        self.time_precision = 0.02
        self.max_length = 448

+    @property
+    def supported_languages(self) -> List[str]:
+        """The languages supported by the model."""
+        return list(_LANGUAGE_CODES) if self.model.is_multilingual else ["en"]
+
+    def _get_feature_kwargs(self, model_path) -> dict:
+        preprocessor_config_file = os.path.join(model_path, "preprocessor_config.json")
+        config = {}
+        if os.path.isfile(preprocessor_config_file):
+            try:
+                with open(preprocessor_config_file, "r", encoding="utf-8") as json_file:
+                    config = json.load(json_file)
+                valid_keys = signature(FeatureExtractor.__init__).parameters.keys()
+                config = {k: v for k, v in config.items() if k in valid_keys}
+            except json.JSONDecodeError as e:
+                self.logger.warning(
+                    "Could not load preprocessor_config.json: %s", str(e)
+                )
+
+        return config
+
    def transcribe(
        self,
        audio: Union[str, BinaryIO, np.ndarray],
@@ -142,6 +190,8 @@ class WhisperModel:
        best_of: int = 5,
        patience: float = 1,
        length_penalty: float = 1,
+        repetition_penalty: float = 1,
+        no_repeat_ngram_size: int = 0,
        temperature: Union[float, List[float], Tuple[float, ...]] = [
            0.0,
            0.2,
@@ -154,7 +204,8 @@ class WhisperModel:
        log_prob_threshold: Optional[float] = -1.0,
        no_speech_threshold: Optional[float] = 0.6,
        condition_on_previous_text: bool = True,
-        initial_prompt: Optional[str] = None,
+        prompt_reset_on_temperature: float = 0.5,
+        initial_prompt: Optional[Union[str, Iterable[int]]] = None,
        prefix: Optional[str] = None,
        suppress_blank: bool = True,
        suppress_tokens: Optional[List[int]] = [-1],
@@ -164,8 +215,14 @@ class WhisperModel:
        prepend_punctuations: str = "\"'“¿([{-",
        append_punctuations: str = "\"'.。,，!！?？:：”)]}、",
        vad_filter: bool = False,
-        vad_parameters: Optional[dict] = None,
-    ) -> Tuple[Iterable[Segment], AudioInfo]:
+        vad_parameters: Optional[Union[dict, VadOptions]] = None,
+        max_new_tokens: Optional[int] = None,
+        chunk_length: Optional[int] = None,
+        clip_timestamps: Union[str, List[float]] = "0",
+        hallucination_silence_threshold: Optional[float] = None,
+        language_detection_threshold: Optional[float] = None,
+        language_detection_segments: int = 1,
+    ) -> Tuple[Iterable[Segment], TranscriptionInfo]:
        """Transcribes an input file.

        Arguments:
@@ -178,6 +235,9 @@ class WhisperModel:
          best_of: Number of candidates when sampling with non-zero temperature.
          patience: Beam search patience factor.
          length_penalty: Exponential length penalty constant.
+          repetition_penalty: Penalty applied to the score of previously generated tokens
+            (set > 1 to penalize).
+          no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable).
          temperature: Temperature for sampling. It can be a tuple of temperatures,
            which will be successively used upon failures according to either
            `compression_ratio_threshold` or `log_prob_threshold`.
@@ -192,7 +252,10 @@ class WhisperModel:
            as a prompt for the next window; disabling may make the text inconsistent across
            windows, but the model becomes less prone to getting stuck in a failure loop,
            such as repetition looping or timestamps going out of sync.
-          initial_prompt: Optional text to provide as a prompt for the first window.
+          prompt_reset_on_temperature: Resets prompt if temperature is above this value.
+            Arg has effect only if condition_on_previous_text is True.
+          initial_prompt: Optional text string or iterable of token ids to provide as a
+            prompt for the first window.
          prefix: Optional text to provide as a prefix for the first window.
          suppress_blank: Suppress blank outputs at the beginning of the sampling.
          suppress_tokens: List of token IDs to suppress. -1 will suppress a default set
@@ -208,14 +271,28 @@ class WhisperModel:
          vad_filter: Enable the voice activity detection (VAD) to filter out parts of the audio
            without speech. This step is using the Silero VAD model
            https://github.com/snakers4/silero-vad.
-          vad_parameters: Dictionary of Silero VAD parameters (see available parameters and
-            default values in the function `get_speech_timestamps`).
+          vad_parameters: Dictionary of Silero VAD parameters or VadOptions class (see available
+            parameters and default values in the class `VadOptions`).
+          max_new_tokens: Maximum number of new tokens to generate per-chunk. If not set,
+            the maximum will be set by the default max_length.
+          chunk_length: The length of audio segments. If it is not None, it will overwrite the
+            default chunk_length of the FeatureExtractor.
+          clip_timestamps: Union[str, List[float]]
+            Comma-separated list start,end,start,end,... timestamps (in seconds) of clips to
+             process. The last end timestamp defaults to the end of the file.
+             vad_filter will be ignored if clip_timestamps is used.
+          hallucination_silence_threshold: Optional[float]
+            When word_timestamps is True, skip silent periods longer than this threshold
+             (in seconds) when a possible hallucination is detected
+          language_detection_threshold: If the maximum probability of the language tokens is higher
+           than this value, the language is detected.
+          language_detection_segments: Number of segments to consider for the language detection.

        Returns:
          A tuple with:

            - a generator over transcribed segments
-            - an instance of AudioInfo
+            - an instance of TranscriptionInfo
        """
        sampling_rate = self.feature_extractor.sampling_rate

@@ -223,19 +300,24 @@ class WhisperModel:
            audio = decode_audio(audio, sampling_rate=sampling_rate)

        duration = audio.shape[0] / sampling_rate
+        duration_after_vad = duration

        self.logger.info(
            "Processing audio with duration %s", format_timestamp(duration)
        )

-        if vad_filter:
-            vad_parameters = {} if vad_parameters is None else vad_parameters
-            speech_chunks = get_speech_timestamps(audio, **vad_parameters)
+        if vad_filter and clip_timestamps == "0":
+            if vad_parameters is None:
+                vad_parameters = VadOptions()
+            elif isinstance(vad_parameters, dict):
+                vad_parameters = VadOptions(**vad_parameters)
+            speech_chunks = get_speech_timestamps(audio, vad_parameters)
            audio = collect_chunks(audio, speech_chunks)
+            duration_after_vad = audio.shape[0] / sampling_rate

            self.logger.info(
                "VAD filter removed %s of audio",
-                format_timestamp(duration - (audio.shape[0] / sampling_rate)),
+                format_timestamp(duration - duration_after_vad),
            )

            if self.logger.isEnabledFor(logging.DEBUG):
@@ -254,20 +336,61 @@ class WhisperModel:
        else:
            speech_chunks = None

-        features = self.feature_extractor(audio)
+        features = self.feature_extractor(audio, chunk_length=chunk_length)

        encoder_output = None
+        all_language_probs = None

        if language is None:
            if not self.model.is_multilingual:
                language = "en"
                language_probability = 1
            else:
-                segment = features[:, : self.feature_extractor.nb_max_frames]
-                encoder_output = self.encode(segment)
-                results = self.model.detect_language(encoder_output)
-                language_token, language_probability = results[0][0]
-                language = language_token[2:-2]
+                if (
+                    language_detection_segments is None
+                    or language_detection_segments < 1
+                ):
+                    language_detection_segments = 1
+                seek = 0
+                detected_language_info = {}
+                content_frames = (
+                    features.shape[-1] - self.feature_extractor.nb_max_frames
+                )
+                while (
+                    seek <= content_frames
+                    and seek
+                    < self.feature_extractor.nb_max_frames * language_detection_segments
+                ):
+                    segment = features[
+                        :, seek : seek + self.feature_extractor.nb_max_frames
+                    ]
+                    encoder_output = self.encode(segment)
+                    # results is a list of tuple[str, float] with language names and
+                    # probabilities.
+                    results = self.model.detect_language(encoder_output)[0]
+                    # Parse language names to strip out markers
+                    all_language_probs = [
+                        (token[2:-2], prob) for (token, prob) in results
+                    ]
+                    # Get top language token and probability
+                    language, language_probability = all_language_probs[0]
+                    if (
+                        language_detection_threshold is None
+                        or language_probability > language_detection_threshold
+                    ):
+                        break
+                    detected_language_info.setdefault(language, []).append(
+                        language_probability
+                    )
+                    seek += segment.shape[-1]
+                else:
+                    # If no language detected for all segments, the majority vote of the highest
+                    # projected languages for all segments is used to determine the language.
+                    language = max(
+                        detected_language_info,
+                        key=lambda lang: len(detected_language_info[lang]),
+                    )
+                    language_probability = max(detected_language_info[language])

                self.logger.info(
                    "Detected language '%s' with probability %.2f",
@@ -275,6 +398,13 @@ class WhisperModel:
                    language_probability,
                )
        else:
+            if not self.model.is_multilingual and language != "en":
+                self.logger.warning(
+                    "The current model is English-only but the language parameter is set to '%s'; "
+                    "using 'en' instead." % language
+                )
+                language = "en"
+
            language_probability = 1

        tokenizer = Tokenizer(
@@ -289,10 +419,13 @@ class WhisperModel:
            best_of=best_of,
            patience=patience,
            length_penalty=length_penalty,
+            repetition_penalty=repetition_penalty,
+            no_repeat_ngram_size=no_repeat_ngram_size,
            log_prob_threshold=log_prob_threshold,
            no_speech_threshold=no_speech_threshold,
            compression_ratio_threshold=compression_ratio_threshold,
            condition_on_previous_text=condition_on_previous_text,
+            prompt_reset_on_temperature=prompt_reset_on_temperature,
            temperatures=(
                temperature if isinstance(temperature, (list, tuple)) else [temperature]
            ),
@@ -305,6 +438,9 @@ class WhisperModel:
            word_timestamps=word_timestamps,
            prepend_punctuations=prepend_punctuations,
            append_punctuations=append_punctuations,
+            max_new_tokens=max_new_tokens,
+            clip_timestamps=clip_timestamps,
+            hallucination_silence_threshold=hallucination_silence_threshold,
        )

        segments = self.generate_segments(features, tokenizer, options, encoder_output)
@@ -312,13 +448,17 @@ class WhisperModel:
        if speech_chunks:
            segments = restore_speech_timestamps(segments, speech_chunks, sampling_rate)

-        audio_info = AudioInfo(
+        info = TranscriptionInfo(
            language=language,
            language_probability=language_probability,
            duration=duration,
+            duration_after_vad=duration_after_vad,
+            transcription_options=options,
+            vad_options=vad_parameters,
+            all_language_probs=all_language_probs,
        )

-        return segments, audio_info
+        return segments, info

    def generate_segments(
        self,
@@ -328,22 +468,73 @@ class WhisperModel:
        encoder_output: Optional[ctranslate2.StorageView] = None,
    ) -> Iterable[Segment]:
        content_frames = features.shape[-1] - self.feature_extractor.nb_max_frames
-        seek = 0
+        content_duration = float(content_frames * self.feature_extractor.time_per_frame)
+
+        if isinstance(options.clip_timestamps, str):
+            TranscriptionOptions.clip_timestamps = [
+                float(ts)
+                for ts in (
+                    options.clip_timestamps.split(",")
+                    if options.clip_timestamps
+                    else []
+                )
+            ]
+        seek_points: List[int] = [
+            round(ts * self.frames_per_second) for ts in options.clip_timestamps
+        ]
+        if len(seek_points) == 0:
+            seek_points.append(0)
+        if len(seek_points) % 2 == 1:
+            seek_points.append(content_frames)
+        seek_clips: List[Tuple[int, int]] = list(
+            zip(seek_points[::2], seek_points[1::2])
+        )
+
+        punctuation = "\"'“¿([{-\"'.。,，!！?？:：”)]}、"
+
+        idx = 0
+        clip_idx = 0
+        seek = seek_clips[clip_idx][0]
        all_tokens = []
        prompt_reset_since = 0

        if options.initial_prompt is not None:
-            initial_prompt = " " + options.initial_prompt.strip()
-            initial_prompt_tokens = tokenizer.encode(initial_prompt)
-            all_tokens.extend(initial_prompt_tokens)
+            if isinstance(options.initial_prompt, str):
+                initial_prompt = " " + options.initial_prompt.strip()
+                initial_prompt_tokens = tokenizer.encode(initial_prompt)
+                all_tokens.extend(initial_prompt_tokens)
+            else:
+                all_tokens.extend(options.initial_prompt)

-        while seek < content_frames:
+        last_speech_timestamp = 0.0
+        # NOTE: This loop is obscurely flattened to make the diff readable.
+        # A later commit should turn this into a simpler nested loop.
+        # for seek_clip_start, seek_clip_end in seek_clips:
+        #     while seek < seek_clip_end
+        while clip_idx < len(seek_clips):
+            seek_clip_start, seek_clip_end = seek_clips[clip_idx]
+            if seek_clip_end > content_frames:
+                seek_clip_end = content_frames
+            if seek < seek_clip_start:
+                seek = seek_clip_start
+            if seek >= seek_clip_end:
+                clip_idx += 1
+                if clip_idx < len(seek_clips):
+                    seek = seek_clips[clip_idx][0]
+                continue
            time_offset = seek * self.feature_extractor.time_per_frame
-            segment = features[:, seek : seek + self.feature_extractor.nb_max_frames]
-            segment_size = min(
-                self.feature_extractor.nb_max_frames, content_frames - seek
+            window_end_time = float(
+                (seek + self.feature_extractor.nb_max_frames)
+                * self.feature_extractor.time_per_frame
            )
+            segment_size = min(
+                self.feature_extractor.nb_max_frames,
+                content_frames - seek,
+                seek_clip_end - seek,
+            )
+            segment = features[:, seek : seek + segment_size]
            segment_duration = segment_size * self.feature_extractor.time_per_frame
+            segment = pad_or_trim(segment, self.feature_extractor.nb_max_frames)

            if self.logger.isEnabledFor(logging.DEBUG):
                self.logger.debug(
@@ -358,12 +549,15 @@ class WhisperModel:
                prefix=options.prefix if seek == 0 else None,
            )

-            if encoder_output is None:
+            if seek > 0 or encoder_output is None:
                encoder_output = self.encode(segment)

-            result, avg_log_prob, temperature = self.generate_with_fallback(
-                encoder_output, prompt, tokenizer, options
-            )
+            (
+                result,
+                avg_logprob,
+                temperature,
+                compression_ratio,
+            ) = self.generate_with_fallback(encoder_output, prompt, tokenizer, options)

            if options.no_speech_threshold is not None:
                # no voice activity check
@@ -371,7 +565,7 @@ class WhisperModel:

                if (
                    options.log_prob_threshold is not None
-                    and avg_log_prob > options.log_prob_threshold
+                    and avg_logprob > options.log_prob_threshold
                ):
                    # don't skip if the logprob is high enough, despite the no_speech_prob
                    should_skip = False
@@ -392,10 +586,33 @@ class WhisperModel:
            previous_seek = seek
            current_segments = []

+            # anomalous words are very long/short/improbable
+            def word_anomaly_score(word: dict) -> float:
+                probability = word.get("probability", 0.0)
+                duration = word["end"] - word["start"]
+                score = 0.0
+                if probability < 0.15:
+                    score += 1.0
+                if duration < 0.133:
+                    score += (0.133 - duration) * 15
+                if duration > 2.0:
+                    score += duration - 2.0
+                return score
+
+            def is_segment_anomaly(segment: Optional[dict]) -> bool:
+                if segment is None or not segment["words"]:
+                    return False
+                words = [w for w in segment["words"] if w["word"] not in punctuation]
+                words = words[:8]
+                score = sum(word_anomaly_score(w) for w in words)
+                return score >= 3 or score + 0.01 >= len(words)
+
+            def next_words_segment(segments: List[dict]) -> Optional[dict]:
+                return next((s for s in segments if s["words"]), None)
+
            single_timestamp_ending = (
                len(tokens) >= 2
-                and tokens[-2] < tokenizer.timestamp_begin
-                and tokens[-1] >= tokenizer.timestamp_begin
+                and tokens[-2] < tokenizer.timestamp_begin <= tokens[-1]
            )

            consecutive_timestamps = [
@@ -467,9 +684,6 @@ class WhisperModel:

                seek += segment_size

-            if not options.condition_on_previous_text or temperature > 0.5:
-                prompt_reset_since = len(all_tokens)
-
            if options.word_timestamps:
                self.add_word_timestamps(
                    current_segments,
@@ -478,21 +692,65 @@ class WhisperModel:
                    segment_size,
                    options.prepend_punctuations,
                    options.append_punctuations,
+                    last_speech_timestamp=last_speech_timestamp,
                )

-                word_end_timestamps = [
-                    w["end"] for s in current_segments for w in s["words"]
-                ]
+                if not single_timestamp_ending:
+                    last_word_end = get_end(current_segments)
+                    if last_word_end is not None and last_word_end > time_offset:
+                        seek = round(last_word_end * self.frames_per_second)

-                if not single_timestamp_ending and len(word_end_timestamps) > 0:
-                    seek_shift = round(
-                        (word_end_timestamps[-1] - time_offset) * self.frames_per_second
-                    )
+                # skip silence before possible hallucinations
+                if options.hallucination_silence_threshold is not None:
+                    threshold = options.hallucination_silence_threshold

-                    if seek_shift > 0:
-                        seek = previous_seek + seek_shift
+                    # if first segment might be a hallucination, skip leading silence
+                    first_segment = next_words_segment(current_segments)
+                    if first_segment is not None and is_segment_anomaly(first_segment):
+                        gap = first_segment["start"] - time_offset
+                        if gap > threshold:
+                            seek = previous_seek + round(gap * self.frames_per_second)
+                            continue

-            encoder_output = None
+                    # skip silence before any possible hallucination that is surrounded
+                    # by silence or more hallucinations
+                    hal_last_end = last_speech_timestamp
+                    for si in range(len(current_segments)):
+                        segment = current_segments[si]
+                        if not segment["words"]:
+                            continue
+                        if is_segment_anomaly(segment):
+                            next_segment = next_words_segment(
+                                current_segments[si + 1 :]
+                            )
+                            if next_segment is not None:
+                                hal_next_start = next_segment["words"][0]["start"]
+                            else:
+                                hal_next_start = time_offset + segment_duration
+                            silence_before = (
+                                segment["start"] - hal_last_end > threshold
+                                or segment["start"] < threshold
+                                or segment["start"] - time_offset < 2.0
+                            )
+                            silence_after = (
+                                hal_next_start - segment["end"] > threshold
+                                or is_segment_anomaly(next_segment)
+                                or window_end_time - segment["end"] < 2.0
+                            )
+                            if silence_before and silence_after:
+                                seek = round(
+                                    max(time_offset + 1, segment["start"])
+                                    * self.frames_per_second
+                                )
+                                if content_duration - segment["end"] < threshold:
+                                    seek = content_frames
+                                current_segments[si:] = []
+                                break
+                        hal_last_end = segment["end"]
+
+                last_word_end = get_end(current_segments)
+                if last_word_end is not None:
+                    last_speech_timestamp = last_word_end

            for segment in current_segments:
                tokens = segment["tokens"]
@@ -502,20 +760,39 @@ class WhisperModel:
                    continue

                all_tokens.extend(tokens)
+                idx += 1

                yield Segment(
+                    id=idx,
+                    seek=seek,
                    start=segment["start"],
                    end=segment["end"],
                    text=text,
+                    tokens=tokens,
+                    temperature=temperature,
+                    avg_logprob=avg_logprob,
+                    compression_ratio=compression_ratio,
+                    no_speech_prob=result.no_speech_prob,
                    words=(
                        [Word(**word) for word in segment["words"]]
                        if options.word_timestamps
                        else None
                    ),
-                    avg_log_prob=avg_log_prob,
-                    no_speech_prob=result.no_speech_prob,
                )

+            if (
+                not options.condition_on_previous_text
+                or temperature > options.prompt_reset_on_temperature
+            ):
+                if options.condition_on_previous_text:
+                    self.logger.debug(
+                        "Reset prompt. prompt_reset_on_temperature threshold is met %f > %f",
+                        temperature,
+                        options.prompt_reset_on_temperature,
+                    )
+
+                prompt_reset_since = len(all_tokens)
+
    def encode(self, features: np.ndarray) -> ctranslate2.StorageView:
        # When the model is running on multiple GPUs, the encoder output should be moved
        # to the CPU since we don't know which GPU will handle the next job.
@@ -532,14 +809,29 @@ class WhisperModel:
        prompt: List[int],
        tokenizer: Tokenizer,
        options: TranscriptionOptions,
-    ) -> Tuple[ctranslate2.models.WhisperGenerationResult, float, float]:
-        result = None
-        avg_log_prob = None
-        final_temperature = None
+    ) -> Tuple[ctranslate2.models.WhisperGenerationResult, float, float, float]:
+        decode_result = None
+        all_results = []
+        below_cr_threshold_results = []

        max_initial_timestamp_index = int(
            round(options.max_initial_timestamp / self.time_precision)
        )
+        if options.max_new_tokens is not None:
+            max_length = len(prompt) + options.max_new_tokens
+        else:
+            max_length = self.max_length
+
+        if max_length > self.max_length:
+            raise ValueError(
+                f"The length of the prompt is {len(prompt)}, and the `max_new_tokens` "
+                f"{max_length - len(prompt)}. Thus, the combined length of the prompt "
+                f"and `max_new_tokens` is: {max_length}. This exceeds the "
+                f"`max_length` of the Whisper model: {self.max_length}. "
+                "You should either reduce the length of your prompt, or "
+                "reduce the value of `max_new_tokens`, "
+                f"so that their combined length is less that {self.max_length}."
+            )

        for temperature in options.temperatures:
            if temperature > 0:
@@ -555,12 +847,13 @@ class WhisperModel:
                    "patience": options.patience,
                }

-            final_temperature = temperature
            result = self.model.generate(
                encoder_output,
                [prompt],
                length_penalty=options.length_penalty,
-                max_length=self.max_length,
+                repetition_penalty=options.repetition_penalty,
+                no_repeat_ngram_size=options.no_repeat_ngram_size,
+                max_length=max_length,
                return_scores=True,
                return_no_speech_prob=True,
                suppress_blank=options.suppress_blank,
@@ -573,44 +866,72 @@ class WhisperModel:

            # Recover the average log prob from the returned score.
            seq_len = len(tokens)
-            cum_log_prob = result.scores[0] * (seq_len**options.length_penalty)
-            avg_log_prob = cum_log_prob / (seq_len + 1)
+            cum_logprob = result.scores[0] * (seq_len**options.length_penalty)
+            avg_logprob = cum_logprob / (seq_len + 1)

            text = tokenizer.decode(tokens).strip()
            compression_ratio = get_compression_ratio(text)

+            decode_result = (
+                result,
+                avg_logprob,
+                temperature,
+                compression_ratio,
+            )
+            all_results.append(decode_result)
+
            needs_fallback = False

-            if (
-                options.compression_ratio_threshold is not None
-                and compression_ratio > options.compression_ratio_threshold
-            ):
-                needs_fallback = True  # too repetitive
+            if options.compression_ratio_threshold is not None:
+                if compression_ratio > options.compression_ratio_threshold:
+                    needs_fallback = True  # too repetitive

-                self.logger.debug(
-                    "Compression ratio threshold is not met with temperature %.1f (%f > %f)",
-                    temperature,
-                    compression_ratio,
-                    options.compression_ratio_threshold,
-                )
+                    self.logger.debug(
+                        "Compression ratio threshold is not met with temperature %.1f (%f > %f)",
+                        temperature,
+                        compression_ratio,
+                        options.compression_ratio_threshold,
+                    )
+                else:
+                    below_cr_threshold_results.append(decode_result)

            if (
                options.log_prob_threshold is not None
-                and avg_log_prob < options.log_prob_threshold
+                and avg_logprob < options.log_prob_threshold
            ):
                needs_fallback = True  # average log probability is too low

                self.logger.debug(
                    "Log probability threshold is not met with temperature %.1f (%f < %f)",
                    temperature,
-                    avg_log_prob,
+                    avg_logprob,
                    options.log_prob_threshold,
                )

+            if (
+                options.no_speech_threshold is not None
+                and result.no_speech_prob > options.no_speech_threshold
+                and options.log_prob_threshold is not None
+                and avg_logprob < options.log_prob_threshold
+            ):
+                needs_fallback = False  # silence
+
            if not needs_fallback:
                break
+        else:
+            # all failed, select the result with the highest average log probability
+            decode_result = max(
+                below_cr_threshold_results or all_results, key=lambda x: x[1]
+            )
+            # to pass final temperature for prompt_reset_on_temperature
+            decode_result = (
+                decode_result[0],
+                decode_result[1],
+                temperature,
+                decode_result[3],
+            )

-        return result, avg_log_prob, final_temperature
+        return decode_result

    def get_prompt(
        self,
@@ -634,6 +955,8 @@ class WhisperModel:
            prefix_tokens = tokenizer.encode(" " + prefix.strip())
            if len(prefix_tokens) >= self.max_length // 2:
                prefix_tokens = prefix_tokens[: self.max_length // 2 - 1]
+            if not without_timestamps:
+                prompt.append(tokenizer.timestamp_begin)
            prompt.extend(prefix_tokens)

        return prompt
@@ -646,7 +969,8 @@ class WhisperModel:
        num_frames: int,
        prepend_punctuations: str,
        append_punctuations: str,
-    ):
+        last_speech_timestamp: float,
+    ) -> None:
        if len(segments) == 0:
            return

@@ -659,6 +983,25 @@ class WhisperModel:
        alignment = self.find_alignment(
            tokenizer, text_tokens, encoder_output, num_frames
        )
+        word_durations = np.array([word["end"] - word["start"] for word in alignment])
+        word_durations = word_durations[word_durations.nonzero()]
+        median_duration = np.median(word_durations) if len(word_durations) > 0 else 0.0
+        median_duration = min(0.7, float(median_duration))
+        max_duration = median_duration * 2
+
+        # hack: truncate long words at sentence boundaries.
+        # a better segmentation algorithm based on VAD should be able to replace this.
+        if len(word_durations) > 0:
+            sentence_end_marks = ".。!！?？"
+            # ensure words at sentence boundaries
+            # are not longer than twice the median word duration.
+            for i in range(1, len(alignment)):
+                if alignment[i]["end"] - alignment[i]["start"] > max_duration:
+                    if alignment[i]["word"] in sentence_end_marks:
+                        alignment[i]["end"] = alignment[i]["start"] + max_duration
+                    elif alignment[i - 1]["word"] in sentence_end_marks:
+                        alignment[i]["start"] = alignment[i]["end"] - max_duration
+
        merge_punctuations(alignment, prepend_punctuations, append_punctuations)

        time_offset = (
@@ -689,10 +1032,51 @@ class WhisperModel:
                saved_tokens += len(timing["tokens"])
                word_index += 1

+            # hack: truncate long words at segment boundaries.
+            # a better segmentation algorithm based on VAD should be able to replace this.
            if len(words) > 0:
-                # adjust the segment-level timestamps based on the word-level timestamps
-                segment["start"] = words[0]["start"]
-                segment["end"] = words[-1]["end"]
+                # ensure the first and second word after a pause is not longer than
+                # twice the median word duration.
+                if words[0]["end"] - last_speech_timestamp > median_duration * 4 and (
+                    words[0]["end"] - words[0]["start"] > max_duration
+                    or (
+                        len(words) > 1
+                        and words[1]["end"] - words[0]["start"] > max_duration * 2
+                    )
+                ):
+                    if (
+                        len(words) > 1
+                        and words[1]["end"] - words[1]["start"] > max_duration
+                    ):
+                        boundary = max(
+                            words[1]["end"] / 2, words[1]["end"] - max_duration
+                        )
+                        words[0]["end"] = words[1]["start"] = boundary
+                    words[0]["start"] = max(0, words[0]["end"] - max_duration)
+
+                # prefer the segment-level start timestamp if the first word is too long.
+                if (
+                    segment["start"] < words[0]["end"]
+                    and segment["start"] - 0.5 > words[0]["start"]
+                ):
+                    words[0]["start"] = max(
+                        0, min(words[0]["end"] - median_duration, segment["start"])
+                    )
+                else:
+                    segment["start"] = words[0]["start"]
+
+                # prefer the segment-level end timestamp if the last word is too long.
+                if (
+                    segment["end"] > words[-1]["start"]
+                    and segment["end"] + 0.5 < words[-1]["end"]
+                ):
+                    words[-1]["end"] = max(
+                        words[-1]["start"] + median_duration, segment["end"]
+                    )
+                else:
+                    segment["end"] = words[-1]["end"]
+
+                last_speech_timestamp = segment["end"]

            segment["words"] = words

@@ -724,7 +1108,16 @@ class WhisperModel:
        words, word_tokens = tokenizer.split_to_word_tokens(
            text_tokens + [tokenizer.eot]
        )
+        if len(word_tokens) <= 1:
+            # return on eot only
+            # >>> np.pad([], (1, 0))
+            # array([0.])
+            # This results in crashes when we lookup jump_times with float, like
+            # IndexError: arrays used as indices must be of integer (or boolean) type
+            return []
        word_boundaries = np.pad(np.cumsum([len(t) for t in word_tokens[:-1]]), (1, 0))
+        if len(word_boundaries) <= 1:
+            return []

        jumps = np.pad(np.diff(text_indices), (1, 0), constant_values=1).astype(bool)
        jump_times = time_indices[jumps] / self.tokens_per_second
@@ -735,22 +1128,6 @@ class WhisperModel:
            for i, j in zip(word_boundaries[:-1], word_boundaries[1:])
        ]

-        # hack: ensure the first and second word is not longer than twice the median word duration.
-        # a better segmentation algorithm based on VAD should be able to replace this.
-        word_durations = end_times - start_times
-        word_durations = word_durations[word_durations.nonzero()]
-        if len(word_durations) > 0:
-            median_duration = np.median(word_durations)
-            max_duration = median_duration * 2
-            if len(word_durations) >= 2 and word_durations[1] > max_duration:
-                boundary = max(end_times[2] / 2, end_times[2] - max_duration)
-                end_times[0] = start_times[1] = boundary
-            if (
-                len(word_durations) >= 1
-                and end_times[0] - start_times[0] > max_duration
-            ):
-                start_times[0] = max(0, end_times[0] - max_duration)
-
        return [
            dict(
                word=word, tokens=tokens, start=start, end=end, probability=probability
@@ -773,7 +1150,8 @@ def restore_speech_timestamps(
            words = []
            for word in segment.words:
                # Ensure the word start and end times are resolved to the same chunk.
-                chunk_index = ts_map.get_chunk_index(word.start)
+                middle = (word.start + word.end) / 2
+                chunk_index = ts_map.get_chunk_index(middle)
                word = word._replace(
                    start=ts_map.get_original_time(word.start, chunk_index),
                    end=ts_map.get_original_time(word.end, chunk_index),
@@ -806,7 +1184,10 @@ def get_compression_ratio(text: str) -> float:
    return len(text_bytes) / len(zlib.compress(text_bytes))


-def get_suppressed_tokens(tokenizer, suppress_tokens):
+def get_suppressed_tokens(
+    tokenizer: Tokenizer,
+    suppress_tokens: Optional[List[int]],
+) -> Optional[List[int]]:
    if not suppress_tokens or -1 in suppress_tokens:
        return suppress_tokens

@@ -827,7 +1208,7 @@ def get_suppressed_tokens(tokenizer, suppress_tokens):
    return sorted(set(suppress_tokens))


-def merge_punctuations(alignment: List[dict], prepended: str, appended: str):
+def merge_punctuations(alignment: List[dict], prepended: str, appended: str) -> None:
    # merge prepended punctuations
    i = len(alignment) - 2
    j = len(alignment) - 1
--- a/faster_whisper/utils.py
+++ b/faster_whisper/utils.py
@@ -1,24 +1,37 @@
 import logging
 import os
+import re

-from typing import Optional
+from typing import List, Optional

 import huggingface_hub
+import requests

 from tqdm.auto import tqdm

-_MODELS = (
-    "tiny.en",
-    "tiny",
-    "base.en",
-    "base",
-    "small.en",
-    "small",
-    "medium.en",
-    "medium",
-    "large-v1",
-    "large-v2",
-)
+_MODELS = {
+    "tiny.en": "Systran/faster-whisper-tiny.en",
+    "tiny": "Systran/faster-whisper-tiny",
+    "base.en": "Systran/faster-whisper-base.en",
+    "base": "Systran/faster-whisper-base",
+    "small.en": "Systran/faster-whisper-small.en",
+    "small": "Systran/faster-whisper-small",
+    "medium.en": "Systran/faster-whisper-medium.en",
+    "medium": "Systran/faster-whisper-medium",
+    "large-v1": "Systran/faster-whisper-large-v1",
+    "large-v2": "Systran/faster-whisper-large-v2",
+    "large-v3": "Systran/faster-whisper-large-v3",
+    "large": "Systran/faster-whisper-large-v3",
+    "distil-large-v2": "Systran/faster-distil-whisper-large-v2",
+    "distil-medium.en": "Systran/faster-distil-whisper-medium.en",
+    "distil-small.en": "Systran/faster-distil-whisper-small.en",
+    "distil-large-v3": "Systran/faster-distil-whisper-large-v3",
+}
+
+
+def available_models() -> List[str]:
+    """Returns the names of available models."""
+    return list(_MODELS.keys())


 def get_assets_path():
@@ -31,16 +44,24 @@ def get_logger():
    return logging.getLogger("faster_whisper")


-def download_model(size: str, output_dir: Optional[str] = None):
+def download_model(
+    size_or_id: str,
+    output_dir: Optional[str] = None,
+    local_files_only: bool = False,
+    cache_dir: Optional[str] = None,
+):
    """Downloads a CTranslate2 Whisper model from the Hugging Face Hub.

-    The model is downloaded from https://huggingface.co/guillaumekln.
-
    Args:
-      size: Size of the model to download (tiny, tiny.en, base, base.en, small, small.en,
-        medium, medium.en, large-v1, or large-v2).
+      size_or_id: Size of the model to download from https://huggingface.co/Systran
+        (tiny, tiny.en, base, base.en, small, small.en medium, medium.en, large-v1, large-v2,
+        large-v3, large), or a CTranslate2-converted model ID from the Hugging Face Hub
+        (e.g. Systran/faster-whisper-large-v3).
      output_dir: Directory where the model should be saved. If not set, the model is saved in
-        the standard Hugging Face cache directory.
+        the cache directory.
+      local_files_only:  If True, avoid downloading the file and return the path to the local
+        cached file if it exists.
+      cache_dir: Path to the folder where cached files are stored.

    Returns:
      The path to the downloaded model.
@@ -48,31 +69,55 @@ def download_model(size: str, output_dir: Optional[str] = None):
    Raises:
      ValueError: if the model size is invalid.
    """
-    if size not in _MODELS:
-        raise ValueError(
-            "Invalid model size '%s', expected one of: %s" % (size, ", ".join(_MODELS))
-        )
+    if re.match(r".*/.*", size_or_id):
+        repo_id = size_or_id
+    else:
+        repo_id = _MODELS.get(size_or_id)
+        if repo_id is None:
+            raise ValueError(
+                "Invalid model size '%s', expected one of: %s"
+                % (size_or_id, ", ".join(_MODELS.keys()))
+            )

-    repo_id = "guillaumekln/faster-whisper-%s" % size
-    kwargs = {}
+    allow_patterns = [
+        "config.json",
+        "preprocessor_config.json",
+        "model.bin",
+        "tokenizer.json",
+        "vocabulary.*",
+    ]
+
+    kwargs = {
+        "local_files_only": local_files_only,
+        "allow_patterns": allow_patterns,
+        "tqdm_class": disabled_tqdm,
+    }

    if output_dir is not None:
        kwargs["local_dir"] = output_dir
        kwargs["local_dir_use_symlinks"] = False

-    allow_patterns = [
-        "config.json",
-        "model.bin",
-        "tokenizer.json",
-        "vocabulary.txt",
-    ]
+    if cache_dir is not None:
+        kwargs["cache_dir"] = cache_dir

-    return huggingface_hub.snapshot_download(
-        repo_id,
-        allow_patterns=allow_patterns,
-        tqdm_class=disabled_tqdm,
-        **kwargs,
-    )
+    try:
+        return huggingface_hub.snapshot_download(repo_id, **kwargs)
+    except (
+        huggingface_hub.utils.HfHubHTTPError,
+        requests.exceptions.ConnectionError,
+    ) as exception:
+        logger = get_logger()
+        logger.warning(
+            "An error occured while synchronizing the model %s from the Hugging Face Hub:\n%s",
+            repo_id,
+            exception,
+        )
+        logger.warning(
+            "Trying to load the model directly from the local cache, if it exists."
+        )
+
+        kwargs["local_files_only"] = True
+        return huggingface_hub.snapshot_download(repo_id, **kwargs)


 def format_timestamp(
@@ -102,3 +147,10 @@ class disabled_tqdm(tqdm):
    def __init__(self, *args, **kwargs):
        kwargs["disable"] = True
        super().__init__(*args, **kwargs)
+
+
+def get_end(segments: List[dict]) -> Optional[float]:
+    return next(
+        (w["end"] for s in reversed(segments) for w in reversed(s["words"])),
+        segments[-1]["end"] if segments else None,
+    )
--- a/faster_whisper/vad.py
+++ b/faster_whisper/vad.py
@@ -3,47 +3,67 @@ import functools
 import os
 import warnings

-from typing import List, Optional
+from typing import List, NamedTuple, Optional

 import numpy as np

 from faster_whisper.utils import get_assets_path

+
 # The code below is adapted from https://github.com/snakers4/silero-vad.
+class VadOptions(NamedTuple):
+    """VAD options.

-
-def get_speech_timestamps(
-    audio: np.ndarray,
-    *,
-    threshold: float = 0.5,
-    min_speech_duration_ms: int = 250,
-    max_speech_duration_s: float = float("inf"),
-    min_silence_duration_ms: int = 2000,
-    window_size_samples: int = 1024,
-    speech_pad_ms: int = 200,
-) -> List[dict]:
-    """This method is used for splitting long audios into speech chunks using silero VAD.
-
-    Args:
-      audio: One dimensional float array.
+    Attributes:
      threshold: Speech threshold. Silero VAD outputs speech probabilities for each audio chunk,
        probabilities ABOVE this value are considered as SPEECH. It is better to tune this
        parameter for each dataset separately, but "lazy" 0.5 is pretty good for most datasets.
      min_speech_duration_ms: Final speech chunks shorter min_speech_duration_ms are thrown out.
      max_speech_duration_s: Maximum duration of speech chunks in seconds. Chunks longer
        than max_speech_duration_s will be split at the timestamp of the last silence that
-        lasts more than 100s (if any), to prevent agressive cutting. Otherwise, they will be
+        lasts more than 100ms (if any), to prevent aggressive cutting. Otherwise, they will be
        split aggressively just before max_speech_duration_s.
      min_silence_duration_ms: In the end of each speech chunk wait for min_silence_duration_ms
        before separating it
      window_size_samples: Audio chunks of window_size_samples size are fed to the silero VAD model.
        WARNING! Silero VAD models were trained using 512, 1024, 1536 samples for 16000 sample rate.
-        Values other than these may affect model perfomance!!
+        Values other than these may affect model performance!!
      speech_pad_ms: Final speech chunks are padded by speech_pad_ms each side
+    """
+
+    threshold: float = 0.5
+    min_speech_duration_ms: int = 250
+    max_speech_duration_s: float = float("inf")
+    min_silence_duration_ms: int = 2000
+    window_size_samples: int = 1024
+    speech_pad_ms: int = 400
+
+
+def get_speech_timestamps(
+    audio: np.ndarray,
+    vad_options: Optional[VadOptions] = None,
+    **kwargs,
+) -> List[dict]:
+    """This method is used for splitting long audios into speech chunks using silero VAD.
+
+    Args:
+      audio: One dimensional float array.
+      vad_options: Options for VAD processing.
+      kwargs: VAD options passed as keyword arguments for backward compatibility.

    Returns:
      List of dicts containing begin and end samples of each speech chunk.
    """
+    if vad_options is None:
+        vad_options = VadOptions(**kwargs)
+
+    threshold = vad_options.threshold
+    min_speech_duration_ms = vad_options.min_speech_duration_ms
+    max_speech_duration_s = vad_options.max_speech_duration_s
+    min_silence_duration_ms = vad_options.min_silence_duration_ms
+    window_size_samples = vad_options.window_size_samples
+    speech_pad_ms = vad_options.speech_pad_ms
+
    if window_size_samples not in [512, 1024, 1536]:
        warnings.warn(
            "Unusual window_size_samples! Supported window_size_samples:\n"
--- a/faster_whisper/version.py
+++ b/faster_whisper/version.py
@@ -0,0 +1,3 @@
+"""Version information."""
+
+__version__ = "1.0.1"
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,5 +1,5 @@
-av==10.*
-ctranslate2>=3.10,<4
+av==11.*
+ctranslate2>=4.0,<5
 huggingface_hub>=0.13
-tokenizers==0.13.*
-onnxruntime==1.14.* ; python_version < "3.11"
+tokenizers>=0.13,<0.16
+onnxruntime>=1.14,<2
--- a/setup.py
+++ b/setup.py
@@ -11,6 +11,14 @@ def get_long_description():
        return readme_file.read()


+def get_project_version():
+    version_path = os.path.join(base_dir, "faster_whisper", "version.py")
+    version = {}
+    with open(version_path, encoding="utf-8") as fp:
+        exec(fp.read(), version)
+    return version["__version__"]
+
+
 def get_requirements(path):
    with open(path, encoding="utf-8") as requirements:
        return [requirement.strip() for requirement in requirements]
@@ -23,13 +31,13 @@ conversion_requires = get_requirements(

 setup(
    name="faster-whisper",
-    version="0.4.1",
+    version=get_project_version(),
    license="MIT",
    description="Faster Whisper transcription with CTranslate2",
    long_description=get_long_description(),
    long_description_content_type="text/markdown",
    author="Guillaume Klein",
-    url="https://github.com/guillaumekln/faster-whisper",
+    url="https://github.com/SYSTRAN/faster-whisper",
    classifiers=[
        "Development Status :: 4 - Beta",
        "Intended Audience :: Developers",
--- a/tests/test_transcribe.py
+++ b/tests/test_transcribe.py
@@ -3,14 +3,26 @@ import os
 from faster_whisper import WhisperModel, decode_audio


+def test_supported_languages():
+    model = WhisperModel("tiny.en")
+    assert model.supported_languages == ["en"]
+
+
 def test_transcribe(jfk_path):
    model = WhisperModel("tiny")
    segments, info = model.transcribe(jfk_path, word_timestamps=True)
+    assert info.all_language_probs is not None

    assert info.language == "en"
    assert info.language_probability > 0.9
    assert info.duration == 11

+    # Get top language info from all results, which should match the
+    # already existing metadata
+    top_lang, top_lang_score = info.all_language_probs[0]
+    assert info.language == top_lang
+    assert abs(info.language_probability - top_lang_score) < 1e-16
+
    segments = list(segments)

    assert len(segments) == 1
@@ -27,12 +39,30 @@ def test_transcribe(jfk_path):
    assert segment.end == segment.words[-1].end


+def test_prefix_with_timestamps(jfk_path):
+    model = WhisperModel("tiny")
+    segments, _ = model.transcribe(jfk_path, prefix="And so my fellow Americans")
+    segments = list(segments)
+
+    assert len(segments) == 1
+
+    segment = segments[0]
+
+    assert segment.text == (
+        " And so my fellow Americans ask not what your country can do for you, "
+        "ask what you can do for your country."
+    )
+
+    assert segment.start == 0
+    assert 10 < segment.end < 11
+
+
 def test_vad(jfk_path):
    model = WhisperModel("tiny")
-    segments, _ = model.transcribe(
+    segments, info = model.transcribe(
        jfk_path,
        vad_filter=True,
-        vad_parameters=dict(min_silence_duration_ms=500),
+        vad_parameters=dict(min_silence_duration_ms=500, speech_pad_ms=200),
    )
    segments = list(segments)

@@ -47,6 +77,9 @@ def test_vad(jfk_path):
    assert 0 < segment.start < 1
    assert 10 < segment.end < 11

+    assert info.vad_options.min_silence_duration_ms == 500
+    assert info.vad_options.speech_pad_ms == 200
+

 def test_stereo_diarization(data_dir):
    model = WhisperModel("tiny")
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,6 +1,12 @@
 import os

-from faster_whisper import download_model
+from faster_whisper import available_models, download_model
+
+
+def test_available_models():
+    models = available_models()
+    assert isinstance(models, list)
+    assert "tiny" in models


 def test_download_model(tmpdir):
@@ -15,3 +21,9 @@ def test_download_model(tmpdir):
    for filename in os.listdir(model_dir):
        path = os.path.join(model_dir, filename)
        assert not os.path.islink(path)
+
+
+def test_download_model_in_cache(tmpdir):
+    cache_dir = str(tmpdir.join("model"))
+    download_model("tiny", cache_dir=cache_dir)
+    assert os.path.isdir(cache_dir)
Author	SHA1	Message	Date
otakutyrant	91c8307aa6	make faster_whisper.assets as a valid python package to distribute (#772 ) (#774 )	2024-04-02 18:22:22 +02:00
Purfview	b024972a56	Foolproof: Disable VAD if clip_timestamps is in use (#769 ) * Foolproof: Disable VAD if clip_timestamps is in use Prevent silly things to happen.	2024-04-02 18:20:34 +02:00
Purfview	8ae82c8372	Bugfix: code breaks if audio is empty (#768 ) * Bugfix: code breaks if audio is empty Regression since https://github.com/SYSTRAN/faster-whisper/pull/732 PR	2024-04-02 18:18:12 +02:00
trungkienbkhn	e0c3a9ed34	Update project github link to SYSTRAN (#746 )	2024-03-27 08:31:17 +01:00
Sanchit Gandhi	a67e0e47ae	Add support for distil-large-v3 (#755 ) * add distil-large-v3 * Update README.md * use fp16 weights from Systran	2024-03-26 14:58:39 +01:00
trungkienbkhn	1eb9a8004c	Improve language detection (#732 )	2024-03-12 15:44:49 +01:00
trungkienbkhn	a342b028b7	Bump version to 1.0.1 (#725 )	2024-03-01 11:32:12 +01:00
Purfview	5090cc9d0d	Fix window end heuristic for hallucination_silence_threshold (#706 ) Removes the wishful heuristic causing more issues than it's fixing. Same as https://github.com/openai/whisper/pull/2043 Example of the issue: https://github.com/openai/whisper/pull/1838#issuecomment-1960041500	2024-02-29 17:59:32 +01:00
Gabriel F	09cd57e7f3	Fix typo 'ditil' (#721 )	2024-02-29 17:08:58 +01:00
trungkienbkhn	16141e65d9	Add pad_or_trim function to handle segment before encoding (#705 )	2024-02-29 17:08:28 +01:00
trungkienbkhn	06d32bf0c1	Bump version to 1.0.0 (#696 )	2024-02-22 09:49:01 +01:00
Purfview	30d6043e90	Prevent infinite loop for out-of-bound timestamps in clip_timestamps (#697 ) Same as https://github.com/openai/whisper/pull/2005	2024-02-22 09:48:35 +01:00
BBC-Esq	22c75d0cc3	Update README.md (#672 ) Add Faster-Whisper-Transcriber to community integrations.	2024-02-21 10:18:11 +01:00
trungkienbkhn	092067208b	Add clip_timestamps and hallucination_silence_threshold options (#646 )	2024-02-20 17:34:54 +01:00
Jordi Mas	6ffcbdfbc2	Fix typos in README.md (#668 )	2024-02-20 17:33:17 +01:00
Purfview	52695567c9	Bumps up PyAV version to support Python 3.12.x (#679 )	2024-02-20 17:31:07 +01:00
IlianP	c6b28ed3a0	Update README.md (#685 ) I'm surprised that WhisperX hasn't made it into this list yet, as it has more stars than faster-whisper itself 🚀	2024-02-20 17:28:00 +01:00
trungkienbkhn	4ab646035f	Upgrade ctranslate2 version to support CUDA 12 (#694 )	2024-02-20 17:26:55 +01:00
Purfview	f144e4c83d	Expands the note for distil-whisper (#659 )	2024-01-28 21:48:40 +01:00
Purfview	3aec421849	Add: More clarity of what "max_new_tokens" does (#658 ) * Add: More clarity of what "max_new_tokens" does	2024-01-28 21:40:33 +01:00
Dominik Macháček	64b9f244bd	Whisper-Streaming mention (#656 ) under community integrations	2024-01-25 18:27:27 +01:00
Purfview	00efce1696	Bugfix: Illogical "Avoid computing higher temperatures on no_speech" (#652 )	2024-01-24 11:54:43 +01:00
metame	ad3c83045b	support distil-whisper (#557 )	2024-01-24 10:17:12 +01:00
Jürgen Fleiß	72ff979a2e	Add GUI faster-whisper project README.md (#554 ) Added aTrain GUI faster-whisper transcription and diarization tool as community project. Co-authored-by: JuergenFleiss <118339672+Juergen-J-F@users.noreply.github.com>	2024-01-18 13:01:02 +01:00
makaveli	615de0d2d9	add WhisperLive to community integration (#647 )	2024-01-18 12:54:14 +01:00
Purfview	44f7e58947	Update whisper-standalone-win description in README.md (#508 ) * Update whisper-standalone-win description in README.md	2023-12-14 13:03:46 +01:00
Purfview	ebcfd6b964	Fix broken prompt_reset_on_temperature (#604 ) * Fix broken prompt_reset_on_temperature Fixing: https://github.com/SYSTRAN/faster-whisper/issues/603 Broken because `generate_with_fallback()` doesn't return final temperature. Regression since PR356 -> https://github.com/SYSTRAN/faster-whisper/pull/356	2023-12-13 13:14:39 +01:00
trungkienbkhn	19329a3611	Word timing tweaks (#616 )	2023-12-13 12:38:44 +01:00
Purfview	65094b779e	Update info on cuBLAS and cuDNN libs in README.md (#513 )	2023-11-27 12:12:47 +01:00
Clayton Yochum	9641d5f56a	Force read-mode in `av.open` (#566 ) The `av.open` functions checks input metadata to determine the mode to open with ("r" or "w"). If an input to `decode_audio` is found to be in write-mode, without this change it can't be read. Forcing read mode solves this.	2023-11-27 10:43:35 +01:00
Dang Chuan Nguyen	e1a218fab1	Bump version to 0.10.0	2023-11-24 23:19:47 +01:00
Oscaarjs	3084409633	Add V3 Support (#578 ) * Add V3 Support * update conversion example --------- Co-authored-by: oscaarjs <oscar.johansson@conversy.se>	2023-11-24 23:16:12 +01:00
Guillaume Klein	5a0541ea7d	Bump version to 0.9.0	2023-09-18 16:21:37 +02:00
Guillaume Klein	e94711bb5c	Add property WhisperModel.supported_languages (#476 ) * Expose function supported_languages * Make it a method	2023-09-14 17:42:02 +02:00
Guillaume Klein	0048844f54	Expose function available_models (#475 ) * Expose function available_models * Add test case	2023-09-14 17:17:01 +02:00
Guillaume Klein	a49097e655	Add some missing typing annotations in transcribe.py	2023-09-12 15:45:54 +02:00
Guillaume Klein	81086f6d33	Always run the encoder at the beginning of the loop (#468 )	2023-09-12 14:44:37 +02:00
Guillaume Klein	f697945691	Update tokenizers requirement to include version 0.14 (#469 )	2023-09-12 14:44:22 +02:00
Guillaume Klein	727ab81f31	Improve error message for invalid task and language parameters (#466 )	2023-09-12 10:02:23 +02:00
Guillaume Klein	0285d46f6f	Add more details about the requirements in the README (#463 )	2023-09-08 14:35:17 +02:00
Guillaume Klein	ad388cd394	Bump version to 0.8.0	2023-09-04 11:56:48 +02:00
Guillaume Klein	4a41746e55	Log a warning when the model is English-only but the language is set to something else (#454 )	2023-09-04 11:55:40 +02:00
Guillaume Klein	1e6eb967c9	Add "large" alias for "large-v2" model (#453 )	2023-09-04 11:54:42 +02:00
Guillaume Klein	f0ff12965a	Expose generation parameter no_repeat_ngram_size (#449 )	2023-09-01 17:31:30 +02:00
Guillaume Klein	5871858a5f	Force the garbage collector to run after decoding the audio with PyAV (#448 )	2023-09-01 15:25:13 +02:00
MinorJinx	e87fbf8a49	Added audio duration after VAD to TranscriptionInfo object (#445 ) * Added VAD removed audio duration to TranscriptionInfo object Along with the duration of the original audio, this commit adds the seconds of audio removed by the VAD to the returned info obj * Chaning naming for duration_after_vad Instead of the property returning the audio duration removed, it now returns the final duration after the vad. If vad_filter is False or if it doesn't remove any audio, the original duration is returned.	2023-08-31 17:19:48 +02:00
Hrishikesh Barman	7b271da035	docs: add wscribe to community integrations (#427 ) wscribe is a utility to generate transcript specifically to make it easy for further manual edits accompanied by the wscribe-editor	2023-08-17 08:50:24 +02:00
Aisu Wata	1562b02345	added repetition_penalty to TranscriptionOptions (#403 ) Co-authored-by: Aisu Wata <aisu.wata0@gmail.com>	2023-08-06 10:08:24 +02:00
Purfview	1ce16652ee	Adds DEBUG log message for prompt_reset_on_temperature (#399 ) Produce DEBUG log message if prompt_reset_on_temperature threshold is met.	2023-08-04 09:06:17 +02:00
Purfview	857be6f621	Rename clear_previous_text_on_temperature argument (#398 ) `prompt_reset_on_temperature` is more clear what it does.	2023-08-03 18:44:37 +02:00
KH	1a1eb1a027	Add clear_previous_text_on_temperature parameter (#397 ) * Add clear_previous_text_on_temperature parameter * Add a description	2023-08-03 15:40:58 +02:00
Guillaume Klein	5c17de1771	Bump version to 0.7.1	2023-07-24 11:10:12 +02:00
Guillaume Klein	0f55c436fe	Invalidate the cached encoder output when no_speech threshold is met (#376 )	2023-07-24 10:57:15 +02:00
KH	e786e26f75	Return result with best log prob when all temperature fallbacks failed (#356 ) * Resolve Inference Selection Bug * Refactor for better readability * Filter out results with compression_ratio * Refactor to avoid variable repetition * Fix incorrect index and perform minor refactoring * Remove final_temperature variable	2023-07-20 16:13:11 +02:00
KH	687db319e0	Remove duplicate code (#359 )	2023-07-18 16:03:01 +02:00
Guillaume Klein	171d90dd1f	Bump version to 0.7.0	2023-07-18 15:23:47 +02:00
Guillaume Klein	0e051a5b77	Prepend prefix tokens with the initial timestamp token (#358 )	2023-07-18 15:22:39 +02:00
Guillaume Klein	2a37390fed	Minor reformatting in code snippet	2023-07-18 15:08:53 +02:00
Hoon	3b4a6aa1c2	Improve timestamp heuristics (#336 ) * Improve timestamp heuristics * Chore	2023-07-05 15:16:53 +02:00
zh-plus	c7cb2aa8d4	Add support for using whisper models from Huggingface by specifying the model id. (#334 ) * Add support for downloading CTranslate-converted models from Huggingface. * Update utils.py to pass Flake8. * Update utils.py to pass black. * Remove redundant usage instructions. * Apply suggestions from code review Co-authored-by: Guillaume Klein <guillaumekln@users.noreply.github.com> --------- Co-authored-by: Guillaume Klein <guillaumekln@users.noreply.github.com>	2023-07-03 17:40:10 +02:00
Guillaume Klein	c0d93d0829	Avoid computing higher temperatures on no_speech segments (#225 ) Port commit `e334ff141d`	2023-07-03 10:20:36 +02:00
Guillaume Klein	19c294f978	Squash long words at window and sentence boundaries (#226 ) Port commit `255887f219`	2023-07-03 10:20:20 +02:00
FlippFuzz	fee52c9229	Allow users to input an Iterable of token ids into initial_prompt (#306 ) * Allow users to input an Iterable of token ids into initial_prompt * Need to check for String first because string is also an Iterable	2023-06-21 14:46:20 +02:00
Guillaume Klein	efc4f61d85	Do not specify the vocabulary file extension in the download pattern (#311 )	2023-06-20 10:53:11 +02:00
kh	ad58ba26ab	Fix typo (#304 ) https://github.com/snakers4/silero-vad/discussions/319#discussion-5081706	2023-06-16 07:37:45 +02:00
zh-plus	20d4e9418b	Add Open-Lyrics as a community project. (#291 )	2023-06-10 08:22:29 +02:00
Antonio Zarauz Moreno	d4222da952	Update README with community repo using FW (#284 ) * Update README with community repo using FW * Minor formatting change --------- Co-authored-by: Guillaume Klein <guillaumekln@users.noreply.github.com>	2023-06-07 11:30:53 +02:00
Guillaume Klein	1bb7e33b93	Reformat code snippet in README	2023-05-24 18:22:44 +02:00
Guillaume Klein	2a00621564	Bump version to 0.6.0	2023-05-24 16:15:01 +02:00
Guillaume Klein	a150adcc19	Enable onnxruntime dependency for Python 3.11 (#260 )	2023-05-24 16:07:54 +02:00
Guillaume Klein	ae1e6d9883	Remove reference to the VAD function from the README	2023-05-24 15:56:21 +02:00
Guillaume Klein	cf7c021573	Export __version__ at the module level (#258 )	2023-05-24 15:50:37 +02:00
Guillaume Klein	4db549b800	Make get_speech_timestamps backward compatible with the previous usage (#259 )	2023-05-24 15:49:36 +02:00
Guillaume Klein	c99feb22dc	Include requirements files in sdist (#240 )	2023-05-24 12:55:15 +02:00
Guillaume Klein	723cb97483	Fix occasional IndexError on empty segments (#227 )	2023-05-24 12:55:04 +02:00
Guillaume Klein	6a2da9a95c	Also catch client-side network exceptions when synchronizing models (#228 )	2023-05-11 15:07:15 +02:00
Guillaume Klein	6a1d331d66	Add CONTRIBUTING.md (#229 )	2023-05-11 15:06:46 +02:00
Guillaume Klein	2d7c984bfc	Reformat function download_model for clarity	2023-05-11 14:47:22 +02:00
Guillaume Klein	8e5c747ab5	Reformat list of community integrations	2023-05-11 12:15:41 +02:00
Purfview	32b962bed8	Adds: whisper-standalone-win (#216 )	2023-05-09 20:20:41 +02:00
David Axelrod	53d247b0bb	retry model download locally if huggingface throws an http error. (#215 ) * rety model download locally if huggingface throws an http error. * appease the linter * key error fix * use non internal lib error Co-authored-by: Guillaume Klein <guillaumekln@users.noreply.github.com> --------- Co-authored-by: Guillaume Klein <guillaumekln@users.noreply.github.com>	2023-05-09 17:20:22 +02:00
Ozan Caglayan	91f948b0d6	transcribe: return all language probabilities if requested (#210 ) * transcribe: return all language probabilities if requested If return_all_language_probs is True, TranscriptionInfo structure will have a list of tuples reflecting all language probabilities as returned by the model. * transcribe: fix docstring * transcribe: remove return_all_lang_probs parameter	2023-05-09 14:53:47 +02:00
FlippFuzz	5d8f3e2d90	Implement VadOptions (#198 ) * Implement VadOptions * Fix line too long ./faster_whisper/transcribe.py:226:101: E501 line too long (111 > 100 characters) * Reformatted files with black * black .\faster_whisper\vad.py * black .\faster_whisper\transcribe.py * Fix import order with isort * isort .\faster_whisper\vad.py * isort .\faster_whisper\transcribe.py * Made recommended changes Recommended in https://github.com/guillaumekln/faster-whisper/pull/198 * Fix typing of vad_options argument --------- Co-authored-by: Guillaume Klein <guillaumekln@users.noreply.github.com>	2023-05-09 12:47:02 +02:00
Mahmoud Ashraf	d889345e07	added whisper-diarize (#193 )	2023-04-28 10:56:13 +02:00
Jordi Mas	5d203d2757	Update Github link to community project (#187 )	2023-04-27 14:53:28 +02:00
Guillaume Klein	a3dcb90081	Bump version to 0.5.1	2023-04-26 17:38:16 +02:00
Guillaume Klein	89a4c7f1f0	Update docstring to clarify download_root and output_dir	2023-04-26 17:37:51 +02:00
Guillaume Klein	6f9d68dd6b	Fix typing of local_files_only	2023-04-26 17:36:24 +02:00
Jordi Mas	68df3214ba	Use cache_dir instead of local_dir (#182 ) * Use cache_dir instead of local_dir * Fix unit test * Use cache_dir and preserve local_dir parameter * Remove blank line at the end * Disable ut * Implement download_root suggestion * Use cache_dir=download_root	2023-04-26 16:35:18 +02:00
Guillaume Klein	67cce3f552	Bump version to 0.5.0	2023-04-25 17:00:41 +02:00
Guillaume Klein	8340e04dc6	Assign words to the speech chunk with the greatest coverage (#180 )	2023-04-25 15:54:31 +02:00
Guillaume Klein	8cf5d5a4b3	Increase the default value of speech_pad_ms to 400 ms (#179 )	2023-04-25 15:54:22 +02:00
Guillaume Klein	32dc625f11	Update README.md	2023-04-25 15:47:38 +02:00
Guillaume Klein	e06511f96b	Rename AudioInfo to TranscriptionInfo (#174 )	2023-04-24 16:29:17 +02:00
Anthony	338a725ff8	fix where the tokens are reset (#175 )	2023-04-24 16:28:47 +02:00
Amar Sood	f893113759	Align segment structure with openai/whisper (#154 ) * Align segment structure with openai/whisper * Update code to apply requested changes * Move increment below the segment filtering --------- Co-authored-by: Guillaume Klein <guillaumekln@users.noreply.github.com>	2023-04-24 15:04:42 +02:00
FlippFuzz	2b51a97e61	Add transcription_options to AudioInfo (#170 ) * Add transcription_options to AudioInfo It would be great if we can include the transcription_options in AudioInfo. My application is only making a few changes but leaving the rest as default. However, I would like to record down all settings (including those that I did not specify) so that the audio can be transcribed again identically in future if need be. * Make TranscriptionOptions appear before AudioInfo * Remove unnecessary whitespace	2023-04-24 15:02:19 +02:00
Jordi Mas	358d373691	Allow specifying local_files_only to prevent checking the Internet everytime (#166 )	2023-04-20 14:26:06 +02:00