heimoshuiyu/faster-whisper
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

Revert "Prevent out of range error in method split_tokens_on_unicode"

Browse Source 
This reverts commit 36160c1e7e.
This commit is contained in:
Guillaume Klein
2023-04-04 10:25:41 +02:00
parent 36160c1e7e
commit 9fa1989073
1 changed files with 0 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
faster_whisper/tokenizer.py
View File

@@ -135,9 +135,6 @@ class Tokenizer:
current_tokens = [] current_tokens = []
unicode_offset += len(decoded) unicode_offset += len(decoded)
if unicode_offset >= len(decoded_full):
break
return words, word_tokens return words, word_tokens
def split_tokens_on_spaces( def split_tokens_on_spaces(