heimoshuiyu/whisper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Use tiktoken (#1044)

Browse Source 
* use tiktoken==0.3.0

* formatting

* tuple should be safer

* Update whisper/tokenizer.py

Co-authored-by: Ruhollah Majdoddin <r.majdodin@gmail.com>

* use tiktoken 0.3.1

* reflecting suggestions

* cleanup

* bypassing load_tiktoken_bpe to avoid blobfile dep

---------

Co-authored-by: Ruhollah Majdoddin <r.majdodin@gmail.com>
This commit is contained in:
Jong Wook Kim
2023-03-13 05:34:16 -04:00
committed by GitHub
parent ad3250a846
commit 839639a223
15 changed files with 100601 additions and 100096 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
MANIFEST.in
View File

@@ -2,6 +2,4 @@ include requirements.txt
include README.md
include LICENSE
include whisper/assets/*
include whisper/assets/gpt2/*
include whisper/assets/multilingual/*
include whisper/normalizers/english.json