Romain Beaumont a2737ac264 Add truncate option to tokenize (#126) ... * Add truncate_text option to tokenize This makes it possible to run tokenize on texts that are longer than the number of tokens that fit the context length without having to try to guess how to cut in number of characters beforehand * add doc, rename to just "truncate", use eot_token Co-authored-by: Jong Wook Kim <jongwook@openai.com>		2021-07-18 20:17:40 -07:00
..
__init__.py	Make the repo installable as a package (#26)	2021-01-30 03:05:01 +09:00
bpe_simple_vocab_16e6.txt.gz	Make the repo installable as a package (#26)	2021-01-30 03:05:01 +09:00
clip.py	Add truncate option to tokenize (#126)	2021-07-18 20:17:40 -07:00
model.py	fixed model loading issue (#66)	2021-03-23 03:05:17 -04:00
simple_tokenizer.py	Make the repo installable as a package (#26)	2021-01-30 03:05:01 +09:00