Romain Beaumont a2f6647a4e Add truncate_text option to tokenize ...

This makes it possible to run tokenize on texts that are longer than the number of tokens
that fit the context length without having to try to guess how to cut in number of 
characters beforehand

2021-07-08 17:34:16 +02:00

..

__init__.py

Make the repo installable as a package (#26)

2021-01-30 03:05:01 +09:00

bpe_simple_vocab_16e6.txt.gz

Make the repo installable as a package (#26)

2021-01-30 03:05:01 +09:00

clip.py

Add truncate_text option to tokenize

2021-07-08 17:34:16 +02:00

model.py

fixed model loading issue (#66)

2021-03-23 03:05:17 -04:00

simple_tokenizer.py

Make the repo installable as a package (#26)

2021-01-30 03:05:01 +09:00