Commit Graph

6 Commits

Author SHA1 Message Date
a2737ac264 Add truncate option to tokenize ()
* Add truncate_text option to tokenize

This makes it possible to run tokenize on texts that are longer than the number of tokens
that fit the context length without having to try to guess how to cut in number of 
characters beforehand

* add doc, rename to just "truncate", use eot_token

Co-authored-by: Jong Wook Kim <jongwook@openai.com>
2021-07-18 20:17:40 -07:00
db20393f4a Using non-JIT by default; compat fix with 1.8+ 2021-07-18 18:45:21 -07:00
fd6c1443c2 add RN101 and RN50x4; update paper URL and model card 2021-03-04 12:30:39 -05:00
4c0275784d Load model from path ()
* Load model from path

* showing download progress in "MiB"

* clip.load() now can take a file path

Co-authored-by: Jong Wook Kim <jongwook@openai.com>
2021-02-16 06:19:42 -05:00
8f6deb52a1 showing download progress in "MiB" 2021-02-15 22:00:33 -05:00
3bee28119e Make the repo installable as a package () 2021-01-30 03:05:01 +09:00