This prevents the following error on Windows (when using
a multi-process DataLoader, for example):
AttributeError: Can't pickle local object '_transform.<locals>.<lambda>'
* Add truncate_text option to tokenize
This makes it possible to run tokenize on texts that are longer than the number of tokens
that fit the context length without having to try to guess how to cut in number of
characters beforehand
* add doc, rename to just "truncate", use eot_token
Co-authored-by: Jong Wook Kim <jongwook@openai.com>
Du besuchst diese Seite mit einem veralteten IPv4-Internetzugang. Möglicherweise treten in Zukunft Probleme mit der Erreichbarkeit und Performance auf. Bitte frage deinen Internetanbieter oder Netzwerkadministrator nach IPv6-Unterstützung.
You are visiting this site with an outdated IPv4 internet access. You may experience problems with accessibility and performance in the future. Please ask your ISP or network administrator for IPv6 support. Weitere Infos | More Information Klicke zum schließen | Click to close