localhorst/CLIP_on_Tesla_K20Xm
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

correctly tokenizing SOT/EOT tokens (fixes #8)

Browse Source
This commit is contained in:
Jong Wook Kim
2021-01-09 02:55:09 +09:00
parent c89e0c16de
commit ebd0e35aac
3 changed files with 47 additions and 344 deletions
Download Patch File Download Diff File Expand all files Collapse all files

373
Interacting_with_CLIP.ipynb
View File

File diff suppressed because one or more lines are too long

14
README.md
View File

@ -41,7 +41,7 @@ with torch.no_grad():
logits_per_image, logits_per_text = model(image, text) logits_per_image, logits_per_text = model(image, text)
probs = logits_per_image.softmax(dim=-1).cpu().numpy() probs = logits_per_image.softmax(dim=-1).cpu().numpy()
print("Label probs:", probs) # prints: [[0.9956 0.002144 0.002213]] print("Label probs:", probs) # prints: [[0.9927937 0.00421068 0.00299572]]
``` ```
@ -83,7 +83,7 @@ Given a batch of images and a batch of text tokens, returns two Tensors, contain
### Zero-Shot Prediction ### Zero-Shot Prediction
The code below performs of zero-shot prediction using CLIP, as shown in Appendix B in the paper. This example takes an image from the [CIFAR-100 dataset](https://www.cs.toronto.edu/~kriz/cifar.html), and predicts the most likely labels among the 100 textual labels from the dataset. The code below performs zero-shot prediction using CLIP, as shown in Appendix B in the paper. This example takes an image from the [CIFAR-100 dataset](https://www.cs.toronto.edu/~kriz/cifar.html), and predicts the most likely labels among the 100 textual labels from the dataset.
```python ```python
import os import os
@ -125,11 +125,11 @@ The output will look like the following (the exact numbers may be slightly diffe
``` ```
Top predictions: Top predictions:
snake: 41.53% snake: 65.31%
turtle: 24.04% turtle: 12.29%
sweet_pepper: 4.18% sweet_pepper: 3.83%
lizard: 3.92% lizard: 1.88%
leopard: 3.69% crocodile: 1.75%
``` ```
Note that this example uses the `encode_image()` and `encode_text()` methods that return the encoded features of given inputs. Note that this example uses the `encode_image()` and `encode_text()` methods that return the encoded features of given inputs.

4
clip.py
View File

@ -121,7 +121,9 @@ def tokenize(texts: Union[str, List[str]], context_length: int = 77):
if isinstance(texts, str): if isinstance(texts, str):
texts = [texts] texts = [texts]
all_tokens = [_tokenizer.encode(text + "<|endoftext|>") for text in texts] sot_token = _tokenizer.encoder["<|startoftext|>"]
eot_token = _tokenizer.encoder["<|endoftext|>"]
all_tokens = [[sot_token] + _tokenizer.encode(text) + [eot_token] for text in texts]
result = torch.zeros(len(all_tokens), context_length, dtype=torch.long) result = torch.zeros(len(all_tokens), context_length, dtype=torch.long)
for i, tokens in enumerate(all_tokens): for i, tokens in enumerate(all_tokens):