DI
distilbert/distilbert-base-uncased-finetuned-sst-2-english
text-classificationtransformersapache-2.070M params · updated Dec 2023
Usage
usage.pytemplate from repo id
from osai import AutoModelForCausalLM, AutoTokenizer
repo = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo, dtype="bfloat16", device_map="auto",
)
msgs = [{"role": "user", "content": "Explain speculative decoding."}]
ids = tok.apply_chat_template(msgs, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0]))Hardware
Computed from 0.07B parameters × bytes per parameter, plus 20% for activations and KV cache. Not measured.
| Precision | Weights | Estimated VRAM | Fits on |
|---|---|---|---|
| bf16 | 128 MB | 0.1 GB | CPU or T4 16 GB |
| int8 | 63.9 MB | 0.1 GB | CPU or T4 16 GB |
| nf4 | 31.9 MB | 0.0 GB | CPU or T4 16 GB |