Guides
Quantisation
Fit a larger checkpoint on smaller hardware. The client supports 8- and 4-bit weight-only quantisation at load time.
Load in 4-bit
NF4 with double quantisation costs roughly one quarter of bf16 memory, typically within a point of the full-precision score.
Load in 4-bit
model = AutoModel.from_pretrained(REPO, quantization="nf4", compute_dtype="bf16")
Trade-offs
Quantisation hurts long-context reasoning and non-English generation first. Re-run the harness on your own task before shipping.