deepseek-ai/DeepSeek-V3

text-generationtransformersdeepseek685B params · updated Mar 2025

Usage

usage.pytemplate from repo id

from osai import AutoModelForCausalLM, AutoTokenizer

repo = "deepseek-ai/DeepSeek-V3"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype="bfloat16", device_map="auto",
)

msgs = [{"role": "user", "content": "Explain speculative decoding."}]
ids = tok.apply_chat_template(msgs, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0]))

Hardware

Computed from 684.53B parameters × bytes per parameter, plus 20% for activations and KV cache. Not measured.

Precision	Weights	Estimated VRAM	Fits on
bf16	1.2 TB	1531 GB	Multi-GPU (2× A100 80 GB or more)
int8	638 GB	766 GB	Multi-GPU (2× A100 80 GB or more)
nf4	319 GB	383 GB	Multi-GPU (2× A100 80 GB or more)