Reference

Streaming datasets

Iterate terabyte corpora without downloading them. Splits are sharded Parquet behind a range-request reader.

Stream a split

Streaming mode yields rows lazily and supports shuffling with a buffer, so training can start immediately.

Stream a split

from osai import load_dataset

ds = load_dataset("HuggingFaceFW/fineweb-edu", split="train", streaming=True)
for row in ds.shuffle(buffer_size=10_000).take(5):
    print(row["score"], row["text"][:80])

Filter server side

Predicates on columns push down to the Parquet reader, so only matching row groups cross the network.

Filter server side

ds = ds.filter(lambda r: r["score"] > 3.5)

← Model cards Quantisation →