Portfolio 09 · run envelope

LLM training time and cost planner

Check per-rank model-state fit, validate topology factors, keep measured throughput beside a clearly labeled theoretical FLOPs envelope, and price checkpoint replay on whole nodes.

Decision this answers: Does the model state fit the selected topology, and what completion time and spend follow from an actual throughput measurement?

Scenario

Decision summary

Per-rank model state

97.79 GiBexact-derived

Fits usable HBM: false.

[2]

Measured runtime

21604.94 hmeasured

Tokens divided by compatible measured throughput.

[4]

Theoretical envelope

55854.30 hupper bound

Peak FLOPS path; never labeled a forecast.

[3]

Whole-node run cost

$2145439.51model estimate

Measured runtime plus expected replay.

[4]

Direct-labeled result profile

Per-rank model state Measured runtime Theoretical envelope Whole-node run cost

The tables below are the accessible source of truth; bar lengths never carry identity alone.

Topology and evidence ledger

Topology and evidence ledger
Candidate	Memory driver	Compute driver	Runtime authority	Checkpoint treatment	Billing unit	Evidence
Default 8-rank plan	70B total parameters	70B active parameters	18k measured tokens/s	Expected replay separate	Whole node-hour	measured[2][4]
Theoretical comparison	Same exact state	6 × active params × tokens	Peak FLOPS × MFU × efficiency	Not included	Envelope only	upper bound[3][1]

Official surface versus this site

Training Compute-Optimal Large Language Models

Use the official surface to confirm native prices, counting rules, limits, and product eligibility.

This offline workbench

The primary paper frames training compute. This workbench adds exact state sharding, usable HBM, measured throughput, topology efficiency, checkpoints, interruption replay, and whole-node billing.

Sources and methodology

Every result-affecting reference is visible here without JavaScript and is retained in the JSON export.

[1] Training Compute-Optimal Large Language Models

Authority: DeepMind
Native identifier: arXiv:2203.15556
Unit: parameters, tokens, FLOPs
Evidence: exact-derived
As of: 2026-08-01
Retrieved: 2026-08-01
Freshness: reference
Normalization: The operation envelope is labeled theoretical and does not predict quality.

Open the authoritative source

[2] PyTorch Fully Sharded Data Parallel

Authority: PyTorch
Native identifier: torch.distributed.fsdp
Unit: sharded model state
Evidence: exact-derived
As of: 2026-08-01
Retrieved: 2026-08-01
Freshness: reference
Normalization: Weight, gradient, and optimizer sharding factors are validated independently.

Open the authoritative source

[3] NVIDIA NeMo FLOPs utilities

Authority: NVIDIA
Native identifier: model FLOPs utilization
Unit: FLOPs
Evidence: upper bound
As of: 2026-08-01
Retrieved: 2026-08-01
Freshness: reference
Normalization: Peak FLOPS, MFU, and scaling efficiency produce an envelope, never delivered throughput.

Open the authoritative source

[4] MLPerf Training benchmark

Authority: MLCommons
Native identifier: MLPerf Training
Unit: time to solution
Evidence: measured
As of: 2026-08-01
Retrieved: 2026-08-01
Freshness: reference
Normalization: Measured throughput is user supplied and benchmark methodology caveats remain visible.

Open the authoritative source

Limits and non-claims

Theoretical FLOPS do not predict delivered throughput or training quality.
Spot interruption and replay are scenario assumptions, not first-party price facts.
Quota, cluster availability, network topology feasibility, and non-CUDA portability are not asserted.