Inference TCO Calculator
At what monthly token volume, on which accelerator, at which provider, does self-hosting an open-weights model beat a managed API — once utilization and platform headcount are honest?
The same chip varies about 7x in price. A B200 rents from $3.75/GPU-hour at the cheapest tracked provider to $27.04 at the dearest. Before optimising anything else, check where you are renting.
Presets
The answer
At tokens a month, the option wins: a month against .
- Cheapest self-hosted x at
- Cheapest managed API ·
- Self-hosted $/M tokens vs on the API
- Per day the deck's sweet spot is $10–$50
That is per employee per month — .
No viable self-hosted configuration for this model at this context length. Lower the context, pick a narrower quantization, or include estimated prices.
Where the lines cross
Monthly token volume against monthly cost, both on log scales. The API lines are rays through the origin; the self-hosted line has a fixed floor and steps as replicas are added. The table below is the source of truth — this chart is a reading aid.
Every option, cheapest first
| Option | $/M tok | $/month | tok/s | Accel. | Price | As of |
|---|---|---|---|---|---|---|
| open weights | — | — |
Not viable here () — shown rather than dropped
Show the math
Every number above, with this configuration's inputs substituted. If it disagrees with sites tco quote, the Go engine is the referee.
- Weights
params_total x bytes_per_param= x B =- KV cache
layers x (kv_lora_rank + rope) x bytes= x ( + ) x 1 B2 x layers x kv_heads x head_dim x bytes= 2 x x x x 1 B= , at tokens- Memory fit
ceil((weights + kv x context) / usable_per_chip)= , leaving room for concurrent streams- Decode throughput
min(c x bandwidth x MBU / (step_bytes + c x kv x ctx), flops x MFU / (2 x active))= min(, ) = at concurrency , bound by- Prefill throughput
flops x MFU / (2 x params_active)=- Fleet size
ceil(busy_seconds / (duty_hours x 3600 x utilization))= ceil( / ) =- Compute
instances x $/hr x duty_hours x redundancy= x x x =- Platform
fte x $/month= x =- Total
- a month, per million tokens
Methodology and honest limits
The discrete, error-prone half of this model — memory fit, minimum accelerator count, KV cache sizing, the mixture-of-experts active-versus-total split, the concurrency knee — runs in Go, is covered by table-driven tests, and is baked into a grid at build time. The browser multiplies those terms by the sliders. It never re-derives them.
What the model gets right
- Total parameters drive memory, active parameters drive compute. Every expert of a mixture-of-experts model is resident even when only a few fire per token. Calculators that size memory from the active count claim a 744B model fits on one accelerator.
- Attention variant changes the cache by multiples. Grouped-query attention stores a key and a value tensor per layer; multi-head latent attention compresses both into one low-rank latent. Hybrid models carry a cache on only some of their blocks.
- Batching is not free forever. Decode throughput is memory-bandwidth bound, and the KV read grows with concurrency. That produces a knee, and then a compute ceiling.
- Dense, not sparse, FLOPS. Vendor datasheets usually quote the sparsity-doubled number; the halves are used here.
What it deliberately does not model
- Speculative decoding, chunked prefill, prefix caching, and disaggregated serving. All of these move the numbers, usually in self-hosting's favour.
- Tensor parallelism beyond four instances. A replica smeared across dozens of nodes does not serve at full speed, so those configurations are reported as out of range rather than costed optimistically.
- Licence cost and porting cost. A research licence or a non-CUDA programming model is a real expense this tool leaves to you.
- Egress, storage, and networking beyond a flat egress term.
Provenance
Every price carries a source link and an as_of date, and every row is marked list, estimate, or unknown. Capable silicon with no public price — TPU v7 Ironwood, B300, MI355X — appears explicitly rather than being dropped, because an honest gap is more useful than a quiet omission. The dataset check warns at 60 days and fails continuous integration at 120, so this page cannot silently rot.
The data is hand-curated TOML, not scraped: read it, and send a pull request when it drifts.