Speed and cost
How fast is it, and what does it really cost, on each provider?
Each model is measured on every provider that serves it: median and 90th percentile latency, decisions per minute, cold start, and billed cost per 1,000 decisions. Rate limits and errors are shown, not hidden.
Small open models we run ourselves on a Mac Mini are added later and reported separately, with memory use and decisions per minute, never ranked against hosted providers.
Latency, median median seconds per decision
Median wall time per decision, timed from our server with the network included.
Latency, p90 90th percentile seconds per decision
90th percentile wall time per decision.
Throughput decisions per minute at concurrency 16
Decisions finished per minute at concurrency 16, or lower where the provider's rate limit stops it, as stated.
Cold start first call after 30 idle minutes
Median latency of the first call after at least 30 minutes idle, from 20 samples. Indicative only.
Cost per 1,000 decisions billed, US dollars
Billed cost per 1,000 decisions from the provider's usage records, next to list price times measured tokens.
Test items and precision
- Items
- 900 timed calls per model and provider, plus throughput and cold-start passes
- Precision
- Latency is timed from one server with the network included, and stated as such. Cold start is indicative only.