Inference Cost

Inference cost is what it costs to run a trained model on a request — usually priced per token for hosted APIs — as distinct from training cost, a one-time or periodic expense most teams building on foundation models never pay directly. It scales with token volume, model choice, and context size, and is the line item that turns a good demo into an uneconomical product if it isn't modeled before shipping. Model-agnostic routing — sending easy requests to a cheaper model and hard ones to a stronger one — is one of the most direct levers for controlling it.

← Back to the full glossary