Cost & ops
Inference Cost
Inference cost is what it costs to run a trained model on a request.
It is priced per token for hosted APIs and is distinct from training cost.
It scales with token volume, model choice, and context size.
Left unmodeled before shipping, it turns a good demo into an uneconomical product.
Model-agnostic routing, sending easy requests to a cheaper model, is a direct lever for controlling it.
← Back to the full glossaryRelated terms
Token A token is the basic unit of text an LLM reads and writes. Model-Agnostic Architecture A model-agnostic architecture lets an application swap or route between LLM providers without a rewrite. Fine-Tuning (vs RAG) Fine-tuning trains a model's weights on your own examples, changing its behavior directly. Notebook-to-Production Notebook-to-production is the gap between a working prototype and a system that runs reliably.
More in Cost & ops
Canary Release A canary release exposes a change to a small, deliberately chosen slice of traffic first. Output Drift Output drift is a gradual change in a live system's output, with no release behind it. Model Deprecation Model deprecation is a provider retiring a model version on its own schedule. Acceptance Rate Acceptance rate is the share of an agent's proposed changes a human reviewer approves.