# Inference Cost

Source: https://customlabs.io/glossary/inference-cost/
Updated: 2026-09-13

Cost & ops

# Inference Cost

Inference cost is what it costs to run a trained model on a request.

It is priced per token for hosted APIs and is distinct from training cost.

It scales with token volume, model choice, and context size.

Left unmodeled before shipping, it turns a good demo into an uneconomical product.

Model-agnostic routing, sending easy requests to a cheaper model, is a direct lever for controlling it.

[Insight: What AI actually costs](https://customlabs.io/insights/what-ai-actually-costs/)[Insight: AI cost benchmark by workload](https://customlabs.io/insights/ai-cost-benchmark-by-workload/)[AI Cost Calculator](https://customlabs.io/tools/cost-calculator/)

[← Back to the full glossary](https://customlabs.io/glossary/)

## Related terms

[Token A token is the basic unit of text an LLM reads and writes.](https://customlabs.io/glossary/token/)[Model-Agnostic Architecture A model-agnostic architecture lets an application swap or route between LLM providers without a rewrite.](https://customlabs.io/glossary/model-agnostic-architecture/)[Fine-Tuning (vs RAG) Fine-tuning trains a model's weights on your own examples, changing its behavior directly.](https://customlabs.io/glossary/fine-tuning/)[Notebook-to-Production Notebook-to-production is the gap between a working prototype and a system that runs reliably.](https://customlabs.io/glossary/notebook-to-production/)

## More in Cost & ops

[Canary Release A canary release exposes a change to a small, deliberately chosen slice of traffic first.](https://customlabs.io/glossary/canary-release/)[Output Drift Output drift is a gradual change in a live system's output, with no release behind it.](https://customlabs.io/glossary/output-drift/)[Model Deprecation Model deprecation is a provider retiring a model version on its own schedule.](https://customlabs.io/glossary/model-deprecation/)[Acceptance Rate Acceptance rate is the share of an agent's proposed changes a human reviewer approves.](https://customlabs.io/glossary/acceptance-rate/)