StackLift
All terms

Glossary

Inference

Running a trained AI model to produce an output: the step that happens every time someone uses the AI.

Inference is the process of running a trained machine learning model on new input to produce an output, such as an answer, a classification or an image. In an LLM application, every request you send to the model is an inference call. It is distinct from training, which builds the model once, and it is where most ongoing running cost and latency come from.

For language models, inference has two phases: reading the prompt, then generating the reply one token at a time. Prompt length, output length, model size and the hardware or service running it all affect speed and cost.

You can buy inference as a hosted API or run it yourself with a serving engine on your own or rented GPUs. Hosted is simpler; self-hosting gives control over data and cost shape but adds operations work. See [private LLM development](/solutions/private-llm-development).

Start in 60 seconds

Put this to work on your project.

Get an AI-assisted, architect-reviewed scope, cost range, and timeline. Free, no login.

The complete guide

Inference: the complete guide

Updated November 18, 2026 ยท Published by , Founder, StackLift AI. Written with AI assistance; how we make content.

Short answer

Inference is running a trained AI model on new input to produce an output, such as an answer, a label or an image. In an LLM application, every request to the model is an inference call. It differs from training, which builds the model once, and it is where most ongoing running cost and response time come from.

What is inference in AI?

Training teaches a model from data. Inference uses the finished model. When you ask a chatbot a question and it replies, that reply is inference. When a photo app recognizes a face or a bank scores a transaction for fraud, that is inference too.

For most software teams, inference is the part they actually touch. You rarely train a model from scratch, but you pay for inference every time a user does something, and users feel its speed on every click.

What is the difference between training and inference?

Training compared with inference
AspectTrainingInference
PurposeBuild or adjust a modelUse a model to produce output
How oftenOccasionallyEvery request
Who does itModel builders, or you when fine-tuningAnyone running the model
Cost shapeLarge, upfront computeOngoing, grows with usage
Main concernData and qualityLatency, throughput and cost
HardwareLarge clusters of acceleratorsOne or a few GPUs, or a hosted API

A useful rule: training is a project, inference is a utility bill. Planning a product around AI means planning for the bill.

How does LLM inference work, step by step?

  1. Your application sends a prompt: instructions, retrieved documents and the user's message.
  2. The text is split into tokens, the units the model reads.
  3. Prefill: the model processes the whole prompt in parallel and builds an internal memory of it, often called the KV cache.
  4. Decode: the model generates the reply one token at a time, each token depending on everything before it.
  5. Sampling settings such as temperature shape which token is picked at each step.
  6. Generation stops at an end marker or a length limit, and the tokens are turned back into text.
  7. The response streams to the user, or returns whole, and your application logs it.

Two phases explain most behavior. Prefill is fast per token and depends on prompt length. Decode is slower per token and depends on output length. That is why a long prompt delays the first word, and a long answer delays the last.

What affects the speed and cost of inference?

Main factors
FactorEffectWhat you can do
Model sizeBigger models are slower and cost more per tokenUse the smallest model that passes your tests; see small language models
Prompt lengthLonger prompts cost more and slow the first tokenTrim instructions, retrieve fewer and better passages, use prompt caching
Output lengthEvery generated token adds time and costAsk for concise or structured output, set a limit
HardwareFaster accelerators and more memory raise throughputMatch hardware to the model and traffic
Batching and concurrencyServing many requests together raises efficiency but can add waitTune the batching in your serving engine
Reasoning modeModels that think before answering use many more tokensUse it only on tasks that need it
Network and regionDistance adds delayHost near your users and data

Most savings come from choosing the right model for each task and sending less text, not from clever hardware.

Should you use a hosted API or run inference yourself?

Hosted API compared with self-hosting
PointHosted APISelf-hosted
SetupAn API key and a few lines of codeServers or GPUs, a serving engine, monitoring
ModelsWhatever the provider offersOpen-weight models you choose and can tune
Data controlGoverned by the provider's termsStays in your environment
Cost shapePay per token, scales to zeroPay for capacity, cheaper only at steady high volume
OperationsProvider's jobYour job: scaling, upgrades, failures
Good forMost products, early stages, variable trafficStrict data rules, steady heavy load, special models

Start hosted unless a rule or a cost model says otherwise. If you do need control over where data goes, see private LLM development and the comparison of an open-source LLM vs. an API.

What does serving a model yourself involve?

A serving engine such as vLLM, Hugging Face Text Generation Inference, NVIDIA Triton or llama.cpp loads the model, batches incoming requests and streams tokens back. Around it you need load balancing, autoscaling, health checks, authentication, logging and a rollout process for new model versions.

  • Pick hardware with enough memory for the model weights and the KV cache of your longest prompts.
  • Consider quantization, which stores weights in lower precision to cut memory and cost, and test that quality holds on your tasks.
  • Plan for spikes: queue, shed load or fall back to a hosted model.
  • Keep a second model or provider ready for outages.
  • Treat model files and endpoints as sensitive assets, and restrict access.

This is real operations work. Teams that run inference themselves treat it like any other production service, with observability, alerts and runbooks.

How do you make inference faster and cheaper?

  1. Measure first: log tokens in and out, latency and cost per request and per feature.
  2. Pick the smallest model that meets quality on your test set, per task.
  3. Route easy requests to a cheaper model and hard ones to a stronger one; see model routing.
  4. Shorten prompts: remove repeated instructions, trim retrieved text, cache stable prefixes.
  5. Limit output length and use structured formats instead of long prose.
  6. Stream responses so users see progress while the model works.
  7. Cache repeated questions and results where answers do not need to be fresh.
  8. Batch background jobs, and run them off-peak where the provider offers lower rates.

These steps are covered in more depth under AI cost optimization and the LLM cost optimization playbook.

What are real-world examples of inference?

  • A support chatbot sends the customer's question and retrieved help articles to a model and streams the reply.
  • An invoice agent calls a model to extract fields from each scanned bill.
  • An ecommerce site scores product reviews for sentiment in nightly batches.
  • A coding assistant completes code as the developer types, where very low delay matters.
  • A clinic's scribe tool turns a recorded visit into a structured note, using a model hosted in a controlled environment.
  • A fraud system runs a small model on every transaction in a few milliseconds.

Notice how different the demands are: the coding assistant needs speed, the nightly job needs low cost, and the clinic needs data control. One inference setup rarely fits all three.

What are latency and throughput, and which matters more?

Latency is how long one user waits. For chat, it is usually split into time to first token and the speed of the rest of the reply. Throughput is how many requests or tokens the system handles at once. They pull against each other: batching more requests raises throughput but can make each one wait longer.

Streaming helps perceived speed a great deal. A reply that starts in under a second and finishes in ten feels faster than one that arrives whole after eight, even though the total time is similar.

Interactive features should be tuned for latency. Background work, such as classifying a backlog of documents, should be tuned for throughput and cost. Set a target for each feature, and measure the user-facing number, not only the average.

What kinds of inference workloads are there?

Workload types
TypeWhat it isWhat matters most
Real-time (online)A user waits for the answerLow latency, steady availability
Batch (offline)A large set of items processed in one jobCost per item and total throughput
StreamingTokens or events returned as they are producedTime to first token, smooth delivery
Edge or on-deviceThe model runs on a phone, browser or local machineSmall models, privacy, offline use
AgenticMany model calls chained with tools in one taskTotal time and cost across the whole run, plus error handling

Agents deserve special mention. One user request can trigger many inference calls, each with its own prompt and tools, so costs and delays multiply. Budget per task, not per call, and cap the number of steps an agent may take. See AI agents.

Many products use more than one type. The chat window is real-time, the nightly summary is batch, and the mobile keyboard might run a tiny model on the device. Choosing the type per feature, instead of one setup for everything, is one of the simplest ways to control both cost and experience.

What are the common inference mistakes?

Mistakes and fixes
MistakeWhat happensFix
Using the biggest model everywhereCost and delay climb with no quality gainTest smaller models per task
Ignoring prompt growthCosts creep up as context is addedTrack tokens per request over time
No timeouts or fallbacksOne slow call hangs the featureSet timeouts and a backup model or message
Self-hosting too earlyOperations work crowds out product workStart hosted and move only with a clear reason
Testing only at low loadSpeed collapses under real trafficLoad test before launch
No usage limitsA bug or abuse produces a surprise billSet budgets, rate limits and alerts

How do you measure inference performance?

Track time to first token, total response time, tokens per second, error and timeout rates, and cost per request and per user action. Break each down by feature and model, so you can see which part of the product drives the bill.

Pair these with quality measures from your test set, so an optimization that saves money is not quietly lowering answer quality. See AI observability and LLMOps.

What does inference mean for project cost and timeline?

There are two cost lines: building the feature, and running it. Building covers the integration, prompts, retrieval, evaluation and safeguards. Running is the inference bill plus hosting and monitoring, and it grows with usage. Cost-saving features such as routing and caching add build effort but reduce what you pay each month.

Check current per-token prices at LLM pricing. For project ranges, see the AI LLM integration cost page and the free estimate.

How StackLift approaches inference

StackLift scopes inference during discovery: which model for which task, where the data may go, what delay users will accept and what the monthly run cost looks like. A senior architect reviews the choices before they become commitments.

Observability, security review and rollback are part of the plan from sprint one. Agents work on feature branches, and engineers own the decisions. See LLM integration.

Questions people ask

What is the difference between training and inference?

Training builds or adjusts a model using data. Inference runs the finished model on new input to produce an output. Training happens occasionally; inference happens on every request.

Why is inference expensive?

Large models need powerful hardware for every request, and cost grows with the number of tokens read and generated. High usage multiplies that, so small per-request costs add up.

What is a token in inference?

A token is a small piece of text, often part of a word, that the model reads and writes. Providers usually price inference by tokens in and tokens out.

What is time to first token?

It is the delay between sending a request and receiving the first part of the reply. It depends mostly on prompt length and how busy the service is.

Is inference the same as an API call?

When you use a hosted model, each API call is one inference request. Running a model on your own servers is also inference, with no external API involved.

Can I run LLM inference on a laptop?

Smaller open-weight models can run on a laptop with tools such as llama.cpp or Ollama, which is useful for testing. Production traffic normally needs dedicated servers or a hosted service.

What is quantization?

Quantization stores a model's weights in lower precision, which reduces memory and can speed up inference. Quality can drop slightly, so test it on your own tasks.

How do I reduce inference cost?

Use the smallest model that passes your tests, shorten prompts, cache what repeats, limit output length and route easy requests to cheaper models. Measure each change.

Sources

Sources and further reading

The standards, rules, research and official documentation this page draws on.