How a model API call works

Key takeaways A model call is an ordinary HTTP request. You send an ordered list of messages, each tagged with a role — system (standing instructions), user (the input), assistant (prior replies) — plus a few parameters. Text is counted in tokens on the way in and out. The reply is generated text plus a usage report. And it’s stateless: the API remembers nothing, so your code resends the whole conversation every time.

You’ve seen when to build on a model versus just use one. This lesson opens the box: what actually goes over the wire when your program talks to a model. It’s simpler than it looks — a request with a list of messages, and a response with generated text. Everything else is detail on top of that shape.

The request: a list of messages

The core of every call is an ordered list of messages. Order matters: the model reads them top to bottom as a conversation so far. Each message carries a role that tells the model what that message is:

  • system — standing instructions: the model’s persona, the rules it should follow, the format you want back. This is set by you, the developer, not the end user, and it frames everything else.
  • user — the input: a question, a command, a chunk of text to work on. In a chat app this is what the person typed.
  • assistant — the model’s own prior replies. You include these so the model can see what it already said and stay consistent across a multi-turn conversation.

Alongside the messages you send parameters that shape the call: which model to use, the maximum number of output tokens to generate, the temperature, and optionally stop sequences. More on those below.

Tokens

The model doesn’t read characters or whole words — it reads tokens. A token is a chunk of text, often a short word or a piece of one. “Antenna” might be one token; “downconverter” might be several. Both what you send (input tokens) and what the model generates (output tokens) are counted this way.

Two consequences follow. First, every model has a context window — a hard cap on how many tokens fit in a single call, input and output combined. Overflow it and the call fails or older content must be dropped. (See the sibling lesson context windows.) Second, tokens are the unit of both cost and latency: you pay per token, and more tokens take longer to process. We’ll dig into that in cost, latency and limits.

The response

The reply comes back with three things worth knowing:

  • Generated content — the text the model produced. (Depending on how you called it, this may instead be a request to call a tool, or a structured object like JSON.)
  • A finish reason — why the model stopped. Common values: stop (finished naturally), length (hit your output-token limit), tool_use (wants to call a tool).
  • A usage block — how many input and output tokens the call consumed, so you can track cost.

Here is a compact, provider-neutral sketch of a request and its response:

// request
{
  "model": "some-model",
  "max_output_tokens": 200,
  "temperature": 0.7,
  "messages": [
    { "role": "system", "content": "You are a concise assistant." },
    { "role": "user",   "content": "Name three SDR sample rates." }
  ]
}

// response
{
  "content": "Common SDR sample rates include 2.4, 2.5, and 10 MS/s.",
  "finish_reason": "stop",
  "usage": { "input_tokens": 24, "output_tokens": 18 }
}

Field names differ between providers, but the shape — messages in, content plus a finish reason plus usage out — is the same everywhere.

It’s stateless — you carry the conversation

Here is the part that surprises people: the API is stateless. It remembers nothing between calls. Each request is judged entirely on the messages you send in that request, and the moment it responds, the server forgets you were ever there.

So how does a chatbot “remember” the last ten turns? Your code does the remembering. To continue a conversation, you resend the entire message history — every prior user and assistant message — appended with the new user message, on every single call. The model re-reads the whole thing each time.

This is why context, and cost, grow as a conversation gets longer: turn twenty sends all nineteen previous turns plus the new one. Nothing is stored for you; the running conversation lives in your application.

Parameters you’ll actually set

A handful of parameters do most of the work:

  • Model — which model to call. Bigger, more capable models cost more and run slower; choosing well is its own topic (see choosing a model for a feature).
  • Maximum output tokens — a ceiling on how much the model may generate. It caps cost and latency, but set it too low and replies get cut off (finish reason length).
  • Temperature — how much randomness the model uses when picking each next token. Lower is more focused and repeatable; higher is more varied. The mechanics are covered in how models decide.
  • Stop sequences — strings that, when generated, make the model halt immediately. Handy for keeping output inside a boundary you define.

Quick check: how does the model "remember" earlier turns of a chat?

Recap

  • A model call is an HTTP request carrying an ordered list of messages plus parameters, and it returns generated text with a usage report.
  • Each message has a role — system (standing instructions), user (the input), assistant (prior replies) — and order matters.
  • Text is measured in tokens on both input and output; the context window caps how many fit, and tokens drive cost and latency.
  • The response carries content, a finish reason, and a usage block.
  • The API is stateless: to continue a conversation your code resends the whole history every call, which is why context and cost grow.
  • The parameters you’ll set most are model, max output tokens, temperature, and stop sequences.

Next up: your first API call.