Your First Model Integration

Calling a model from code is three lines. Calling it in a way that survives a bad afternoon at the provider is the rest of this lesson. The difference is not model expertise. It is the ordinary discipline you would apply to any third-party HTTP dependency that is slow, occasionally unavailable, and billed per use.

Treat it that way and the surprises are small. The parts people skip are always the same four: where the key lives, what happens when the call hangs, what happens when you are rate limited, and whether anyone can tell afterwards what it cost.

import logging
import os
from openai import OpenAI, APITimeoutError, RateLimitError
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential

log = logging.getLogger(__name__)

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],   # fails at start-up if absent
    timeout=20.0,
    max_retries=0,                          # one visible retry policy, below
)

@retry(
    retry=retry_if_exception_type((RateLimitError, APITimeoutError)),
    wait=wait_exponential(multiplier=1, min=1, max=20),
    stop=stop_after_attempt(4),
    reraise=True,
)
def answer_policy_question(question: str, extract: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        temperature=0,
        max_completion_tokens=250,
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer only from the POLICY block. If it does not cover the "
                    "question, reply exactly: Not covered by the supplied policy."
                ),
            },
            {"role": "user", "content": f"POLICY\n{extract}\n\nQUESTION\n{question}"},
        ],
    )

    choice = response.choices[0]
    log.info(
        "policy question answered",
        extra={
            "model": response.model,
            "prompt_tokens": response.usage.prompt_tokens,
            "completion_tokens": response.usage.completion_tokens,
            "finish_reason": choice.finish_reason,
        },
    )
    if choice.finish_reason == "length":
        log.warning("reply was truncated at the token cap")

    return choice.message.content

• os.environ with square brackets raises immediately if the variable is missing, so a misconfigured deployment fails at start-up with an obvious message rather than later, in front of a user, as a confusing 401.
• timeout=20.0 is the line most often absent. Without it a request can hang for as long as the connection stays open, holding a worker thread and a person who thinks the page is broken.
• max_retries=0 switches off the SDK's built-in retries. Two retry mechanisms stacked on top of each other multiply, and the one you did not know about is the one that surprises you in the logs.
• The retry applies to rate limits and timeouts only. Both are transient. A malformed request returns the same error four times in a row and the backoff just delays the failure.
• wait_exponential spaces the attempts roughly 1, 2, 4 and 8 seconds apart. Retrying instantly makes a rate limit worse, because your retries are part of the rate you are exceeding.
• response.usage is where the token counts live. Log them with the model name on every call. Without that line you cannot answer why the bill moved, or which feature is responsible for most of it.
• finish_reason tells you whether the reply ended naturally or hit max_completion_tokens. A truncated answer displayed as a finished one is a bug your users will report as the assistant being cut off mid-sentence.

def stream_policy_answer(question: str, extract: str):
    stream = client.chat.completions.create(
        model="gpt-4.1-mini",
        temperature=0,
        max_completion_tokens=250,
        stream=True,
        messages=build_messages(question, extract),
    )
    for event in stream:
        piece = event.choices[0].delta.content
        if piece:
            yield piece

• The reply arrives in fragments as the tokens are produced, so the first words appear in a fraction of a second instead of after the whole answer is ready.
• Total time is unchanged. What changes is the waiting, which stops being a blank screen. For a two-sentence answer this matters little; for a long summary it is the difference between usable and abandoned.
• The trade-off is real: you are showing output before you have seen all of it. Anything you intend to validate, parse, or check against a guardrail cannot be streamed straight to the user, because the check needs the whole response.
• Usage figures may arrive differently when streaming, sometimes only in a final event and sometimes needing an extra option on the request. Check the current reference — and keep logging them, because a streamed call costs exactly the same as a buffered one.