2.1.2. Chat Completions and Prompt Engineering

💡 First Principle: A chat completion is a list of role-tagged messages (system, user, assistant) plus sampling parameters. The system message sets durable behavior; user/assistant messages carry the turn-by-turn exchange; parameters like temperature tune randomness, not intelligence.

The system message is your highest-leverage prompt-engineering tool: it persists across the whole exchange and defines role, constraints, tone, and output format. Few-shot examples (showing a couple of input→output pairs in the messages) steer format and style far more reliably than describing the format in prose. Parameters matter for control: temperature and top_p govern diversity (low = deterministic, high = varied), max_tokens caps response length, and these affect cost and predictability — not the model's underlying capability.

response = client.chat.completions.create(
    model="my-gpt4o-deployment",          # deployment name
    messages=[
        {"role": "system", "content": "You are a concise compliance assistant. Answer only from provided context."},
        {"role": "user", "content": "Summarize the refund window."},
    ],
    temperature=0.2,
    max_tokens=300,
)
print(response.choices[0].message.content)

⚠️ Exam Trap: "Raise the temperature to get a smarter/more accurate answer." Temperature controls randomness, not competence — higher values increase variety and hallucination risk. For factual, repeatable tasks (classification, extraction, compliance), you want low temperature; reach for higher values only when you genuinely want creative variation.

Reflection Question: A summarization feature gives slightly different output every run, frustrating QA. Which single parameter change makes it repeatable, and what do you trade away?