AI assistant gave different answers to the same prompt: fix drift

TL;DR — If your AI assistant gives two different answers to the same question, the most common reason is that the requests were not actually identical: chat history, model settings, connected knowledge sources, or retrieval (document lookup) changed between runs. Start by comparing the exact prompt, conversation history, model name, temperature, and attached sources in your provider dashboard; in most cases, pinning those settings and using a saved prompt fixes the inconsistency. Reading time: ~6 min

The scenario

It is a normal Tuesday afternoon and you are trying to answer a customer email quickly. You ask your AI assistant, "Can our refund policy cover annual plans after 30 days?" and it says no. Ten minutes later, you paste what looks like the same question into the same assistant, and now it says yes, with a different explanation and a different citation. Your team starts asking which answer is correct, and you need a practical way to work out whether this is a settings issue, a knowledge-source issue, or a real bug.

Symptoms

• The same or nearly the same prompt returns different wording, different facts, or different citations.
• One answer says something like: "Based on your policy documents, annual plans are non-refundable after 30 days," while another says: "Customers may request a prorated refund after 30 days."
• The assistant cites different documents or sections on each run.
• The answer changes when asked in a new chat versus the existing chat.
• The answer changes after a document sync, content update, or connector refresh.
• In logs or request history, you may see different values for settings such as:

{
  "model": "...",
  "temperature": 0.7,
  "top_p": 1,
  "system_prompt": "...",
  "tools": ["search", "retrieval"]
}

• If your provider exposes retrieval logs, the list of returned chunks (small document excerpts) differs between runs.

Likely causes

Step-by-step diagnosis

1. Compare the exact prompt text.

   • Dashboard path: your AI provider's dashboard → Request history/logs → open both requests → copy the user message from each.
   • Paste both into any text diff tool.
   • This is your problem if you find small differences like added context, punctuation, a date, or attached files.
   • Jump to: Fixes → Prompt was not actually identical.

2. Compare the chat history and instructions.

   • Dashboard path: Request history/logs → open request A and request B → expand full conversation and system/developer instructions.
   • This is your problem if one request includes earlier messages such as "answer briefly" or "use policy docs only," or if one was run in an old chat and the other in a new chat.
   • Jump to: Fixes → Different conversation history or hidden instructions.

3. Compare model and randomness settings.

   • Dashboard path: Request history/logs → request details → compare model, temperature, top_p, seed (if available), and tool settings.
   • This is your problem if the model changed, temperature is above 0, or one run has a fixed seed and the other does not.
   • Jump to: Fixes → Temperature or model settings changed.

4. Compare citations and document versions.

   • Dashboard path: open each answer → Citations/Sources → note document title, last updated time, and version.
   • This is your problem if the two answers cite different files, different versions of the same file, or one answer has no citations at all.
   • Jump to: Fixes → Different knowledge sources or document versions were used.

5. Check retrieval/debug output.

   • Dashboard path: request details → Retrieval/Debug/Trace (name varies by provider).
   • This is your problem if the top returned chunks differ between runs even though the prompt and sources are the same.
   • Jump to: Fixes → Retrieval returned different document chunks.

6. Check moderation, policy, and tool execution.

   • Dashboard path: request details → Safety/Moderation and Tools.
   • This is your problem if one run shows blocked content, disabled web/search/retrieval tools, or a failed tool call.
   • Jump to: Fixes → Safety/policy filters or tool availability changed.

7. If you use an API and need a direct comparison, export the raw requests.

   • CLI option if your provider supports API logs export; otherwise copy the JSON from the dashboard.
   • Compare these fields side by side:

{
  "model": "...",
  "messages": [],
  "temperature": 0,
  "top_p": 1,
  "seed": 12345,
  "tools": [],
  "attachments": [],
  "metadata": {}
}

• If any of those differ, treat that difference as the cause and use the matching fix below.

Fixes

Different conversation history or hidden instructions

Use a fresh chat for repeatability, or save a prompt template that includes the same instructions every time.

• Dashboard path: your AI provider's dashboard → Prompts/Templates (or equivalent) → create a saved prompt with fixed instructions.
• Put the instruction in one place, for example:

You are a support assistant. Answer using only the attached policy documents. If the answer is not in the documents, say "I don't know based on the provided sources." Quote the exact section title when possible.

• For testing, always ask the question in a new chat with no prior messages.

Verify it worked: ask the same question twice in two new chats using the saved prompt; the answer and citations should match closely.

Temperature or model settings changed

For factual support answers, reduce randomness and pin the model.

• Dashboard path: Assistant settings → Model/Generation settings.
• Set:

{
  "model": "your-chosen-model",
  "temperature": 0,
  "top_p": 1
}

• If your provider supports a seed (a fixed random starting value), set one fixed value for testing:

{
  "seed": 12345
}

• If you use the API, send the same values on every request:

curl https://api.example.com/v1/responses \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model":"your-chosen-model",
    "temperature":0,
    "top_p":1,
    "seed":12345,
    "input":"Can our refund policy cover annual plans after 30 days?"
  }'

Verify it worked: request history shows the same model and generation settings on both runs.

Different knowledge sources or document versions were used

Pin the source set and re-index after updates.

• Dashboard path: Knowledge base/Documents → review connected files and last updated times.
• Remove duplicate or outdated files, then upload the current canonical (official) version only.
• If your provider has a "sync" or "re-index" action, run it after replacing documents.
• Name documents clearly, for example:

refund-policy-v2026-07-15.pdf
terms-of-service-v2026-07-15.pdf

• In the assistant settings, restrict answers to that source set if your provider offers a source selector.

Verify it worked: both answers cite the same document names and version dates.

Retrieval returned different document chunks

Tighten retrieval so the assistant pulls the same evidence more reliably.

• Dashboard path: Knowledge base/Retrieval settings (name varies).
• If available, increase the number of returned chunks slightly and prefer semantic + keyword search together.
• Add a short retrieval instruction to your system prompt:

When answering policy questions, prefer chunks from documents with titles containing "policy" or "terms". If multiple chunks conflict, quote both and say the policy needs review.

• If your provider allows metadata filters, filter to the right document type or collection.

Verify it worked: retrieval/debug view shows the same top chunks for repeat questions.

Prompt was not actually identical

Standardize the wording and remove accidental context.

• Use a saved prompt or a form field in your app instead of free typing.
• Good repeatable prompt:

Question: Can our refund policy cover annual plans after 30 days?
Instructions: Answer from the attached policy documents only. Include the exact section title and one short quote.

• Do not paste screenshots, extra notes, or previous replies into one run but not the other.

Verify it worked: a text diff shows the two prompts are exactly the same.

Safety/policy filters or tool availability changed

Align the environment so both runs have the same tools and policy path.

• Dashboard path: Assistant settings → Tools → enable the same tools for both test runs.
• Dashboard path: Request history → Safety/Moderation → compare whether one request was filtered.
• If a tool failed, retry after fixing the connector or source permission.

⚠️ If you change moderation or safety settings, do not weaken them broadly just to force consistency. Test in a limited environment first, because looser settings can create compliance and trust problems.

• For API-based apps, log tool availability with each request:

{
  "tools_enabled": ["retrieval"],
  "knowledge_base": "support-policies",
  "environment": "production"
}

Verify it worked: both runs show the same enabled tools and no blocked tool calls.

Prevention

• Pin the model and generation settings in code or assistant config:

{
  "model": "your-chosen-model",
  "temperature": 0,
  "top_p": 1,
  "seed": 12345
}

• Save one canonical support prompt and reuse it everywhere instead of ad-hoc typing:

Answer using only approved policy documents. If the answer is not present, say so. Include source title and section.

• Version your knowledge files and remove duplicates before syncing:

policy-refund-v2026-07-15.pdf
policy-refund-v2026-08-01.pdf

Keep only the current approved version in the live knowledge base.

• Turn on request logging for these fields: prompt, full message history, model, temperature, seed, tools, citations, and document version. This gives you a clean side-by-side comparison during incidents.
• Add a simple regression test in CI (continuous integration, automated checks before release) that asks the same question twice and fails if citations differ unexpectedly:

{
  "question": "Can our refund policy cover annual plans after 30 days?",
  "expected_sources": ["refund-policy-v2026-07-15.pdf"]
}

• After every document update, run a small acceptance test set of 5-10 common questions and confirm the returned citations before your team uses the assistant with customers.

If you work through the checks in this order, you will usually find that the assistant was not operating under the same conditions both times. Once you pin the prompt, history, model settings, and source set, "different answers to the same question" becomes much rarer and much easier to explain when it does happen.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.