RAG Prompt Architecture & Context Stuffing

1. Anatomy of a Production RAG Prompt

A high-reliability RAG prompt must clearly separate developer policy, retrieved reference passages, and the user's query:

2. The "Lost in the Middle" Phenomenon

Stanford researchers (Liu et al., 2023) discovered a universal characteristic of Transformer attention: retrieval performance forms a distinct U-shaped curve across long context windows.

3. Enforcing Exact Citations & Strict Refusal

To prevent models from filling information gaps with unverified training weights, require verbatim quotes alongside numeric citations:

# Production RAG Prompt Template
rag_system_prompt = """
You are a factual enterprise knowledge assistant.
Your task is to answer user inquiries strictly using the information provided in the <context> block.

Strict Operating Rules:
1. Grounding: Answer ONLY based on direct statements in the context. Never extrapolate or assume.
2. Citations: Every statement or claim in your response MUST be followed by its bracketed citation, e.g., [Doc 1] or [Doc 3].
3. Refusal: If the provided documents do not explicitly contain the complete answer, respond verbatim with:
   'I am sorry, but the provided documentation does not contain sufficient information to answer this question.'
4. Quotes: Whenever possible, support key figures with a short direct quote from the source document.
"""

4. Production Python RAG Prompt Builder

Here is a reusable Python module that formats retrieved vector search hits into an optimized prompt:

from typing import List, Dict
from openai import OpenAI

client = OpenAI()

def build_rag_prompt(query: str, retrieved_chunks: List[Dict[str, str]]) -> List[Dict[str, str]]:
    """
    Constructs an optimized RAG prompt with XML-tagged documents,
    applying Lost-in-the-Middle re-ordering.
    """
    # 1. Lost-in-the-Middle optimization: alternating placement
    reordered_chunks = []
    left, right = 0, len(retrieved_chunks) - 1
    toggle = True
    while left <= right:
        if toggle:
            reordered_chunks.append(retrieved_chunks[left])
            left += 1
        else:
            reordered_chunks.append(retrieved_chunks[right])
            right -= 1
        toggle = not toggle

    # 2. Build XML Document Block
    context_str = "<context>\n"
    for idx, chunk in enumerate(reordered_chunks, start=1):
        context_str += (
            f'<document id="{idx}" source="{chunk["source"]}">\n'
            f'{chunk["content"]}\n'
            f'</document>\n\n'
        )
    context_str += "</context>"

    user_payload = (
        f"{context_str}\n\n"
        f"Question: {query}\n"
        f"Answer with citations:"
    )

    return [
        {"role": "system", "content": rag_system_prompt},
        {"role": "user", "content": user_payload}
    ]

# Demonstration Data
sample_chunks = [
    {"source": "hr_policy_2026.pdf", "content": "Employees are eligible for 24 days of paid time off (PTO) annually."},
    {"source": "equipment_guide.md", "content": "Remote workers receive a $1,000 home office hardware stipend after 90 days."}
]

messages = build_rag_prompt("How many PTO days do remote workers receive?", sample_chunks)
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    temperature=0.0
)

print(response.choices[0].message.content)