Prompt Evaluation, LLM-as-a-Judge & DSPy

1. Moving Beyond "Vibe Checks": The Evaluation Triad

When you modify a prompt to fix one edge case, you frequently degrade performance across three other tasks without realizing it. A production evaluation suite consists of three layers:

2. LLM-as-a-Judge: The G-Eval Rubric Pattern

Asking an LLM judge "Is this answer good?" yields noisy, subjective ratings. To achieve high correlation with human experts, provide a discrete 1-to-5 scoring rubric with explicit criteria:

# LLM-as-a-Judge Evaluation Rubric
judge_system_prompt = """
You are an impartial expert evaluator for AI customer support responses.
Evaluate the model's response based strictly on the provided Context and User Inquiry.

Score Criteria (Faithfulness to Context):
- Score 1: Direct Hallucination. The response contradicts the context or invents non-existent policies.
- Score 2: Unsupported claims. The response includes facts not mentioned in the context.
- Score 3: Partially supported. The main answer is accurate, but minor details are assumed.
- Score 4: Fully faithful. Every single claim is directly backed by the context, with minor wordiness.
- Score 5: Flawless. 100% grounded in the context, perfectly concise, and directly answers the query.

Provide your output in JSON:
{
  "reasoning": "<detailed step-by-step critique>",
  "score": <integer from 1 to 5>
}
"""

3. Automated Prompt Compilation: Stanford DSPy

Developed by Stanford NLP, DSPy (Declarative Self-improving Python) introduces a revolutionary paradigm:stop writing raw prompt strings. Instead, define your task as code signatures and let teleprompter optimizers synthesize the optimal few-shot exemplars and instructions:

import dspy

# 1. Configure the Language Model
turbo = dspy.LM('openai/gpt-4o-mini', api_key="sk-...")
dspy.configure(lm=turbo)

# 2. Define Input / Output Signatures (No string prompts!)
class QAWithContext(dspy.Signature):
    """Answer technical questions based on the provided reference context."""
    context = dspy.InputField(desc="Documentation snippets")
    question = dspy.InputField(desc="User inquiry")
    answer = dspy.OutputField(desc="Precise, grounded answer")

# 3. Define a Modular Pipeline
class RAGPipeline(dspy.Module):
    def __init__(self):
        super().__init__()
        # Chains of Thought reasoning module
        self.generate_answer = dspy.ChainOfThought(QAWithContext)

    def forward(self, context, question):
        return self.generate_answer(context=context, question=question)

# 4. Automated Prompt Optimization (MIPRO / BootstrapFewShot)
from dspy.teleprompt import BootstrapFewShot

# Train/validation dataset
trainset = [ ... ]

# Define validation metric (e.g. semantic match or LLM judge)
def validate_answer(example, pred, trace=None):
    return example.answer.lower() in pred.answer.lower()

# The Optimizer tunes instructions and selects optimal exemplars automatically!
teleprompter = BootstrapFewShot(metric=validate_answer, max_bootstrapped_demos=4)
compiled_rag = teleprompter.compile(RAGPipeline(), trainset=trainset)

4. Production Cost Optimization: Prompt Caching

When building production chatbots or RAG systems with massive context (such as a 50,000-token employee handbook), re-sending those tokens on every user turn is costly and slow. Modern providers (Anthropic Prompt Caching, OpenAI Cached Tokens) reuse pre-computed attention keys and values: