RAG Evaluation, Metrics & Monitoring

1. The Evaluation Crisis: Why ROUGE & BLEU Fail at RAG

Traditional machine translation metrics like BLEU and ROUGE rely on exact lexical n-gram overlaps between a model's output and a human reference sentence. In generative AI, an answer can be 100% factually accurate while using completely different vocabulary than the ground truth.

2. The RAG Triad

Developed by TruLens and adopted across the AI industry, the RAG Triad identifies the three critical failure modes of any retrieval-augmented system:

3. Standard Ragas Metrics Explained

Ragas (Retrieval Augmented Generation Assessment) is the industry-standard open-source framework for quantifying pipeline performance:

4. Hands-On Python: Automated RAG Pipeline Evaluation with Ragas

Here is a complete script demonstrating how to run an automated evaluation over test questions using ragas and datasets:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
)

# Prepare an evaluation dataset consisting of questions, contexts, answers, and ground truths
eval_data = {
    "question": [
        "What is the maximum token context window of text-embedding-3-small?",
        "How does HNSW navigate multi-layer graphs?"
    ],
    "answer": [
        "The text-embedding-3-small model has an 8,191 token context window.",
        "HNSW uses a multi-layer graph where upper layers have sparse, long-range links and layer 0 has dense local connections."
    ],
    "contexts": [
        [
            "OpenAI text-embedding-3-small supports up to 8,191 input tokens and outputs 1536 dimensions.",
            "Pricing for text-embedding-3-small is $0.02 per 1M tokens."
        ],
        [
            "HNSW organizes vectors into hierarchical layers inspired by skip-lists. Top layers execute long express jumps.",
            "In layer 0, all nodes are present with dense connections to find exact local nearest neighbors."
        ]
    ],
    "ground_truth": [
        "The text-embedding-3-small model supports a context window of 8,191 tokens.",
        "HNSW searches top layers with long-range hops and transitions down to layer 0 for local neighbor search."
    ]
}

dataset = Dataset.from_dict(eval_data)

# Run automated evaluation using LLM-as-a-judge
results = evaluate(
    dataset=dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall
    ]
)

df_results = results.to_pandas()
print("=== RAG Evaluation Scorecard ===")
print(df_results[["question", "faithfulness", "answer_relevancy", "context_precision", "context_recall"]])

5. Production Observability & Feedback Loops

In production, evaluating static test sets is not enough. Continuous observability tools (such as LangSmith, Arize Phoenix, and TruLens) monitor live telemetry:

• Latency Tracing: Tracking embedding generation time vs. vector search time vs. LLM token streaming time.
• User Feedback Logging: Capturing thumbs-up/thumbs-down ratings and linking them back to retrieved chunk IDs.
• Automated CI/CD Gates: Running Ragas evaluation in GitHub Actions. If a prompt or chunking tweak causes Faithfulness to drop below 0.90, the PR build fails automatically.