Evaluating RAG with RAGAS and DeepEval

Why this matters

If you cannot measure your RAG, you cannot improve it. Every senior interview eventually asks "How did you measure RAG quality?" The boring, professional answer — golden set + RAGAS metrics + LLM-as-judge with calibration — beats every vague "we tested it manually" reply by miles.

Learning objectives

1. Build a golden evaluation set the right way.
2. Use RAGAS to score Faithfulness, Answer Relevancy, Context Precision, Context Recall, Factual Correctness, Semantic Similarity.
3. Use DeepEval as a CI-friendly alternative.
4. Calibrate LLM-as-judge against human labels.
5. Catch regressions in CI before merging.

1. Golden datasets — the foundation

A golden dataset is input + expected output (+ optionally retrieved contexts). Without it, every "improvement" is wishful thinking.

How to build one

• 50-200 pairs is enough to start.
• Cover all important query types: factual, multi-hop, ambiguous, out-of-corpus, adversarial.
• Include expected sources (chunk IDs) when you can — needed for Context Recall/Precision.
• Tag each example: category, difficulty, domain. So you can slice metrics.

{"q":"How long is parental leave?","a":"12 weeks paid","sources":["hr-policy#L42-L48"],"tags":["hr","factual"]}
{"q":"What is GST 27ABCDE1234F1Z5?","a":"That is a vendor's GSTIN, used in invoices","sources":["finance-glossary#L210"],"tags":["exact-id"]}

Synthetic generation (the trick everyone uses)

You can ask an LLM to write QA pairs from your corpus. RAGAS itself ships a synthesizer.

# uv add ragas
from ragas.testset import TestsetGenerator
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator = TestsetGenerator.from_langchain(
    generator_llm=ChatOpenAI(model="gpt-4.1-mini"),
    critic_llm=ChatOpenAI(model="gpt-4.1"),
    embedding_model=OpenAIEmbeddings(model="text-embedding-3-small"),
)
testset = generator.generate_with_langchain_docs(documents, testset_size=50)

Important: review synthetic Q/A by hand. ~20% will be junk; toss them.

2. The RAGAS metrics (current 2026 API)

Modern RAGAS (v0.2+) uses class-based metrics passed to evaluate:

# uv add ragas datasets langchain-openai
from ragas import EvaluationDataset, evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.metrics import (
    Faithfulness, AnswerRelevancy,
    LLMContextPrecisionWithReference, LLMContextRecall,
    FactualCorrectness, SemanticSimilarity,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4.1-mini"))
emb = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-small"))

samples = [
    {
        "user_input":      q["q"],
        "response":        my_rag_answer(q["q"]),
        "retrieved_contexts": [c["text"] for c in my_rag_retrieve(q["q"])],
        "reference":       q["a"],            # ground truth answer
    }
    for q in golden
]

ds = EvaluationDataset.from_list(samples)
result = evaluate(
    dataset=ds,
    metrics=[
        Faithfulness(llm=llm),
        AnswerRelevancy(llm=llm, embeddings=emb),
        LLMContextPrecisionWithReference(llm=llm),
        LLMContextRecall(llm=llm),
        FactualCorrectness(llm=llm),
        SemanticSimilarity(embeddings=emb),
    ],
)
print(result)
df = result.to_pandas()
df.to_csv("ragas_run.csv", index=False)

What each metric tells you

Use Faithfulness + Context Recall as your primary two metrics. They catch the most common failures.

Pitfall: what counts as "retrieved_contexts"?

Only the actual retrieved chunks. Not your system prompt, not chat history, not the LLM's reasoning. Mistakes here silently break Context Precision/Recall.

3. DeepEval — the CI-friendly alternative

DeepEval treats evals like pytest. You write assert_test(...) and it runs in your CI/CD.

# uv add deepeval
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric, AnswerRelevancyMetric,
    ContextualPrecisionMetric, ContextualRecallMetric,
)

def test_rag_faithfulness():
    case = LLMTestCase(
        input="How long is parental leave?",
        actual_output=my_rag("How long is parental leave?"),
        retrieval_context=[c["text"] for c in retrieve_for("How long is parental leave?")],
        expected_output="12 weeks paid",
    )
    assert_test(case, [
        FaithfulnessMetric(threshold=0.85),
        AnswerRelevancyMetric(threshold=0.8),
        ContextualPrecisionMetric(threshold=0.8),
        ContextualRecallMetric(threshold=0.8),
    ])

Run with deepeval test run. Failing thresholds = failed CI.

4. LLM-as-judge calibration (do not skip)

LLM judges have biases: they prefer longer answers, prefer their own outputs, are inconsistent across runs. Calibrate them.

Process:

1. Sample 100 (input, output) pairs.
2. Have 2 humans label each as correct/incorrect.
3. Run your LLM judge on the same pairs.
4. Compute Cohen's kappa between human and judge.
5. If kappa < 0.6, your judge is bad — improve its rubric or use a stronger judge model.

# uv add scikit-learn
from sklearn.metrics import cohen_kappa_score
print(cohen_kappa_score(human_labels, judge_labels))

For high-stakes evals, consider G-Eval (chain-of-thought judging) or pairwise comparisons. Both are supported by RAGAS and DeepEval.

5. Wiring evals into CI

# .github/workflows/eval.yml
name: rag-eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run pytest tests/eval -q
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

In tests/eval/test_rag.py:

def test_quality_thresholds():
    result = evaluate(dataset=ds, metrics=[Faithfulness(llm=llm), LLMContextRecall(llm=llm)])
    assert result["faithfulness"] >= 0.85, "faithfulness regressed!"
    assert result["context_recall"] >= 0.80, "recall regressed!"

A failing PR builds is the social pressure that keeps quality up.

6. Production "online" evals

After deploy you score live traffic too. Sample 5-10% of requests, run a cheap LLM judge, log scores to LangSmith / Langfuse / Arize.

Alarms: faithfulness drops 10% week-over-week → investigate. New "I do not know" rate spikes → maybe corpus drift.

Hands-on lab (4 hours)

Take your advanced RAG from Lesson 2.4.

1. Build / synthesise a golden set of 50 Q/A pairs from your corpus.
2. Run RAGAS with all 6 metrics.
3. Pick a baseline (vector-only) and a candidate (full advanced pipeline). Compare.
4. Wire DeepEval into a pytest tests/eval/.
5. Add a GitHub Actions workflow that runs the suite on PRs.
6. Make a README.md "evaluation" section with a table:

   | metric | baseline | advanced | Δ |
   | --- | --- | --- | --- |
   | faithfulness | 0.78 | 0.91 | +0.13 |

Acceptance criteria:

• All 6 metrics computed.
• A bad PR (intentional regression) fails CI.
• README is interview-grade.

Common pitfalls

1. No reference answers — Context Recall and Factual Correctness need them.
2. Judging with the same model that generated the answer — bias. Use a different (often stronger) judge.
3. Not stratifying — single average hides which query types fail. Always group by tag.
4. Tiny eval set (5 questions) — high variance, useless. ≥ 30; aim for 100+.
5. Metric goal-hacking — optimising one metric until the others tank. Track all 6.

Self-check

1. Why is Faithfulness more important than Answer Relevancy in regulated industries?
2. Why must retrieved_contexts exclude the system prompt?
3. What is the difference between Context Precision with and without reference?
4. How does G-Eval reduce LLM-as-judge inconsistency?
5. What kappa is "good enough" for an LLM judge to be trusted?

References

• RAGAS docs (current)
• RAGAS metrics list
• DeepEval docs
• LLM-as-judge (Anthropic)
• G-Eval paper
• LangSmith evaluations