aiml
SearchMy learning
Menu

LLM Evals

DeepEval 4.3

Test LLM features with pytest and DeepEval, measure RAG and tool-call quality per stage, and gate prompt, model and dataset changes on regressions in CI.

Prerequisites
Python and pytest basics: test functions, assert and fixtures. Live runs use a Gemini API key or a local Ollama model; replaying the recorded runs needs neither.
Duration
about 2 hours of reading
Cost
Free, no sign-up
Start course

What you'll learn

  • Build a pytest eval harness with a versioned dataset, deterministic checks, a calibrated LLM-as-judge metric and record/replay for CI.
  • Score an agent's tool calls, retrieval and grounded answers separately with DeepEval metrics, and name the stage that failed.
  • Write perturbation and prompt-injection tests, and find flips that a single pass rate hides.
  • Gate a prompt, model or dataset change on regressions in GitHub Actions, and refuse runs that can't be compared.
  • Check dataset splits for leakage and match recorded cases and stages to a local release manifest.

Lessons

  1. Build an eval harness with pytestRead

    Build a versioned pytest eval harness for an LLM triage function with structured-output checks, a calibrated DeepEval G-Eval judge, pass^k and offline replay.

    25 min readLabQuiz

  2. Measure RAG and tool-call qualityRead

    Evaluate a LangChain docs agent stage by stage with DeepEval, score its tool calls, retrieval and answer faithfulness, and report the first stage that fails.

    25 min readLabQuiz

  3. Write robustness tests for an LLM appRead

    Extend a pytest eval suite with meaning-preserving perturbations and prompt-injection cases, run it on Gemini and Gemma, and read per-perturbation pass counts.

    20 min readLabQuiz

  4. Gate releases on eval regressions in CIRead

    Gate prompt, model and dataset changes by comparing a pytest eval run with the main baseline, block new failures and run it in GitHub Actions without a model.

    22 min readLabQuiz

  5. Check evaluation datasets for leakageRead

    Keep related tracker issues in one split, review word overlap and copied prompt text, and detect changes after a held-out dataset is frozen.

    22 min readLabQuiz

  6. Bind an eval run to a release manifestRead

    Hash a tracker docs-assistant bundle, compare its settings with a recorded eval run, and reject changed files or missing evaluated cases.

    20 min readLabQuiz