EvalForge

Portable evaluation evidence and policy gates for RAG applications and AI assistants.

EvalForge turns AI evaluation results into reviewable release evidence. Import promptfoo, Ragas, or supported DeepEval results, enforce a versioned policy, and publish JSON, JUnit, SARIF, and Markdown reports in your existing CI system. The gate runs offline with two direct dependencies and no hosted account.

Status: v0.4.0 alpha. The gate and offline evaluator are usable today. Integrations have explicit, tested format boundaries; the project is still seeking independently verified downstream adoption.

Why EvalForge?

AI projects can calculate useful metrics and still lack a shared answer to a maintainer's release question: what changed, which limits were enforced, and where is the machine-readable evidence? Evaluation frameworks use different result shapes, while CI platforms understand established formats such as JUnit and SARIF.

EvalForge provides that missing, deliberately narrow interoperability layer. It does not try to replace Ragas, DeepEval, promptfoo, or a team's custom evaluator. A flat JSON metric summary can be wrapped as a versioned artifact, compared with a baseline, evaluated by an explicit policy, and translated into reports that existing developer tooling already understands. The built-in RAG evaluator makes the complete workflow reproducible without a model key or hosted service.

The 30-second story

1. Import documents plus golden questions with labeled relevant sources.
2. Run a baseline and candidate across retrieval/model configurations.
3. Compare multi-dimensional deltas on the same dataset fingerprint.
4. Apply explicit quality, security, latency, and cost thresholds.
5. Emit a portable artifact plus JSON, JUnit, and SARIF reports; return a non-zero exit code on regression.

Read the CI cookbook for a reproducible dataset-mismatch failure and a complete GitHub workflow, or the case study for the RAG evaluator's engineering trade-offs.

Quick start

Quality gate (no server or model key)

Install from PyPI:

pip install evalforge-ci

For the committed examples and source development:

git clone https://github.com/jsdhwfmax/EvalForge.git
cd EvalForge
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

evalforge gate examples/candidate_summary.json \
  --policy examples/quality_policy.json \
  --baseline examples/baseline_summary.json \
  --json build/evalforge-report.json \
  --junit build/evalforge-junit.xml \
  --sarif build/evalforge.sarif \
  --markdown build/evalforge-summary.md

The committed example passes required checks and reports one advisory answer-quality regression. Gate exit codes are 0 for pass, 1 for a failed/error check, and 2 for invalid input or policy configuration.

The candidate can also be any flat numeric summary:

{"faithfulness": 0.91, "latency_ms": 420, "cost_usd": 0.003}

See the portable artifact and policy specification.

Import promptfoo evidence

Convert a promptfoo JSON output file into the same small artifact used by the gate, without copying prompts, responses, configuration, or traces:

promptfoo eval --output build/promptfoo-results.json
evalforge import promptfoo build/promptfoo-results.json \
  --output build/evalforge-artifact.json \
  --source-revision "$GITHUB_SHA"

evalforge gate build/evalforge-artifact.json \
  --policy examples/promptfoo_policy.json

The adapter requires promptfoo results schema 3 and versioned producer metadata. It is verified against promptfoo 0.122.2; see the exact metric mapping and privacy boundary in the interoperability specification.

Import Ragas or DeepEval evidence

Export Ragas results with result.to_pandas().to_json(..., orient="records"). Choose score columns explicitly so sample content never becomes a metric:

evalforge import ragas tests/fixtures/ragas/evaluation-result-records.json \
  --producer-version 0.2.12 --metric faithfulness --metric answer_relevancy \
  --dataset-fingerprint synthetic-ragas-fixture-v1 \
  --metric-version ragas-demo-rubric-v1 --output build/ragas.json
evalforge gate build/ragas.json --policy examples/ragas_policy.json

DeepEval imports the JSON serialization of EvaluationResult using a verified producer version. Raw inputs, outputs, reasons, traces, and custom metric names are excluded from the portable artifact:

evalforge import deepeval tests/fixtures/deepeval/evaluation-result-4.2.2.json \
  --producer-version 4.2.2 --output build/deepeval.json
evalforge gate build/deepeval.json --policy examples/deepeval_policy.json

These fixtures contain synthetic scores and demonstrate interoperability, not model quality. Missing or errored selected evidence fails the import. DeepEval metrics whose meaning changed between v3 and v4 receive separate names, so opposite score directions cannot silently share a baseline. See the precise Ragas and DeepEval contracts and their upstream fixture provenance.

Require comparable evidence

A higher score on a different test set does not establish an improvement. Add the optional comparison policy object to require matching dataset fingerprints, producer name/version, or metric-definition versions. Imports accept --dataset-fingerprint and --metric-version for these explicit identities. Missing or mismatched required identity blocks the gate, even for warning checks.

The CI cookbook includes passing and blocked examples, migration guidance, and the digest specification. Existing policies retain their behavior unless these comparison options are enabled.

Docker (recommended)

git clone https://github.com/jsdhwfmax/EvalForge.git
cd EvalForge
docker compose up --build

Open:

• Dashboard: http://localhost:8501
• API documentation: http://localhost:8000/docs
• Health check: http://localhost:8000/health

The API container idempotently loads the demo dataset and two configurations. PostgreSQL data persists in a named volume.

Local Python

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[rag,dashboard,dev]"
cp .env.example .env
evalforge seed
uvicorn evalforge.api:app --reload

In a second terminal:

source .venv/bin/activate
streamlit run dashboard/app.py

Run both demo configurations from the CLI:

evalforge run baseline_top1 --name "Baseline" --output build/baseline.json
evalforge run hybrid_top3 --name "Candidate" --output build/candidate.json

Run the same release gate used by GitHub Actions:

evalforge check hybrid_top3 --name "PR candidate" --report-dir artifacts

The command prints the measured summary, writes a portable evaluation artifact plus JSON, JUnit, and SARIF reports, and exits non-zero if any threshold fails.

GitHub Action

- uses: jsdhwfmax/EvalForge@v0.4.0
  with:
    candidate: build/candidate.json
    baseline: build/baseline.json
    policy: evalforge-policy.json

The Action writes evalforge-report.json, evalforge-junit.xml, evalforge.sarif, and evalforge-summary.md. It appends the summary to the GitHub job page even when the gate blocks a release. Use if: always() when uploading reports; the complete workflow shows how. Pin a full commit SHA where your supply-chain policy requires immutable Actions.

Using EvalForge in another public repository? Please open an issue or pull request to add it to ADOPTERS.md. Projects are listed only with a maintainer's consent and a public, verifiable integration link.

The base evalforge-ci distribution installs only the gate dependencies. The built-in RAG API is available through the rag extra; the UI uses the dashboard extra. This keeps the reusable gate small for downstream CI jobs.

How an experiment works

flowchart LR
    D[Documents] --> I[Hashing embeddings]
    Q[Golden questions] --> R{Retriever}
    I --> R
    C[Configuration] --> R
    R --> M[Model provider]
    C --> M
    M --> E[Metric engine]
    Q --> E
    E --> DB[(PostgreSQL + pgvector)]
    S[Adversarial suite] --> M
    E --> G{Release gate}
    G --> J[JSON + JUnit + SARIF reports]
    J --> CI[GitHub Actions]
    E --> API[FastAPI]
    DB --> API
    API --> UI[Streamlit dashboard]

Every test result stores the answer, citations, retrieved document IDs, quality scores, latency, token counts, cost, and provider metadata. Aggregate results are a cache for comparison; the test-level evidence remains available.

Each experiment summary also stores a stable 16-character dataset fingerprint, a complete configuration snapshot, and a metric-version identifier. EvalForge flags experiments with different dataset fingerprints as non-comparable.

The gate path is independent of the API, dashboard, database, and model provider:

flowchart LR
    A[EvalForge, Ragas, DeepEval, promptfoo, or custom evaluator] --> J[JSON metrics]
    J --> P[Portable artifact v1]
    B[Baseline artifact] --> G{Policy gate}
    P --> G
    G --> O[Exit code]
    G --> R[JSON]
    G --> U[JUnit]
    G --> S[SARIF]

Metrics

These deterministic metrics are deliberately transparent and stable enough for CI regression gates. Production teams should add an LLM-as-judge or human review layer for semantic correctness and entailment. See metric definitions.

Default release thresholds require Recall@K ≥ 0.80, correctness ≥ 0.50, citation support ≥ 0.80, hallucination ≤ 0.10, and a 100% security pass rate. Every value is configurable through the API, Dashboard, or CLI.

Measured demo result

On the committed five-question demo set, a real local run measured 90% Recall@K for BM25 top-1 and 100% for hybrid top-3. Answer token-F1 moved from 75.92% to 54.85% because the larger context made the extractive baseline more verbose—an intentional example of a quality trade-off that aggregate retrieval scores alone would miss. See the full reproducible benchmark and limitations; hosted-model latency and cost are deliberately not invented.

API example

Create a configuration:

curl -X POST http://localhost:8000/api/v1/configs \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Hybrid top-5",
    "retrieval_method": "hybrid",
    "top_k": 5,
    "provider": "local"
  }'

Run one or more configurations against all stored test cases:

curl -X POST http://localhost:8000/api/v1/experiments/run \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "release-2026-08-24",
    "config_ids": ["baseline_top1", "hybrid_top3"],
    "include_security": true
  }'

Import data:

curl -X POST http://localhost:8000/api/v1/datasets/upload \
  -F file=@examples/demo_dataset.json

The full contract is always available at /docs and /openapi.json.

Compare two completed experiments:

curl --get http://localhost:8000/api/v1/experiments/compare \
  --data-urlencode "baseline_id=$BASELINE_ID" \
  --data-urlencode "candidate_id=$CANDIDATE_ID"

Apply a release gate:

curl -X POST "http://localhost:8000/api/v1/experiments/$EXPERIMENT_ID/gate" \
  -H 'Content-Type: application/json' \
  -d '{"retrieval_recall_at_k":0.9,"answer_correctness":0.6,"hallucination_rate":0.1}'

Use a real model

EvalForge works with endpoints that implement the OpenAI Chat Completions shape:

export OPENAI_API_KEY='...'

curl -X POST http://localhost:8000/api/v1/configs \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Production candidate",
    "provider": "openai_compatible",
    "model": "your-model-name",
    "api_base": "https://your-provider.example/v1",
    "api_key_env": "OPENAI_API_KEY",
    "retrieval_method": "hybrid",
    "top_k": 5,
    "input_cost_per_million": 1.0,
    "output_cost_per_million": 4.0
  }'

Keys are read from environment variables at execution time and are never stored in the database.

Dataset format

{
  "documents": [
    {
      "id": "refund_policy",
      "title": "Refund policy",
      "content": "Customers may request a refund within 30 days.",
      "source": "help-center/refunds",
      "metadata": {"version": "2026-08"}
    }
  ],
  "test_cases": [
    {
      "id": "refund_window",
      "question": "How long is the refund window?",
      "expected_answer": "The refund window is 30 days.",
      "relevant_document_ids": ["refund_policy"],
      "tags": ["support"]
    }
  ]
}

Document and test IDs should be stable across dataset versions so experiments remain comparable.

Development

make install
make lint
make test

The test suite covers portable artifacts, strict comparison identities, digest stability, four CI report formats, CLI exit codes, three evaluator adapters, retrieval, metrics, security grading, persistence, and complete multi-configuration experiments. CI enforces at least 85% branch-aware coverage; see the linked CI run for the current measured result.

CI runs on Python 3.9 and 3.12, builds the package and Docker image, and executes both portable-policy and seeded RAG release gates.

Maintainers should follow the release checklist so tags, package metadata, distributions, checksums, and public claims remain consistent.

Deployment

• docker-compose.yml provides API + Dashboard + PostgreSQL/pgvector.
• render.yaml is a starting blueprint for two web services and managed Postgres.
• Secrets belong in the deployment provider's environment settings; never commit .env.
• Protect the API with an identity-aware proxy or API gateway before exposing private datasets.

See architecture and production notes.

Ecosystem role

EvalForge's ecosystem value is interoperability and auditability, not a claim that one deterministic score can certify an AI system. Its artifact keeps metric values, units, direction, producer, and source revision portable; its policy makes release decisions reviewable in Git; its reporters reuse CI standards instead of creating another proprietary dashboard requirement.

The project records adoption evidence conservatively—stars, downstream integrations, released versions, issues, and external contributors are never invented. Read the ecosystem rationale and success measures, governance, and maintainer responsibilities.

Roadmap

• Pluggable LLM-as-judge rubrics and calibrated human review
• Chunking and embedding provider experiments
• Dataset/version lineage and statistical significance
• Durable workers, progress streaming, and experiment cancellation
• Adapters and fixtures for widely used evaluator result formats
• Signed evaluation provenance and SLSA-compatible attestations
• GitLab and Jenkins integration examples
• Multi-turn agent/tool-call evaluation
• Role-based access control and tenant isolation
• React UI once workflows stabilize

Contributions are welcome. Read CONTRIBUTING.md before opening a pull request.

License

MIT © EvalForge Contributors