> ## Documentation Index
> Fetch the complete documentation index at: https://emartai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Quality gate

> How evalflow decides whether a run passes, fails, or errors — and what that means for your CI pipeline.

A quality gate is the rule that decides whether an evalflow run should block a pull request. When you run `evalflow eval`, the process exits with a numeric code your CI system can act on.

## Run outcomes

Every run ends in one of three states:

| Outcome | Meaning                                         |
| ------- | ----------------------------------------------- |
| `PASS`  | Quality is at or above the configured threshold |
| `FAIL`  | Quality dropped below the threshold             |
| `ERROR` | Setup, config, or provider execution failed     |

```text theme={null}
Quality Gate: PASS
Baseline: saved
Failures: 0
Duration: 2.8s
```

## Exit codes

evalflow maps each outcome to a standard exit code:

| Exit code | Outcome | Cause                                           |
| --------- | ------- | ----------------------------------------------- |
| `0`       | PASS    | All test cases met the threshold                |
| `1`       | FAIL    | One or more test cases fell below the threshold |
| `2`       | ERROR   | Config, provider, or execution error            |

<Note>
  CI systems (GitHub Actions, GitLab CI, CircleCI) treat any non-zero exit code as a failure. Exit code `1` blocks the PR because quality regressed. Exit code `2` blocks it because something is broken. Both deserve attention, but for different reasons.
</Note>

Separating quality failures (`1`) from setup errors (`2`) lets you write different triage steps in your pipeline — for example, alerting on errors while treating quality failures as expected review signals.

## Baselines and regression detection

A baseline is the saved result of the last accepted run for the same dataset. evalflow compares every new run against the baseline so it catches regressions rather than just low absolute scores.

```text theme={null}
Baseline: saved
Δ overall: +0.03
```

When you run `evalflow eval`, the result is compared per test case against the stored baseline. If any case drops below its previous score by more than the configured threshold, the run fails.

<Tip>
  Save a new baseline explicitly with `evalflow eval --save-baseline` after you accept a result as the new standard. The baseline advances only when you tell it to.
</Tip>

The baseline is stored locally in `.evalflow/`. Commit that directory to version control so every developer and your CI runner share the same reference point.

## What a prompt regression looks like

A prompt regression is when a change improves one case but silently harms another. This is the failure mode evalflow is designed to catch before a change ships.

```text theme={null}
✓ summarize-short-article    0.91   (was 0.88)
✗ answer-with-context        0.61   (was 1.00)

Quality Gate: FAIL
Failures: 1
```

In this example, the summarization test improved, but the answer quality test dropped from a perfect score to 0.61. Without a quality gate that tracks both, the regression in `answer-with-context` would have merged undetected.

<Warning>
  Improving one prompt can degrade another, especially when prompts share a system instruction or when a model update affects different task types differently. Always run the full dataset, not just the cases you changed.
</Warning>