> ## Documentation Index
> Fetch the complete documentation index at: https://emartai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# GitHub Actions

> Block pull requests automatically when prompt quality regresses.

<Steps>
  <Step title="Install evalflow in your repo">
    Run the following commands at the root of your repository to install evalflow and create the initial config and dataset files.

    ```bash theme={null}
    pip install evalflow
    evalflow init
    ```

    `evalflow init` creates two files:

    ```text theme={null}
    evalflow.yaml created
    evals/dataset.json created
    ```

    Run `evalflow doctor` locally to confirm your setup is valid before pushing.
  </Step>

  <Step title="Create the workflow file">
    Create `.github/workflows/evalflow.yml` in your repository with the following content:

    ```yaml theme={null}
    # .github/workflows/evalflow.yml
    name: LLM Quality Gate

    on:
      pull_request:
        paths:
          - "prompts/**"
          - "evals/**"
          - "**.py"
          - "evalflow.yaml"

    jobs:
      eval:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - uses: actions/setup-python@v5
            with:
              python-version: "3.11"
          - run: pip install evalflow
          - run: evalflow doctor --no-provider-check
          - run: evalflow eval
            env:
              OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ```

    The `paths` filter limits the workflow to pull requests that touch prompts, evals, or your evalflow config. Remove or adjust those filters to run on every pull request.

    <Tip>
      Run `evalflow doctor --no-provider-check` before `evalflow eval` so setup errors surface as a distinct step in the GitHub Actions log.
    </Tip>
  </Step>

  <Step title="Add repository secrets">
    Store your provider API key as a GitHub Actions secret so it is never visible in logs or workflow files.

    1. Go to your repository on GitHub.
    2. Click **Settings** → **Secrets and variables** → **Actions**.
    3. Click **New repository secret**.
    4. Set the name and value:

    ```text theme={null}
    Name:  OPENAI_API_KEY
    Value: your real provider key
    ```

    If you use a different provider, set the variable name that matches your `evalflow.yaml` configuration — for example `GROQ_API_KEY` or `ANTHROPIC_API_KEY` — and reference it the same way in your workflow file.
  </Step>
</Steps>

## How blocking merges works

evalflow uses exit codes to communicate the result of a run:

```text theme={null}
0  — all evals passed
1  — quality regression detected
2  — setup or provider error
```

GitHub Actions treats any non-zero exit code as a workflow failure. When the `evalflow eval` step exits with `1` or `2`, the workflow fails and GitHub blocks the pull request from merging — no additional configuration required.

To enforce this as a required status check, go to **Settings** → **Branches** → **Branch protection rules** and add the `eval` job as a required check for your default branch.

## Security

<Warning>
  Never hardcode API keys in your workflow YAML. Anyone with read access to the repository can see workflow files.
</Warning>

* Always store provider keys in GitHub Secrets or organization secrets.
* Keep `.env` out of version control by adding it to `.gitignore`.
* Use `evalflow doctor` locally before pushing so CI failures reflect model quality, not missing setup.