GitLab CI

Run the evalflow quality gate in GitLab CI and fail the pipeline on regressions.

Add the following job to your .gitlab-ci.yml file. It installs evalflow, validates your setup, and runs the quality gate on every pipeline.

stages:
  - eval

evalflow:
  stage: eval
  image: python:3.11
  script:
    - pip install evalflow
    - evalflow doctor --no-provider-check
    - evalflow eval
  variables:
    OPENAI_API_KEY: $OPENAI_API_KEY

The variables block maps a GitLab CI/CD variable to the environment variable evalflow reads at runtime. If you use a different provider, replace OPENAI_API_KEY with the variable name that matches your evalflow.yaml configuration — for example GROQ_API_KEY or ANTHROPIC_API_KEY.

Storing API keys

Never commit API keys to .gitlab-ci.yml. GitLab CI/CD variables keep secrets out of your repository.

Store your provider key as a CI/CD variable:

1. Go to your project in GitLab.
2. Click Settings → CI/CD → Variables.
3. Click Add variable.
4. Set the key (for example OPENAI_API_KEY) and paste your provider key as the value.
5. Check Mask variable to prevent the value from appearing in job logs.

How the pipeline fails

evalflow exits with a non-zero code when a run does not pass:

0  — all evals passed
1  — quality regression detected
2  — setup or provider error

GitLab CI marks a job as failed when its script exits with a non-zero code. When evalflow exits with 1 or 2, the evalflow job fails and the pipeline fails — blocking the merge request from merging if you have pipeline success required for merging.

Run evalflow doctor locally before pushing to confirm your setup is valid. This keeps CI failures focused on model quality rather than configuration problems.