Skip to main content
Add the following to your .circleci/config.yml file. It runs evalflow in a dedicated job that fails the workflow when quality regresses. 
version: 2.1

jobs:
  evalflow:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run: pip install evalflow
      - run: evalflow doctor --no-provider-check
      - run: evalflow eval

workflows:
  version: 2
  quality-gate:
    jobs:
      - evalflow
The evalflow job runs in CircleCI’s Python convenience image. Adjust the image tag if you need a different Python version.

​
Storing API keys

Never put API keys directly in config.yml. Use CircleCI project environment variables to keep secrets out of your repository.
Store your provider key as a project environment variable:
  1. Go to your project in CircleCI.
  2. Click Project Settings → Environment Variables.
  3. Click Add environment variable.
  4. Set the name (for example OPENAI_API_KEY) and paste your provider key as the value.
CircleCI automatically injects project environment variables into every job. evalflow reads OPENAI_API_KEY (or whichever key your evalflow.yaml is configured to use) from the environment at runtime.

​
How the workflow fails

evalflow exits with a non-zero code when a run does not pass: 
0  — all evals passed
1  — quality regression detected
2  — setup or provider error
CircleCI marks a job as failed when any step returns a non-zero exit code. When evalflow exits with 1 or 2, the evalflow job fails and the quality-gate workflow fails.
Run evalflow doctor locally before pushing to confirm your setup is valid. This keeps CI failures focused on model quality rather than configuration problems.