CircleCI

Run the evalflow quality gate in CircleCI and block merges when quality regresses.

Add the following to your .circleci/config.yml file. It runs evalflow in a dedicated job that fails the workflow when quality regresses.

version: 2.1

jobs:
  evalflow:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run: pip install evalflow
      - run: evalflow doctor --no-provider-check
      - run: evalflow eval

workflows:
  version: 2
  quality-gate:
    jobs:
      - evalflow

The evalflow job runs in CircleCI’s Python convenience image. Adjust the image tag if you need a different Python version.

Storing API keys

Never put API keys directly in config.yml. Use CircleCI project environment variables to keep secrets out of your repository.

Store your provider key as a project environment variable:

1. Go to your project in CircleCI.
2. Click Project Settings → Environment Variables.
3. Click Add environment variable.
4. Set the name (for example OPENAI_API_KEY) and paste your provider key as the value.

CircleCI automatically injects project environment variables into every job. evalflow reads OPENAI_API_KEY (or whichever key your evalflow.yaml is configured to use) from the environment at runtime.

How the workflow fails

evalflow exits with a non-zero code when a run does not pass:

0  — all evals passed
1  — quality regression detected
2  — setup or provider error

CircleCI marks a job as failed when any step returns a non-zero exit code. When evalflow exits with 1 or 2, the evalflow job fails and the quality-gate workflow fails.

Run evalflow doctor locally before pushing to confirm your setup is valid. This keeps CI failures focused on model quality rather than configuration problems.