> ## Documentation Index
> Fetch the complete documentation index at: https://emartai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CLI

> Complete reference for every evalflow command and flag.

## `evalflow init`

Set up evalflow in your project. By default, runs an interactive wizard that prompts for a provider, model, and API key environment variable name. Use `--non-interactive` for CI, Docker, or scripted environments.

**Syntax**

```bash theme={null}
evalflow init [--provider PROVIDER] [--model MODEL] [--non-interactive] [--force] [--list-providers]
```

**Flags**

<ParamField path="--provider" type="string">
  LLM provider to configure. Accepted values: `openai`, `anthropic`, `groq`, `gemini`, `ollama`. Defaults to `openai` in non-interactive mode.
</ParamField>

<ParamField path="--model" type="string">
  Model name to use. Defaults to the provider's recommended model when omitted.
</ParamField>

<ParamField path="--non-interactive" type="boolean">
  Skip all prompts. Also accepted as `--yes` or `-y`. Required when stdin is not a TTY (CI, Docker).
</ParamField>

<ParamField path="--force" type="boolean">
  Overwrite an existing `evalflow.yaml` without asking.
</ParamField>

<ParamField path="--list-providers" type="boolean">
  Print supported providers and their default models, then exit. Does not write any files.
</ParamField>

**Files created**

`evalflow init` writes the following files on first run:

* `evalflow.yaml` — project configuration
* `evals/dataset.json` — starter dataset with one example test case
* `prompts/` — empty directory for prompt YAML files
* `.env.example` — template for required environment variables
* `.evalflow/` — local storage directory (runs database, cache)

**Examples**

Interactive setup:

```bash theme={null}
evalflow init
```

```text theme={null}
Choose provider [openai, anthropic, groq, gemini, ollama]: openai
Choose model [gpt-4o-mini]:
API key env var name [OPENAI_API_KEY]:
  evalflow initialized

  Next steps:
  1. Add your API key to your environment:
     export OPENAI_API_KEY="your-key-here"

  2. Run your first eval:
     evalflow eval
```

Non-interactive setup for CI:

```bash theme={null}
evalflow init --provider groq --model llama-3.1-8b-instant --non-interactive
```

List supported providers:

```bash theme={null}
evalflow init --list-providers
```

```text theme={null}
Provider    Default model
openai      gpt-4o-mini
anthropic   claude-3-5-haiku-20241022
groq        llama-3.1-8b-instant
gemini      gemini-1.5-flash
ollama      llama3.2
```

***

## `evalflow eval`

Run the LLM quality gate against your dataset. Each test case is scored and compared to the saved baseline. The command exits with code `0` on pass, `1` on quality failure (which blocks CI), and `2` on an unexpected error.

**Syntax**

```bash theme={null}
evalflow eval [--provider PROVIDER] [--model MODEL] [--dataset PATH] [--tag TAG]
              [--offline] [--debug] [--save-baseline] [--concurrency N]
```

**Flags**

<ParamField path="--provider, -p" type="string">
  Override the provider set in `evalflow.yaml`. Accepted values: `openai`, `anthropic`, `groq`, `gemini`, `ollama`.
</ParamField>

<ParamField path="--model, -m" type="string">
  Override the model set in `evalflow.yaml` for this run.
</ParamField>

<ParamField path="--dataset, -d" type="string">
  Path to a dataset JSON file. Defaults to `evals/dataset.json` (or the value in `evalflow.yaml`).
</ParamField>

<ParamField path="--tag, -t" type="string">
  Run only test cases whose `tags` array contains this value. Useful for running a `smoke` or `critical` subset.
</ParamField>

<ParamField path="--offline" type="boolean">
  Use cached LLM responses instead of making live API calls. Useful for re-scoring without consuming API quota.
</ParamField>

<ParamField path="--debug" type="boolean">
  Print full exception tracebacks on unexpected errors. Not recommended for production use.
</ParamField>

<ParamField path="--save-baseline" type="boolean">
  Save this run as the new baseline. Future runs are compared against it. The first run always saves a baseline automatically.
</ParamField>

<ParamField path="--concurrency" type="number" default="5">
  Maximum number of test cases to run in parallel. Minimum value is `1`.
</ParamField>

**Example**

```bash theme={null}
evalflow eval --provider openai --model gpt-4o-mini --dataset evals/dataset.json --tag critical --save-baseline
```

```text theme={null}
Running 5 test cases against gpt-4o-mini...

✓ summarize-short-article      0.91
✓ classify-support-priority    1.00
✗ answer-with-context          0.61
✓ extract-key-dates            0.87
✓ rewrite-formal-tone          0.83

Quality Gate: FAIL
Baseline: saved
Failures: 1
Duration: 4.2s
Run ID: 20260325-a3f9c2d81b4e
```

<Note>
  `evalflow eval` exits with code `1` when the quality gate fails. CI systems interpret a non-zero exit code as a pipeline failure, blocking the PR automatically.
</Note>

***

## `evalflow doctor`

Check your local evalflow setup. Verifies the installation, configuration file, dataset, environment variables, storage, and optional provider health. Run this first when something isn't working.

**Syntax**

```bash theme={null}
evalflow doctor [--fix] [--check-providers | --no-provider-check] [--validate-config]
```

**Flags**

<ParamField path="--fix" type="boolean">
  Automatically apply supported fixes, such as adding required entries to `.gitignore`.
</ParamField>

<ParamField path="--check-providers / --no-provider-check" type="boolean">
  Run a live health check against each configured provider. Off by default to avoid unnecessary API calls.
</ParamField>

<ParamField path="--validate-config" type="boolean">
  Validate `evalflow.yaml` syntax, field values, and configured API key environment variables, then exit.
</ParamField>

**Example**

```bash theme={null}
evalflow doctor --check-providers
```

```text theme={null}
✓ evalflow 0.1.2 installed
✓ Python 3.11.9
✓ evalflow.yaml found
✓ evalflow.yaml valid
✓ dataset.json found (3 test cases)
✓ dataset.json valid
✓ .evalflow directory exists
✓ SQLite database accessible
✓ Response cache: 0 entries
✓ Git repository detected
✓ OPENAI_API_KEY set
✓ openai health check
! sentence-transformers not installed (optional - needed for embedding_similarity)
✗ .gitignore has .env entry

1 issue found. Run evalflow doctor --fix to resolve.
```

***

## `evalflow runs`

List recent eval runs stored in the local SQLite history. Each row shows the run ID, date, provider, model, overall score, and pass/fail status.

**Syntax**

```bash theme={null}
evalflow runs [--limit N] [--since WINDOW] [--failed-only]
```

**Flags**

<ParamField path="--limit, -n" type="number" default="20">
  Maximum number of runs to display.
</ParamField>

<ParamField path="--since" type="string">
  Show only runs newer than a time window. Accepted formats: `7d` (days) or `24h` (hours).
</ParamField>

<ParamField path="--failed-only" type="boolean">
  Show only runs that failed the quality gate.
</ParamField>

**Example**

```bash theme={null}
evalflow runs --limit 20 --since 7d --failed-only
```

```text theme={null}
Run ID                  Date         Provider  Model         Score  Status
20260325-a3f9c2d81b4e   2026-03-25   openai    gpt-4o-mini   0.89   pass
20260324-b4c2e1f39d7a   2026-03-24   openai    gpt-4o-mini   0.71   fail
```

***

## `evalflow compare RUN_A RUN_B`

Compare two runs side by side. Shows the overall score change and highlights which test cases improved or regressed. Partial run IDs are accepted as long as they resolve to a unique run.

**Syntax**

```bash theme={null}
evalflow compare RUN_A RUN_B
```

**Arguments**

<ParamField path="RUN_A" type="string" required>
  The first run ID (or a unique prefix of at least 8 characters).
</ParamField>

<ParamField path="RUN_B" type="string" required>
  The second run ID (or a unique prefix of at least 8 characters).
</ParamField>

**Example**

```bash theme={null}
evalflow compare 20260325-a3f9 20260324-b4c2
```

```text theme={null}
Overall score: +0.04

summarize-short-article     improved    0.87 → 0.91
answer-with-context         regressed   0.88 → 0.61
classify-support-priority   unchanged   1.00 → 1.00
```

<Tip>
  Use `evalflow runs` first to find the run IDs you want to compare.
</Tip>

***

## `evalflow prompt create NAME`

Create a new prompt YAML file in the `prompts/` directory. The name must be in lowercase kebab-case.

**Syntax**

```bash theme={null}
evalflow prompt create NAME
```

**Example**

```bash theme={null}
evalflow prompt create summarization
```

```text theme={null}
✓ Created prompts/summarization.yaml
```

***

## `evalflow prompt list`

List all prompts in the registry with their current version, status, author, and creation date.

**Syntax**

```bash theme={null}
evalflow prompt list
```

**Example**

```bash theme={null}
evalflow prompt list
```

```text theme={null}
ID              Version  Status      Author   Created
summarization   1        draft       unknown  2026-03-25
classification  2        staging     unknown  2026-03-20
```

***

## `evalflow prompt diff NAME V1 V2`

Show a diff between two stored versions of a prompt. Lines removed in the newer version are prefixed with `-`, and lines added are prefixed with `+`.

**Syntax**

```bash theme={null}
evalflow prompt diff NAME V1 V2
```

**Arguments**

<ParamField path="NAME" type="string" required>
  The prompt name, matching the file in `prompts/`.
</ParamField>

<ParamField path="V1" type="number" required>
  The older version number.
</ParamField>

<ParamField path="V2" type="number" required>
  The newer version number.
</ParamField>

**Example**

```bash theme={null}
evalflow prompt diff summarization 1 2
```

```text theme={null}
- You are a helpful assistant.
+ You are a concise summarization assistant. Always respond in one sentence.
```

***

## `evalflow prompt promote NAME --to STATUS`

Promote a prompt version to `staging` or `production`. Run `evalflow eval` before promoting to `production` to confirm quality has not regressed.

**Syntax**

```bash theme={null}
evalflow prompt promote NAME --to STATUS
```

**Flags**

<ParamField path="--to" type="string" required>
  Target status. Accepted values: `staging`, `production`.
</ParamField>

**Example**

```bash theme={null}
evalflow prompt promote summarization --to production
```

```text theme={null}
Consider running evalflow eval before promoting to production.
summarization promoted to production
```

<Warning>
  Promoting to `production` does not automatically run an eval. Always verify quality with `evalflow eval` first.
</Warning>

***

## `evalflow dataset lint`

Validate your dataset file without running a full eval. Checks structure, field types, and per-test-case quality rules.

**Syntax**

```bash theme={null}
evalflow dataset lint [PATH]
```

**Arguments**

<ParamField path="PATH" type="string" default="evals/dataset.json">
  Path to the dataset JSON file to validate. Defaults to `evals/dataset.json`.
</ParamField>

**Example**

```bash theme={null}
evalflow dataset lint
```

```text theme={null}
✓ dataset.json valid (3 test cases)
✓ summarize-release-notes: id kebab-case
✓ summarize-release-notes: input non-empty
✓ summarize-release-notes: expected_output reasonable length
...

Dataset lint passed.
```

If the dataset contains issues, `evalflow dataset lint` exits with code `2` and prints the failing checks.

<Tip>
  Run `evalflow dataset lint` after editing your dataset to catch schema errors before they surface during a full `evalflow eval` run.
</Tip>