> ## Documentation Index
> Fetch the complete documentation index at: https://emartai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Configuration

> Reference for evalflow.yaml, the main project configuration file.

`evalflow.yaml` lives at the root of your project and controls which providers to use, where to find your dataset, scoring thresholds, and local storage settings. Run `evalflow init` to generate it automatically, or copy the example below and edit it by hand.

<Warning>
  Store environment variable **names** in `evalflow.yaml`, not the API key values themselves. Never commit real API keys to version control.
</Warning>

## Complete example

```yaml theme={null}
version: "1.0"
project: my-llm-app

providers:
  openai:
    api_key_env: OPENAI_API_KEY
    default_model: gpt-4o-mini
  groq:
    api_key_env: GROQ_API_KEY
    default_model: llama-3.1-8b-instant

eval:
  dataset: evals/dataset.json
  baseline_file: .evalflow/baseline.json
  default_provider: openai
  consistency_runs: 3

thresholds:
  task_success: 0.80
  relevance: 0.75
  hallucination_max: 0.10
  consistency_min: 0.85

judge:
  provider: groq
  model: llama-3.1-8b-instant

prompts:
  directory: prompts/

storage:
  store_raw_outputs: true
  max_output_chars: 2000
```

***

## Top-level fields

<ResponseField name="version" type="string" required>
  Schema version. Use `"1.0"`.
</ResponseField>

<ResponseField name="project" type="string">
  A human-readable name for the project. Used in run history and output headers. Defaults to the directory name when generated by `evalflow init`.
</ResponseField>

***

## `providers`

Configure one or more LLM providers. You can define multiple providers and switch between them using `--provider` on the command line. At least one provider must be configured.

```yaml theme={null}
providers:
  openai:
    api_key_env: OPENAI_API_KEY
    default_model: gpt-4o-mini
```

<ResponseField name="providers.PROVIDER_NAME" type="object">
  Replace `PROVIDER_NAME` with one of: `openai`, `anthropic`, `groq`, `gemini`, `ollama`.

  <Expandable title="properties">
    <ResponseField name="api_key_env" type="string" required>
      The name of the environment variable that holds the API key. evalflow reads this variable at runtime — it does not read the key from the config file itself.
    </ResponseField>

    <ResponseField name="default_model" type="string" required>
      The model to use when `--model` is not passed on the command line. Provider defaults are shown by running `evalflow init --list-providers`.
    </ResponseField>
  </Expandable>
</ResponseField>

<Note>
  For Ollama (local models), `api_key_env` is still required by the schema but the variable does not need to hold a real value. Set it to any non-empty string.
</Note>

***

## `eval`

Controls how `evalflow eval` loads and runs your dataset.

<ResponseField name="eval.dataset" type="string" default="evals/dataset.json">
  Path to the dataset JSON file, relative to the project root.
</ResponseField>

<ResponseField name="eval.baseline_file" type="string" default=".evalflow/baseline.json">
  Path where the baseline run is stored. The baseline is used to detect regressions between runs.
</ResponseField>

<ResponseField name="eval.default_provider" type="string" default="openai">
  The provider to use when `--provider` is not passed on the command line. Must match a key under `providers`.
</ResponseField>

<ResponseField name="eval.consistency_runs" type="number" default="3">
  Number of times to call the model when using the `consistency` eval method. Higher values improve accuracy at the cost of more API calls.
</ResponseField>

***

## `thresholds`

Minimum acceptable scores for each quality dimension. A test case fails when its score falls below the relevant threshold.

<ResponseField name="thresholds.task_success" type="number" default="0.80">
  Minimum score for the overall task to be considered successful. Range: `0.0`–`1.0`.
</ResponseField>

<ResponseField name="thresholds.relevance" type="number" default="0.75">
  Minimum relevance score. Range: `0.0`–`1.0`.
</ResponseField>

<ResponseField name="thresholds.hallucination_max" type="number" default="0.10">
  Maximum allowed hallucination score. A run fails if hallucination exceeds this value. Range: `0.0`–`1.0`.
</ResponseField>

<ResponseField name="thresholds.consistency_min" type="number" default="0.85">
  Minimum consistency score when using the `consistency` eval method. Range: `0.0`–`1.0`.
</ResponseField>

***

## `judge`

Settings for the LLM-as-judge used when a test case sets `eval_config.judge: true`.

<ResponseField name="judge.provider" type="string" default="groq">
  Provider for the judge model. Must be one of the supported providers.
</ResponseField>

<ResponseField name="judge.model" type="string" default="llama-3.1-8b-instant">
  Model to use as judge. A fast, low-cost model is recommended here since the judge runs once per test case.
</ResponseField>

***

## `prompts`

Settings for the prompt registry.

<ResponseField name="prompts.directory" type="string" default="prompts/">
  Directory where prompt YAML files are stored. Created automatically by `evalflow init`.
</ResponseField>

***

## `storage`

Controls what evalflow stores locally after each run.

<ResponseField name="storage.store_raw_outputs" type="boolean" default="true">
  When `true`, the raw model output for each test case is saved to the local database. Set to `false` to reduce disk usage.
</ResponseField>

<ResponseField name="storage.max_output_chars" type="number" default="2000">
  Maximum number of characters to store per model output. Outputs longer than this are truncated in storage (not in scoring).
</ResponseField>

***

## Security

evalflow reads API keys from environment variables at runtime. The config file stores only the variable **name**.

```yaml theme={null}
# Correct — stores the variable name
providers:
  openai:
    api_key_env: OPENAI_API_KEY

# Wrong — never do this
providers:
  openai:
    api_key_env: sk-abc123...
```

Keep `.env` out of version control. `evalflow init` adds the following entries to `.gitignore` automatically:

```text theme={null}
.env
.env.local
.env.*
!.env.example
.evalflow/
*.evalflow.db
```