> ## Documentation Index
> Fetch the complete documentation index at: https://emartai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataset schema

> Reference for evals/dataset.json, the file that defines your test cases.

Your dataset lives at `evals/dataset.json` by default (configurable in `evalflow.yaml`). It is a JSON object that contains a version string and an array of test cases. evalflow validates the file on every run and reports clear errors when something is wrong.

## Complete example

The following dataset uses three different eval methods across three test cases:

```json theme={null}
{
  "version": "1.0",
  "test_cases": [
    {
      "id": "summarize-release-notes",
      "description": "Summarize short product notes in one sentence",
      "task_type": "summarization",
      "input": "Summarize in one sentence: Evalflow now stores baselines locally and prints run IDs in the summary.",
      "expected_output": "Evalflow stores baselines locally and shows run IDs in summaries.",
      "context": "",
      "tags": ["smoke", "summary"],
      "eval_config": {
        "methods": ["embedding_similarity"],
        "judge": false,
        "weight": 1.0
      }
    },
    {
      "id": "classify-support-priority",
      "description": "Classify a support issue as urgent",
      "task_type": "classification",
      "input": "Classify as urgent or normal: Customer cannot access production after migration.",
      "expected_output": "urgent",
      "context": "",
      "tags": ["smoke", "classification"],
      "eval_config": {
        "methods": ["exact_match"],
        "judge": false,
        "weight": 1.0
      }
    },
    {
      "id": "answer-billing-question",
      "description": "Answer a billing question using provided context",
      "task_type": "qa",
      "input": "What plan includes audit logs?",
      "expected_output": "The business plan includes audit logs.",
      "context": "Pricing: Starter has email support. Business includes audit logs and SSO.",
      "tags": ["smoke", "qa"],
      "eval_config": {
        "methods": ["embedding_similarity"],
        "judge": true,
        "weight": 1.5
      }
    }
  ]
}
```

***

## Top-level fields

<ResponseField name="version" type="string" required>
  Dataset schema version. Use `"1.0"`.
</ResponseField>

<ResponseField name="test_cases" type="object[]" required>
  Array of test case objects. Must contain at least one entry. All `id` values must be unique.
</ResponseField>

***

## Test case fields

<ResponseField name="id" type="string" required>
  Unique identifier for the test case. Must be lowercase kebab-case (e.g. `summarize-release-notes`). Used in run output and comparisons.
</ResponseField>

<ResponseField name="description" type="string" required>
  Short human-readable description of what this test case checks. Displayed in run history and diffs.
</ResponseField>

<ResponseField name="task_type" type="string" required>
  The type of task the model is performing. Accepted values:

  | Value            | When to use                                              |
  | ---------------- | -------------------------------------------------------- |
  | `summarization`  | The model condenses a longer input                       |
  | `classification` | The model assigns a category label                       |
  | `extraction`     | The model pulls specific information from a text         |
  | `qa`             | The model answers a question, optionally using `context` |
  | `generation`     | The model produces free-form content                     |
  | `rewrite`        | The model rephrases or transforms an existing text       |
</ResponseField>

<ResponseField name="input" type="string" required>
  The prompt or instruction sent to the model. Write this exactly as you would pass it to the LLM.
</ResponseField>

<ResponseField name="expected_output" type="string" required>
  The ideal model response. Scoring methods compare the actual model output against this value.
</ResponseField>

<ResponseField name="context" type="string">
  Optional additional context passed alongside `input`. Use this for retrieval-augmented generation (RAG) test cases where the model must answer from a specific document or data source. Set to `""` when not needed.
</ResponseField>

<ResponseField name="tags" type="string[]">
  Optional list of labels. Use tags to group related test cases and run a subset with `evalflow eval --tag TAG`. Common patterns: `smoke`, `critical`, `regression`, `slow`.
</ResponseField>

<ResponseField name="eval_config" type="object">
  Per-test evaluation settings. If omitted, evalflow uses default scoring behaviour.

  <Expandable title="eval_config properties">
    <ResponseField name="methods" type="string[]">
      List of scoring methods to apply. You can combine multiple methods for a single test case. Accepted values:

      | Method                 | Description                                                                                                                                                             |
      | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
      | `exact_match`          | Passes when the model output matches `expected_output` after normalization (lowercased, whitespace stripped). Best for classification labels and structured extraction. |
      | `embedding_similarity` | Semantic similarity score using sentence embeddings. Requires `sentence-transformers` to be installed.                                                                  |
      | `consistency`          | Runs the model multiple times (controlled by `eval.consistency_runs` in `evalflow.yaml`) and scores how consistent the outputs are.                                     |
      | `llm_judge`            | Uses a second LLM to judge quality. Configured via the `judge` section of `evalflow.yaml`.                                                                              |
    </ResponseField>

    <ResponseField name="judge" type="boolean" default="false">
      When `true`, the LLM-as-judge (configured in `evalflow.yaml`) scores this test case in addition to any `methods`. Use for test cases where semantic correctness is hard to capture with automated metrics alone.
    </ResponseField>

    <ResponseField name="weight" type="number" default="1.0">
      Relative importance of this test case in the overall run score. Must be greater than `0.0`. A test case with `weight: 2.0` counts twice as much as one with `weight: 1.0`. Use higher weights for critical paths.
    </ResponseField>
  </Expandable>
</ResponseField>

***

## Validation rules

evalflow validates your dataset before running and exits with a clear error if any rule is violated:

* `version` must be present.
* `test_cases` must contain at least one entry.
* Every `id` must be unique and in lowercase kebab-case.
* Every test case must have `input` and `expected_output`.
* `task_type` must be one of the accepted values.
* `eval_config.methods` entries must be unique within a test case.
* `eval_config.weight` must be greater than `0.0`.

<Tip>
  Run `evalflow doctor` to validate your dataset without running a full eval.
</Tip>