Gemini

Configure evalflow to run evals against Google Gemini models.

Use Gemini when you want Google’s latest models with a free tier and generous rate limits during development. It is a natural fit for teams already on Google Cloud, and the free tier is sufficient for routine CI eval runs.

Configure evalflow.yaml

Add a gemini block under providers and set default_provider to gemini:

providers:
  gemini:
    api_key_env: "GEMINI_API_KEY"
    default_model: "gemini-1.5-flash"

eval:
  default_provider: "gemini"

api_key_env is the name of the environment variable that holds your key — evalflow reads the variable at runtime and never stores the key itself.

Set your API key

Get a free key at aistudio.google.com, then export it:

export GEMINI_API_KEY="your-key-here"

Add this line to your shell profile (~/.bashrc, ~/.zshrc, etc.) or a .env file so you do not have to re-export it each session. Never commit your .env file to version control.

Verify the connection

Run evalflow doctor to confirm evalflow can see the key before running any evals:

evalflow doctor

✓ GEMINI_API_KEY set

Run evals

evalflow eval --provider gemini

Running test cases against gemini-1.5-flash...
Quality Gate: PASS

If eval.default_provider is already set to gemini in your evalflow.yaml, you can omit the --provider flag:

evalflow eval

Available models

Set default_model in your evalflow.yaml to any of these model names:

gemini-1.5-flash   — fast, low cost, good for CI
gemini-1.5-pro     — higher quality, slower
gemini-2.0-flash   — latest generation

Provider notes

• Default model: gemini-1.5-flash. This model is optimized for speed and is well-suited for high-volume CI runs.
• Free tier: Google AI Studio provides a free tier with rate limits that cover most routine eval pipelines.
• Judge model: By default, evalflow uses Groq as the LLM judge. If you want Gemini to serve as both the model under test and the judge, update the judge block in evalflow.yaml:

judge:
  provider: "gemini"
  model: "gemini-1.5-flash"

Best for

• Teams already using Google Cloud infrastructure
• High-volume eval runs on the free tier
• Experimenting with Google’s latest model generations