Testing

Overview

Testing helps you create test data, define how your AI should be evaluated, run model comparisons, and use those results to build intelligent routers. You can use it to see which models best meet your requirements, review past runs, and refine your scoring setup.

Getting Started

1. In the sidebar, click Design My AI.
2. Select Testing.
3. You’ll land on the Model Testing area, where you can create test data, configure evaluation, run tests, and review results.

Actions

Create and define test requirements

1. Go to Model Testing.
2. Click Requirements.
3. You’ll see a page for defining what your AI needs to be good at.
4. Add or review the questions your models will be tested on.
5. Save your changes.
6. If available, click ↓ Export JSON to download your requirements.

Generate new test questions

1. Go to Requirements.
2. Click + New Run or the option to create new requirements.
3. You’ll see suggested or generated questions based on your setup.
4. Review the questions and make any needed adjustments.
5. Click Next → to continue.

Design a test

1. Go to Design a Test.
2. Describe your use case.
3. You’ll see guidance that helps suggest what to test.
4. Review the suggested test setup.
5. Click Next → to proceed.

Configure evaluation

1. Go to Evaluation.
2. Choose an evaluator model.
3. Set your scoring criteria so model responses are scored the way you want.
4. If needed, click + New Evaluator to create one.
5. Open an existing evaluator to edit its instructions.
6. Save your configuration.

Run model tests

1. Go to Measure Models.
2. Select the models you want to compare.
3. Start the test run.
4. You’ll see the run begin and the page will show progress while models, questions, responses, and API calls are processed.
5. Wait for the run to complete, then review the results.

Review run history

1. Go to Run History.
2. You’ll see a list of your past test runs and their results.
3. Open a completed run to review details.
4. If a run needs to be repeated, click ← Try again.

Build an intelligent router

1. Go to Your AI.
2. Click Build Your Intelligent Router.
3. Select one or more completed test runs.
4. Review the scores used to determine how requests should be routed.
5. Save the router.
6. Open an existing router from the list to review its details.

Review judge calibration

1. Go to Judge Calibration Registry.
2. Review the judge quality indicators and calibration results.
3. Use these results to help you choose reliable evaluators.

Requirements

• Plan: All plans
• Role: Admin
• Some pages are restricted to Super admin only, including Template Judge Config.

Troubleshooting

Integrity check failed — results quarantined

1. Review your test requirements and evaluation setup.
2. Try running the test again.
3. If the issue continues, contact support.

Composite scores do not match axis scores × weights

1. Review the affected test run.
2. Wait for re-aggregation if prompted.
3. Do not publish or generate a report until the mismatch is resolved.
4. If the problem persists, contact support.

Access denied. Super admin only.

1. Confirm you have super admin access.
2. If you do not, ask a super admin to open the page or adjust your permissions.

Report generation blocked

1. Review the test run for scoring or integrity issues.
2. Resolve any warnings before trying again.
3. Contact support if reporting remains blocked.

Partial coverage: not all models scored all tasks

1. Check whether every model was tested against the same set of tasks.
2. Rerun the test if needed to improve consistency.
3. Review rankings carefully, since only commonly scored tasks are used for fairness.

Not configured

1. Check whether the evaluator, requirements, or router settings were completed.
2. Finish the missing setup step.
3. Save and try again.

Suggested based on your description — adjust as needed

1. Review the suggested questions or setup.
2. Edit the content so it matches your intended use case.
3. Save the updated configuration.

Important: AI-generated ground truth introduces noise

1. Review the generated ground truth carefully.
2. Adjust any questionable items before running or publishing.
3. Use your own validation to confirm the results.