1. Define the job before comparing AI models
Start with the decision the comparison must support. A customer-support team may need accurate policy answers, safe escalation and a complete ticket update. A coding team may care about repository-level changes that pass tests. A research team may need claims that trace to authoritative sources. Those are different jobs, so they should not inherit one universal winner. Write the workflow, the users, the allowed tools, the consequences of a mistake and the minimum acceptable result before naming candidate models. Sources: NIST; OpenAI.
Turn the workflow into tasks with observable end states. Instead of asking whether a model writes a good refund reply, require it to identify the policy, calculate the eligible amount, update the correct test record and produce a customer message that stays within the rule. For a research task, require a complete answer, primary-source citations and a list of unresolved uncertainty. The test should measure the system the user experiences, including retrieval, tools and prompts, not an isolated chat response when the production system does more. Sources: Anthropic; OpenAI.
- Name one workflow and one user group.
- Write the successful final state before running a candidate.
- List errors that are unacceptable even when the answer sounds good.
- Include the same tools, context limits and approval rules used in production.
2. Freeze a representative test set and the exact model versions
Build a small but deliberate set from real work. Include common cases, costly edge cases, ambiguous requests, long inputs, missing information and at least one case that should be refused or escalated. Remove private data or create faithful synthetic equivalents. Keep the set unchanged while candidates are compared. If one model sees easier examples, extra context or a different tool, the scores describe different tests and cannot support a side-by-side decision. Sources: Anthropic; Google Cloud.
Record the exact model identifier, date, API or product surface, system instructions, temperature or reasoning setting, tool definitions, retrieval configuration and retry policy. Model names alone are not enough because providers update aliases and expose different capabilities through different surfaces. Google's model endpoint, for example, exposes version and token-limit metadata. Save the actual identifier returned by the platform so the comparison can be reproduced after an alias changes. Sources: Google AI for Developers.
Run more than one trial when outputs vary. The right number depends on the cost of the workflow and the gap between candidates, so avoid a fake universal sample size. Ten to twenty representative tasks with repeated trials can expose obvious failures, but a close or high-stakes decision needs more coverage and human review. Keep the raw failures. A single average hides whether a model fails randomly or fails the same important case every time. Sources: Anthropic.
3. Score quality, evidence and risk as separate dimensions
Choose a few dimensions that map directly to the workflow. Useful candidates include final-state completion, factual accuracy, source quality, instruction following, completeness, tone, safety, privacy handling and recovery from tool errors. Define what earns each score with examples. A rubric such as correct, partially correct and failed is more useful when each label names observable conditions. Reviewers should not need to guess what good means after seeing the outputs. Sources: OpenAI; Stanford CRFM.
Keep dimensions separate until the decision. Stanford HELM was created around broad, transparent evaluation across scenarios and metrics because one number cannot explain all capability and risk tradeoffs. NIST similarly frames AI risk work around governing, mapping, measuring and managing in context. A candidate with slightly better prose but worse source support may be wrong for research. A slower candidate with better escalation may be right for a regulated queue. A weighted total is useful only after the weights have been tied to the business consequence. Sources: Stanford CRFM; NIST; NIST.
| Dimension | Evidence to keep | Failure signal |
|---|---|---|
| Task completion | Final record, file or navigation result | Claims success without the required state |
| Factual support | Source URL beside each checkable claim | Cites a page that does not support the sentence |
| Safety and scope | Refusal, escalation and permission decisions | Acts outside the stated boundary |
| Quality | Rubric score plus reviewer note | Smooth prose hides an omitted requirement |
| Operations | Latency, token use, cost and error receipt | Averages hide a long tail or repeated provider failure |
4. Verify end-to-end outcomes, not the model's own claim
Grade the thing the workflow was meant to change. If an agent books a test appointment, confirm that the calendar contains the right event. If it edits a document, open the rendered document. If it researches a claim, open the cited primary source and check that the passage supports the claim. A transcript saying completed is not proof that the external state changed. This distinction becomes more important when agents take several tool actions and an early error can propagate through later steps. Sources: Anthropic.
Use code-based checks where the answer is objective, human review where judgment matters, and calibrated model graders only where their own error is understood. Google's evaluation guidance recommends comparing judge-model scores with human ratings when a model is used as a grader. OpenAI's GDPval uses detailed expert-written rubrics and expert comparisons for real-world deliverables, and notes that its automated grader is experimental rather than a replacement for experts. The practical lesson is to pair cheap automated checks with sampled human review instead of treating either as complete. Sources: Google Cloud; OpenAI.
- Read back every external state change.
- Open the final artifact in the same modality the user receives.
- Inspect failure examples, not only the aggregate score.
- Calibrate model graders against human judgments on your own task set.
5. Compare AI model cost, latency and operational fit
Measure cost per successful task, not price per token in isolation. Include input, output, reasoning or cache charges, retries, tool calls and the human time required to repair failures. A cheaper request can cost more when it needs repeated attempts or frequent review. Record both the median and slow tail for latency. A model that usually responds in two seconds but regularly takes forty may be unsuitable for an interactive workflow even when its average appears acceptable. Sources: MLCommons.
Check operational constraints before choosing a winner: rate limits, regional availability, data retention terms, supported input types, context limits, structured output reliability and whether the necessary model version can be pinned. Run the real payload sizes. A context window advertised in documentation does not prove that your largest documents produce accurate answers. Likewise, an inference benchmark measures the hardware and rules it names, not the total latency of your retrieval, network and tool chain. Sources: Google AI for Developers; MLCommons.
6. Source-check every AI model comparison claim
For every row in a comparison, link the source that proves it and record the access date. Prefer provider documentation for context limits, supported inputs, availability and pricing. Prefer the benchmark's own methodology and result files for scores. Prefer system cards or technical reports for safety claims. A search snippet, reseller table or undated chart is a discovery lead, not evidence. If the source covers a preview model but the table names a production version, the row fails the check. Sources: Google AI for Developers; OpenAI.
Read the benchmark conditions before copying a rank. Check the task set, model version, prompt or scaffold, number of trials, judge, date, excluded failures and whether results were submitted by vendors or reproduced independently. MLCommons publishes rules alongside its inference results, and HELM exposes scenarios and metrics for transparency. These details explain why two credible leaderboards can disagree without either being fraudulent. They also show whether a result transfers to your environment. Sources: MLCommons; Stanford CRFM.
- Does the source name the exact model version in the comparison?
- Does the benchmark test the same task type and modality?
- Are prompts, tools, scoring rules and dates visible?
- Can a reader distinguish measured results from the author's inference?
- Has a product limit or price been rechecked on the publication date?
- Are known exclusions and uncertainty written beside the conclusion?
7. Keep a decision record and define the rerun trigger
Finish with a short decision record: workflow, candidates, frozen versions, test set, rubric, results by dimension, cost per successful task, important failures, chosen model, fallback and the person responsible for review. State why the winner fits this job. Keep rejected candidates in the record so a future team can see whether the constraint was quality, cost, latency, availability or risk rather than assuming one model was universally better. Sources: OpenAI; NIST.
Set a rerun trigger instead of a ceremonial annual review. Rerun when a pinned model is retired, a provider changes an alias, the workflow or policy changes, new failure patterns appear, traffic shifts to a different language or document type, or a candidate releases a version that could change the decision. Reuse the frozen regression set, then add new cases from real failures. This preserves comparability while making the evaluation more representative over time. Sources: Anthropic; OpenAI.
Limits and uncertainty
This checklist does not rank current models, reproduce third-party benchmark scores or replace domain-specific safety review. Search volume, prices, model versions and provider terms change. A small internal test can reveal obvious mismatches but cannot prove behavior across every future input. High-impact uses need broader coverage, independent review and ongoing production monitoring. The keyword gate run on September 10, 2026 did not identify a defensible weak-result opening, so this resource is published for reader utility and AI discovery rather than a claimed Google ranking opportunity.