Choose a model for the work it must perform, not from a popularity list. This worksheet turns a concrete need into a reproducible comparison of quality, latency, context, data handling, modality, integration, and budget, with a result you can test again.
When to use it and when not to
Use it before integrating a model into support, analysis, writing, classification, or automation. Do not use it to declare a universal winner or copy rankings, prices, or specifications that may change. Consider a business audit only when comparison is recurring, involves a team, has meaningful volume or spend, processes sensitive data, or creates a material bottleneck. Otherwise, reuse this worksheet or use the general Quest assistant to prepare the test protocol.
Decision worksheet
Define the task and assign weights totaling 100:
| Criterion | Question | Weight |
|---|---|---|
| Task | What must it produce or decide? | |
| Required quality | Which errors are acceptable? | |
| Latency | How fast must the user receive an answer? | |
| Context | What input length and memory are needed? | |
| Data handling | What retention, controls, and residency are required? | |
| Modality | Text, image, audio, files, or combinations? | |
| Integration | Which APIs, formats, tools, and observability? | |
| Budget | What monthly or per-task band is sustainable? |
Record mandatory constraints, estimated volume, languages, available human evaluation, and review date. Use cost bands such as low, medium, or high, and confirm actual cost in the selected configuration. Do not present exact prices as stable facts.
Workflow
- Describe one use case and a small set of representative inputs without sensitive data.
- Define an acceptable output and examples of critical, minor, and tolerable errors.
- Treat the task as a shared requirement, weight the seven scored criteria, and mark any other mandatory requirements.
- Select two or three plausible candidates without assuming reputation determines the result.
- Check the documentation available for your account for required capabilities. Record missing information as unknown.
- Run the same test in each candidate's own environment or API, keeping the input, configuration, and context comparable where possible.
- Measure quality with a rubric, observed latency, refusal rate, cost per task, and integration effort.
- Score each criterion from 1 to 5 and calculate a weighted score.
- Create a shortlist of one or two candidates and record selection risks.
- Define an acceptance test, review date, and rollback condition.
Example: replies for an online shop
An online shop receives 600 questions per month in Spanish and English. It needs accurate replies based on supplied policies, latency of roughly a few seconds, and exclusion of payment data. Quality weighs 35, data handling 20, integration 15, latency 10, context 8, budget 9, and modality 3, while the task itself is a shared requirement. The weights total 100. Two candidates score similarly. The first handles ambiguous cases better; the second requires less integration work. The shortlist proposes a one-week test with both using 50 anonymized questions. No winner is claimed until accuracy, cost per task, and escalations are reviewed.
Copyable final artifact: comparison sheet
# Model comparison sheet
Task:
Team and date:
Estimated volume:
Allowed data:
Mandatory criteria:
## Weights
- Quality: __%
- Latency: __%
- Context: __%
- Data handling: __%
- Modality: __%
- Integration: __%
- Budget: __%
## Candidates
| Candidate | Quality 1-5 | Latency 1-5 | Context 1-5 | Data 1-5 | Modality 1-5 | Integration 1-5 | Budget 1-5 | Weighted score |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| A | | | | | | | | |
| B | | | | | | | | |
| C | | | | | | | | |
## Cost band
Low / medium / high:
Volume and configuration assumptions:
What remains to confirm:
## Shortlist and test
Candidates:
Acceptance cases:
Metrics:
Review date:
Rollback condition:
Failure and safety checks
Do not use real customer data for comparison without authorization; anonymize and minimize inputs. Control randomness and retain prompts, evaluation versions, and dates. Check that a quality improvement does not hide higher latency, cost, or data exposure. Treat undocumented or untested capabilities as unknown. Do not confuse a stated maximum context with useful quality for your task. Repeat tests with edge cases, languages, and real formats. If information is missing, test each candidate in its own environment or API and record observations instead of claiming a winner. Use the general Quest assistant to review the protocol, not to simulate multiple models. Review the decision periodically because capabilities, terms, and costs change.
As a restrained next step, run the acceptance test with the shortlist and archive the sheet. Request a business audit only if comparison is recurring, collaborative, high-volume or materially costly, sensitive, or materially blocking.