language benchmark
COLLIE Leaderboard
COLLIE is a grammar-based framework for constructing constrained text generation tasks, with the COLLIE-v1 dataset containing 2,080 instances across 13 constraint structures.
Updated Sep 5, 2026
Models10
Model coverage10
MetricScore
EvidenceB
COLLIE Ranking
Higher score ranks better on this benchmark.
Sort by
Rank | Model | Score | Percentile | Participants | Evidence | Evaluated |
|---|
| Rank01 | ModelOP | Score99.00% | Percentile100.00% | Participants10 | EvidenceC | Evaluated |
| Rank02 | ModelOP | Score98.70% | Percentile88.89% | Participants10 | EvidenceC | Evaluated |
| Rank03 | ModelOP | Score98.40% | Percentile77.78% | Participants10 | EvidenceC | Evaluated |
| Rank04 | ModelMA | Score95.80% | Percentile66.67% | Participants10 | EvidenceC | Evaluated |
| Rank05 | ModelOP | Score72.30% | Percentile55.56% | Participants10 | EvidenceC | Evaluated |
| Rank06 | ModelOP | Score65.80% | Percentile44.44% | Participants10 | EvidenceC | Evaluated |
| Rank07 | ModelMA | Score62.90% | Percentile33.33% | Participants10 | EvidenceC | Evaluated |
| Rank08 | ModelOP | Score61.00% | Percentile22.22% | Participants10 | EvidenceC | Evaluated |
| Rank09 | ModelOP | Score54.60% | Percentile11.11% | Participants10 | EvidenceC | Evaluated |
| Rank10 | ModelOP | Score42.50% | Percentile0.00% | Participants10 | EvidenceC | Evaluated |
COLLIE Highlights
The leading models and scores on this benchmark.
COLLIE Score Distribution
A closer view of the leading scores on this benchmark.
The Top AI Models for COLLIE
The first five results on this benchmark, with official price and output speed added where the model identity can be matched.
What is COLLIE?
What COLLIE measures and how its scores work.
COLLIE is a grammar-based framework for systematically constructing constrained text generation tasks across language understanding, logical reasoning, and semantic planning.
COLLIE reports a Score measured as a ratio.
Scores are shown in ratio. This benchmark is not independently verified and has an evidence level of B.
- Family
- COLLIE
- Modality
- text
- Primary category
- language
- Score direction
- higher
- LLMBoard eligible
- No
- Evaluation key
- overall
LLM Stats. Benchmark scores retain their original unit. Overall score eligibility is shown separately.
FAQ
Common questions about COLLIE.
Which model scores highest on COLLIE?
GPT-5 is currently ranked first with 99.00%.
What are the top three models on COLLIE?
The current leaders are GPT-5 (99.00%), o3-mini (98.70%), and o3 (98.40%).
Which COLLIE model has the lowest official input price?
GPT-4.1 nano has the lowest matched official input price at $0.10 input / $0.40 output per 1M tokens.
Which models are fastest among COLLIE results?
The fastest matched records are GPT-4.1 mini (467.39 tok/s via OpenAI), Mistral Small 4 (163.71 tok/s via Mistral AI), and GPT-4.1 nano (138.14 tok/s via OpenAI).
Does the highest score result prove overall model quality?
No. This benchmark measures one defined capability or task. The overall LLMBoard score uses a separate aggregation across eligible benchmark evidence.