CTT-Bench v1.0
Does a published chat template survive the tool-call shapes that harnesses actually feed it? This is a spec-compatibility table, not a model-quality ranking.
Read the write-up — 31 of 40 popular chat templates crash on OpenAI-style tool calls.
Click a row for the error or the rendered snippet. Click a header to sort.
The runner ships with the dataset. To re-score the same 41 repos:
pip install transformers==5.22.2
python scan.py --repos repos.json --out-dir .
Or one OpenAI-style string call, the case most templates fail (C2):
import json
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("openbmb/MiniCPM5-2B")
tools = [{"type": "function", "function": {
"name": "get_weather", "description": "Get weather",
"parameters": {"type": "object", "properties": {
"city": {"type": "string", "description": "City"},
"metric": {"type": "boolean", "description": "Metric units"},
}, "required": ["city"]},
}}]
messages = [
{"role": "user", "content": "Weather in Paris?"},
{"role": "assistant", "content": "", "tool_calls": [{
"id": "c1", "type": "function",
"function": {"name": "get_weather",
"arguments": json.dumps({"city": "Paris", "metric": True})},
}]},
{"role": "tool", "tool_call_id": "c1", "name": "get_weather",
"content": '{"temp": 21}'},
]
print(tok.apply_chat_template(messages, tools=tools, tokenize=False))
On many popular templates that raises UndefinedError: 'str object' has no attribute 'items' or TypeError: Can only get item pairs from a mapping. Passing a dict instead of the string is the documented shape (case C1) and usually works. In a harness, json.loads every arguments value before you template, and reject a tool name that is not in your declared list before you write it into history.
Population: top 500 text-generation models by Hub downloads, updated in the last 180 days, minus quants, abliterated forks, and tiny test repos, keeping those whose chat template references tool_calls. That leaves 41 repos (33 distinct template hashes). Scoring uses tokenizer.apply_chat_template(..., tokenize=False) only — no weights, no generation. Markers and snippets are taken from the post-user region of the prompt so a name that already appears in the tool schema is not counted as a successful call.
A clear error is not a pass. It is a better failure mode than a crash, but it is still not OpenAI-string compatible. No template in this snapshot passes all 10 cases; the best score is 9/10, failing only tool-name escaping.
Related reports: Qwen #240, MiniCPM5 #30, Granite #7, Apertus PR #36.