CTT-Bench v1.0

Chat-template tool-call compatibility

Does a published chat template survive the tool-call shapes that harnesses actually feed it? This is a spec-compatibility table, not a model-quality ranking.

Measured 16 Nov 2026, 22:16 IST · transformers 5.22.2 · 41 repos, 33 distinct templates, 10 cases

Read the write-up — 31 of 40 popular chat templates crash on OpenAI-style tool calls.

pass rendered correctly crash obscure Jinja / TypeError clear honest “deserialize first” error double silently double-encoded verbatim tool name pasted unescaped

Click a row for the error or the rendered snippet. Click a header to sort.

The ten cases

How to reproduce

The runner ships with the dataset. To re-score the same 41 repos:

pip install transformers==5.22.2
python scan.py --repos repos.json --out-dir .

Or one OpenAI-style string call, the case most templates fail (C2):

import json
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("openbmb/MiniCPM5-2B")
tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Get weather",
    "parameters": {"type": "object", "properties": {
        "city": {"type": "string", "description": "City"},
        "metric": {"type": "boolean", "description": "Metric units"},
    }, "required": ["city"]},
}}]
messages = [
    {"role": "user", "content": "Weather in Paris?"},
    {"role": "assistant", "content": "", "tool_calls": [{
        "id": "c1", "type": "function",
        "function": {"name": "get_weather",
                     "arguments": json.dumps({"city": "Paris", "metric": True})},
    }]},
    {"role": "tool", "tool_call_id": "c1", "name": "get_weather",
     "content": '{"temp": 21}'},
]
print(tok.apply_chat_template(messages, tools=tools, tokenize=False))

On many popular templates that raises UndefinedError: 'str object' has no attribute 'items' or TypeError: Can only get item pairs from a mapping. Passing a dict instead of the string is the documented shape (case C1) and usually works. In a harness, json.loads every arguments value before you template, and reject a tool name that is not in your declared list before you write it into history.

Method, in brief

Population: top 500 text-generation models by Hub downloads, updated in the last 180 days, minus quants, abliterated forks, and tiny test repos, keeping those whose chat template references tool_calls. That leaves 41 repos (33 distinct template hashes). Scoring uses tokenizer.apply_chat_template(..., tokenize=False) only — no weights, no generation. Markers and snippets are taken from the post-user region of the prompt so a name that already appears in the tool schema is not counted as a successful call.

A clear error is not a pass. It is a better failure mode than a crash, but it is still not OpenAI-string compatible. No template in this snapshot passes all 10 cases; the best score is 9/10, failing only tool-name escaping.

Related reports: Qwen #240, MiniCPM5 #30, Granite #7, Apertus PR #36.