Seven promises, one request each.
The OpenAI chat format is a list of promises: set this option and that will happen. A gateway that speaks the format takes those promises on. Spec check tries each one.
§1The promises
| Promise | What is sent | Kept when |
|---|---|---|
| System prompt | A system message that asks for one made-up word, then an unrelated question. | The reply is that word. |
| Output cap | "Count to 300" with max_tokens set to 24. | The reply stops at about 24 tokens. |
| Stop sequence | "Write alpha beta gamma delta" with gamma as the stop word. | The reply ends before gamma. |
| JSON mode | A request for a small object with response_format set to json_object. | The reply parses as JSON. |
| Tool calling | A weather question, with a weather tool on offer. | The reply calls the tool with a city. |
| Streaming | "Write the numbers 1 to 30" with stream switched on. | The reply arrives in more than one piece. |
| Token counts | A plain request. | The reply reports tokens in and tokens out. |
§2Also noticed
- Prompt padding
- How many tokens a seven-word message was counted as. A large number means something is added to your prompt before the model sees it, and you are billed for it.
- Model name returned
- Whether the reply names the model you asked for. It is the endpoint's own word, nothing more.
- Repeatable at temperature 0
- Whether the same request twice gives the same reply. Many hosts do not promise this; it is shown so you know.
§3Gateway or model?
- Some promises are the model's to keep. A model that was never trained to call tools will fail that test on every host.
- So read a broken promise next to what the model's maker says the model supports. If the maker's own API keeps it and this endpoint does not, the endpoint dropped it.
- The quickest way to be sure is to run spec check twice: once here, once against the maker.
§4Where it stops
- One request per promise. A gateway that keeps a promise most of the time and breaks it sometimes can pass.
- It tests the options most code depends on, not every option in the format.
- OpenAI chat format only.
→Keep reading
- Next
- How the drift log works · each endpoint against its own first record
- Before this
- How unmask works
- All pages
- The docs