Haiku 5.5’s lower listed rates need independent task tests

A token is a unit a model uses to process text. Anthropic says Haiku 5.5’s tokenizer, which divides text into tokens, produces about 30% more tokens for the same text than Haiku 4.5. Simon Willison reported a count about 1.25 times higher for one long prompt using his token-counter tool. That individual observation is not a broad workload study, and the two figures describe different measures. Neither tells buyers how many tokens a particular application will use.

Anthropic estimates that Haiku 5.5’s average running costs are about 75% lower than Haiku 4.5’s, accounting for the tokenizer change. That estimate can help buyers decide whether to run a comparison. But the practical measure is cost per successfully completed task, which also depends on how much text is processed, how many model calls are needed and whether the output meets the task’s requirements.

Benchmarks can help buyers shortlist, not decide

Anthropic positions Haiku 5.5 for high-volume, latency-sensitive work such as classification, extraction and routing. SiliconANGLE reported scores of 72.4% for Haiku 5.5 and 48.9% for GPT-6 Luna on OSWorld 2.1’s offline subset. On Terminal-Bench 4.0, it reported scores of 39.2% and 16.4%, respectively. The reported benchmark results describe performance on those tests; they do not establish how either model will perform across a business’s own tasks.

The material reviewed here does not establish independent replication of those scores. That does not make them useless. They can help buyers identify tasks worth testing, but benchmark results are no substitute for measuring accuracy, errors, latency and cost on the work a buyer needs done.

Anthropic also published customer evaluations. Asana reported more than 30% lower task-completion latency and up to 2.5 times faster inference per agent turn on its evaluation suite, compared with the model it had been using. Inference is the model’s processing of a request. Asana’s results are specific to its tests; they are not an independent or generally applicable evaluation of Haiku 5.5.

There is a practical case for using these signals: businesses need to narrow their options before spending time on their own tests, and posted prices are concrete. Anthropic’s benchmarks and Asana’s experience can help form a shortlist. But evidence that helps decide what to test does not prove which model will deliver the best value on a different workload.

Cache savings depend on actual use

Anthropic lowered Sonnet 5.5’s cache-read price from $0.20 to $0.10 per million tokens. Cache reads are reused prompt material billed separately from standard input and output. Anthropic says cached tokens account for a large share of consumption in many agentic tasks and estimates that the lower rate will reduce the cost of most such tasks by around 20%.

The cut applies to cache reads; the estimated reduction in total task cost depends on how much a customer uses the cache and on the tokens and calls needed to complete the task. The change is a real reduction in one listed rate, but it does not establish a 20% saving on every user’s bill. Buyers need their own usage data to see what the lower rate means in practice.

We urge Anthropic to support independent evaluations of Haiku 5.5 and Sonnet 5.5 across varied workloads, with evaluators setting the methods and reporting task success, errors, latency and total cost per completed task—including cached and uncached token use. Until those comparisons are available, buyers should treat price cuts as a reason to test a model, not proof that it delivers more for the money.