Thinking Machines’ Inkling 2 Small pairs stronger benchmarks with higher task costs

Inkling 2 Small’s benchmark gains come with a clear tradeoff: the model performs substantially better than its predecessor on several coding, vision and agent tasks, but uses far more output tokens and declines to answer more often. Thinking Machines presents it as a base model for fine-tuning, a design that could suit teams adapting it to specialized work.

The model comes from Thinking Machines, the startup founded by former OpenAI CTO Mira Murati. Its mixture-of-experts architecture has 594 billion parameters in total, with 17 billion active at a time. It accepts text and images, and users can adjust its thinking depth to choose between faster responses and more extended reasoning.

Thinking Machines says Inkling 2 Small is its second model generation. The company’s central case for it rests on performance across several kinds of work, rather than one headline benchmark. On the Artificial Analysis Intelligence Index, it scores 31, up from 13 for Inkling Small and roughly level with Google’s Gemini 3.8 Flash, according to the company.

Agent performance improves as response rates fall

The strongest reported changes appear in tasks that require a model to carry out multiple steps. On AutomationBench-AA, Inkling 2 Small scores 46 percent, compared with 5 percent for its predecessor. Its FrontierCode 2.0 score rises from 5 to 25 percent. On AA-Briefcase, a benchmark for knowledge work, the model reaches 1,527 Elo, 615 points above Inkling Small.

Those results need to be read alongside a substantial change in how often the model responds. Its reported hallucination rate drops from 61 to 27 percent, but Artificial Analysis attributes much of that decline to a lower response rate: Inkling 2 Small answers 53 percent of questions, compared with 77 percent for its predecessor. That makes the lower hallucination figure less straightforward to interpret as a standalone measure of accuracy. For teams evaluating it, response behavior matters as much as benchmark scores.

Thinking Machines says it trained the model with reinforcement learning on hundreds of thousands of long-horizon coding tasks. An automated evaluation system assessed whether the code worked and how the model approached each task. The company reports scores of 25.4 percent on FrontierCode 2.0, 72.1 percent on DeepSWE, and 86.3 percent on SWEBench Pro v2 (Hard).

These scores don’t put Inkling 2 Small ahead on every coding test. DeepSeek-V4.1-Flash scores 28.5 percent on FrontierCode and 77 percent on DeepSWE, according to the comparison in the source material. The results point to a model with strong coding performance, while leaving room for competitors to lead on particular evaluations.

Vision results and a higher price per task

Thinking Machines also reports strong results on visual tasks. Inkling 2 Small scores 25.2 percent on GDP.pdf, which tests question answering across complex PDF documents, the highest result among the models compared. It reaches 81.6 percent on MMMU Pro, a benchmark covering academic tasks that involve diagrams and figures, and 64.7 percent on PerceptionBench for visual perception.

For website design, the model receives a 1,340 Elo rating on DesignArena Website, where users compare generated websites head to head. Thinking Machines says that is the highest score among open-weight models. The company also reports progress in 3D modeling, though the source gives no benchmark score for that capability. A separate demonstration had the model identify three wines in 19 out of 20 attempts from photos alone, a striking result that should be treated as a narrow demonstration rather than evidence of broad visual reliability.

Higher benchmark scores come with higher inference costs. Inkling 2 Small generates about 150,000 output tokens per task on average, roughly four times the predecessor’s usage. Artificial Analysis estimates its cost at $1.18 per task, nearly six times the cost of Inkling Small. That remains below the cited estimates for Gemini 3.8 Flash at $7.77 and Claude Sonnet 5.5 at $10.02. The comparison is useful, but teams should assess costs against their own workloads, since token use and task mix affect the bill.

Training scale, limitations and access

Thinking Machines says the full training process used about 12 trillion tokens. The reinforcement-learning phase used 3 trillion tokens on 2,048 GPUs over nine days. Those figures offer a glimpse of the training effort, though they don’t by themselves establish how the model will perform after fine-tuning or under different deployment conditions.

The company describes several practical weaknesses. Inkling 2 Small can communicate inefficiently, may choose unsuitable tools when working with large codebases, and isn’t optimized for chat. It doesn’t support audio, may overreact to health-related topics, and can produce cluttered output. These limits matter for teams considering it as a base model: specialized tuning may help adapt behavior, but the source does not report how much work that would require.

For now, the model is available through the Tinker platform. Thinking Machines plans to publish the weights under the Apache 2.0 license after external safety testing is complete. The company says additional sizes in the model family are also planned, but gives no release dates. The next useful test will be whether the reported gains hold up when developers can evaluate and adapt the weights on their own tasks.

Source: Thinking Machines. Benchmark comparison: Artificial Analysis.

Related comparison: DeepSeek-V4.1-Flash benchmark results. DesignArena score: DesignArena.