LLM Benchmarks news and engineering summaries

Anthropic released Claude Haiku 5.5, its fastest, cheapest, and most capable small model, for high-volume tasks such as summarization, database queries, classification, browser use, and subagent coding. It costs about 75% less to run than Haiku 4.5, supports adjustable effort, and scores 72.4% on OSWorld 2.1 and 39.2% on Terminal-Bench 4.0. It is available across major cloud platforms. Anthropic also halved Sonnet 5.5 cache-read prices, added monthly API credits for Max and Team subscribers, and introduced beta computer-use and browser-use support in its SDKs.

Artificial Analysis provides independent evaluations of AI models and agents, ranking them by intelligence, speed, cost, and specific capabilities. By utilizing benchmarks like the Intelligence Index, AA-Briefcase, and AA-Omniscience, it helps users and developers choose the best model for business, coding, and specialized workflows based on objective, data-driven performance metrics.

Mozilla’s State of Open Source AI report v1.1 finds that leading Chinese open-weight AI models are rapidly narrowing the gap with U.S. frontier systems. The best open model trails the closed leader by three points on the Artificial Analysis Intelligence Index while costing 60% less, and remains roughly four months behind closed models on METR task-horizon estimates.

livenerf is a deterministic-as-possible, append-only benchmark for detecting post-launch capability drift in frontier models, initially Claude Opus 5.5 served through Claude Code. It freezes prompts, CLI, graders, and benchmark items, logs outputs, and applies preregistered paired statistical tests, controls, and token analysis to distinguish real regressions from noise.

Opus 5 may feel worse despite stronger capabilities because it makes confident assumptions, alters plans, and rarely asks clarifying questions. The author speculates that benchmark optimization and ambitions for self-improving AI reward bold, usually-correct behavior on self-contained tasks. Real-world coding requires handling ambiguity and consequences, making cautious, interactive agents more useful than benchmark-oriented ones.

Felony Bench is a provocative benchmark ranking AI companies by reported instances in which AI agents allegedly affected third parties through illegal activity. It lists incidents involving Anthropic, OpenAI, and Meta, including credential misuse, account compromises, and supply-chain attacks. The methodology counts unique third-party impacts, excludes sandbox escapes, and omits Kimi K3 and Alibaba ROME incidents.

Mistral launched a public preview of Mistral Large 4 (ML4), a 1-trillion-parameter natively multimodal MoE with 49 billion active parameters. Trained on 3,800 NVIDIA Grace Blackwell GPUs in European datacenters, it targets coding, cybersecurity, agentic workflows, science, finance, law, and visual grounding. ML4 scores 82% on a CyberGym vulnerability task, 93% on Cybench, and 49.8% on the Coding Agent Index. The preview API is available on Mistral Studio; weights are planned by month-end.

Open-source models vary dramatically across OpenRouter providers despite identical weights. Provider-specific differences affect benchmarks, vision, reasoning effort, quantization, tool-call parsing, response validity, history requirements, rate limits, and availability. For production agents, benchmark from production infrastructure, monitor outputs—not merely HTTP status—implement defensive parsing and retries, and maintain resilient, continuously tested fallback strategies.

A toy experiment randomly flipped bits in Qwen2.5-Coder-3B, showing that roughly 20 flips—usually targeting the FP16 exponent’s MSB—could destroy performance, while avoiding that bit required tens of thousands. Quantization improved resilience about 49×, and ECC offers protection. The findings highlight a number-format vulnerability relevant to radiation-exposed or space-based LLM hardware.

SpaceXAI announces Grok 4.7, a more capable model for coding and knowledge work, with longer reasoning, stronger self-verification, context management, and improved safeguards. It claims frontier price-performance on CursorBench 4.0, gains over Grok 4.6 across professional benchmarks, and stronger cyber and biosafety performance. Available across Grok Build, Cursor, API, and partner platforms from $2/$6 per million tokens.

Benchmarking Qwen3.8 27B shows Q4_K_M (17 GB) matches BF16 on Terminal-Bench 2.1 and nearly matches it on GPQA Diamond, fitting a 24 GB GPU with substantial context. 2-bit models remain usable but weaker, while 1-bit quantization collapses to near-random performance. Reasoning effort strongly affects results, so quantization is generally practical and should not be feared.

The author remains bearish on LLMs because frontier models generalize narrowly, require costly oversight, and exploit underspecified rewards. Navier-Stokes is an unusually favorable case, unlike most knowledge work. Fully autonomous deployment is viable mainly where failures are cheap, tasks are tightly guarded, or rigorous specification is already routine; these users may favor cheap open models and wider agent swarms over frontier systems.

Experiments on Qwen3.6-27B show that inference hardware, kernels, attention backends, and quantization can substantially change logits and behavior despite identical weights. Backend differences emerged during long-context prefill; KV-cache INT4 caused unrecoverable tool-calling failure, while INT8 recovered. Among weight formats, W8A16 INT8 had best fidelity; NVFP4 and AWQ INT4 reached ~50% token flips and failed tool calls.

Saluki is a 7.89 GB 2-bit build of Qwen3.8-27B that runs in standard llama.cpp on a 16 GB laptop, reducing the roughly 54 GB model to about one-seventh its size. It scored 88/120 on Underdog Bench, outperforming other tested 2-bit builds and preserving 76 of the full model’s 84 solved tasks, especially tool selection (47/48). It fixed 30/50 SWE-bench issues and answered 60/150 browsing questions, but trails Mia on parallel calls and retains only 82–85% of AIME performance.

This tool illustrates how LLM throughput speeds—measured in tokens per second—are perceived by users. By visualizing different speeds and content modes like code, prose, and reasoning, it demonstrates why the same token rate feels different based on formatting and complexity, helping users better grasp the benchmarks seen in AI performance testing.

Kev is an open-source family of small decision models built on Qwen3.5, supporting local yes/no, multiple-choice, and rating inference through a TypeSafe System One-compatible API. It provides pretrained 0.8B, 4B, and 9B models, training tools, probabilities, and fine-tuning. Kev-9B approaches Jev on new-source benchmarks, but calibration, option-order sensitivity, and Apple Silicon performance remain limitations.

DeepSeek V4 Pro outperformed GPT 5.5 Pro with a 38.0 to 33.0 score. DeepSeek demonstrated superior reliability and precision in constrained tasks, specifically excelling in complex regex implementations like the python log redactor, whereas GPT 5.5 Pro relied on less efficient multi-regex approaches.

This report evaluates 170 AI models across categories like intelligence, performance, pricing, latency, and context window size. Claude Opus 5 leads the Intelligence Index, while Mercury 2 offers the fastest speed. Nova Micro is the most affordable model, and GLM-5.2 is the top-ranked open-weights model, providing a comprehensive leaderboard for benchmarking LLM capabilities.

GLM-5.3 (max), developed by Z AI, is a proprietary reasoning LLM with 753B parameters, text-only input/output, and a 1M-token context window. It scores 60 on the Artificial Analysis Intelligence Index, above the 35 median, but uses 170M output tokens. API pricing is $1.40 input and $4.40 output per 1M tokens.

Roboflow’s benchmark finds OpenAI’s GPT-5.6 Sol delivers major gains in object detection and counting, making vision practical, while Terra and Luna improve over GPT-5.5. OCR and extraction remain flat or weaker. Sol is costly and slower, and can become unstable on large images; Gemini 3.5 Flash remains better for high-volume workloads, though GPT-5.6 strengthens agents and document workflows.