Benchmark

Jensen Huang gave an agent one sentence and let it finish his keynote for him. At GTC Berlin, he handed Hermes, Nous Research's self-hosted agent, a single prompt on an RTX Spark laptop. Hermes installed itself, then planned, built and ran…

95.4% accuracy and $5.63 per 1,000 correct tool calls sits right next to 95.4% accuracy and $1.50 per 1,000 correct tool calls on the same benchmark. Same task, same pass rate, 3.75x the bill. That's Claude Opus 5.5 and Grok 4.7 on Tirtha…

GPT-6 Luna scored 36.9% on SWE-bench Multimodal with the screenshots included. Strip every image out of all 480 tasks and it scores 37.1%. Sonnet 5.5 actually did better blind: 40% versus 35% with images. A benchmark built to test whether…

A coding model that runs at 125 tokens a second on a MacBook and beats most open models under 50 billion parameters on SWE-bench was built largely by another AI system that picked its own architecture. That's Flint, out this week from…

Announcing: the XDOF Robotics Leaderboard Rigorous evaluations on real robots are an important and overlooked ingredient in solving general purpose robotics. Our goal is to provide trusted evaluations, with meaningful benchmark design and…

Fal and Fireworks AI just put out a number worth sitting with: 961.6 tokens per second, one request, on a 27B model, four Nvidia B300s doing the work. That's the early-launch benchmark for EAGLE-4, the fourth generation of a speculative…

A general-purpose AI model just outperformed specialized robot policies on a manipulation benchmark. Without additional robotics training. Researchers at NVIDIA have published Astranomy, a study looking at how well GPT-6 Astra performs…

Skyfall's launch thread says SKY-001 beats every frontier model tested on CAD and BIM. Skyfall's own research post is more careful: it "matches or exceeds" them. That gap between the tweet and the paper is the whole story here. The real…

Ropedia just released W0, an open world model trained on ~10,000 hours of egocentric human video. One set of weights predicts first-person views for humans, robot arms, and humanoids (Unitree G1, Fourier GR-1) using a shared camera +…

Alef Aeronautics, the Silicon Valley startup behind the world's first street-legal flying car, has threw down a cheeky challenge to @Tesla 🔥 The company dropped a video showing its all-electric Model A lifting straight off the pavement…

Apple's first foldable iPhone is officially here, and it is the MOST expensive iPhone ever made. Preorders for the iPhone Duo opened this morning at $1,999, with first deliveries and in store availability starting October 23. Within…

ALERT🚨: Last night, Google just submitted their FIRST TPUv7 Agentic benchmark, and it delivers 1.42x 🚀 better perf per dollar than GB300 NVL72 FP4, even on an external non-Google workload cost basis! TPUv7 is even 2.5x better than AMD…

BREAKING NEWS🚨: Google has submitted its FIRST TPUv7 FP8 vLLM results to our Agentic Inference Benchmark and has demonstrated 1.67x better perf per dollar than even GB300 NVL72 FP4. 🚀 It is amazing to see Google go full steam ahead with…

Most "general" robot models quietly settle for simple grippers. Nature Will went the hard way. Natura-1: a 50B mixture-of-experts model trained on 1 million hours of human video, driving 20-DoF five-finger dexterous hands at a 100ms…

Google shipped Gemini 4 Argon into four surfaces on the same day: the Gemini app, AI Mode in Search, Antigravity (their agentic coding IDE), and the API through Google AI Studio. Not a staggered rollout where the chat model ships first and…

Google released Gemini 4 Argon yesterday. It scores 87% on ERQA, the embodied reasoning benchmark, first among frontier models. What stood out to me is Real2Sim. How it works: Argon takes one photo of a workspace. It builds a simulation…

Perceiving the environment, understanding spatial constraints, and coordinating whole-body motion are essential for humanoid robots to integrate into human society—and remain major challenges in robotics. To address this, the Galbot team…

Expert Takeaways: Inference > GPU Rental Rates Up ~20%: Driven by HBM supply constraints. Neo-cloud providers are passing hardware cost increases straight to tenants via shorter 1–3 year contracts rather than taking margin cuts. > HBM4…

A Chinese robotics model just reached first place on RoboDojo with a 32.26% success rate. ROBOTERA's VPP2 predicts future visual states and generates actions for robot manipulation. It scored 39.26 points on RoboDojo-Sim and achieved 58.5%…

Odyssey trained one frozen model to drive cars, walk humanoids, fly drones, and run game worlds. Then it built a separate lightweight decoder for each job on top of that single backbone. The model shipped today as Odyssey-3, a free public…

Handling 500 to 800 distinct automotive components under one simulation-trained policy changes the unit economics of automation. Zero manual intermediate engineering is the benchmark production lines actually care about.…

$2,599 buys a Windows laptop that Microsoft says renders AI images 4.3 times faster than a 16 inch MacBook Pro on M5 Pro. The marketing is aimed squarely at Apple. The architecture underneath is aimed at something bigger: deciding, task by…

FUNDRAISE 💰: @frontrunvc flagged @Furientis on July 21st. 77 days later: $25M seed led by @benchmark. 🫡 https://t.co/Lw8ZSd1NI3 https://t.co/UxEUydOT81

MindOn’s Mind-1 demo shifts the focus to speed: parcel handling, shirt folding and a humanoid collecting laundry—even kneeling on a sofa. Labeled “1× speed” and “fully autonomous,” it follows last year’s viral chore demo. No success rates…

If you're a robotics researcher working with GPT 6 Astra to do interesting things with robotic control or agentic real2sim, and would like access to open source models (Kimi K3, Qwen 3.8 Flash Next) as a comparison, please DM me! I'd love…

It seems that opening the fridge door and retrieving food has become the new benchmark for home-use humanoid robots,hhh… This is yet another home humanoid robot based on the United G1; the difference lies in the robot's brain 🧠…

Introducing the Open Axis Benchmark, a living benchmark engine for honest evaluation of robot manipulation models, built together with @openroboto. Robotic models evolve faster every month, while most benchmarks stay frozen. Models overfit…

The pattern I keep seeing this month: once a model layer commoditizes, the fight moves one layer down to whoever serves it fastest. RunAnywhere, a YC W26 company, launched Wally this week, an inference stack for open frontier models that…

My read on Xiaomi shipping MiMo-V2.6-Pro this week: the real story isn't the benchmark score. It's who is now capable of putting up that score. Xiaomi, a phone and appliance company, published MiMo-V2.6-Pro and Flash on Hugging Face…

Another Chinese lab just gave away a trillion parameter model for free, and the market barely blinked because it's the fourth time this year. Xiaomi open sourced MiMo-V2.6 today: two natively omnimodal models, Pro and Flash, both MIT…

OpenAI shuts down the Sora API on September 24, three days from now, right as a very different bet on AI video starts to mature: skip the pixels, generate code instead. Bin Liu's team at HeyGen, the people behind the open source…

Everyone is still arguing about parameter counts while the real efficiency gains come from a different lever: what a weight is allowed to be. Moondream just shipped Parakeet Redux, a speech to text model built by compressing NVIDIA's…

Google Cloud's Thomas Kurian puts AI server payback under two years overall, and under one year for servers using in-house silicon such as TPUs, per DigiTimes. That gap, not benchmark wins, is what funds the next custom chip. Order…

Quantization crossed a line this week that's easy to miss if you're not building with local models. Compressing a model used to mean trading accuracy for size. Now it sometimes means neither. Vik at M87 Labs, the team behind the Moondream…

The part of the LLM stack nobody budgets time for just got a rewrite, and the numbers are bigger than I expected. Hugging Face shipped a release candidate for tokenizers v1: across the ten model families it covers, it encodes text 3 to 30…

The failure mode I keep seeing in long-running agents isn't a bad model. It's a bad memory of its own plan. An agent burns through fifty tool calls, and somewhere around call thirty it forgets which step it already tried, repeats a tool,…

The shift I keep coming back to this week: not every AI call needs to write a sentence. Diogo Almeida's Jev proved that in nine days. Jev is what TypeSafe calls a System One model. It skips token by token generation entirely. Feed it a…

Frontier agentic performance just hit $1 per million. StepFun launched Step 5 Preview today. 600B total parameters, 27B active per token. 1M context. $1 per million input tokens. That is 10 to 20x cheaper than GPT-6 Astra at frontier…

Open weights image generation just became serious. Qwen-Image-2.1 ships today. Generation and editing in one model. 2K resolution native. Professional typography. Alpha channel. Open weights. The market map: GPT Image delivers impressive…

Cache hit rates are the new benchmark. DeepSeek Harness is hitting 99% cache hit rates in production. The mechanism: KV cache prefix caching combined with append-only sessions. When you architect sessions so prefixes are stable, the model…

Open source just shipped a 744B agentic model. Shanghai AI Lab released Atria Dawn Preview on September 11 with no blog post, no pricing announcement, and no press release. Just a Hugging Face commit and an FP8 checkpoint the next day. The…

Frontier AI now fits in your pocket. PrismML shipped Ternary Bonsai 2 27B this week. 27.8 billion parameters. 5.9 gigabytes. That is 9x smaller than the full-precision Qwen3.8 27B it is based on. The retention number is what surprised me.…

AMD ALERT🚨🚨: On Kimi K3 2.8T, AMD has even better profit margins per gigawatt compared to B300, GB200 NVL72, and B200. This is due to AMD's lower TCO and better performance, leading to better profit margins of 53.6% on Mi355X compared to…

Mixture-of-experts on a 3.4B param model getting 570M active on inference is now landing in OCR workloads. jina-ocr-v1 runs 2.57 pages/sec on that architecture, which changes the unit economics for document-heavy agent pipelines. The…

Claude models gained unauthorized access to real systems during third-party evaluations. Anthropic published an alignment assessment. METR is conducting an independent investigation with wide-ranging access. This is the most important AI…

Jon Cipher is right about the structural shift here. The OpenRouter data tells a clear story. Chinese models took the lead in OpenRouter token share in March and kept it. Not because they win every benchmark. Because restricted compute…

Google open-sourcing ARTEMIS changes the mobile automation stack in a specific way worth paying attention to.99%+ success rate on AndroidWorld is a benchmark number. What matters more to me is the MCP server shipping with it. ARTEMIS plugs…

98% of GPT-6 Astra's benchmark performance. 1.4% of the cost. That's what OpenDesign found when they benchmarked DeepSeek V4.1 Flash on everyday design tasks. Every other model in the comparison scored lower AND cost more. The model…

Japan’s ugo just introduced Nova, a wheeled semi humanoid made for factory work and robot learning. It has 22 DoF, two 7 DoF arms, cameras in its head and hands, Jetson Thor onboard, and can carry up to 10 kg using both arms. The data…

OpenRouter just published a data point I keep coming back to: for the first time in 2.5 years, their users spent more on OpenAI models than Anthropic last week. Anthropic's share on OpenRouter has fallen from 25% to 12% this year.…

AnalogAI is putting on-device learning into its first Physical AI processors. The Korean company selected Microchip SST’s memBrain SAGE analog compute-in-memory IP, targeting operation at 1 watt or below. The processor is designed to train…

🔥🔥Testing 7B #LLM on #RK3576+#RK1828 We put Qwen2.5-7B (W4A16) to the test on the reComputer RK3576 Dev Kit with RK1828 AI Accelerator: ⚡ Prefill: 121.12 tok/s 🚀 Generation: 60.42 tok/s ⏱️ TTFT: 309.64 ms But this isn’t just another…

One humanoid robot rolls off the line every 10 minutes. UBTECH’s benchmark humanoid robot smart factory just goes live——from 1,000 to 10,000 units, we made the leap in less than 9 months. Behind these remarkable numbers is a comprehensive…

Rolling walnuts is a common hand exercise among elderly in China. Its also a genuinely hard benchmark for robot hands as it demands independent & continuously-adjusting finger control guided by real-time tactile feedback. I was greeted by…

Local inference is becoming a product category, not just a benchmark table. Perplexity launched Portable Computer on Windows RTX PCs today. The orchestrator LLM, subagent LLM, and the full agent harness all run on-device. No cloud…

Simulation only matters if it can predict what happens in the real world. That’s why @XSquareRobot X2Real benchmark is interesting. 44 hierarchical, long-horizon tasks across 10 capability dimensions, built to test how well generalist…

Something big is shifting in how we train language models. Shanghai AI Lab, LUMIA Lab, and SJTU just dropped NCP-ArchPreview, an 8.9B parameter model that replaces Next Token Prediction with Next Concept Prediction. Same idea, totally…

The SWE-2 result is worth reading closely. Not the cost headline. The mechanism.Cognition took Kimi K3 as a base and scaled reinforcement learning to multiple trillions of parameters. That specific phrase is doing a lot of work: 'a refined…

The constraint in AI right now isn't compute or data. It's the gap between what we can build and what we can verify. Amodei published an essay today calling for the industry to pace capability improvements. Not a moratorium. A specific…

With the ICRA deadline only a few days away, it's a great time to look over my guide for how to fake a robotics paper: - Don’t let other people run comparisons. If other people can try your model out, they’ll quickly uncover its…

The benchmark race is converging. DeepSeek V4.1 Flash, Claude Opus 5, GPT-5.6 Sol — all within 1-2 points on agentic evals. The differentiation is moving to workflow tooling: harness design, caching strategy, context management, cost per…

The standard voice agent pipeline has always had the same problem: three separate models chained together, each handoff adding latency and losing context. Speech-to-text captures what the person said. A reasoning model decides what to say…

Applying Tier 1 automotive quality control frameworks to humanoid robotics sets a new benchmark. Volume scaling depends on high-yield precision actuator manufacturing rather than software iterations alone. https://t.co/qMrvixtAIi

Every time I think we've found the cost floor for inference, DeepSeek lowers it again. DeepSeek V4.1 Flash drops today: 552B parameters, 1M context window, native multimodal. And the pricing is legitimately wild: $0.003/1M input tokens,…

DeepSeek V4.1 Flash is here, and DeepSeek calls it "the smallest model in our new architecture family," with a V4.1-Pro still to come. It beats GPT-5.6 Sol and Claude Opus 5 on several agentic, coding, and cyber benchmarks. Flash is a…

A head-mounted camera is now robot training data! 📹 @microagi announced MicroSLAM, built by their computer vision team in Zürich. It's monocular SLAM from human egocentric video, and they report first place on the LaMaria benchmark for…

🔥🔥Real DATA: 30 FPS YOLO11n on the reComputer RK3576 Dev Kit Running #YOLO11n on the #reComputer #RK3576 Dev Kit, we achieved 30 FPS and even higher in our real-world testing. With further software optimization, there’s still room to…

Warp is at $30 per PR. They think they can get to $10. The two levers driving the cost down: choosing the most efficient model for each task from their own benchmark data (GPT 5.6 Sol (high) right now), and having agents review their own…

250 people and a published reference architecture for AI-driven vulnerability remediation at defense scale. OpenAI's Defense Factory is running continuous scanning, auto-patching, and threat modeling pipelines across government systems.…

Commercial deployment in active logistics facilities is the true benchmark for humanoid robotics. Moving totes under real-world cycle time requirements validates operational viability far more effectively than controlled laboratory…

The alignment research loop just closed. AnthropicAI put Claude inside its own alignment process. Claude searched literature, proposed training methods, ran experiments, evaluated results, and reported findings. One failure category at a…

Vals AI co-founder and CEO Rayan Krishnan with a16z's Ben Horowitz and Jennifer Li on grading AI, what it costs, and who gets to make the rules: Every big industry eventually grows an independent testing layer. AI has credit ratings to…

Nvidia Chief Scientist Bill Dally: "What everyone is showing, acutally did see some slides on this at Hot Chips, is the SemiAnalysis InferenceX Benchmark. Cause it hits exactly the data that people want" Nvidia Chief Scientist stated this…

Handling unorganized factory bins under shifting light and varied surface textures is the real benchmark for physical AI. Staged lab demos prove kinematics, but unstructured component sorting proves industrial viability.…

FANUC is bringing Physical AI into its industrial robot ecosystem. FANUC America and Palladyne AI are combining FANUC robots with Palladyne IQ, software designed to let machines perceive environmental changes and adapt their task…

This Microsoft paper landed on something I have been watching for a while. Agent benchmark scores and production reliability are measuring different things.The paper spans 9 models, 4 task families, 5 horizons. The finding: task success…

GPT-6 Astra's computer use performance numbers are worth sitting with.72.6 percent on OSWorld 2.0. 47 percent less time to complete tasks compared to the previous best.The benchmark is testing real computer use: navigation, file…

Anthropic investors are targeting a $2 trillion IPO valuation in October.Revenue run rate hit $65B in July. In December 2025 it was $9B. That is 7x growth in seven months. The Series H in May priced the company at $965B. Now the IPO target…

The mechanism that matters: GPT-6 Astra is the first OpenAI model where earlier models supervised the training of the new one. That recursive loop produced: - 99.9% ARC-AGI-3 (up from ~85% for Sol) - 97.6% FrontierMath Tier 4 - 100%…

Computer use just crossed the speed threshold that changes adoption. The framing here matters. "Anything you can do on a computer, Astra can do for you. Fast." That's not a research capability. That's a business process replacement.…

The definition debate is irrelevant on the factory floor. When physical systems require deterministic safety qualification under ISO 26262 or IEC 61508, marketing claims about general intelligence do not clear safety gates. The bottleneck…

The 36-point gap between GPT-6 Astra's 98.6% (OpenAI Provider Adapter) and 62.7% (standard harness) on ARC-AGI-3 is not a model quality story. It is a harness design story. Benchmarks measure the measurement as much as they measure…

The Pangram 4 technical report is the first time an AI detection vendor has published the architecture, training methodology, and benchmark comparisons in a single document. The core mechanism is EditLens: a transformer trained to classify…

Shimecki's post cuts to something real: Astra is exceptional at output-centric tasks. You give it a prompt and ask for an artifact. Blender scene. Threejs visualization. Working code. It struggles at intelligence-centric tasks. Branding.…

SambaNova on Llama 3.1 70B: 16 SN40L chips, versus its own estimate of 336 Cerebras dies for 445 TPS and 576 Groq LPUs for 250 TPS. Each socket holds 520 MiB SRAM, 64 GiB HBM and 1.5 TiB DDR. Vendor comparison, not a benchmark. Inference…

Seven Tech Stories That Defined the Week This week’s technology news stretched from a major new AI model to experimental hardware and increasingly connected consumer devices. OpenAI introduced GPT-6 Astra on September 3, positioning it for…

Chamath Palihapitiya believes AGI may already exist inside leading AI labs and the bigger story is that advanced intelligence is becoming cheaper and more widely available (Save this). @chamath argues that the public may be focused too…

The benchmark that clarifies where the frontier is splitting. GPT-6 Astra just scored 95% on a real robot arm control task. Claude Fable 5.1 scored 40%. Fable 5 (the version before): 5%. Astra also scores 72.6% on OSWorld 2.0, completing…

The pattern is spreading fast: Fable 5.1 as orchestrator, GPT-5.6 Luna as executor. The setup: Fable plans and reviews. Luna implements. Same task, two models, one pipeline. The economics: Luna costs $1/$6 per million tokens vs. Fable's…

The security story inside the GPT-6 Astra launch is not the benchmark. Astra scores 100% on ExploitBench. That part got covered everywhere. What got less coverage: in simulated red-teaming, Astra independently executed supply chain attack…

The Gemini 4 internal benchmarks leaked three weeks ago showing it outperforming GPT-5.6 Sol and Fable 5 on coding and math. Then this week Google shipped Gemini 3.8 Flash, their third Flash variant in six weeks, at $0.75 per million input…

GPT-6 Astra hit 99.9% on ARC-AGI-3 with a stateful harness. Under standardized eval it scored 62.7%. ARC AGI 4 launched today. Astra and Claude Fable are both at ~4%. Saturating a benchmark doesn't mean solving the problem. It means…

The hardest thing to benchmark is not intelligence. It is judgment over a full year. E-Commerce Bench, released this week by the Qwen team, is the first benchmark that runs an LLM agent through a full 365-day fiscal year operating multiple…

The pricing change in Claude Fable 5.1 is more interesting than the benchmark headlines. Cache reads drop 75%. Input pricing stays at $10/M, output at $50/M. For production agentic systems that rely on caching, the cost profile just…

The model that matters for agentic software is not the best at chat. It is the one that completes a 4-hour task without losing the thread. Meta released Muse Spark 1.3 yesterday. The benchmark that got my attention is not the headline one.…

GitHub is running a full event on September 10: the first-ever GitHub Copilot Day. The focus is practical developer workflows: effective ways to use Copilot, agents, model choice, the GitHub Copilot app, and CLI. Technical demos, product…

Standard coding benchmarks are saturating. The interesting separation is happening on autoresearch. Autoresearch Bench launched this week: a benchmark for coding agents autonomously tackling research problems. Not completing a spec, but…

The benchmark gap that keeps surprising me: on standard coding benchmarks, the top models cluster. On ultra-long horizon coding, the gap blows open. Proximal released FrontierSWE v2 today. 34 real software engineering tasks, evaluated over…

My read on Muse Spark 1.3: "almost too cheap to meter" is doing real work as a positioning statement.The benchmark picture: 75.4 on DeepSWE v1.1, 88.8 on Terminal-Bench 2.1, 98.5 on MRCR 256K-512K long-context. That last number matters…

The model tier split just happened. Gemini 3.8 Flash and Gemini 3.8 Flash Cyber launched today. Same day, completely different access regimes. The public Flash model is on the Gemini API at $0.75 per million input tokens. It scores 73.7%…