vLLM

A coding model that runs at 125 tokens a second on a MacBook and beats most open models under 50 billion parameters on SWE-bench was built largely by another AI system that picked its own architecture. That's Flint, out this week from…

NVIDIA already has TensorRT-LLM. It also has Dynamo. This week it shipped a third local inference server anyway, and the part worth sitting with is why. Flare runs on Rust, not Python, and it's built narrow: just the GPUs with the…

Every hyperscaler racing to dent Nvidia's lead just converged on the same move within a day of each other. Google open sourced TorchTPU at the PyTorch Conference in San Jose this week, co-built with Meta's PyTorch team, letting existing…

Model weights are free. Running them at scale cheaply is not, and that's the part DeepSeek open sourced for everyone this week. DeepInfer is the inference engine serving DeepSeek V4.1 Flash in production right now, and it's fully open,…

BREAKING NEWS🚨: @Inferact & Google submit their FIRST preliminary TPUv7 Kimi K3 2.8T Model Submission, and it is already delivering strong perf per TCO against GB200 NVL72, B300, and MI355 in the ultra-high-interactivity regime due to…

TOKENMAXXING: @vllm_project and @inferact achieve 1.3 BILLION tokens per $ of TCO (hyperscaler volume pricing) via disaggregated serving on AgentX. This result is 4x better than the B300 aggregated configuration. https://t.co/jVevqcSNiG

BREAKING NEWS🚨: Google has submitted its FIRST TPUv7 FP8 vLLM results to our Agentic Inference Benchmark and has demonstrated 1.67x better perf per dollar than even GB300 NVL72 FP4. 🚀 It is amazing to see Google go full steam ahead with…

ALERT🚨🚨: @vllm_project, @Inferact & NVIDIA TOOK CO-DESIGN TO THE EXTREME WITH A GB300 NVL72 INFERENCE OPTIMIZATION TO GET 🚀 7.2x 🚀 BETTER PERF PER TCO THAN LITERALLY EVERY OTHER AI CHIP, using an inference co-design optimization called…

ALERT🚨: AMD JUST LANDED SUPPORT FOR MEGAKERNEL DECODE IN UPSTREAM VLLM WELL BEFORE NVIDIA. This marks one of the first times that a major vLLM feature is only supported on ROCm and not on CUDA yet. MegaKernels is a technique that is…

HOLY ALERT🚨: NVIDIA vLLM INFERENCE RUBIN IS 3.2x BETTER IN PROFIT💰️ PER GIGAWATT & HAS UP TO 🚀 10x BETTER PERF PER DOLLAR THAN EVEN GB300 NVL72. This is on the widely used production LLM engine called @vllm_project. We explain…

The pattern I keep seeing this month: once a model layer commoditizes, the fight moves one layer down to whoever serves it fastest. RunAnywhere, a YC W26 company, launched Wally this week, an inference stack for open frontier models that…

Great blog from Kevin Lu at @vllm_project about quality control at the inference engine. Unfortunately, @AnushElangovan has not provided enough stable AMD QA clusters to vLLM, which leads to an order of magnitude worse software quality on…

POWER OF CUDA MOAT ALERT🚨: 2 days after CUDA vLLM supported DeepSeekv4.1 Flash, AMD finally publicly released its DeepSeek v4.1 Flash image. Functionally, it works out of the box, but performance-wise, it is currently up to 14.8x worse…

On the Day 0 release of DeepSeekv4.1 Flash, NVIDIA vLLM works out of the box with zero issues across all 6 SKUs: H100, H200, B200, B300, GB200, GB300! Amazing work by the NVIDIA & Inferact teams! In comparison, AMD vLLM still does not work…

PATTON maps production LLM KV-cache serving onto commodity PIM DRAM: 1.95x average speedup and 4.83x better energy efficiency over baselines, with no changes to PIM processing units. Hit rate stays comparable to vLLM on GPU. Memory vendors…

ALERT🚨 GB300 NVL72 NVLINK BACKPLANE ENABLES 13x BETTER PERF PER DOLLAR THAN HOPPER ON AGENTIC INFERENCE Through extreme co-design, GB300 NVL72 uses an innovative copper backplane to expand the scale-up domain to 72 versus 8 on Hopper.…

Every time an agentic system routes a task from one model to another, the receiving model pays a prefill tax. It has to re-read the entire conversation from scratch, driving up both cost and latency. NVIDIA's new paper eliminates that.…

Spec Decoding (DSpark) finally works with Pipeline Parallelism in vLLM!! 🔥 A lot of the GPU poor/proletarian have been asking for this feature for a while now, but it took the recent Kimi K3's massive 2.8T parameters affecting the GPU…

On total tokens per $ TCO, a new AMD MI355x submission beats B300 at lower interactivity ranges on AgentX. Shoutout to vLLM, AMD, and LMCache engineers. https://t.co/1oYYeYYu9S

On total tokens per $ TCO, a new AMD MI355x submission beats B300 at lower interactivity ranges on AgentX. Both high throughput configs use disaggregated setups. Shoutout to vLLM, AMD, and LMCache engineers. https://t.co/Sc35IXkXY8