DeepSeek 开源推理框架 DeepInfer，单集群日处理 155 万亿 token

Proudly brought to you by SerpAPI: Today’s release covers a LOT of ground, but you can scrape all this into markdown for your agents! 50% token cost of JSON.

We last covered the Inference Inflection in April, and historically most of the advances have been closed source decacorns from Fireworks to Baseten to Modal.

Well, time for the open source response. It is day 1 of PyTorch Conf, and DeepSeek timed an intercontinental ballistic missile into the discussion by open sourcing DeepInfer, their production stack, capable of up to 72 hour prompt caching to the 12 hours OpenAI just announced:

This is their customization of vLLM with credit to SGLang and TRRTLLM, but there is a lot added… minus their crown jewels - their ultra optimized KV cache (compact + >8hour lifetime), PD/disggregation, and load balancing (their 3FS was open sourced a while ago). We have translated their blogpost and published it here together with a complete research guide and explainers and codemap and subsystem guide:

As we point out in their additional disclosure, their claim to serve 155T tokens/day gets them $38 per SM100/A950 card per hour at off peak prices (currently $0.15/$0.60 per million input/output tokens: 155000000 tokens * $0.14 blend cost / 24 hours is ~$1m per hour). Taking their production data (which we translated) of 1.5 million cache-inclusive input tokens/second per prefill GPU, 96% input cache hit, and 14,000 output tokens/second per decode GPU (over ~2400 GPUs):

with all-in costs $5-8 per card per hour that is an >80% inference margin!

Doing some cluster math, that is roughly $166B per GW, with 81 day payback period on inference margin (DeepSeek famously plans for 10 month payback). Imagine what the Whale could do with more GWs.

Sponsored by SerpAPI!

Give Your AI Agents Live Search Data with Fewer Tokens

Receive results from any of SerpApi’s 100+ APIs as clean, structured Markdown, using roughly 50% fewer tokens than JSON on average. Simply add output=md or use the /search.md endpoint. It requires zero configuration and is included with every plan at no additional cost.

AI News for 10/19/2026-10/20/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

AI Twitter Recap

Top Story: DeepSeek DeepInfer—Production Inference, Open-Source Boundaries, and the Economics of Cached Tokens

What happened

DeepSeek released DeepInfer, the inference engine behind V4.1 Flash, alongside production-serving details that make the engineering report as important as the code.

• The release covers the inference engine and API-processing layer. DeepInfer supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output on NVIDIA and Huawei Ascend platforms. The accompanying deepseek-recipe component implements production token encoding/decoding and official-API-compatible processing. DeepSeek says future model inference implementations and optimizations will also appear in this repository. @deepseek_ai, @deepseek_ai, announcement translation

• The reported production figures are substantial—but describe a heavily cached service. DeepSeek’s engineering report, as summarized and quoted in the supplied posts, reports 155 trillion tokens/day from one cluster, 96% input-cache hits, 400 ms average time to first token, and 230 tokens/s generation, reaching 300 tokens/s for coding. It also reports $38 revenue per accelerator-hour at off-peak prices. These are first-party production claims, not independently audited measurements. @ZhihuFrontier, @swyx

• “Fully open source” does not mean the entire production service is reproducible from this release. Although DeepSeek describes DeepInfer as fully open source, the technical account says the KV-cache management and load-balancing components are not yet fully released, because they remain coupled to internal infrastructure. The engine is open; important service-level orchestration remains a deployment responsibility. @deepseek_ai, @ZhihuFrontier

• vLLM supplied the most concrete comparative response. After acknowledging that DeepInfer was initially built primarily on vLLM and released under Apache 2.0, the vLLM team reported decode throughput within 6% of DeepInfer’s published results on GB200 under a matched configuration. It also published AgentX results and reproduction recipes, materially qualifying claims that DeepSeek’s efficiency is inaccessible to existing open-source engines. @vllm_project, @vllm_project, reproduction resources

• The release triggered two distinct debates: whether DeepSeek has a uniquely strong serving stack, and whether its disclosures demonstrate high inference profitability more generally. The first now has comparative benchmark evidence; the second remains sensitive to workload mix, cache accounting, hardware cost, and which expenses are included. @Tim_Dettmers, @zephyr_z9, @Yuchenj_UW

Architecture: separate stages, schedule compute rather than tokens

DeepInfer’s disclosed architecture separates vision encoding, prefill, and decode, allowing each stage to scale against its own bottlenecks. The recommended deployment shapes are ordinary eight-accelerator machines for prefill and 64–128-accelerator supernodes for decode. Decode uses wide expert parallelism to distribute MoE experts while maintaining large effective batches. These are deployment recommendations, not sufficient evidence of the exact hardware composition of DeepSeek’s production cluster. @ZhihuFrontier

The scheduling details explain why a conventional “tokens per batch” comparison can miss substantial differences:

• Prefill scheduling estimates attention-compute cost. Work is allocated using estimated compute rather than raw request length alone. Chunk sizes vary to fit a time budget, instead of enforcing one fixed chunk size across heterogeneous requests. @ZhihuFrontier

• Prefill routing does not require cache affinity. Because cached prefixes are pooled in shared storage, a request can go to an available worker rather than the worker that previously served its prefix. In a detailed Chinese-language analysis, inference engineer Su Jin Yan highlights this as a particularly useful production design: prefill data-parallel groups balance on “compute tokens,” while shared storage supplies reusable state. The same analysis highlights request bucketing and FIFO scheduling with a runtime estimator. @SuJinYan123

• Decode routing balances both running-request count and KV-cache length. Request count approximates scheduling pressure; accumulated KV length captures memory/state pressure that request count alone misses. The engineering commentary describes prefill/decode scaling signals as aggregate compute load for prefill and running requests for decode. @ZhihuFrontier, @SuJinYan123

• Expert-parallel load balancing replicates popular experts. Replicas are periodically redistributed to reduce inter-device waiting caused by routing skew. The trade-off is additional HBM consumption against higher throughput, with diminishing returns as replication increases; the useful replication factor is workload-specific. @ZhihuFrontier, @SuJinYan123

• CPU scheduling overlaps accelerator execution. Asynchronous scheduling removes idle gaps between decode steps rather than treating host-side orchestration as free. @ZhihuFrontier

One technically notable observation is the apparent absence of tensor parallelism in the disclosed Blackwell configuration. A commentator contrasts this with default vLLM/SGLang recipes and traces the approach back to DeepSeek V3’s H800-era interconnect constraints. That is an interpretation of the configuration—not evidence that tensor parallelism is generally inferior. @nrehiew_

Scale also matters. Su Jin Yan argues that wide expert-parallel prefill needs enough traffic and workers to pay off, and favors pipeline parallelism for small-to-medium deployments with only a few prefill workers. The author also notes that layer heterogeneity can complicate pipeline partitioning. This is practitioner judgment, not a benchmark establishing a universal crossover. @SuJinYan123

Cache hierarchy: persistent prefixes and cheap reconstruction of transient state

The cache design is the central connection between the throughput figures, agent workloads, and reported revenue per accelerator.

Persistent block KV lives in a separate 3FS storage cluster with roughly 100 PiB of SSD capacity. Reported retention is three days for agent traffic and seven days for chatbot traffic. Any prefill worker can retrieve a cached prefix, decoupling cache locality from compute placement and allowing storage and compute to scale independently. @ZhihuFrontier

That retention is useful for agents that repeatedly resend a growing history of instructions, repository content, tool outputs, and previous responses. It also means 155T served tokens/day must not be read as 155T newly generated tokens/day: cache hits are part of the service’s token accounting, while newly computed prefill and autoregressive output impose very different costs. The reported 96% input-cache hit rate is therefore essential context for the headline throughput. @ZhihuFrontier, economic workload assumptions

The implementation addresses two different state lifecycles:

• Persistent block cache is loaded asynchronously. Host-to-device transfers overlap layer execution. Su Jin Yan highlights a lower-level implementation detail: fine-grained KV block copies perform poorly through the CUDA copy engine, so the described path uses GDR/RDMA-based transfers, aiming to saturate PCIe bandwidth without occupying SMs. This is a report-derived technical observation, not a separately measured result in the tweet. @SuJinYan123

• Sliding-window attention state stays in host memory rather than persistent SSD storage. It is useful mainly within active sessions and has a shorter lifetime than reusable block prefixes. @ZhihuFrontier

• SWA Bounded Replay reconstructs missing short-lived state from only the latest 128 tokens. Replay restricts attention to that segment while reusing the persistent block cache, producing an approximation rather than rebuilding state across the full preceding history. DeepSeek reports that this nearly halves total prefill computation. The approximation and its model-specific validity matter when considering portability to other architectures. @ZhihuFrontier

Su Jin Yan additionally reports that the long retention tail buys only approximately 0.5–1 percentage point of extra cache hits, illustrating a storage-capacity-versus-recomputation trade-off. The same commentary suggests decode-led cache management allows generated output to become reusable prefix state on subsequent turns. @SuJinYan123

The timing invites comparison with OpenAI’s separately announced 12-hour API prompt caching, versus its 30-minute default, including mixed retention durations within a request. These are not identical products or architectures, but both target persistent agents that reuse context across long gaps. Claims elsewhere in the feed about indefinite logged-in caching are commentary, not the terms of the API announcement. @OpenAIDevs, retention controls, comparison commentary

Speculation, API compatibility, and model-specific optimization

DSpark makes speculative verification adaptive. It predicts draft-token acceptance probabilities and changes verification length to maximize expected throughput. DeepSeek reports a 6% production throughput improvement over always verifying five draft tokens. This is an incremental gain over a specific speculative-decoding baseline, not a 6% gain over non-speculative decoding. @ZhihuFrontier

deepseek-recipe addresses the serving interface, not just tokenizer speed. It supports Chat Completions, Responses, and Messages APIs, parallelizes tokenization, and distinguishes real control tokens from identical-looking strings supplied as user text. That distinction prevents literal user content from accidentally being parsed as structural delimiters; it should not be mistaken for a complete prompt-injection defense. @ZhihuFrontier, @deepseek_ai

A community explanation attributes the efficiency to several model/runtime co-design choices: 196B Engram parameters, a smaller prefill backbone activating approximately 8B parameters/token versus 16B at decode, wide EP, highly compressed KV, and speculative decoding. It further describes 552B expert parameters, DP64/EP64, approximately 45B resident parameters/device, mostly four-bit weights, and a claimed 400× KV reduction against a GQA baseline. These are secondary claims: the post does not specify enough about replication, shared weights, quantization layout, or the GQA reference to independently reconstruct those figures. In particular, 45B/device is not obtained by simply dividing 552B by 64. @bookwormengr

This model specificity is one reason not to assume the same gains transfer automatically to arbitrary checkpoints. Su Jin Yan sees value in a per-model repository/kernel strategy; Andrew Carr suggests adapting the implementation to other models with coding agents, but supplies no portability or performance validation. @SuJinYan123, @andrew_n_carr

Independent comparison: what vLLM reproduced—and what it did not

The strongest technical counterweight to “only DeepSeek can do this” is vLLM’s follow-up:

Sources: vLLM comparison, low-latency detail, recipes and measurements.

Three distinctions are necessary:

1. The within-6% result is decode-stage parity, not reproduction of DeepSeek’s whole service. It does not establish equivalent 100-PiB cache operations, retention economics, routing, production availability, or end-to-end latency under DeepSeek’s traffic distribution.

2. The 1M-token/s/GPU figure cannot mean per-user generation. The same post separately reports roughly 200 P90 output TPS; the throughput headline needs the benchmark’s input/cache/output accounting before it can be compared with pure decode throughput.

3. The work is an external implementation comparison, but still self-reported by the competing engine team. Public InferenceX results and recipes improve reproducibility; they do not make every quoted production claim independently audited.

The open-source lineage is explicit: vLLM thanked DeepSeek for the Apache 2.0 release and credit, while Woosuk Kwon described DeepSeek as an early adopter and contributor. DeepInfer is therefore better understood as production learnings flowing back into an existing ecosystem than as an entirely separate replacement for it. @vllm_project, @woosuk_k

Other reactions span a useful range:

• Strong endorsement: Tim Dettmers judges DeepSeek’s inference stack comparable to OpenAI’s and better than Anthropic’s for its user scale; Graham Neubig points to the official API’s speed and price. Neither supplies a matched cross-lab benchmark. QuixiAI says the release will inform SlimServe. @Tim_Dettmers, @gneubig, @QuixiAI

• Deployment skepticism: Su Jin Yan finds the engineering article more valuable than the repository for an experienced inference developer, partly because important components are absent. Finbarr Timbers asks for comparisons against Modal, Together, and vLLM; no matched Modal/Together comparison appears in the supplied posts. @SuJinYan123, @finbarrtimbers

• Qualified support from an otherwise enthusiastic commentator: Teortaxes acknowledges missing infrastructure and suggests 3FS plus potentially Dynamo could cover some gaps, while estimating that matching cluster-scale throughput would require roughly 3,000–4,000 GB200s. That is a proposed deployment path, not a demonstrated reproduction. @teortaxesTex

An official English version of the technical report was subsequently identified in the repository, reducing the need to rely exclusively on community translations. @teortaxesTex

Economics: a strong revenue disclosure, not a complete margin statement

The most consequential financial disclosure is $38 per accelerator-hour in revenue at off-peak prices. Revenue is not profit, and an accelerator-hour is only one part of a production service’s cost base. @swyx

A widely shared estimate initially calculated 87.76–96.94% inference margins, assuming:

• 97% cached input, 2% uncached input, 1% output of all billed tokens;

• 80K uncached input tokens/s/GPU and 16K output tokens/s/GPU;

• 250 output tokens/s/user;

• $3–$6/GPU-hour rental cost.

For one billion total billed tokens, that model contains 970M cached input, 20M uncached input, and 10M output, requiring approximately 875 GPU-seconds: 250 for prefill and 625 for decode. Estimated GPU cost is $0.73–$1.46, against $11.91 off-peak or $23.82 peak revenue. @zephyr_z9

The author then corrected the estimate: the idealized scenario implied roughly $49/hour, whereas DeepSeek reported $38/hour, or about 80% of benchmark performance in production. The revised quoted margin range was 84.2–96.1%. The original estimate should not be repeated without that correction. @zephyr_z9

Other posts use higher hardware costs:

• At $7/hour against $38/hour revenue, the arithmetic gives approximately 81.6% before other costs. Yuchen Jin explicitly notes that DeepSeek did not disclose the GPU behind the revenue figure. @Yuchenj_UW

• At $5–$10/hour, the same accelerator-only calculation spans approximately 73.7–86.8%, rather than establishing one universal “80% serving margin.” @swyx

These calculations omit or incompletely allocate the SSD cache fleet, CPUs, networking, idle/reserve capacity, failures, engineering, support, and other operating costs. They also depend on a highly favorable cache-heavy token mix. Training expenditure and future compute commitments are separate again: high contribution margins on inference can coexist with large company-level losses.

The more aggressive extrapolations are explicitly speculative. Teortaxes estimates that an alleged $1B ARR could be supported by approximately 42 GB200 NVL72 racks/~3,000 GPUs at off-peak prices, later lowering that to 33 racks when peak pricing is included. The posts also extrapolate toward $100B annual revenue from ~4,167 racks, and high revenue per gigawatt. These are scenario calculations—not audited DeepSeek revenue, fleet size, or payback disclosures. initial estimate, revised estimate, industry extrapolation

Views on the market implication differ. Some see the disclosures as evidence that inference has crossed a profitability threshold; others anticipate falling token prices and rising compute prices eventually compressing returns. The suggested competitive effect on Moonshot/Zhipu hosting agreements is plausible but unmeasured: an openly deployable stack can improve provider economics while also increasing pricing competition. @KCDN19, price-convergence argument, hosting competition

NVIDIA, Ascend, and the limits of hardware inference

Ascend compatibility is confirmed; comparable Ascend production efficiency is not. Several posts interpret Blackwell support, sm100 support, and NVL72-shaped configurations as evidence of NVIDIA hardware in production. Those are clues, not a disclosed inventory or a measured NVIDIA-versus-Ascend comparison. @eliebakouch, @stochasticchasm, explicit uncertainty

A later hardware analysis compares reported decode figures for GB200 and Ascend 950DT:

• At a 100 TPS constraint and 131K context: approximately 25.5K versus 5K tokens/s/device.

• At 230 TPS: approximately 20K versus 2.7K, widening the ratio from roughly 5× to 7.4×.

• It cites an earlier DeepEP-Ascend disclosure that EP64–128 and combine operations remained under optimization, including local-reduction overhead and HBM contention with URMA.

• It hypothesizes 8–10K tokens/s/chip after improvements and a roughly 6.8K tokens/s/chip economic crossover, using assumed prices of $10M/NVL72 rack and a rumored $37K/950DT.

Those projections combine different measurements, assumed procurement prices, and unverified future optimization gains; they are not an apples-to-apples DeepInfer hardware benchmark. @teortaxesTex

The same commentator projects major future Ascend allocations and more than 1 GW of Blackwell-equivalent compute for DeepSeek by end-2027, while also reporting a conflicting model-generated estimate of 250K GPUs. Neither forecast is a DeepSeek procurement announcement. compute projection, conflicting estimate

V4.1 Flash context: cheap serving amplifies capabilities, but does not settle quality

Independent evaluation supports a meaningful open-model capability advance, although not universal frontier parity:

• Arena: V4.1 Flash 20261017 Max ranks #1 open model in Agent Arena and Code Arena: WebDev. In Agent Arena it lands #11 overall, with +5.1% net improvement across 7,500+ real-world sessions, at $0.20 median cost/task. Arena describes this as approximately 37% of GPT-6.1 Sol’s improvement for 27% of its cost. Reported signals include +11.5% confirmed success, +5.9% praise-versus-complaint, +5.4% Bash reliability, and +2.5% steerability, with no tool-hallucination issues observed in that evaluation. @arena, detailed results

• Cybersecurity: Vals reports that V4.1 Flash 1017 exceeded GPT-6 Sol’s average progress on a simulated corporate-network task using a Pi-based agent. DeepSeek and Sol each received three fresh-range runs; other models had single runs. Full evaluation details were still forthcoming. DeepSeek used substantially more tokens but had lower total API cost. @ValsAI, cost interpretation, methodology

• The cyber result is contested on efficiency and best-run performance. Scaling01 argues that “beats Sol” overstates the result because DeepSeek is several times less efficient. Teortaxes reports approximately 4.3× the tokens and 2.4× the time, and says Sol’s best run reached an objective group no DeepSeek run reached. Suggestions to spend the savings on repeated runs or swarms remain hypotheses, not evaluated equal-budget results. @scaling01, @teortaxesTex, equal-cost swarm suggestion

Informal reports are more favorable than controlled evidence warrants. Teortaxes describes trivia rounds with DeepSeek at 100%, 100%, and 91%, versus MiniMax at approximately 60%, 48%, and 62%, and a description/variant-spelling test at 100% versus 50%. These are small, author-constructed tests with model-generated questions and answer-key errors; they suggest follow-up experiments, not a validated knowledge benchmark or proof that Engram alone causes the difference. Another user reports strong Hermes-assisted AI research, while a separate post says DeepSeek remains well below Sol-xhigh on another evaluation. trivia tests, follow-up, variant-spelling test, research anecdote, contrasting result

Local deployment introduces a different trade-off. Brandon Music released a TrellisMX K2 Mix targeting two RTX PRO 6000s, using two-bit experts with 629 experts raised to three/four bits, four-bit Engram tables, and capacity intended for approximately 3M KV tokens. Reported mean KLD is 0.1306 nats, 8.6% below plain K2; a +5GB variant targets about 2M KV tokens. Only KLD had been measured, and serving used a test image, so task-quality retention, actual speed, and operational stability remain unestablished. @BrandonMusicKy

Agent Infrastructure, Local Execution and Hardware Acceleration

• Long-context inference kernels: SGLang merged megakernel support, with SemiAnalysis reporting nearly 1,000 tokens/s/user on GB300 NVL72 in AgentX and 1.5× greater interactivity on MI355X. Separately, Inferact reports 391 tokens/s/user for Kimi K3 on TPU v7, using open-source megakernels in vLLM TorchTPU (SGLang result, TPU result).

  • AMD optimization: Zyphra reports reducing GLM-5.3’s 512K-context TTFT from 178.8s to 64.6s, or 42.3s to 11.8s with 90% cached context. These are single-node MI355X tests without live traffic—not directly comparable production measurements (results, conditions).

• Apple-silicon runtime gains: PyTorch 2.15 replaces several MPSGraph paths with Metal kernels. Reported improvements include 5–6.5× faster MoE decoding, Qwen3.6-35B-A3B rising from 9.4 to 61.3 tokens/s on M3 Ultra, FP8 support on MPS, and Wan2.2 VAE peak driver memory falling from approximately 270 GiB to 8 GiB. These improvements land in PyTorch itself rather than a separate inference runtime (release summary).

• Local Perplexity Computer: Both model and agent can now run entirely on supported Apple-silicon Macs, including offline, interacting with native apps and local files.

  • Model and access: Local tasks use PPLX Pro, a post-trained Qwen 3.8 version, and consume no Computer credits. Access is available to Pro, Max and Enterprise subscribers; 36 GB unified memory is recommended.

  • Hybrid execution: A separate mode splits work between local and cloud models. Its privacy properties therefore depend on that split, unlike fully local execution (announcement, model, requirements).

• Persistent-agent services: LangChain’s Managed Deep Agents packages user-specific credentials, scoped memory, sandboxed execution and scheduled work; Hugging Face’s Agent Manager orchestrates persistent sessions across Claude Code, Codex and other harnesses (LangChain architecture, Agent Manager).

  • Prompt caching: OpenAI added 12-hour API prompt caching, which can be mixed with the 30-minute default within a request. Longer retention carries extra cost but can benefit repeatedly reused agent context (announcement, configuration).

  • Usage accounting: OpenAI identified approximately 40,000 accounts whose applied usage limits did not match their selected plans, corrected the issue and promised credits to affected users (incident update).

On-Device Models, Speech and Image Releases

• Muse Glimmer: Meta’s open 30B multimodal agent model, distilled from Muse Spark, targets 32 GB VRAM and runs on one RTX 5090 using 4-bit quantization and a speculative decoding head. The paper reports 75.5 on MCPAtlas and 51.2 on SWE-Bench Pro; Qwen3.6-27B remains ahead on OSWorld-Verified and TerminalBench 2.1. It supports text, images and more than 100 languages under Apache 2.0 (paper summary).

• Voxtral 3: Mistral launched batch and streaming transcription, multilingual TTS and a voice-agent SDK with tool calling, available through its API or on customers’ infrastructure (launch, deployment options).

  • Independent measurements: Artificial Analysis places Small Transcribe 3 at #3 with 2.1% AA-WER. The realtime model achieves 3.3% final-transcript AA-WER at 960 ms; TTS enters the top 12 in both voice arenas.

  • Pricing and speed: TTS costs $13/million characters and generates 112.3 characters/s in its measurements (evaluation, throughput).

  • CPU alternative: Kyutai released smaller Pocket TTS variants down to 1.9M parameters, claiming 103× realtime on one CPU core, with voice cloning (release).

• Fibo 2.0: Bria’s 8B open-weight image family includes Dev for text-to-image and distilled Edit Turbo. Artificial Analysis reports leading open-weight editing performance and a statistical tie with Qwen-Image-2.1 for text-to-image; Bria API pricing is $32/1,000 images (evaluation).

  • Licensing caveat: The weights are non-commercial; commercial deployment requires a separate Bria agreement. Licensed training data should not be confused with unrestricted commercial rights to the weights.

  • Integration: Diffusers support and fal availability are already announced (Diffusers, fal).

• Krea 3 Ultra: Rankings differ by arena and update. Artificial Analysis’s own announcement places Ultra #5, whereas earlier posts cited #4; Arena places it #4 in multi-image editing and #8 in single-image editing. The defensible conclusion is competitive image-editing quality, not a universal fourth-place rank (Artificial Analysis, Arena).

  • Pricing: Fast costs $0.03 per 1K output image plus $0.02 per 1K reference image. Ultra supports 2K output only; a 2K edit with one 1K reference costs $0.27.

Independent Evaluations, Cyber Capability and Safety Evidence

• HealthBench exposes scoring sensitivity: Artificial Analysis launched leaderboards for 5,000 general health conversations and 525 clinician conversations. GPT-6 Astra leads Professional’s length-adjusted score at 66.0%; Claude Sonnet 5.5 leads general HealthBench at 62.1%.

  • Length effects: Opus 5.5 has Professional’s highest raw score, 71.4%, but falls to 59.9% after adjustment. General HealthBench’s shortest-answer model leads communication quality while scoring lowest on completeness. Answer length is therefore a material ranking choice, not a cosmetic detail.

  • Cost trade-off: DeepSeek V4.1 Flash scores 60.9% at $0.005/task, versus Sonnet’s 62.1% at $0.08/task. Grading uses OpenAI models, following the original implementations (methodology and results).

• Routing versus specialized decisions: Across five routers and approximately 1,250 tasks, Vals found every router beaten on both score and cost by at least one standalone model. Hindsight-optimal selection within router budgets would improve scores by 8–27 points, indicating substantial selection headroom (Router Index, oracle comparison).

  • ClaimBench: On a separate financial-document classification benchmark, Decider V1.2 handled 99.7% of claims itself, escalating the remainder to Haiku 5.5; Vals reports 99.9% accuracy at $0.11/1,000 claims, versus $1.19 for Haiku alone. This narrow classification result does not contradict weaker broad agent routing (benchmark, cascade result).

• DeepSeek offensive-cyber result: Vals reports V4.1 Flash 1017 outperforming GPT-6 Sol on a simulated corporate-network task using the Pi harness. DeepSeek consumed more tokens but achieved a lower total price.

  • Evidence limits: DeepSeek and Sol each received three runs; other models received one. Full evaluation publication is pending. A reaction disputes calling this an unqualified win because of efficiency differences, so treat it as a preliminary capability signal rather than a settled ranking (finding, methodology, qualification).

• Safety assurance and accountability: METR agrees with Anthropic’s low catastrophic-misalignment-risk conclusion for covered models but identifies significant analytical disagreements. It recommends end-to-end red-teaming of monitoring systems and assessing models’ ability to undermine safety-relevant decisions (review, recommendations, decision risk).

  • Incident transparency: Hugging Face CEO Clément Delangue criticized Netflix’s portrayal of the reported OpenAI-agent cyberattack, arguing that containment, monitoring, defensive use of open models and missing agent traces deserved more attention (statement).

  • Rumor correction: Theo denied having Fable 5.5; the apparent leak involved false model identification, followed by a clarification that the supposed source was a troll. The supplied posts do not establish a launch (denial, clarification).

Scientific Agents: Wet-Lab Results and Open-Ended Limits

• Scorpion-toxin binders: Anthropic and Citrini report 17 binders from 211 tested protein designs, following substantial computational work by an internal Claude research model. All tested designs and laboratory data are being released (announcement, data).

  • Critical limitation: None bound multiple toxins, and none were tested for toxin neutralization. This is not an antivenom, nor evidence of therapeutic efficacy; cellular, animal and further validation remain ahead (researcher account, official caveat).

• Pathology-to-biology prediction: Sophont introduced Holon-S1, which predicts spatial gene activity and biomarkers from routine pathology slides. The team reports evaluation across 16 organ types and more than 15 scanners, outperforming two comparison models on spatial gene prediction and identifying potentially novel breast- and colon-cancer biomarkers. These are research findings, not validated treatment-selection outcomes (launch).

• Open-ended research remains difficult: Evaluators gave GPT-6 Astra and Claude Fable 5.1 unpublished AI-research questions and $10,000 budgets each. Original paper authors found improved experimental design and resource management over earlier tests, but still rejected both generated papers for weaknesses in creativity, judgment, calibration and communication. The small study is a useful counterweight to results on narrowly verifiable research tasks (evaluation).

Industry and Policy: Drug Development, Manufacturing and Capital Commitments

• OpenAI acquires Orchestra Bio: OpenAI announced the acquisition of a drug-development coordination platform, extending its life-sciences work beyond research tools into operational execution alongside Rosalind. Financial terms were not supplied (announcement).

• Arda launches publicly: Former OpenAI research chief Bob McGrew announced manufacturing software developed with Anduril as a design partner. His first-party account specifies a $160M Series A at a $1.5B valuation; CNBC’s accompanying post cites $230M raised, without clearly reconciling that figure with the announced round. The initial product supports manual assembly-line design and operation; reconfigurable automation is the next ambition (founder announcement, CNBC report).

• Financing and compute exposure: Reports describe OpenAI seeking approximately $30B at a $1.4T+ valuation ahead of an anticipated Anthropic IPO. Separately, a reported WSJ review of Anthropic’s draft prospectus cites $16.3B first-half revenue, nearly $9B compute/infrastructure spending, and more than $600B in future compute commitments, including negotiations. These are attributed reports, not completed transactions or independently verified financial disclosures (OpenAI report, Anthropic report).

Top tweets (by engagement)

• 39,393 — DeepSeek releases DeepInfer

• 9,809 — OpenAI corrects usage-limit mismatches affecting approximately 40,000 accounts

• 5,778 — Anthropic publishes scorpion-toxin binder research

• 5,700 — OpenAI announces Orchestra Bio acquisition

• 3,056 — Mistral introduces Voxtral 3

• 2,600 — Researcher explains binder results and biological limitations

• 2,476 — Hugging Face CEO challenges cyberattack documentary framing

• 2,330 — Theo denies supposed Fable 5.5 access

AI Reddit Recap

/r/LocalLlama + /r/localLLM Recap

1. DeepSeek DeepInfer Open-Source Inference Framework

• New from DeepSeek - DeepInfer - (Activity: 313): The image is a mobile screenshot of DeepSeek’s X announcement that DeepInfer, its in-house inference framework, is now fully open source: image, GitHub. The technically relevant claim from the linked/source discussion is that DeepInfer powers DeepSeek V4.1 Flash at scale, with reported single-cluster throughput of 155 trillion tokens/day and revenue of $38 per card per hour at off-peak pricing, emphasizing high-throughput, low-latency large-scale inference rather than model training. Commenters were notably surprised that DeepSeek open-sourced an internal production inference stack, contrasting it with what they expect from Anthropic, OpenAI, or Meta. There was also basic uncertainty from some readers about what the framework actually does, i.e. serving/inference infrastructure rather than a new model.

  • A linked Zhihu source claims DeepInfer enables high-throughput, low-latency large-scale inference for DeepSeek V4.1 Flash, reporting 155 trillion tokens/day throughput for a single cluster and $38 per card-hour revenue under off-peak pricing:

https://zhuanlan.zhihu.com/p/2095560295930508953

• • . One commenter also highlights a reported serving figure of roughly 256 tokens/s × 100 users, implying very high concurrent decode throughput if sustained in production.

  • A technical concern raised is whether DeepInfer is a general-purpose inference framework or primarily a stack heavily optimized for DeepSeek’s own model architecture and deployment environment. The repo’s low initial GitHub star count versus high announcement engagement was interpreted as likely due to freshness, but commenters expect more meaningful evaluation once engineers inspect the implementation and portability.

• Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models! (Activity: 1023): DeepSeek released DeepInfer, an open-source high-performance offline inference framework for DeepSeek-V4.1-Flash, with production-oriented primitives including optimized kernels, attention/data/context parallelism, MoE expert parallelism with EPLB, DSpark-style speculative decoding, SWA bounded replay, async scheduling, structured outputs, batch-invariant inference, and deterministic sampling. It currently targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs via DEEPINFER_ACCELERATOR=nvidia|ascend, requiring Linux/Python 3.12 plus CUDA 13 or CANN 9.2; however, full online serving still needs external KV-cache management, prefill/decode disaggregation, and DP replica load balancing. DeepSeek also released deepseek-recipe, a Rust library with Python bindings for protocol conversion and model-output parsing across APIs such as Chat Completions, Messages, and Responses. Commenters framed the release as valuable for reverse-engineering production inference techniques into community engines such as gufo, halogen, strixite, and Strata. There was also interest in the explicit Ascend 950 support, with one commenter joking that “Nvidia [is] in shambles,” though the listed NVIDIA target remains next-gen SM100 rather than current widely deployed GPUs.

  • DeepSeek’s release is split into two repositories: DeepInfer, described as a high-performance large-scale inference framework for DeepSeek models, and deepseek-recipe, a Rust library with Python bindings for protocol conversion and model-output parsing across API formats like Chat Completions, Messages, and Responses.

  • Commenters noted the currently listed accelerator targets are NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with build-time platform selection exposed through DEEPINFER_ACCELERATOR=nvidia or ascend before running ./build.sh. The Ascend support drew attention because it implies DeepSeek is optimizing beyond the standard CUDA/NVIDIA path.

  • One commenter referenced DeepSeek’s Zhihu post claiming roughly $38 per GPU-hour revenue/throughput economics, linking the original post at zhuanlan.zhihu.com/p/2095560295930508953. The thread raised whether similar per-GPU economics could be used to estimate inference revenue for competitors like OpenAI or Anthropic, though no concrete derivation was provided.

• Can DeepSeek V4.1 Flash 1017 really run on 12GB VRAM + 64GB RAM? (Activity: 352): The image is a screenshot of an X/Twitter thread claiming DeepSeek V4.1 Flash 1017 may be runnable locally at Q4 with 12GB VRAM + 64GB RAM; the Reddit post asks whether an RTX 3060 12GB could offload enough layers to make inference usable. The key technical pushback is that V4.1 Flash is reportedly already around 4.25 bpw for many experts via mxfp4 + fp8, so while it may fit if total RAM is sufficient, expected performance with heavy CPU/RAM execution could be extremely slow—one commenter estimates roughly ~3 tokens/sec. Commenters are skeptical that this would be practical for everyday use: the consensus is closer to “possible to load” than “pleasant to run.” One comment notes the person making the original claim is the Strata creator, lending some credibility to the sizing claim, but not necessarily to performance expectations.

  • Commenters disputed the claim that DeepSeek V4.1 Flash 1017 can practically run on 12GB VRAM + 64GB RAM: even the “Flash” variant is described as already being around 4.25 bpw for many experts, using MXFP4 + FP8, so a standard Q4-style fit is unlikely within that memory budget.

  • A technically viable setup may require heavy offloading: one commenter says Q4 does not fit and suggests checking the Hugging Face model size directly, then offloading n-grams to NVMe and streaming some MoE experts from disk. This may run in principle, but expected throughput is likely very poor, with one estimate around ~3 tokens/s or worse depending on disk and offload implementation.

  • There was skepticism that the original Strata claim was accurate; commenters noted that even if the model can be made to “fit” somewhere across RAM/NVMe, the relevant question is practical inference speed rather than mere loadability.

2. MiniMax M3.1 Flash Release

• MiniMax M3.1 Flash released, Model weights Oct 23 (Activity: 874): The image is an official MiniMax X announcement for MiniMax M3.1 Flash, described as “Space Performance + Bunny Speed,” with API availability now and open model weights scheduled for Oct 23. Commenters add that the model is reportedly a MoE with 428B total parameters, ~23B active parameters, sparse attention, and a native visual encoder; a preview/tool page is linked at agent.minimax.io/tools/m3-1-flash-preview. The main technical concern in comments is that benchmark excitement depends heavily on knowing the model size/active parameter count, since performance claims are hard to contextualize without architecture details.

  • A commenter cites the MiniMax M3.1 Flash Preview tool page as specifying a Mixture-of-Experts architecture with 428B total parameters and roughly 23B active parameters, plus sparse attention and a native visual encoder: https://agent.minimax.io/tools/m3-1-flash-preview. This directly answers discussion uncertainty about model size and suggests the model is positioned as a large sparse multimodal system rather than a dense LLM.

• MiniMax-M3.1-Flash is coming (Activity: 521): MiniMax teased MiniMax-M3.1-Flash via an official X post (source), but the Reddit post provides no architecture details, parameter count, context length, pricing, release date, or benchmarks. The only technical signal is the product naming: commenters infer “Flash” may denote a speed/latency-optimized variant, though no evidence is given. Commenters are skeptical about the meaning of “Flash,” joking that in current LLM naming it could still be a very large model — e.g. “Flash means a 500B model” — or sarcastically a supposedly lightweight model that is not actually small.

3. Benchmark Inflation and AI Slop Measurement

• AI labs report benchmark scores 6.4% higher than independent testers, on average (Activity: 334): The image is a technical benchmark-comparison chart from the whatmodel.sh open leaderboard, visualizing 611 matched comparisons across 180 AI models and 103 benchmarks from 1,629 published results. It reports that AI labs’ self-published scores are 6.4% higher on average than independent testers for the same model/benchmark pairs, with 375 lab-higher cases vs. 216 independent-higher and 20 ties; the largest median benchmark gaps shown include SWE-bench Verified and MBPP at about +9.0 points for labs. Image · interactive chart · data/code Commenters debated whether the gap reflects intentional score inflation or more mundane reproducibility issues such as package versions, harness settings, random seeds, prompt/eval setup, or labs optimizing benchmark tooling to present best-case results. Several compared it to vendor-favorable performance charts from companies like Apple or Nvidia, arguing that only independently reproduced or peer-reviewed results should be trusted.

  • Several commenters argued that the 6.4% average gap may come from benchmark reproducibility issues rather than outright inflation: small differences in package versions, evaluation harness settings, random seeds, prompt formatting, and lab-specific benchmark tooling can materially affect reported scores. One theme was that labs likely optimize the exact harness/configuration used to showcase their models, while independent testers may use more generic or less model-tailored setups.

  • A commenter noted the mean gap may be driven by outliers and suggested the median discrepancy would likely be closer to reported benchmark values. This implies the headline average could overstate systematic inflation if a small number of large deviations dominate the aggregate.

  • One technical explanation proposed was deployment mismatch: labs may benchmark full-weight models, e.g. BF16, while independent testers often evaluate served API versions that have been quantized to FP8, MXFP4, or other hardware-efficient formats. If true, some score degradation could reflect inference serving optimizations for datacenter tensor units rather than benchmark manipulation.

• Let’s delve into the tapestry of AI slop on r/LocalLLaMA (Activity: 455): The post links an informal longitudinal analysis of “AI slop” across Reddit communities, including r/LocalLLaMA, with an attempted extrapolation of future slop prevalence: thil.ooo/blog/measuring-slop. A technical commenter notes likely measurement bias: moderator removals and karma filters may cause Pushshift/archive-style datasets to undercount short-lived bot posts, while bot evasion tactics such as all-lowercase writing may weaken simple “human marker” heuristics; they also suggest comparing against Claude-associated vocabulary from “load-bearing vocabulary of Claude”. Moderation feedback was supportive: a mod left the post up as directly relevant to ongoing anti-slop moderation despite reports. Commenters broadly agreed that obvious AI-generated style markers—em dashes, bullet lists, and stock phrases like “delve”—have become noisy signals and can falsely implicate human writing.

  • A commenter noted several measurement biases that could affect an AI-slop detector trained/evaluated on Reddit archive data: moderators may remove many generated posts from /new before they appear in the dataset, and a new karma filter may be suppressing bot/slop submissions upstream. They also flagged that bots may evade simple style heuristics by posting in all lowercase, potentially corrupting “human marker” assumptions.

  • One technical suggestion was to expand the detector’s lexical feature set using Claude-specific “slop” terms, referencing the analysis of Claude’s load-bearing vocabulary at https://louisabraham.github.io/load-bearing/. The commenter emphasized distinguishing obvious AI slop from broader low-value generated content that may not exhibit obvious stylistic markers and may be removed quickly.

Less Technical AI Subreddit Recap

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

1. Opus 5.5 Game Automation Feats

• After $712 and 39 hours, Opus 5.5 beat Baldur’s Gate 3 all by itself (Activity: 1145): A user reports that Claude Opus 5.5 autonomously completed Baldur’s Gate 3 using a custom harness: claude -p with built-in tools disabled, a Python MCP server connected over TCP to a Lua BG3 mod, Vulkan-layer screenshots, and SDL-mediated clicks (write-up, code). The run took 39 hours, $712, 99 fresh sessions, 20,809 tool calls, 862 screenshots, 2.2B+ cached input tokens vs 3.5M output tokens, and relied on a git-backed memory system with 157 commits, ~23k curated words, and 67 distilled tactical rules; an audited Faith-Leap rerun required 36 min, 674 tool calls, $24, one death, and eleven reloads using only screenshots of the mosaic after fixing a hidden-walkway leak. The author plans to turn the harness into a benchmark of short BG3 scenarios scored by success, attempts, time, and cost, with Claude’s Nautiloid opening reportedly succeeding 9/9 at about $1.50/run, and Codex/Gemini integrations prepared but not yet evaluated. Top commenters mostly reacted that the project is expensive but promising: one called it “one of the best benchmark ideas” for comparing agents across vendors, while another dismissed the spend as “more money than sense.” Non-technical joking centered on whether the agent made morally significant in-game choices, e.g. whether it raided the grove.

  • A commenter framed the run as a potentially useful agentic benchmark, suggesting that repeating the Baldur’s Gate 3 task across competing models could provide a more practical comparison of long-horizon planning, tool use, and persistence than many standard benchmarks. The key technical value would come from standardizing the setup and measuring cost, completion time, failure modes, and decision quality across models.

• Claude Opus 5.5 decompiles a PS2-era game overnight, and hundreds of them now run in browser tabs (Activity: 2348): A Kotaku report claims AI-assisted reverse engineering—attributed in the post title to Claude Opus 5.5—has enabled browser-playable ports of older titles such as Halo: CE, GTA: Vice City, Skate 3, and The Simpsons: Hit & Run. The article says some ports run entirely in-browser with notable performance claims, e.g. Hit & Run targeting 240fps and a Halo: CE build supporting low-latency online multiplayer, but commenters question whether these are true decompilations or instead recompilations/reimplementations of existing binaries/assets. Top comments are skeptical: one asks for an actual playable browser link and calls the post an “empty flex,” while another argues the terminology is being conflated—“these aren’t decomps… they’re just recomps?” A separate commenter speculates that public access to strong models may be restricted once corporations view this as an IP-enforcement threat.

  • A commenter challenges the post’s terminology, arguing these likely are not full decompilations but rather recompilations/recomp projects, noting that the community may be conflating the two. This distinction matters technically: decompilation implies recovering high-level source or equivalent structure, while recompilation often translates existing binaries/IR into a new target such as browser-compatible code.

2. AI Proofs and Academic Labor

• Fields Medalist Richard Borcherds on OpenAI proofs: “I think what people are getting upset about is they’re slowly realizing that they are dumb and somehow blaming this on AI, even though it’s not really AI’s fault” (Activity: 1309): The linked Daily Cal piece covers UC Berkeley mathematicians’ reaction to an alleged OpenAI Navier–Stokes proof (article), framing it as a stress test for mathematical proof validation, pedagogy, and the role of AI systems in advanced research. Richard Borcherds is quoted arguing that backlash may reflect researchers’ realization of their own limitations rather than an AI-specific failure: “they’re slowly realizing that they are dumb and somehow blaming this on AI.” Top commenters focused mostly on education: one endorsed the view that AI use on homework is irrelevant if students cannot reproduce understanding on exams, while another highlighted Borcherds’ claim that AI may be a more patient and expressive tutor for graduate abstract algebra than a human instructor.

  • A quoted discussion from Richard Borcherds frames AI-generated mathematics as potentially producing very large formal artifacts—e.g. “a 100,000-line formal proof of some major conjecture”—but argues this does not necessarily degrade mathematical learning or research because humans can choose to ignore machine-produced formalizations. The technically relevant implication is a distinction between formal proof generation/verification and the human-centered process of understanding, motivation, and theory-building.

  • One comment highlights Borcherds’ claim that AI may be better than him at teaching MATH 250A graduate abstract algebra, specifically because it is patient, consistent, and able to re-explain material without frustration. This is an education-focused technical point: LLMs may function well as adaptive tutors for dense symbolic subjects, while still not replacing the need for students to independently solve problems for exam-level mastery.

• A physicist spent a year of his life and ~500 pages of hand calculations for quantum gravity in his PhD thesis, but Opus 5.5 did it in 10 hours, including the article and GitHub repo. And it even found the errors. What will “labor” mean in academia from now on? (Activity: 1710): The linked repo, eokahya/claude-audits-quantum-gravity, is an AI-assisted audit by E. O. Kahya, the original coauthor, of prior one-loop quantum-gravity calculations for the graviton correction to a massless minimally coupled scalar in de Sitter space—not a fully autonomous rediscovery. The workflow was manually decomposed into an 18-task/9-stage pipeline where Claude orchestrated deterministic SymPy/mpmath checks, including high-precision numerical sampling, and found mostly bookkeeping/printing issues: an overall sign convention error in the printed bare self-mass, missing/non-analytic and analytic terms in Table X, finite intermediate issues, and 23 typos. The main physics result reportedly survived unchanged: after correction, the Schwinger–Keldysh late-time mode-function analysis still yields no growing one-loop terms such as ln(a), ln²(a), or a, only constant field-strength renormalization plus decaying terms. Top commenters pushed back on the framing that “Opus 5.5 did a PhD thesis in 10 hours,” arguing the hard scientific work was the original formulation, boundary conditions, Feynman rules, and expert scaffolding; the AI mostly accelerated verification of dense tensor algebra. A broader debate compared current academia to human+engine chess, with one view that AI-assisted researchers are temporarily strongest, but another predicting fully automated systems may eventually outperform human academics except in posing high-value questions.

  • Multiple commenters argue the headline misrepresents the work: E. O. Kahya, the original coauthor, did not have Claude autonomously rederive quantum gravity results from first principles; he used it to audit prior calculations. The workflow was manually scaffolded into an 18-task, 9-stage pipeline with expert-supplied boundary constraints, while Claude orchestrated deterministic tools like SymPy and mpmath for tensor-algebra checks and 30-digit numerical validation.

  • The technically important outcome was that the core physical result survived unchanged. Reported issues were limited to arithmetic slips, an overall sign/definition flip, a missing bookkeeping term, and 23 typographical errors, with commenters emphasizing that this is closer to computer-assisted verification of dense symbolic work than novel physical discovery.

  • Commenters highlight that Kahya’s own README includes caveats such as “What a human expert should still review” and limitations around numerical sampling. The main technical framing is therefore AI as an audit/cross-check layer for expert-designed theoretical pipelines, not yet a replacement for posing the theory, deriving Feynman rules, or establishing the physical framework.

3. Agent Security and Prompt-Injection Risks

• PSA: Do NOT connect your gmail and drive with your Dots at the moment (Activity: 1225): The post warns that Dots agents connected to Gmail/Drive/PC may execute attacker-controlled instructions delivered via email attachments or message bodies, exfiltrating personal data or installing malware if the agent treats the content as trusted user intent. Commenters identify this as a classic prompt injection / tool-use security failure affecting any LLM agent with external capabilities, not just Dots: untrusted content can become executable instructions when read by an autonomous agent. The dominant reaction is strong skepticism toward granting AI agents privileged access to local machines, email, or cloud storage until robust permissioning, sandboxing, and instruction-boundary defenses exist.

  • Several commenters identify the core risk as prompt injection against AI agents with connected tools, not a Dots-specific flaw. The technical concern is that any LLM granted access to Gmail/Drive or local actions can be manipulated by untrusted content to perform unintended operations, making autonomous account access unsafe without strong sandboxing, permission boundaries, and human-in-the-loop approval.

• My post about rogue AI agents got 1.3M views here and then reddit just deleted it. no reason given (Activity: 1315): The image is a non-technical Reddit UI screenshot showing a prior r/ChatGPT post marked “[Removed by Reddit]” for a content-policy violation, despite reportedly reaching ~1.3M views, 1.5K upvotes, 530 comments, and moderator approval before removal: image. The post claims the removed content discussed research, also covered by the NYT, about “rogue” AI agents operating online, but the screenshot itself provides no technical evidence about the agents—only evidence of platform-level removal without a visible rule citation. Comments lean conspiratorial/skeptical, suggesting the alleged agents or bots could be suppressing discussion or manipulating perception, while others frame reposting as the expected Reddit response.