By Jack White | May 13, 2026

The Hook

A 48% increase in response failures.

Let that number settle. In a field where benchmark improvements of 2-3% are celebrated as breakthroughs, where a single point on MMMU can shift market perception of an entire model family, Tencent's research team has published findings that suggest something fundamentally broken in how we evaluate multimodal AI systems.

The paper, reported by Crypto Briefing, reveals that non-thinking mode — the fast-response configuration that powers most consumer-facing AI products — increases response failures by up to 48% compared to thinking mode. This is not a marginal degradation. This is not a rounding error in quality metrics. This is a systemic failure mode hiding in plain sight.

And here's what bothers me most: the industry's entire evaluation infrastructure — the benchmarks, the leaderboards, the procurement checklists — cannot see it.

The Context: An Evaluation System Built on a False Premise

For the past three years, the AI industry has operated on a remarkably simple assumption: model quality is a fixed property. You measure it once, you get a number, and that number tells you how good the model is. MMMU says 72%. MMBench says 81%. OpenCompass aggregates these into a ranking. Procurement teams copy the numbers into spreadsheets. Done.

The problem is that this assumption collapsed the moment inference configurations became variable. When a model can operate in thinking mode — generating chain-of-thought reasoning, verifying intermediate steps, reviewing context repeatedly — or non-thinking mode — direct output with minimal internal deliberation — the notion of a single "correct" evaluation becomes meaningless.

Based on my experience auditing DeFi protocols, I recognize this pattern. It's the same mistake we made in crypto when we treated TVL as a stable measure of protocol health, ignoring that the same protocol could behave entirely differently under varying market conditions. Evaluation metrics that ignore configuration sensitivity are not just incomplete — they're actively misleading.

The industry's shift toward faster, cheaper inference has been relentless. Every major AI lab has introduced "turbo" or "flash" variants. Every API provider offers tiered pricing based on reasoning depth. The economics are obvious: non-thinking mode costs less, responds faster, and feels snappier to users. What Tencent's research suggests is that this cost optimization comes with a hidden tax — one that's being paid by end users who don't know they're receiving degraded service.

The Core: Anatomy of a Silent Failure

The 48% Discrepancy Is Not What You Think

Let me be precise about what this finding does and doesn't mean. The 48% figure likely doesn't indicate that half of all non-thinking responses are factually wrong. That interpretation would be dramatic but probably inaccurate. What it more likely reflects is a broader definition of "failure" — encompassing incomplete answers, semantic incoherence, deviation from user intent, and logical inconsistency, alongside factual errors.

This distinction matters because it shifts the nature of the problem from "the model is wrong" to "the model is unreliable." Wrong answers are detectable. Users find them. Developers patch them. But unreliable output — responses that sound plausible while being subtly incorrect, incoherent, or contextually inconsistent — is insidious. It erodes trust gradually, without triggering alarm.

The Mechanism: Why Non-Thinking Mode Fails

Multimodal reasoning is not single-step processing. When a model processes a complex visual-language task — visual question answering, spatial reasoning, chart interpretation — it needs to execute a sequence of operations: feature alignment, semantic mapping, cross-modal inference, and verification. Each step builds on the previous one. Errors compound.

Non-thinking mode skips or shortens these intermediate steps. The model produces output without generating the reasoning chain that would catch inconsistencies. In technical terms, it's operating with reduced computational depth. For simple tasks — recognizing an object, classifying an image — this works fine. For tasks requiring precise spatial reasoning or multi-step logical deduction, the model is essentially performing guesswork dressed up as confident response.

This explains why the effect is particularly pronounced in multimodal tasks: the failure rate isn't uniform across task types. The more steps required in the reasoning chain, the more likely non-thinking mode is to break. This is a vulnerability profile that current benchmarks systematically fail to capture.

The Evaluation Paradigm Shift

Tencent's paper doesn't just document this problem — it proposes a solution. The research advocates moving from correctness-centric evaluation to a dual-dimensional framework of consistency and quality.

This is more significant than it might appear. Correctness-based evaluation assumes there is a single right answer and measures whether the model finds it. Coherence-based evaluation asks whether the output is internally consistent, contextually appropriate, and logically structured. These are fundamentally different questions. A model can be correct 95% of the time on individual questions while producing deeply incoherent multi-turn dialogues. Current benchmarks would rank it excellent. Users would experience it as frustrating.

The shift to consistency and quality evaluation represents a maturation of the field — a recognition that AI output quality is not binary (right or wrong) but exists on a spectrum of reliability. This aligns with work coming out of Anthropic's HHH framework and OpenAI's coherence research, but Tencent's contribution is notable for attempting to systematize and quantify these dimensions.

The Contrarian Angle: What Tencent Isn't Telling You

The 48% Number Is a Weapon

Here's where I get suspicious. Tencent chose to publish this research through Crypto Briefing — not arXiv, not a peer-reviewed AI journal, not even a mainstream tech publication. The choice of outlet matters. Crypto Briefing's readership cares about technology trends but isn't equipped to scrutinize experimental methodology. This is a PR play masquerading as research dissemination.

Why does this matter? Because Tencent isn't just sharing knowledge — it's positioning itself. The company is one of China's leading AI developers through its Hunyuan model family. By publishing research that challenges existing evaluation frameworks, Tencent is simultaneously:

1. Setting the terms of the debate: "Consistency and quality" are dimensions where Tencent can claim expertise. By proposing the framework, they become the reference point for subsequent discussions.

1. Potentially deflecting from their own weaknesses: If Tencent's models have known consistency issues compared to GPT-4V or Claude 3.5, publishing research that emphasizes the importance of consistency creates a narrative where evaluation is contested territory rather than settled fact.

1. Building evaluation infrastructure as competitive advantage: The team that defines measurement standards gains outsized influence over how all models are judged. Google did this with Gemini benchmarks. OpenAI reshaped competition by emphasizing human preference alignment. Tencent's research is a low-cost attempt to do the same.

The 48% Figure Lacks Critical Context

The original article provides frustratingly little experimental detail. Which model was tested? What tasks were included? What was the baseline for comparison? Was this measured on public benchmarks or internal test sets?

These details matter enormously. If the 48% figure comes from a narrowly constructed internal test set optimized to demonstrate this effect, it overstates the general problem. If it comes from comprehensive evaluation across diverse tasks, it's an industry-wide alarm. Without this context, the number is essentially uninterpretable — and its prominence in headlines suggests either deliberate ambiguity or sloppy reporting.

The Cost-Performance Tradeoff Nobody Wants to Discuss

Tencent's research exposes a deeper tension in the AI industry: the economics of inference push toward non-thinking mode, but quality demands thinking mode. This isn't a technical problem — it's a business model problem.

Every major AI company has invested heavily in inference optimization to reduce costs. Faster models mean lower prices mean more users mean more market share. The "turbo" variants are central to this strategy. If the research community converges on the conclusion that non-thinking mode produces systematically unreliable output, these companies face a difficult choice: raise prices to cover thinking-mode costs, accept degraded quality at current prices, or find ways to make non-thinking mode more reliable (which requires more research and potentially more compute).

Based on my experience modeling risk-adjusted returns in DeFi, this is a classic adverse selection problem. The market is pricing AI services based on thinking-mode quality while delivering non-thinking-mode reliability. The gap between what's promised and what's delivered is a latent liability that will surface when enterprise customers start measuring actual outcomes rather than benchmark scores.

The industry is running a quality arbitrage — and end users are the exit liquidity.

The Systemic Implications: From Model Competition to Infrastructure Race

Evaluation as Infrastructure

If Tencent's framework gains traction, we'll see a fundamental shift in how AI models are evaluated and procured. This has ripple effects across the entire stack:

Third-party evaluation services: The rise of consistency and quality as standard evaluation dimensions will create demand for independent auditors — entities that can verify not just benchmark scores but real-world reliability. This is the AI equivalent of smart contract audits in crypto. The current landscape of leaderboard rankings will be supplemented by reliability certifications.

Procurement transformation: Enterprise buyers will shift from selecting models based on aggregate scores to requiring scenario-specific validation. "Show me how model X performs on my specific use cases, in my specific configuration" replaces "What's the MMMU score?" This is analogous to how DeFi investors shifted from looking at APY alone to examining underlying liquidity mechanics.

Competitive recalibration: Models that excel on multiple-choice benchmarks but struggle with coherent multi-turn interaction will face reputational corrections. The current ranking system has created a distorted view of model capabilities. New evaluation dimensions will expose capabilities — and deficiencies — that current metrics hide.

The Agent Economy Amplifies the Risk

The failure rate problem becomes dramatically more severe when applied to autonomous agents. An LLM-based agent executing a multi-step task — say, navigating a web interface to complete a purchase or analyzing a series of images to generate a report — makes multiple inference calls. If each call has a 5% failure rate in non-thinking mode, a ten-step task has a 40% cumulative failure probability. The reliability that seems adequate for single responses becomes catastrophic for multi-step operations.

This is the same systemic fragility pattern I documented in my 2021 liquidity analysis, where NFT wash trading artificially inflated perceived demand while draining actual liquidity. The market was measuring the wrong thing — trading volume instead of genuine value transfer. Similarly, the current evaluation system measures single-response correctness instead of multi-step reliability. The mismatch between measured and actual performance creates vulnerability that only surfaces when something fails.

The Regulatory Blind Spot

Regulatory frameworks — the EU AI Act, China's Generative AI Management Measures — focus on training data compliance, content safety filters, and algorithmic registration. None address inference-configuration-dependent quality variation.

This is a significant gap. If a model provider offers a service in non-thinking mode that performs 48% worse than its thinking-mode equivalent, and the provider doesn't disclose this difference, that's arguably a form of deceptive practice. Users believe they're getting the model they evaluated. They're getting a degraded version.

The transparency requirement should be obvious: model providers should be required to disclose performance characteristics across configuration modes. This is the AI equivalent of financial disclosure requirements — the same logic that requires protocols to disclose their liquidity mechanics and risk parameters.

The Takeaway: Rethinking What We Measure

Tencent's research, despite its opaque presentation, points toward an unavoidable conclusion: the AI industry's measurement infrastructure has fallen behind its deployment complexity. We've been treating model quality as a fixed property when it's actually a variable that depends on configuration, task type, and context.

For developers and enterprises building on AI, the practical implications are immediate:

Treat evaluation as configuration-dependent. Don't rely on aggregate benchmark scores. Test models in the specific configuration you'll deploy — non-thinking mode, thinking mode, or something in between. Measure performance on your use cases, not generic benchmarks.

Demand transparency from providers. Ask about performance characteristics across modes. If a vendor can't or won't disclose this information, that's a red flag.

Design for reliability degradation. If your application uses non-thinking mode, build in verification mechanisms — confidence checks, consistency validations, fallback to thinking mode for high-stakes operations.

The question for the industry is whether we're willing to confront the uncomfortable truth that Tencent has surfaced: our evaluation systems are telling us a story about AI capability that isn't true in production. The gap between benchmark performance and actual reliability is growing — and it's being systematically hidden by an evaluation infrastructure designed for a simpler era.

I don't have a clear answer for what comes next. But I'm certain that the models being deployed today are less reliable than their benchmark scores suggest — and that gap will surface when we least expect it.

The 48% number is just the first visible symptom of a deeper problem. The question is whether we'll treat it as a reason to rebuild our evaluation infrastructure — or as an inconvenient finding to be buried under the next benchmark release.

The chain always tells the truth. So does a well-constructed evaluation. It's the interfaces — and the benchmarks — that lie.

Jack White is a digital asset fund manager focused on the intersection of AI, crypto, and macro trends. He has spent the past decade building quantitative frameworks to evaluate protocol reliability and systemic risk. This article is based on publicly available research and should not be construed as investment advice.

Key Takeaways for the Discerning Reader

The hidden cost of speed: Non-thinking mode is the default configuration for most consumer AI products because it's cheaper and faster. Tencent's research suggests this optimization comes with a 48% reliability penalty — a tradeoff that's invisible to current evaluation benchmarks.

Evaluation infrastructure is obsolete: The industry's benchmark system — MMMU, MMBench, OpenCompass — measures single-response correctness. It cannot capture configuration-sensitive reliability, multi-turn coherence, or cross-modal consistency. Tencent's proposed shift to "consistency and quality" evaluation is the first systematic attempt to address this gap.

The agency amplification problem: For multi-step autonomous agents, per-step failure rates compound. A reliability issue that seems minor in single-response evaluation becomes catastrophic in agent deployments. This is the deepest vulnerability — and the one the industry is least prepared to address.

Watch the incentives: Tencent's choice to publish through Crypto Briefing rather than academic venues suggests strategic positioning rather than pure knowledge sharing. The evaluation framework Tencent proposes will shape how competitors' models are judged — a classic standard-setting play disguised as research.

The regulatory gap: No current regulatory framework requires disclosure of performance variation across inference modes. This is a transparency obligation waiting to be established — comparable to financial disclosure requirements in securities markets.