Gemma 4 31B vs Qwen3.5 27B: Inference Speed, Token-Efficiency, Accuracy, and Memory Consumption
A data-driven guide to choosing which of these open models to run locally.
Qwen3.5 27B has established itself as one of the strongest LLMs under 100B parameters, delivering top-tier results across many tasks and even outperforming some much larger MoE models. As we showed in a previous article, it is also highly robust to quantization, especially when the attention layers are preserved. As a result, it quickly became the default choice for local AI.
Then Google released Gemma 4 31B, a slightly larger model that fits on the same GPUs as Qwen3.5 27B and appears to offer similar performance. Google notably avoided publishing direct comparisons with Alibaba’s models, but third-party benchmarks and community feedback quickly made it clear that Gemma 4 31B was a serious contender to Qwen3.5 27B.
So how good is it, exactly, and at what cost?
I ran multiple benchmarks several times and report results on accuracy, token efficiency, inference speed, latency, and memory consumption. I evaluated both models with thinking enabled and disabled.
Note: This article focuses exclusively on the BF16 checkpoints. I also ran evaluations on several quantized variants, including INT4, NVFP4, and FP8, but I am still analyzing those results. I will publish a follow-up article next week.
Gemma 4 31B vs Qwen3.5 27B: Accuracy with Thinking Enabled
Gemma 4 31B achieves higher accuracy on most of the benchmarks I ran.
The only exceptions are MMLU Pro and GPQA Diamond, two multiple-choice benchmarks on which Qwen models have traditionally been very strong. Even there, however, the gap is small.
The chart above does not show this particularly well, but it became very clear during testing and after reviewing the results across multiple runs: Gemma 4 31B is also remarkably consistent in its answers. This is especially notable because Google recommends using a relatively high temperature of 1.0 and a top-k of 64, settings that would normally make a model more variable across runs by allowing more relaxed sampling.
This consistency is also helped by the model’s relatively short reasoning traces for a 2026 model. As we will see in the next section, it rarely generates more than 20k tokens, especially compared with Qwen3.5, which often overthinks and can exceed 100k tokens.
Because the model is more consistent, the difference between pass@1 and pass@k remains limited. On LiveCodeBench, for example, Qwen3.5 significantly narrows the accuracy gap between itself and Gemma 4 31B when moving from pass@1 to pass@4.
And yes, LLM providers almost certainly monitor benchmark accuracy during training and model development. That introduces at least some bias. For example, published checkpoints may be selected partly for their performance on specific benchmarks without generalizing as well elsewhere. In more extreme cases, contamination can occur when benchmark data, or even just benchmark-specific formats, are used directly in training.
To assess how “benchmaxxed” a model is, I like to use CoDeC. This metric was originally designed to detect whether a model has already seen benchmark data, with a CoDeC score above 80 considered a red flag. I have never observed a score that high myself. Still, I think the metric, and the way it was designed, also makes it a useful tool for measuring how much affinity a model has for a given benchmark style/data. A higher CoDeC score is a good proxy for how comfortable the model is with that benchmark, in other words, how much it may be relying on memorization rather than generalization.
I explain how CoDeC works in more detail here:
TL;DR
CoDeC checks benchmark contamination by measuring whether a model becomes more or less confident on a sample after seeing a few in-context examples from the same dataset. If the dataset is truly unseen, those examples usually help by giving useful distribution cues; if the dataset was likely seen during training, they can interfere with memorized patterns and lower confidence instead. It then reports a simple dataset-level score: the percentage of samples whose log-likelihood drops when same-dataset context is added.
Put simply, the goal is to achieve high benchmark accuracy while keeping the CoDeC score as low as possible.
Viewed through that lens, Gemma 4 31B wins by a wide margin:
Gemma 4 31B appears to generalize much better, which means its benchmark scores are more likely to reflect its actual capabilities.
