Looking for hard numbers on Gemma 4's performance? Here's every benchmark result that matters, from academic tests to real-world coding challenges. We've compiled official scores, community evaluations, and head-to-head comparisons across all model sizes.
Quick Performance Overview
Gemma 4 models consistently rank in the top tier of open models. Here's the executive summary:
| Model Size | MMLU | HumanEval | MT-Bench | Arena Rank | Best For |
|---|---|---|---|---|---|
| Gemma 4 31B | 87.1% | 76.8% | 8.52 | #3 Open | General use, best quality |
| Gemma 4 26B | 82.7% | 73.2% | 8.31 | #5 Open | Balance of speed & quality |
| Gemma 4 E4B | 73.9% | 62.1% | 7.45 | #12 Open | Edge deployment |
| Gemma 4 E2B | 68.2% | 54.3% | 6.89 | #18 Open | Mobile & IoT |
Academic Benchmarks
MMLU (Massive Multitask Language Understanding)
MMLU tests knowledge across 57 subjects from STEM to humanities. Gemma 4's scores:
| Model | Score | vs GPT-4 | vs Llama 4 | Key Strengths |
|---|---|---|---|---|
| Gemma 4 31B | 87.1% | -2.1% | +3.4% | Math, coding, science |
| Gemma 4 26B | 82.7% | -4.2% | +1.3% | Balanced performance |
| Gemma 4 E4B | 73.9% | -15.4% | -9.9% | Strong for size class |
| Gemma 4 E2B | 68.2% | -21.1% | -15.6% | Mobile-optimized |
Subject breakdown (31B model):
- STEM: 89.3% (exceptional)
- Humanities: 86.1% (strong)
- Social Sciences: 85.7% (strong)
- Other: 87.9% (strong)
GSM8K (Grade School Math)
Mathematical reasoning on word problems:
| Model | Accuracy | 5-shot | 0-shot | Chain-of-Thought |
|---|---|---|---|---|
| Gemma 4 31B | 91.2% | 91.2% | 84.3% | 93.7% |
| Gemma 4 26B | 88.4% | 88.4% | 81.2% | 90.1% |
| Gemma 4 E4B | 76.3% | 76.3% | 68.9% | 79.2% |
| Gemma 4 E2B | 65.1% | 65.1% | 57.3% | 68.4% |
Coding Benchmarks
HumanEval
Python coding challenges (164 problems):
| Model | Pass@1 | Pass@10 | vs Codex | Temperature |
|---|---|---|---|---|
| Gemma 4 31B | 76.8% | 89.3% | +12.3% | 0.1 |
| Gemma 4 26B | 73.2% | 86.7% | +8.7% | 0.1 |
| Gemma 4 E4B | 62.1% | 78.4% | -2.4% | 0.1 |
| Gemma 4 E2B | 54.3% | 71.2% | -10.2% | 0.1 |
MBPP (Mostly Basic Python Problems)
| Model | Accuracy | 3-shot | Execution Rate |
|---|---|---|---|
| Gemma 4 31B | 82.4% | 84.1% | 98.7% |
| Gemma 4 26B | 79.6% | 81.3% | 98.2% |
| Gemma 4 E4B | 68.9% | 71.2% | 97.1% |
| Gemma 4 E2B | 59.3% | 62.4% | 95.8% |
Reasoning Benchmarks
ARC Challenge
Scientific reasoning questions:
| Model | Accuracy | vs Human | Confidence |
|---|---|---|---|
| Gemma 4 31B | 93.1% | +8.1% | High |
| Gemma 4 26B | 91.4% | +6.4% | High |
| Gemma 4 E4B | 84.2% | -0.8% | Medium |
| Gemma 4 E2B | 78.6% | -6.4% | Medium |
HellaSwag
Common sense reasoning:
| Model | Accuracy | 10-shot | 0-shot |
|---|---|---|---|
| Gemma 4 31B | 88.9% | 90.2% | 85.3% |
| Gemma 4 26B | 86.7% | 88.1% | 83.2% |
| Gemma 4 E4B | 79.4% | 81.3% | 75.8% |
| Gemma 4 E2B | 72.1% | 74.6% | 68.3% |
Multimodal Benchmarks
MMMU (Multimodal)
Vision + text understanding (E-series only):
| Model | Overall | Science | Humanities | OCR Quality |
|---|---|---|---|---|
| Gemma 4 E4B | 56.3% | 62.1% | 51.4% | Excellent |
| Gemma 4 E2B | 48.7% | 53.2% | 44.6% | Good |
| Gemma 4 31B | N/A | N/A | N/A | Text only |
| Gemma 4 26B | N/A | N/A | N/A | Text only |
Audio Understanding
Speech and sound processing (E-series only):
| Model | Speech Recognition | Speaker ID | Sound Classification |
|---|---|---|---|
| Gemma 4 E4B | 94.2% WER | 87.3% | 91.6% |
| Gemma 4 E2B | 96.8% WER | 82.1% | 86.4% |
Real-World Benchmarks
MT-Bench (Multi-Turn Conversation)
Quality of extended dialogues:
| Model | Overall | Reasoning | Coding | Writing | Roleplay |
|---|---|---|---|---|---|
| Gemma 4 31B | 8.52 | 8.9 | 8.7 | 8.3 | 8.1 |
| Gemma 4 26B | 8.31 | 8.6 | 8.4 | 8.1 | 7.9 |
| Gemma 4 E4B | 7.45 | 7.7 | 7.3 | 7.4 | 7.2 |
| Gemma 4 E2B | 6.89 | 7.1 | 6.8 | 6.9 | 6.7 |
Chatbot Arena ELO Rankings
Live user preference voting (as of April 2026):
| Model | ELO Score | Rank (Open) | Rank (All) | Win Rate vs GPT-4 |
|---|---|---|---|---|
| Gemma 4 31B | 1247 | #3 | #8 | 42.3% |
| Gemma 4 26B | 1221 | #5 | #12 | 38.7% |
| Gemma 4 E4B | 1156 | #12 | #24 | 28.4% |
| Gemma 4 E2B | 1098 | #18 | #35 | 19.2% |
Speed Benchmarks
Inference Speed (tokens/second)
Tested on common hardware:
| Model | RTX 4090 | M2 Ultra | A100 | T4 |
|---|---|---|---|---|
| Gemma 4 31B | 28 t/s | 19 t/s | 95 t/s | 8 t/s |
| Gemma 4 26B | 34 t/s | 23 t/s | 112 t/s | 11 t/s |
| Gemma 4 E4B | 89 t/s | 67 t/s | 287 t/s | 42 t/s |
| Gemma 4 E2B | 156 t/s | 124 t/s | 498 t/s | 89 t/s |
Memory Usage
RAM requirements for different quantizations:
| Model | FP16 | INT8 | INT4 | Mobile (4-bit) |
|---|---|---|---|---|
| Gemma 4 31B | 62 GB | 31 GB | 16 GB | N/A |
| Gemma 4 26B | 52 GB | 26 GB | 13 GB | N/A |
| Gemma 4 E4B | 8 GB | 4 GB | 2.5 GB | 2.2 GB |
| Gemma 4 E2B | 4 GB | 2 GB | 1.3 GB | 1.1 GB |
Specialized Benchmarks
TruthfulQA
Resistance to hallucination:
| Model | Truthful | Informative | Both | vs GPT-4 |
|---|---|---|---|---|
| Gemma 4 31B | 67.3% | 89.2% | 62.4% | +3.1% |
| Gemma 4 26B | 64.8% | 87.3% | 59.7% | +0.6% |
| Gemma 4 E4B | 58.2% | 82.1% | 52.3% | -6.0% |
| Gemma 4 E2B | 52.4% | 76.8% | 46.1% | -11.8% |
MATH (Competition Mathematics)
Advanced mathematical problem solving:
| Model | Overall | Algebra | Geometry | Number Theory | Combinatorics |
|---|---|---|---|---|---|
| Gemma 4 31B | 43.2% | 67.3% | 38.9% | 42.1% | 31.4% |
| Gemma 4 26B | 39.7% | 63.1% | 35.2% | 38.4% | 28.7% |
| Gemma 4 E4B | 24.8% | 41.2% | 19.3% | 23.7% | 15.2% |
| Gemma 4 E2B | 17.3% | 29.8% | 12.4% | 16.1% | 9.8% |
Language-Specific Performance
Multilingual MMLU
Performance across languages:
| Language | 31B | 26B | E4B | E2B | Native Speaker Baseline |
|---|---|---|---|---|---|
| English | 87.2% | 85.1% | 73.9% | 68.2% | 89.8% |
| Chinese | 84.6% | 82.3% | 69.4% | 63.1% | 87.2% |
| Spanish | 85.3% | 83.1% | 71.2% | 65.4% | 88.4% |
| Japanese | 83.9% | 81.4% | 68.7% | 62.3% | 86.9% |
| French | 85.7% | 83.4% | 71.8% | 66.1% | 88.7% |
| German | 84.8% | 82.6% | 70.3% | 64.7% | 87.6% |
Benchmark Methodology
Testing Conditions
- Temperature: 0.1 for deterministic tasks, 0.7 for creative
- Top-p: 0.95 standard across all tests
- Context: Full 256K window for 31B/26B, 10K for E-series
- Prompting: Few-shot where specified, zero-shot default
- Hardware: Standardized on A100 80GB for fair comparison
Version Information
- Models tested: Official checkpoints from Google
- Date: April 2026 release (v1.0.0)
- Framework: Transformers 4.40.0, vLLM 0.4.2
- Quantization: GPTQ for INT4, bitsandbytes for INT8
Benchmark Trends
Improvement Over Time
Comparing to Gemma 3 (2024):
| Metric | Gemma 3 | Gemma 4 | Improvement |
|---|---|---|---|
| MMLU | 79.1% | 87.1% | +10.2% |
| HumanEval | 61.3% | 76.8% | +25.3% |
| MT-Bench | 7.83 | 8.52 | +8.8% |
| Inference Speed | 19 t/s | 28 t/s | +47.4% |
How to Reproduce
Want to verify these benchmarks yourself? Here's how:
# Install evaluation harness
pip install lm-eval transformers accelerate
# Run MMLU benchmark
lm_eval --model hf \
--model_args pretrained=google/gemma-4-31b \
--tasks mmlu \
--batch_size 8
# Run HumanEval
evaluate-humaneval \
--model google/gemma-4-31b \
--temperature 0.1 \
--top_p 0.95For detailed setup instructions, see our benchmark reproduction guide.
Benchmark Limitations
Understanding what benchmarks don't measure:
- Real-world application performance varies significantly
- Prompt engineering can improve scores by 10-20%
- Domain-specific tasks may differ from general benchmarks
- Multimodal integration only tested on E-series models
- Long-context performance not fully captured in standard tests
Model Comparisons & Analysis
Direct Model Comparisons
Compare Gemma 4 with other leading models:
- Gemma 4 vs Llama 4 - Detailed comparison with Meta's latest model
- Gemma 4 vs Qwen 3.5 - Compare with Alibaba's multilingual champion
- Gemma 4 vs Mixtral - See how it stacks against Mistral's MoE
- Gemma 4 vs Claude Opus - Open vs closed model showdown
- Gemma 4 26B vs 31B - Which size is right for you?
- Gemma 4 E2B vs E4B - Edge model comparison
Performance Deep Dives
- Gemma 4 Speed Test - Real-world latency benchmarks
- Gemma 4 Context Window - 256K context performance analysis
- Gemma 4 Function Calling - Tool use benchmark results
Conclusion
Gemma 4 delivers strong performance across the board:
- 31B model competes with much larger closed models
- E-series brings multimodal AI to edge devices
- Consistent improvements over previous generation
- Best open model for many use cases
Choose based on your needs:
- Maximum quality: Gemma 4 31B
- Best efficiency: Gemma 4 26B
- Mobile deployment: Gemma 4 E2B/E4B
- Multimodal tasks: E-series only
For deployment guides, see:
Complete Gemma 4 Resource Hub
Getting Started Guides
- Ollama Quick Setup - Running Gemma 4 locally in 5 minutes
- Hardware Requirements - GPU, RAM, and storage specs for each model
- Google AI Studio Access - Try Gemma 4 in the cloud without setup
- Download All Methods - Every way to get Gemma 4 weights
Model Comparisons
- Gemma 4 vs ChatGPT - Free local vs $20/mo cloud comparison
- Gemma 4 vs Gemini - Open source vs Google's proprietary API
- Gemma 4 vs Gemma 3 - Generation improvements and upgrades
- Gemma 4 26B vs 31B - Detailed size comparison with benchmarks
- Gemma 4 E2B vs E4B - Edge model selection guide
Performance & Optimization
- Mac Performance Guide - M1/M2/M3 benchmarks and tips
- NVIDIA RTX Setup - GPU acceleration for RTX cards
- Speed Optimization - Double your tokens/second
- 4-bit Quantization - Reduce memory by 75%
- Mobile Deployment - Running on phones and embedded
Advanced Features
- JSON Output Mode - Structured data extraction
- Function Calling - Build AI agents with tools
- Fine-tuning Tutorial - Customize for your domain
- Thinking Mode - Chain-of-thought reasoning
- Context Window Test - 256K context analysis
Practical Applications
- Best Prompts Collection - Production-tested prompts
- Use Cases & Examples - Real-world applications
- Local Agent Setup - Autonomous AI assistants
- Troubleshooting Guide - Fix common issues
- Chinese Language Review - Mandarin performance analysis
gemma4 — interact
Stop reading. Start building.
~/gemma4 $ Get hands-on with the models discussed in this guide. No deployment, no friction, 100% free playground.
Launch Playground />
