Best LLM for Coding
Best LLM for Coding
This coding LLM leaderboard compares the latest models on engineering-specific benchmarks including SWE-Bench, LiveCodeBench, Aider Polyglot, BFCL tool use, and more. The data comes from model providers as well as independently run evaluations by Vellum or the open-source community.
Top models for coding
Best in Live CodeBench
| Model | Score |
|---|---|
| DeepSeek V4 Pro | 93.5% |
| DeepSeek V4 Flash | 91.6% |
| Kimi K2 Thinking | 83.1% |
| Gemini 3 Pro | 79.7% |
| Grok 3 [Beta] | 79.4% |
Best in Agentic Coding (SWE Bench)
| Model | Score |
|---|---|
| GPT-5.6 Sol | 96.2% |
| Claude Mythos 5 | 95.5% |
| Claude Fable 5 | 95% |
| GPT-5.6 Luna | 93% |
| Claude Opus 4.8 | 88.6% |
Best in Tool Use (BFCL)
| Model | Score |
|---|---|
| GPT-4.5 | 69.9% |
| OpenAI o3-mini | 65.1% |
| Qwen2.5-VL-32B | 62.8% |
| Gemma 3 27b | 59.1% |
| DeepSeek V3 0324 | 58.5% |
Model Comparison
| Models | LiveCodeBench | SWE Bench | MATH 500 | BFCL | Aider Polyglot |
|---|---|---|---|---|---|
| Claude Mythos 5.1 | n/a | n/a | n/a | n/a | n/a |
| Claude Fable 5.1 | n/a | n/a | n/a | n/a | n/a |
| Gemini 3.7 Flash | n/a | n/a | n/a | n/a | n/a |
| Claude Opus 5 | n/a | n/a | n/a | n/a | n/a |
| GPT-5.6 Luna | n/a | 93% | n/a | n/a | n/a |
| GPT-5.6 Sol | n/a | 96.2% | n/a | n/a | n/a |
| GPT-5.6 Terra | n/a | n/a | n/a | n/a | n/a |
| DeepSeek V4 Flash | 91.6% | 79% | n/a | n/a | n/a |
| DeepSeek V4 Pro | 93.5% | 80.6% | n/a | n/a | n/a |
| Gemini 3.1 Pro | n/a | 80.6% | n/a | n/a | n/a |
| Gemini 3.5 Flash | n/a | n/a | n/a | n/a | n/a |
| GLM 5.2 | n/a | n/a | n/a | n/a | n/a |
| Claude Sonnet 5 | n/a | 85.2% | n/a | n/a | n/a |
| MiniMax M3 | n/a | 80.5% | n/a | n/a | n/a |
| Claude Mythos 5 | n/a | 95.5% | n/a | n/a | n/a |
| Claude Fable 5 | n/a | 95% | n/a | n/a | n/a |
| Claude Opus 4.8 | n/a | 88.6% | n/a | n/a | n/a |
| GPT-5.5 | n/a | 58.6% | n/a | n/a | n/a |
| GPT-5.5 Pro | n/a | n/a | n/a | n/a | n/a |
| Claude Opus 4.7 | n/a | 87.6% | n/a | n/a | n/a |
| Claude Opus 4.6 | 76% | 80.8% | 97.6% | n/a | n/a |
| Claude Sonnet 4.6 | 72.4% | 79.6% | 97.8% | n/a | n/a |
| GPT-5.3 Codex | n/a | n/a | n/a | n/a | n/a |
| DeepSeek V3 0324 | 41% | 38.8% | 94% | 58.5% | n/a |
| Qwen2.5-VL-32B | n/a | 18.8% | 82.2% | 62.8% | n/a |
| OpenAI o1-mini | n/a | n/a | 90% | 52.2% | n/a |
| OpenAI o3-mini | 74.1% | 61% | 97.9% | 65.1% | n/a |
| DeepSeek-R1 | 64.3% | 49.2% | 97.3% | 57.5% | n/a |
| Claude 3.7 Sonnet [R] | n/a | 70.3% | 96.2% | 58.3% | n/a |
| GPT-4.5 | n/a | 38% | n/a | 69.9% | n/a |
| Claude 3.7 Sonnet | n/a | 62.3% | 82.2% | 58.3% | n/a |
| Gemini 2.5 Pro | 69% | 59.6% | n/a | n/a | n/a |
| Grok 3 [Beta] | 79.4% | n/a | n/a | n/a | n/a |
| Gemma 3 27b | n/a | 10.2% | 89% | 59.1% | n/a |
| Llama 4 Maverick | 41% | n/a | n/a | n/a | n/a |
| Llama 4 Scout | 32.8% | n/a | n/a | n/a | n/a |
| Llama 4 Behemoth | 49.4% | n/a | 95% | n/a | n/a |
| GPT-4.1 | 52% | 55% | n/a | n/a | n/a |
| GPT-4.1 mini | n/a | 23.6% | n/a | n/a | n/a |
| GPT-4.1 nano | n/a | n/a | n/a | n/a | n/a |
| Claude 4 Sonnet | n/a | 72.7% | n/a | n/a | n/a |
| Claude 4 Opus | n/a | 72.5% | n/a | n/a | n/a |
| GPT oss 120b | 69% | n/a | n/a | n/a | n/a |
| GPT oss 20b | 69% | n/a | n/a | n/a | n/a |
| Claude Opus 4.1 | n/a | 74.5% | n/a | n/a | n/a |
| GPT-5 | n/a | 74.9% | n/a | n/a | n/a |
| GPT 5.1 | n/a | 76.3% | n/a | n/a | n/a |
| Kimi K2 Thinking | 83.1% | 71.3% | n/a | n/a | n/a |
| Gemini 3 Pro | 79.7% | 76.2% | n/a | n/a | n/a |
| Claude Sonnet 4.5 | n/a | 82% | n/a | n/a | n/a |
| Claude Opus 4.5 | n/a | 80.9% | n/a | n/a | n/a |
| GPT 5.2 | n/a | 80% | n/a | n/a | n/a |
Context window, cost and speed comparison
| Models | Context Window | Input Cost / 1M tokens | Output Cost / 1M tokens | Speed (tokens/second) | Latency |
|---|---|---|---|---|---|
| Claude Fable 5.1 | 1000000 | $10 | $50 | n/a | n/a |
| Claude Mythos 5.1 | 1000000 | $10 | $50 | n/a | n/a |
| Claude Opus 5 | 1,000,000 | $5 | $25 | n/a | n/a |
| Claude Mythos 5 | 1,000,000 | $10 | $50 | n/a | n/a |
| Claude Sonnet 5 | 1,000,000 | $3 | $15 | 56.3 t/s | 20.69 seconds |
| GLM 5.2 | 1,000,000 | $0.95 | $3 | 347 t/s | 1.14 seconds |
| DeepSeek V4 Flash | 1000000 | $0.14 | $0.28 | 107.9 t/s | 1.42 seconds |
| DeepSeek V4 Pro | 1,000,000 | $0.435 | $0.87 | 174.9 t/s | 1.2 seconds |
| GPT-5.6 Sol | 1,050,000 | $5 | $30 | n/a | n/a |
| Gemini 3.1 Pro | 1,000,000 | $2 | $12 | 136.2 t/s | 20.34 seconds |
| Gemini 3.5 Flash | 1,000,000 | $1.5 | $9 | 175.4 t/s | 23.16 seconds |
| Gemini 3.7 Flash | 1,048,576 | $0.75 | $3.75 | n/a | n/a |
| GPT-5.6 Luna | 1,050,000 | $0.2 | $1.2 | n/a | n/a |
| GPT-5.6 Terra | 1,050,000 | $2 | $12 | n/a | n/a |
| MiniMax M3 | 1,048,576 | $0.6 | $2.4 | 98.6 t/s | 0.85 seconds |
Coding benchmark glossary
LiveCodeBench Continuously updated competitive programming problems sourced after model training cutoffs. Measures genuine code generation on unseen tasks.
Aider Polyglot Multi-language code editing benchmark using the Aider coding assistant. Tests the ability to correctly modify existing code across languages.
SWE-Bench Verified Real GitHub issues from popular Python repos that the model must resolve end-to-end. Measures agentic software engineering ability.
BFCL Berkeley Function Calling Leaderboard testing structured tool and function call accuracy. Evaluates how reliably a model invokes APIs.
GRIND Adaptive reasoning benchmark requiring iterative problem decomposition. Tests a model's ability to break down and solve multi-step coding challenges.