If you write Kotlin and you pick an AI coding agent based on a benchmark score, you have a problem. Almost every popular benchmark is Python. Your agent might be great at Python and mediocre at Kotlin, and the score would never tell you.
JetBrains just shipped a fix: the Kotlin Benchmark, a public leaderboard that scores AI coding agents on real Kotlin work. Here is what it measures, what it does not measure, and what the current results actually say.
Why SWE-bench Does Not Help Kotlin Developers
SWE-bench is the benchmark most AI labs quote when they claim their model is “good at coding.” It pulls real GitHub issues from open-source projects, gives an agent the issue and the repository, and checks if the agent’s patch passes the project’s own tests.
The problem: the original SWE-bench dataset is built from 12 Python repositories — projects like Django, Matplotlib, and scikit-learn. Every task is Python. A model can top the SWE-bench leaderboard while never touching a .kt file.
There is a multilingual version, SWE-bench Multilingual, that adds 300 tasks across 9 languages: C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust. Look at that list again — no Kotlin. Even the “multilingual” version skips it.
So if you write Android apps, backend services, or Kotlin Multiplatform code, no existing SWE-bench variant tells you anything about your actual workload. That is the gap JetBrains built the Kotlin Benchmark to close.
What “SWE-bench-Style” Actually Means
JetBrains describes the Kotlin Benchmark as based on the SWE-bench methodology. In plain terms, here is what that means for each task:
- The agent gets a real issue or pull request description — the kind a maintainer would actually write.
- The agent gets the repository at the commit right before the fix — the real codebase, not a toy snippet.
- The agent must generate a patch that resolves the issue, using its own tools (search the code, edit files, run commands).
- The patch runs against the project’s hidden test suite, captured from the real human-written fix.
- The task counts as resolved only if the tests pass. Partial credit does not exist — it is pass or fail.
This is a big step up from something like Kotlin HumanEval, which tests small isolated functions. SWE-bench-style tasks test whether an agent can work inside a real codebase: find the right file among hundreds, understand existing patterns, and not break anything else.
What the Benchmark Actually Contains
Verified against JetBrains’ benchmark methodology page and the announcement post:
- 105 tasks total, built from real, merged pull requests.
- Tasks are sourced from 8 open-source Kotlin repositories, selected by GitHub stars and contributor activity:
| Repository | Tasks |
|---|---|
| ktlint/ktlint | 43 |
| detekt/detekt | 28 |
| oss-review-toolkit/ort | 12 |
| Hannah-Sten/TeXiFy-IDEA | 8 |
| ankidroid/Anki-Android | 6 |
| Kotlin/dataframe | 5 |
| square/okhttp | 2 |
| GradleUp/shadow | 1 |
- The evaluation runs on the open-source Multi-SWE-bench harness, extended for Kotlin, using containerized Docker environments so results are reproducible.
- Full dataset and test harness are public on GitHub.
Notice the skew: ktlint and detekt alone make up 71 of the 105 tasks. Both are linters and static-analysis tools — heavy on parsing, AST work, and rule logic. That is a specific kind of Kotlin code, not a cross-section of “all Kotlin.” An agent that handles ktlint-style tasks well is not guaranteed to be equally good at, say, Jetpack Compose UI code or Ktor networking code — those domains barely show up in this first version.
JetBrains says this openly: this is “the first public iteration,” and the plan is to grow coverage toward Android and Kotlin Multiplatform code over time.
The Current Leaderboard
The results below are pulled directly from kotlinlang.org/benchmark — resolution rate is the percentage of the 105 tasks each agent+model setup solved:
| Rank | Agent + Model | Resolved | Resolution Rate |
|---|---|---|---|
| 1 | Claude Code + Opus 4.7 (xhigh) | 90 / 105 | 85.71% |
| 2 | Junie + Opus 4.7 (max) | 86 / 105 | 81.90% |
| 2 | Codex + GPT-5.5 (xHigh) | 86 / 105 | 81.90% |
| 4 | Claude Code + Opus 4.6 (max) | 84 / 105 | 80.00% |
| 5 | Codex + GPT-5.3-codex (xHigh) | 82 / 105 | 78.10% |
| 6 | Junie + Opus 4.6 | 81 / 105 | 77.14% |
| 6 | Codex + GPT-5.4 (xHigh) | 81 / 105 | 77.14% |
| 16 | Gemini CLI + Gemini 3.1 Pro | 69 / 105 | 65.71% |
| 19 | Gemini CLI + Gemini 3 Flash | 47 / 105 | 44.76% |
(Full 19-entry table, including every mid-tier Codex, Junie, and Claude Code configuration, is on the live leaderboard — it updates as JetBrains adds runs.)
Two things stand out:
- JetBrains’ own agent, Junie, does not win. Claude Code with Opus 4.7 tops the board, and JetBrains published that result on its own benchmark. That is a point in favor of the benchmark’s credibility — a vendor-run benchmark that doesn’t quietly favor the vendor’s own product is worth more.
- The gap between #1 and #19 is huge — 85.71% down to 44.76%. Model and agent choice clearly matters for Kotlin specifically, not just “AI coding in general.”
The leaderboard also lists average token usage and latency for each setup, and this is where the top scores get expensive. The winning entry’s “Avg. latency” column reads 10h 37m, next to roughly 10.6M tokens. Both numbers are labelled as averages, though JetBrains does not document exactly what the latency clock includes — so read it as “this configuration is extremely slow and extremely expensive,” not as a precise wall-clock figure you can budget against.
The pattern holds across the table: the setups at the top run at maximum reasoning effort, and they pay for it in tokens and time. A benchmark win is not free.
What This Benchmark Cannot Tell You
Be honest about the limits before you pick a tool based on this table:
- It does not test your codebase. Your app’s structure, your team’s conventions, your internal libraries — none of that is in this dataset. A model that resolves 85% of ktlint issues might struggle with your custom Compose design system.
- It does not measure code quality, only test-passing. JetBrains says this directly: cost, maintainability, and code quality are not yet measured — they are listed as future work.
- It skews toward linters and static analysis. Two of eight repos supply about 68% of all tasks. Android and Kotlin Multiplatform, the two things most Kotlin developers actually build, are thin or absent in this first version.
- It is a snapshot in time. Every row has a date. Models get updated, and new ones ship constantly. The benchmark only launched in July 2026, so today’s ranking is an early reading — not a settled result.
A benchmark score is a signal, not a guarantee. Use it to shortlist agents worth trying on your own repo — not as the final answer.
One More Data Point: X’s Android App Is Now 100% Kotlin
In the same roundup post where JetBrains covered the benchmark, they mention that X’s Android app was rebuilt from scratch and is now written entirely in Kotlin. X Chat also uses Kotlin Multiplatform across Android, iOS, and web for encryption, storage, and sync. It’s a small item in the post, but it’s a real, large-scale production example of Kotlin (and Kotlin Multiplatform) at a company with hundreds of millions of users — not just Android boilerplate.
What to Actually Do With This
If you are choosing an AI coding agent for Kotlin work:
- Check the live leaderboard yourself — it updates over time, so treat this article’s numbers as a snapshot from August 2026.
- Don’t pick blind. Run your top 2-3 candidates on a real ticket from your own backlog before committing. See our guide on what AI coding agents are and how they work if you’re new to this.
- If you’re comparing the big three assistants for daily use, not just Kotlin specifically, read our Cursor vs Claude Code vs GitHub Copilot comparison.
- Cost matters as much as the score. Higher-resolution setups in this benchmark also burned more tokens and took longer per task — factor that into your own budget.