Will coding agents from frontier labs forever outclass their open weights counterparts?
This question is surprisingly hard to answer. What follows is my attempt to answer part of the question: is the raw “coding power” of closed models pulling ahead of their open-weight counterparts?
SWE-Bench
The canonical benchmark for coding agents is SWE-bench Verified, which measures an agent’s ability to successfully resolve issues in GitHub repositories by generating code. This is the coding benchmark you’ll most often see reported.
However, there are three problems with it:
- Contamination. Recent models will likely have “seen” a lot of the repositories during their training.
- Scaffolding confounds the analysis. A coding agent consists of two parts: the underlying LLM and the “scaffolding” - the code which wraps the model and turns it into an agent. The agents’ capabilities stem from both of these. SWE-bench Verified requires submissions to provide their own agentic scaffolding, meaning one cannot distinguish between the relative contributions of the scaffolding and the model.
- Self-reporting. The SWE-bench Verified leaderboard shows submissions from teams who have run the test pack locally and uploaded their results for scoring. Leading labs tend to self-report their SWE-bench verified scores, meaning there are no independent checks on the numbers and no guarantee that results have not been cherry-picked from different model runs or checkpoints.
SWE-rebench
Many of these problems have been addressed by the SWE-rebench benchmark, which works as follows:
- Each month, a fresh batch of GitHub issues are selected and the benchmark models are evaluated against them.
- Nearly all models are evaluated using the same scaffolding. (A few agents like Claude Code are included in the leaderboard using their native scaffolding. Interestingly, they often perform worse than the same underlying model + the SWE-rebench harness - likely due to their scaffolding being optimised for human-interactive coding rather than headless, fully-autonomous coding.)
- All results on SWE-rebench are reported by the maintainers - no self-reporting to merge in.
Here are the SWE-rebench results, as of the time of writing (1st May 2026), taken from the website:

Note the variation, month-by-month. This reflects the varying difficulty of the GitHub issues selected for the evaluation.
The following chart shows the results of my analysis of this data. Here’s what I did:
- I removed all rows from the underlying dataset where the models were being evaluated on potentially contaminated issues (there are some of these even in SWE-rebench).
- I fitted a statistical model (a binomial GLM with mixed effects, since you ask) to estimate the mean “resolved rate” (% of issues successfully resolved by the agents’ code) - explicitly accounting for the varying difficulty of the tasks in each month. (The model has a pseudo-R^2 of 0.972, so it’s a good fit to the data.)
- For each month since the launch of SWE-rebench, I picked the leading closed model and the leading open-weights model.
The resulting chart shows the “Pareto frontier” for open and closed models across SWE-rebench:

What do we see?
- There is a consistent gap between the leading open and closed models in favour of the former.
- Both are still improving and the benchmark is not yet near saturation.
- There is no clear pattern to suggest that closed models are pulling ahead and generating an unrecoverable lead.
There have been 7 closed models at the Pareto frontier since SWE-bench Verified launched. The performance of 5 out of these 7 have already been matched by open models. A quick (and somewhat shaky) Kaplan-Meier estimate suggests that the median time for open models to catch up with leading closed models is 214 days (around 7 months).
What does this mean?
What this analysis does not do is calculate the contribution of the scaffolding to an agent’s coding prowess. It is likely substantial. (I may try to tease it out in a follow-up analysis.) If so, the current trend augers for a marketplace of competing scaffolding with a bring-your-own-model vibe.
Frontier labs have an advantage here: they can post-train their models to work well with the specific tools and behavioural patterns baked in to their scaffolding code. But third parties could play this game too: curating their own post-training datasets for the open weights models. Since curation and post-training are expensive processes, this would only be worth doing if they were confident that the raw capabilities of open models will converge with those of closed models (i.e. as the benchmarks saturate). At present, the Pareto-curves indicate neither convergence nor divergence - but it is the latter which is most notable.
One of the big questions in AI today is whether the majority of future tokens will be generated by a small handful of super-models or a “long tail” of specialised models, working in small niches. Right now, it seems perfectly plausible that the market for coding agents could look much more like the latter.
