GPT-6.1 Sol in the Sonar test: more code, fewer errors per line, but more errors overall

New artificial intelligence models are often presented with a single number, such as the percentage of tasks solved. For developers, however, what matters more is what happens to the code the model writes and how much work remains for humans to review. An independent evaluation by Sonar published on October 9, 2026, using OpenAI’s GPT-6.1 Sol model, shows that the picture is not clear-cut: more code, fewer errors per line, but more errors overall.

What Sonar Tested

Sonar compared the GPT-6.1 Sol model with its predecessor, GPT-6 Sol, on a set of 4,444 Java programming tasks at a medium level of “reasoning” (the extent to which the model thinks before responding). It then analyzed the resulting code using its own quality and security tools. This is a single-language test set and may not reflect everyday work in other languages or projects.

What the results show

More code and higher success rates

• The test success rate reached 85.29%, which is 2.2 percentage points higher than that of its predecessor.
• The model generated 669,155 lines of code compared to 480,125, an increase of 39.4%.
• According to Sonar, the consumption of “reasoning” tokens decreased by 51.4%.

Lower search result density

All metrics for the density of findings per thousand or million lines of code have decreased. Vulnerability density fell by 32.4%, and code complexity remained roughly the same (cognitive complexity decreased by 1.6%). The output is therefore cleaner per line of code.

But more work to review

The absolute numbers paint a different picture, because there is more code:

• There were 407 bugs, compared to 302—a 34.8% increase.
• Code smells increased by 32.1%.
• There were 129 vulnerabilities compared to 137, a decrease of 5.8%.
• The total number of findings rose from 9,859 to 12,982.
• The number of critical findings related to reliability increased from 21 to 46.

The largest category of errors continued to be concurrency and threading issues: their number rose from 116 to 196, accounting for 48% of all errors, compared to 38% previously.

How to Read This

Sonar itself points out that the numbers should be interpreted in conjunction with the volume of code: the model is cleaner on a per-line basis, but because there is more code, the amount of work that needs to be reviewed increases. For the team, this means budgeting capacity for review and not relying solely on the assumption that “lower error density” means less work.

Other independent measurements vary

Another independent error comparison tool, which tracks the actual deployment of agents in the code, ranked GPT-6.1 In mid-October 2026, Sol ranked fourth out of seven with a relative error rate of 1.23—23% more errors than the average. Sol had previously reported that its predecessor, GPT-5.6, had fewer errors per line. The methodologies differ: one test evaluates a set of tasks in Java, while the other evaluates actual changes in code repositories. Therefore, it is not possible to conclude from a single number that the model is “better” in general.

What we haven't confirmed

• “Cheap and massive code generation.” We didn’t find the model’s price or a cost comparison in the Sonar evaluation.
• “Fewer errors after switching from GPT-6 Sol to GPT-6.1 Sol.” Per line, yes; overall, absolutely not. In this test, the number of errors increased by a third.
• That the results apply to languages other than Java. The test does not cover them.

What does this mean for developers?

• When evaluating the model, consider not only the success rate but also the absolute number of findings and their severity.
• Run the generated code through static analysis and review, especially regarding concurrency and threads, which, according to measurements, are the most common sources of errors.
• Before modifying the model, test it on your own project—don’t rely solely on general results.

Conclusion

According to Sonar, GPT-6.1 Sol improved both the success rate and density of findings, but at the cost of a significantly larger code volume and more errors in absolute terms. This is a useful reminder that “per line” and “overall” metrics tell different stories. How good a model is at programming is ultimately determined by measuring it on your code, not by a single percentage from a table.