r/singularity • u/HamsterUnfair6313 • 4h ago

AI Real-SWE 1.1 Benchmark — Specific Labs

https://withspecific.com/benchmarks/real-swe

Sonnet 5.5 is reigning in Real SWE benchmark why is that?

3 Upvotes

64% Upvoted

3 comments sorted by

2

u/Proper_Actuary2907 Spooky Machine Intelligence 2030 4h ago

Sonnet 5.5 is reigning in Real SWE benchmark why is that?

Maybe it went through the most up to date post-training and you can squeeze a lot of extra performance there for software tasks

2

u/Middle_Bullfrog_6173 3h ago

They quote percentages with two decimal places, but have error bars that look to be about ±4%?