When will a single model first achieve 10@k solve rate >= 90% on the CodeContests dataset?
Basic
9
Ṁ3922032
6%
1%
Before 2023
4%
2023-2024
12%
2024-2025
36%
2025-2026
20%
2026-2027
18%
2027-2032
3%
2032-never
10@k - unlimited samples may be generated, but at most 10 code samples may be submitted per problem.
I will post my own responses and pick whichever one of those turns out to be correct.
CodeContests dataset: https://github.com/deepmind/code_contests
Close date updated to 2050-01-01 11:59 pm
Apr 8, 4:16pm: Options are written as [a, b), so 2024-2025 means occurring after the start of 2024 and before the start of 2025.
Close date updated to 2032-01-01 11:59 pm
This question is managed and resolved by Manifold.
Get
1,000and
3.00
1,0003.00Sort by:
@ValeryCherepanov Yeah the before 2023 option seems like free money to me, I think the market is just old and people don't have it on their radar.
Related questions
Related questions
Top score on codeforces by an AI model at the end of 2025
Will an AI model achieve superhuman ELO on Codeforces by the 31 December 2025?
35% chance
Will any AI model score >80% on Epoch's Frontier Math Benchmark in 2025?
3% chance
Will OpenAI models achieve ≥90% on SimpleBench by the end of 2025?
7% chance
In what year will AI achieve a score of 95% or higher on the SWE-bench Verified benchmark?
-
By when will SAT solvers be 10x faster (than 2023 on same hardware)?
In what year will an AI achieve 90%+ on Sudoku bench?
AI model achieves superhuman ELO on Codeforces by June 1st 2027?
87% chance
When Will 9x9 Go Be Solved?
Benchmark Gap #9: Once a model solves current software engineering benchmarks, how long until humans don't code?
-