Top AI model (LLM) for coding in Papers With Code's "Code Generation on HumanEval" ranking at the end of 2024
Market Rules
For the sake of this market, the "top AI model" is defined as the model holding the number one position on the “Models with highest Pass@1’” metric of Code Generation on HumanEval leaderboard made by Papers With Code, as of 11:59 p.m. (UTC-3) on December 31, 2024.
Any updates or changes to the leaderboard after this time will not affect the market resolution. If the Code Generation on the HumanEval leaderboard is unavailable or ceases to be updated, an alternative credible source, such as a widely recognized AI research publication or authority, will be used to determine the top AI model.
"Other" refers to any AI model not covered by a brand or responsible organization listed in the other outcomes.
In case the top AI model is a derivation of one of the models described in an outcome (a modification made by developers, for instance), the original model outcome will be the winning resolution of this market.