ML//benchmark//HumanEval

HumanEval is OpenAI's benchmark of 164 Python programming problems with unit tests. It measures \(\text{pass@}k\), the probability of getting at least one correct solution in \(k\) attempts, and it is more objective than multiple choice: the code either passes the tests or it does not.

HumanEval is OpenAI's benchmark of 164 Python programming problems with unit tests. It measures pass@k\text{pass@}kpass@k, the probability of getting at least one correct solution in kkk attempts, and it is more objective than multiple choice: the code either passes the tests or it does not.