AlphaCode's core objective is to solve competition-level programming problems, which are far more complex than simple code completion or generating code from comments. They typically involve: complex natural language understanding, algorithmic reasoning ability, and creative problem-solving. To accurately evaluate AlphaCode's capabilities in these areas, the paper evaluates on Codeforces, with an evaluation process that simulates the real-world scenario of a human participant taking a Codeforces competition: the natural language description of the problem (including examples) is fed into AlphaCode; AlphaCode does not generate just a single answer, but leverages its large-scale model to generate a massive number of candidate code solutions for each problem (on the order of thousands to tens of thousands); the vast majority of generated solutions are incorrect, and the system filters out most of the obviously wrong ones by running them against example test cases and similar checks, ultimately retaining only a very small number (e.g., around 10) of the most likely correct solutions for submission per problem. This mimics the behavior of a human contestant who tests their solutions before submitting and has a limited number of submission attempts.