Definition
Item response theory (IRT) is a psychometric framework that models the probability of a correct answer as a function of a test-taker’s latent ability and properties of each question, such as its difficulty and how sharply it discriminates between strong and weak respondents. In ML evaluation it is used to place models and benchmark items on a shared scale, which helps explain effects raw accuracy hides—for example, why upgrading to a stronger model pays off most on mid-difficulty questions and yields little on items that are already trivially easy or still out of reach.