Can You Trust AI with a MIT Exam?

The Leaderboard

Measured accuracy spans 42 percentage points across the ten models evaluated, ranging from o3 at 92.1% to DeepSeek-R1 at 50.9%. The problem corpus was drawn from MIT 8.033 (Special and General Relativity, Fall 2025) and 6.1220 (Design and Analysis of Algorithms, Fall 2023), representing genuine undergraduate coursework rather than purpose-built benchmark items.

* DeepSeek-R1 produced no hallucinated formulas because it declined to respond to 123 of 270 questions (45.6%). Abstention precludes both correct answers and formula fabrication.

o3 is the only model whose self-reported confidence falls below its actual accuracy. Every other model in the evaluation exhibits overconfidence of varying severity. The practical consequence is substantial: a system that reports 99.3% certainty while achieving 50.9% accuracy is supplying downstream consumers with confidence signals that are not merely imprecise but systematically inverted in their informational value.

Gemini 2.5 Flash processed all 270 questions at a total cost of $0.31 and achieved an accuracy of 85.4%. Claude Opus 4 processed the same corpus at a total cost of $23.11 and achieved 81.8% accuracy. This represents a 74-fold cost differential in favour of the less expensive model, which also achieved a higher score, rendering the premium pricing economically indefensible on this evaluation.

The cost-accuracy Pareto frontier has a well-defined structure. Gemini 2.5 Flash occupies the budget tier with a cost-per-accuracy-point of $0.004. o4-mini offers the most favourable balance of quality and expenditure among mid-tier models. o3 and Gemini 2.5 Pro justify their higher costs through genuine accuracy improvements over the rest of the field. The remaining models fall inside the frontier, meaning they are dominated on at least one dimension by a less expensive or more accurate alternative.

GPT-4o produced 11 hallucinated formulas across the evaluation corpus, generating equations that do not appear in physics and applying them with apparent confidence to problems for which they are inappropriate. On responses containing a hallucinated formula, the average score fell to approximately 1 out of 5, indicating that formula fabrication is not a minor perturbation but a near-total failure mode. No other quantitative failure was more reliably catastrophic.

DeepSeek-R1 exhibits a qualitatively distinct failure pattern. Rather than producing incorrect answers, it declined to respond to 123 of 270 questions, representing 45.6% of the evaluation corpus. No other model abstained more than three times. On the 147 questions it did attempt, DeepSeek-R1 reported an average confidence of 99.3% while achieving an accuracy of 50.9% — a 48-point miscalibration gap that is the largest observed in the study. The combination of widespread abstention and severe overconfidence on attempted questions suggests a fundamental limitation in the model’s capacity to sustain coherent multi-step reasoning over long derivations.

Disaggregating performance by course reveals a consistent and theoretically informative pattern. Models achieving the highest overall accuracy perform measurably better on algorithmic problems than on physics problems, whereas models in the lower half of the accuracy distribution perform worse on algorithms than on physics.

We attribute this interaction to a structural difference between the two domains. Algorithm problems possess a formal, syntactic character that is well suited to the pattern recognition capabilities central to large language model training. Physics problems, by contrast, require the construction of an intuitive model of a physical situation and the creative selection of appropriate principles before any formal machinery can be applied. A model with only superficial competence can partially simulate physics performance through formula matching, retrieving equations that look appropriate for a given scenario without genuine understanding of their domain of validity. The same strategy fails on algorithm problems, where precise logical reasoning is non-negotiable and surface similarity provides insufficient guidance. This suggests that physics benchmarks may offer a more discriminating test of genuine reasoning ability than algorithm benchmarks alone.

Physics benchmarks may constitute a more discriminating test of genuine machine reasoning than algorithmic benchmarks. The relativistic scenarios constructed for MIT 8.033 cannot be solved through memorization of surface patterns; they require an integrated understanding of physical principles and the ability to select among them appropriately. Algorithmic problems, by contrast, are susceptible to high performance through template recognition, which may overestimate a model’s underlying reasoning capability.

The most frequent error type across all ten models is the conceptual error, defined as a misidentification of which physical principle or framework the problem calls for, as opposed to a computational mistake made within an otherwise correct approach.

Even o3, the highest-performing model in the evaluation, produced 20 conceptual errors compared to 14 algebraic errors. The distinction is significant for downstream trust. An algebraic error indicates that the model correctly identified the relevant physical framework and made an arithmetic or algebraic slip in execution — a class of mistake that is often recoverable by inspection. A conceptual error indicates that the model applied an inappropriate physical principle entirely: for example invoking classical momentum in a context requiring relativistic momentum, misidentifying the reference frame in which a quantity is measured, or constructing a solution to a different problem than the one posed. Such errors are not recoverable by checking the arithmetic; they require recognising that the foundational framing is wrong.

Conceptual understanding, rather than computational fluency, constitutes the primary bottleneck for frontier models on STEM reasoning tasks. This finding is in tension with the prevailing assumption that scaling model size and training compute is sufficient to improve technical reasoning; if conceptual errors dominate even at the frontier, the limitation appears to reside in the nature of how physical understanding is encoded during training rather than in the raw expressive capacity of the models.

Beyond Physics: AI as Essay Grader

Essay evaluation employs different metrics than the physics benchmark. Quadratic Weighted Kappa (QWK) measures inter-rater agreement between model-assigned and human-assigned scores, weighting disagreements by the square of their magnitude so that large discrepancies are penalised more heavily than small ones. Mean Absolute Error (MAE) provides a complementary measure of average score deviation in raw units. Human-human inter-rater agreement on this dataset is approximately QWK = 0.74, representing the practical ceiling against which model performance should be interpreted.

Claude Opus 4 achieves the highest QWK of 0.679, remaining 0.06 below the human-human agreement threshold of 0.74. The rank ordering on essay quality is strikingly different from the physics rank ordering. o3, which leads the physics evaluation by a substantial margin, ranks ninth on essay grading with a QWK of 0.510. This reversal suggests that the capabilities underlying strong quantitative reasoning and those underlying reliable subjective writing assessment are meaningfully distinct, and that practitioners selecting models for language-quality tasks should not assume that physics or mathematics benchmarks are predictive.

Gemini 2.5 Flash’s QWK of 0.617 at a total evaluation cost of $0.02 is a notable result on the cost-efficiency axis. For institutions in which the acceptable performance threshold falls between 0.60 and 0.65, and where budgetary constraints are acute, the Flash model represents an option that is economically transformative relative to alternatives costing an order of magnitude more for comparable agreement scores.

What This Means for AI Trust

These findings bear directly on the problem that motivates Trust’s research programme. The central challenge in AI deployment is not establishing that models achieve acceptable average accuracy; it is enabling users to know, at the moment of an individual response, whether that particular output is reliable. Our results sharpen several dimensions of this problem.

Methodology

@misc{trust2026stemeval,
  author = {Trust Research},
  title  = {Can You Trust AI with a MIT Exam? STEMEval Benchmark v2.0},
  year   = {2026},
  month  = {March},
  url    = {https://www.trytrust.ai/research/stemeval}
}