Each contestant was asked to start from the Einstein field equations Gμν = 0 (vacuum) and derive the Schwarzschild radius from first principles, assuming spherical symmetry and staticity of the metric.
The canonical path: write the most general static, spherically symmetric line element; compute the Christoffel symbols; evaluate the non-trivial Ricci tensor components; impose Rμν = 0; integrate; match to the weak-field Newtonian limit to fix the integration constant. The coordinate-singular surface gtt(r) = 0 gives the Schwarzschild radius r_s = 2GM / c².
This is textbook general relativity — the kind of derivation every graduate student runs through before they're let near anything harder. Every one of the seven contestants reached the correct final formula. What distinguishes them is the integrity of the intermediate algebra.
| Model | Response Length | Claims | Survival Rate | Confirmed / Flagged |
|---|---|---|---|---|
| Claude | 5,918 ch | 52 | 0.903 | 1 / 3 |
| GPT-5.4 | 4,861 ch | 52 | 0.946 | 0 / 2 |
| o3-mini | 4,070 ch | 35 | 1.000 | 0 / 0 |
| Gemini | 5,140 ch | 40 | 0.842 | 0 / 3 |
| DeepSeek R1 | 5,003 ch | 40 | 0.931 | 0 / 2 |
| Qwen | 4,492 ch | 42 | 0.926 | 2 / 2 |
| Grok | 4,730 ch | 38 | 0.917 | 1 / 2 |
Of 14 errors flagged by single adversaries, 4 held up under independent re-review by a second adversary from a different model family. Here are the real physics errors that survived cross-family confirmation.
Claude's expression for the Ricci tensor component R_rr came out off by an overall factor of 2 compared to the standard derivation. The error propagates into the algebra that fixes eλ but happens to cancel by the final step — the formula still lands on r_s = 2GM/c², which is why a final-answer check misses it.
Qwen wrote eλ = 1 / (1 + k/r) with the wrong sign of the integration constant. The standard derivation yields 1 / (1 - C/r); the sign choice depends on how the integration constant is defined relative to the mass parameter, but Qwen's statement conflicts with its own subsequent identification C = 2GM/c².
A second independently-confirmed Qwen claim in the same derivation — reviewers from two different families flagged an inconsistency in the step that integrates the vacuum field equation to yield the metric coefficients. See raw Layer 4/4b output on GitHub for the full verdict text; the error is of the same "sign-or-factor-drift-that-cancels" family as the first one.
Grok invoked R_tr to derive the constraint ∂_r(A'/A + B'/B) = 0. The problem: for a static, spherically symmetric metric of the form Grok wrote, R_tr vanishes identically — no such constraint arises from it. The needed relationship comes from the combination of R_tt and R_rr, not from an off-diagonal component that was zero to begin with.
Read this before citing any number on this page. These limitations are ordered roughly by how much they should move your priors.
Every verdict in this pipeline is produced by an LLM reading LaTeX. There is no SymPy check, no theorem prover. "Confirmed error" means two LLMs from different families independently flagged a step as wrong; it does not mean the step is mathematically wrong. Layer 3 (symbolic verification) is the top roadmap item.
Running Layer 4b twice on the same output can produce different confirmation counts due to LLM sampling variance (observed ±20% across reruns). Reported numbers are single-run.
Approximately 33% of Layer 4b calls returned unparseable responses (primarily from o3-mini on OpenRouter). Parse failures are treated as non-confirmations — which systematically under-counts real errors.
Confirmed errors should be manually spot-checked before use. Not all "confirmed" errors are unambiguous physics errors — some are notation or convention disagreements between adversaries.
The final-answer check is a regex for the formula the problem statement asked the model to derive. Passing this check means the model produced a matching formula, not that the derivation constitutes a valid proof.
Every Layer 1 response, the full claim decomposition, every Layer 4 verdict, and every Layer 4b confirmation for this problem is on GitHub:
github.com/themultivac/multivac-evaluation/tree/main/data/physics_synthesis/PHYSICS-SYNTH-schwarzschild-v0-20260418-160451If you find an error in our methodology or our physics, email yash@themultivac.com. We would rather be corrected than be wrong.