What I did
I used twenty open-ended, thought-experiment style questions (for example “The Civilization Without Memory”, “The Perfect Prediction Machine”, “The World Without Lies”). Each asks for reasoning rather than facts: hidden assumptions, second-order effects, counter-arguments and the limits of what can be known. English is the original; the Ukrainian, Polish and Russian versions are translations of the same questions.
The full list of questions, in the original English:
- The Civilization Without Memory: Imagine a civilization where everyone loses their personal memories every 30 days but retains their knowledge and skills. What institution would become essential for its survival, and what unexpected problems would that institution create?
- The Perfect Prediction Machine: A machine predicts human decisions with 100% accuracy, provided its predictions remain secret. How would you design an experiment to determine whether it predicts decisions or secretly influences them?
- The Intelligence That Cannot Explain Itself: Two AI systems solve scientific problems. One is correct 95% of the time but cannot explain its reasoning. The other is correct 80% of the time and always provides convincing explanations. Which would you invest in, and what evidence could reverse your decision?
- The World Without Lies: Tomorrow, every human permanently loses the ability to deliberately communicate false information. What major institution would become less effective rather than more effective, and why?
- The Invisible Improvement: You discover a technology that improves everyone’s reasoning ability by 20%, but nobody can perceive the improvement in themselves or others. How would you demonstrate that the technology works without relying on conventional intelligence tests?
- The Society of Perfect Memory: Everyone remembers every experience perfectly, including emotions and insignificant details. What human capability might deteriorate as a direct consequence, and how would society compensate?
- The One-Minute Democracy: A country replaces elections with a system in which every citizen can change their political representative once per minute. What would happen to political decision-making over the next decade, and what rules would you introduce to prevent the worst consequences?
- The Disappearing Expert: An organization discovers that whenever an employee becomes exceptionally good at their job, their expertise automatically transfers to a randomly selected colleague, leaving the original employee with average skills. How should the organization redesign its incentives and training?
- The Perfectly Rational Stranger: You encounter someone who always makes decisions that maximize their long-term well-being, yet their behavior appears consistently irrational to everyone else. How could you determine whether they are genuinely rational without knowing their goals?
- The City That Learns: A city can automatically redesign its roads, buildings, and public spaces every night based on the previous day’s behavior. What should it optimize for, and how could optimizing that metric eventually make the city worse?
- The Price of Certainty: You can eliminate uncertainty from one area of human life—medicine, economics, relationships, or politics—but doing so doubles uncertainty in another area. Which trade-off would produce the greatest long-term benefit, and how would you evaluate it?
- The Civilization of Specialists: Every person can master exactly one skill to an extraordinary level but becomes incapable of learning anything outside that specialty. Could such a civilization outperform ours technologically? What would determine its limits?
- The Reverse Scientific Method: Scientists discover that every theory explaining an observation makes that observation slightly less likely to occur in the future. How would you redesign scientific research to distinguish genuine discoveries from effects caused by studying them?
- The Market Without Ownership: Imagine an economy where nobody can own anything for longer than 24 hours, but everyone can freely exchange goods, services, and temporary usage rights. Which economic activities would flourish, which would collapse, and what new institutions would emerge?
- The Knowledge That Cannot Be Shared: You discover a scientific principle that can be understood through experimentation but cannot be communicated through language, mathematics, images, or demonstrations. How could you organize a research community to build upon this discovery?
- The Perfectly Fair Algorithm: An algorithm distributes resources so that everyone agrees the distribution procedure is fair, yet inequality continuously increases. How would you investigate whether the problem lies in the algorithm, the definition of fairness, or the behavior of participants?
- The World With Delayed Consequences: Every action produces its normal consequences exactly ten years later, while people remain aware of what they did. How would this change education, justice, business, and personal responsibility? Which existing institution would struggle most?
- The Intelligence Exchange: People can temporarily exchange their reasoning abilities with others, but their memories, personalities, and knowledge remain unchanged. How would you design an experiment to separate intelligence from expertise, and what results would challenge your understanding of intelligence?
- The Civilization That Cannot Measure Time: Humanity loses every reliable way to measure time, including clocks, calendars, astronomical cycles, and biological rhythms. People can still perceive sequences of events. How would civilization coordinate complex activities, and which technologies would become impossible?
- The Last Unsolved Problem: Humanity develops an AI capable of solving every precisely formulated problem, but it cannot decide which problems are worth solving. What would become the most valuable human intellectual ability, how would you recognize it, and could that ability itself eventually be automated?
Every question, in every language, was sent once to six model configurations through their command-line tools:
- A – Claude Sonnet
- B – Codex Terra
- C – Agy Gemini Flash
- D – Claude Opus 5.5
- E – Codex Sol (high)
- F – Claude Haiku 5.5
That gives 20 questions × 4 languages × 6 models = 480 answers. The models received only the question text; length, format and language of the answer were left to them.
How it was measured
Because one language effect can hide behind another, the answers were examined along seven separate dimensions:
- D1 Length – characters and words per answer; the change versus English for the same model and the same question, corrected for the fact that the translated question itself is a different length.
- D2 Structure – headings, list items, bold spans and table rows per answer.
- D3 Language fidelity – does an answer stay in the language of the question? (Latin letters in Cyrillic text, letters that exist only in the neighbouring Slavic language, English function words.)
- D4 Quality – a judge model rated each answer on seven criteria from 0 to 5: depth of understanding, logical reasoning, creativity and originality, systems thinking, cognitive flexibility, clarity and precision, epistemic maturity. The total is out of 35.
- D5 Variance – how much of the variation in scores and lengths is explained by the question, the model, the language and the model × language interaction.
- D6 Consistency – whether model rankings and question difficulty stay the same across languages.
- D7 Judge reliability – whether the result is an artefact of the judge: agreement between judges, the language shift under each judge, and whether longer answers simply score higher.
Statistics. Every “change versus English” is a paired difference over the questions that have an answer in both languages (same model, same question). Confidence intervals are percentile bootstrap intervals with 10,000 resamples of the questions; a result is called distinguishable from zero when its 95% interval excludes zero. With 20 questions and one answer per cell, small effects cannot be detected, so model-level intervals are wide and the pooled estimates are more informative.
Judges. Answers were scored by three AI judges: codex (gpt-5.6-terra), claude (sonnet) and agy (gemini-3.8-flash). The codex and claude judges scored all 480 answers; agy scored only a part, so it is used as a cross-check only. The scoring prompt is the same for every language, and scores are attributed to a language by the text of the question, not by the file name.
Length: some models write less in every non-English language
Average answer length by language and model is shown below. The ordering of models is almost identical in every language (the verbose Opus and Agy at the top, the terse Codex Terra at the bottom; only Sonnet and Haiku swap places in Ukrainian), and English is the longest language for every model.

Table view: average answer length, words
| Model | English | Ukrainian | Polish | Russian |
|---|---|---|---|---|
| Claude Sonnet | 401 | 320 | 367 | 340 |
| Codex Terra | 158 | 134 | 155 | 153 |
| Agy Gemini Flash | 501 | 381 | 446 | 432 |
| Claude Opus 5.5 | 536 | 426 | 451 | 446 |
| Codex Sol (high) | 225 | 203 | 207 | 208 |
| Claude Haiku 5.5 | 354 | 333 | 338 | 330 |
Raw differences mix two things: the models writing less, and the translated question itself being shorter or longer. The Ukrainian question is on average 5.5% shorter than the English one in characters, the Polish one 2.8% longer. Dividing the answer ratio by the question ratio gives a length change that is independent of the translation. Pooled over all models it is −7.1% for Ukrainian, −6.8% for Polish and −5.4% for Russian (characters); every interval excludes zero.
Structure: formatting is a model trait
Formatting survives translation much better than answer length. The number of headings, lists, bold spans and tables is overwhelmingly a model trait rather than a language trait: Claude Opus 5.5 and Agy Gemini Flash consistently use the most headings, while Codex Terra and Codex Sol remain terse in every language. Translation did not systematically flatten or enrich the answers’ structure.
Language fidelity: no drift into English or Russian
All models stayed in the language of the question. No response drifted into English or into a neighbouring Slavic language. The only visible exception was Agy Gemini Flash, which occasionally retained Latin-script technical terms in Cyrillic answers; these accounted for roughly 2% of letters and did not alter the answer language.
Answer quality: small, model-dependent, partly length-mediated
On the primary codex judge’s 35-point scale, the pooled shift versus English was −0.68 points for Ukrainian, −0.38 for Polish and −0.76 for Russian. These are small differences relative to the scale and to variation between questions. The observed pattern was strongest for the models whose answers became shorter outside English.
| Language vs English | Mean score shift | 95% confidence interval |
|---|---|---|
| Ukrainian | −0.68 | [−1.18, −0.17] |
| Polish | −0.38 | [−0.89, +0.12] |
| Russian | −0.76 | [−1.27, −0.23] |
Longer answers score higher even after accounting for model and language (partial correlation of log length with total score: r = 0.29, p < 0.001). Adding answer length as a covariate shrinks the estimated penalties to −0.24 for Ukrainian, −0.17 for Polish and −0.47 for Russian. Roughly half of the apparent score loss outside English may therefore be a consequence of the judge rewarding length rather than a loss of reasoning quality.
Where the variance comes from
Language explains only 0.7% of score variance. By comparison, the model explains 30.7% and the question explains 30.4%; the model × language interaction explains 1.0% and is not distinguishable from noise. For answer length, model choice explains 60.9%, question 24.0%, and language 1.1%.
| Term | Score share | Length share |
|---|---|---|
| Question | 30.4% | 24.0% |
| Model | 30.7% | 60.9% |
| Language | 0.7% | 1.1% |
| Model × language | 1.0% | 0.4% |
| Unexplained | 37.1% | 13.5% |
Consistency: the same models win, the same questions are hard
Model rankings carry over almost perfectly between English and the other languages (rank correlation at least 0.94). Question difficulty is also stable across languages (correlation 0.85–0.91). Claude Opus 5.5 ranks first and Agy Gemini Flash last in every language. In other words, the language of the prompt does not meaningfully reorder the models or which questions they find difficult.
Judge reliability: is the language effect an artefact?
The judges correlate moderately, but they are not interchangeable. Claude’s scores correlate with codex at about r = 0.67 across the full-coverage languages and are on average roughly six points lower on the same answers. Crucially, both judges reproduce a Ukrainian drop, but they do not agree on Polish or Russian: claude reports −0.95 for Ukrainian, −0.04 for Polish and +0.03 for Russian. Therefore, only the Ukrainian result is robust across the two full-coverage judges.
Model profiles
| Model | English words | Translation-adjusted length change | Profile |
|---|---|---|---|
| Claude Opus 5.5 | 536 | −11.3% | language-sensitive |
| Agy Gemini Flash | 501 | −13.9% | language-sensitive |
| Claude Sonnet | 401 | −11.3% | language-sensitive |
| Claude Haiku 5.5 | 354 | +0.2% | language-robust |
| Codex Terra | 158 | −1.3% | language-robust |
| Codex Sol (high) | 225 | +0.5% | language-robust |
The language-sensitive group writes longer, more heavily formatted English answers and loses 10–19% of its text in the other languages. Codex Terra, Codex Sol and Claude Haiku have stable length and scores, making them safer when outputs must be comparable across languages.
Limitations and recommendations
- There is one answer per model, question and language, so sampling noise cannot be separated from the language effect.
- The translations were not independently validated; prompt register and difficulty may differ.
- Quality was judged by models, not humans, and the judges show a measurable verbosity bias.
- The scoring prompt was Ukrainian for every language and judge-family preference was not controlled.
- The 0–5 clarity criterion has a ceiling effect.
Practical advice. Normalise output length by translated-prompt length; report characters alongside words; treat differences under roughly one point out of 35 as noise at this sample size; ask explicitly for the desired length and structure when completeness matters; and use more than one judge.
Summary
- Length is the main language effect. After prompt-length correction, answers are 7.1% shorter in Ukrainian, 6.8% in Polish and 5.4% in Russian.
- Structure and language fidelity survive translation. Formatting is a model trait, and no answer switched languages.
- The quality penalty is small and judge-dependent. Only Ukrainian shows a consistent drop across the two complete judge runs.
- Language explains very little of the variance. Model and question matter far more.
- Part of the observed penalty is verbosity bias. Controlling for length shrinks language penalties by 38–64%.
- Rankings and question difficulty carry over. The same models lead and the same prompts remain hard.
What this study cannot say. It cannot cleanly separate sampling noise, translation effects, judge bias and genuine language effects on reasoning. The next useful study would repeat prompts, validate translations, include human ratings, and compare original and back-translated answers.
Additional figures and supporting tables

| Model | Ukrainian | Polish | Russian |
|---|---|---|---|
| Claude Sonnet | −20.2% | −8.5% | −15.1% |
| Codex Terra | −7.8% | −0.6% | −1.5% |
| Agy Gemini Flash | −20.7% | −13.7% | −7.4% |
| Claude Opus 5.5 | −17.2% | −12.2% | −12.8% |
| Codex Sol (high) | −4.2% | +1.1% | +4.6% |
| Claude Haiku 5.5 | −0.4% | +1.5% | −0.4% |

| Model | Ukrainian | Polish | Russian |
|---|---|---|---|
| Claude Sonnet | −1.30 [−2.4, −0.1] | −0.65 [−1.6, +0.3] | −1.05 [−1.9, −0.1] |
| Codex Terra | −0.55 [−1.2, +0.1] | +0.10 [−0.6, +0.8] | +0.55 [−0.8, +2.0] |
| Agy Gemini Flash | −0.35 [−2.0, +1.2] | −0.25 [−1.8, +1.4] | −1.60 [−3.1, −0.1] |
| Claude Opus 5.5 | −0.95 [−1.9, −0.1] | −0.90 [−1.7, +0.0] | −1.70 [−2.7, −0.8] |
| Codex Sol (high) | −0.40 [−1.4, +0.7] | +0.00 [−1.2, +1.2] | −0.45 [−1.6, +0.8] |
| Claude Haiku 5.5 | −0.50 [−2.2, +1.1] | −0.60 [−2.4, +1.1] | −0.30 [−1.5, +1.1] |

| Criterion | Ukrainian | Polish | Russian |
|---|---|---|---|
| Depth of Understanding | −0.15 [−0.25, −0.05] | −0.12 [−0.23, −0.03] | −0.12 [−0.23, −0.03] |
| Logical Reasoning | −0.08 [−0.21, +0.04] | −0.05 [−0.17, +0.07] | −0.03 [−0.15, +0.08] |
| Creativity & Originality | −0.09 [−0.19, +0.01] | −0.07 [−0.17, +0.04] | −0.09 [−0.20, +0.02] |
| Systems Thinking | −0.09 [−0.19, +0.01] | −0.03 [−0.14, +0.07] | −0.07 [−0.18, +0.03] |
| Cognitive Flexibility | −0.14 [−0.30, +0.02] | −0.06 [−0.19, +0.07] | −0.21 [−0.36, −0.06] |
| Clarity & Precision | −0.04 [−0.10, +0.02] | −0.01 [−0.07, +0.05] | −0.08 [−0.14, −0.03] |
| Epistemic Maturity | −0.07 [−0.20, +0.05] | −0.04 [−0.17, +0.09] | −0.14 [−0.28, −0.01] |

| Judge | Language vs English | n | Mean shift (95% CI) |
|---|---|---|---|
| codex | Ukrainian | 120 | −0.68 [−1.18, −0.17] |
| codex | Polish | 120 | −0.38 [−0.89, +0.12] |
| codex | Russian | 120 | −0.76 [−1.27, −0.23] |
| claude | Ukrainian | 119 | −0.95 [−1.43, −0.48] |
| claude | Polish | 119 | −0.04 [−0.48, +0.40] |
| claude | Russian | 119 | +0.03 [−0.45, +0.53] |
Full table views
Change in answer length versus English, translation-adjusted
Characters, paired by question and adjusted for the translated prompt length. Red = shorter; blue = longer.
| Model | Ukrainian | Polish | Russian |
|---|---|---|---|
| Claude Sonnet | −13% | −10% | −12% |
| Codex Terra | −6% | 0% | +2% |
| Agy Gemini Flash | −19% | −12% | −11% |
| Claude Opus 5.5 | −11% | −13% | −11% |
| Codex Sol (high) | +2% | −1% | 0% |
| Claude Haiku 5.5 | +6% | −4% | −1% |
Headings / list items / bold spans per answer
| Model | English | Ukrainian | Polish | Russian |
|---|---|---|---|---|
| Claude Sonnet | 4.2 / 13.9 / 15.4 | 4.5 / 14.2 / 14.2 | 4.9 / 15.3 / 16.4 | 4.1 / 14.8 / 13.5 |
| Codex Terra | 0.0 / 5.4 / 1.2 | 0.0 / 5.8 / 0.7 | 0.0 / 6.0 / 1.7 | 0.0 / 5.6 / 0.0 |
| Agy Gemini Flash | 7.0 / 18.4 / 21.8 | 6.2 / 15.5 / 15.7 | 6.5 / 16.4 / 19.4 | 7.0 / 17.2 / 17.9 |
| Claude Opus 5.5 | 6.0 / 14.1 / 18.4 | 6.7 / 13.0 / 17.0 | 6.8 / 14.2 / 16.6 | 6.0 / 13.2 / 16.8 |
| Codex Sol (high) | 0.1 / 8.3 / 4.2 | 0.6 / 9.0 / 5.0 | 0.2 / 8.3 / 4.8 | 0.2 / 8.6 / 3.3 |
| Claude Haiku 5.5 | 1.8 / 9.1 / 11.3 | 2.6 / 9.6 / 13.7 | 2.8 / 11.9 / 14.0 | 2.5 / 9.9 / 12.9 |
Fidelity checks
| Model | Latin letters in Ukrainian | Latin letters in Russian | Wrong-language letters / 1,000 | English function words / 1,000 (uk / pl / ru) |
|---|---|---|---|---|
| Claude Sonnet | 0.6% | 0.3% | 0.0 | 0.0 / 0.1 / 0.0 |
| Codex Terra | 0.1% | 0.0% | 0.0 | 0.0 / 0.0 / 0.0 |
| Agy Gemini Flash | 2.5% | 2.2% | 0.0 | 0.8 / 0.8 / 0.6 |
| Claude Opus 5.5 | 0.2% | 0.3% | 0.0 | 0.0 / 0.0 / 0.0 |
| Codex Sol (high) | 0.1% | 0.1% | 0.0 | 0.0 / 0.0 / 0.0 |
| Claude Haiku 5.5 | 0.6% | 0.6% | 0.0 | 0.0 / 0.1 / 0.2 |
Mean total score by model and language
Higher scores are shown in deeper blue.
| Model | English | Ukrainian | Polish | Russian |
|---|---|---|---|---|
| Claude Sonnet | 30.60 | 29.30 | 29.95 | 29.55 |
| Codex Terra | 27.60 | 27.05 | 27.70 | 28.15 |
| Agy Gemini Flash | 25.70 | 25.35 | 25.45 | 24.10 |
| Claude Opus 5.5 | 32.10 | 31.15 | 31.20 | 30.40 |
| Codex Sol (high) | 29.90 | 29.50 | 29.90 | 29.45 |
| Claude Haiku 5.5 | 29.15 | 28.65 | 28.55 | 28.85 |
Question × language
Mean total score (max 35) over the models, per question. Darker is higher.
| Question | English | Ukrainian | Polish | Russian |
|---|---|---|---|---|
| 1. The Civilization Without Memory | 29.8 | 30.2 | 28.7 | 29.3 |
| 2. The Perfect Prediction Machine | 29.2 | 28.3 | 29.2 | 29.5 |
| 3. The Intelligence That Cannot Explain Itself | 32.3 | 32.0 | 32.7 | 30.5 |
| 4. The World Without Lies | 24.8 | 24.7 | 26.0 | 24.2 |
| 5. The Invisible Improvement | 29.0 | 27.8 | 27.5 | 29.0 |
| 6. The Society of Perfect Memory | 26.8 | 25.7 | 25.3 | 26.3 |
| 7. The One-Minute Democracy | 29.0 | 30.0 | 29.7 | 28.2 |
| 8. The Disappearing Expert | 27.7 | 26.0 | 27.0 | 25.5 |
| 9. The Perfectly Rational Stranger | 26.7 | 27.3 | 28.3 | 27.0 |
| 10. The City That Learns | 31.5 | 31.0 | 32.0 | 30.7 |
| 11. The Price of Certainty | 29.3 | 28.0 | 27.5 | 27.0 |
| 12. The Civilization of Specialists | 29.7 | 28.5 | 28.8 | 28.0 |
| 13. The Reverse Scientific Method | 30.5 | 29.0 | 31.3 | 30.7 |
| 14. The Market Without Ownership | 30.7 | 29.3 | 28.8 | 30.5 |
| 15. The Knowledge That Cannot Be Shared | 30.5 | 28.0 | 28.8 | 30.3 |
| 16. The Perfectly Fair Algorithm | 31.3 | 31.0 | 30.0 | 31.5 |
| 17. The World With Delayed Consequences | 27.2 | 26.7 | 26.5 | 24.3 |
| 18. The Intelligence Exchange | 31.2 | 31.3 | 31.8 | 30.7 |
| 19. The Civilization That Cannot Measure Time | 26.7 | 25.5 | 27.0 | 27.0 |
| 20. The Last Unsolved Problem | 29.7 | 29.7 | 28.8 | 28.2 |
Question × model
Mean total score over the scored languages, per question and model. A dash means no scores.
| Question | Claude Sonnet | Codex Terra | Agy Gemini Flash | Claude Opus 5.5 | Codex Sol (high) | Claude Haiku 5.5 |
|---|---|---|---|---|---|---|
| 1. The Civilization Without Memory | 30.0 | 27.5 | 29.2 | 32.5 | 29.0 | 28.8 |
| 2. The Perfect Prediction Machine | 30.5 | 29.8 | 23.2 | 31.2 | 30.5 | 29.0 |
| 3. The Intelligence That Cannot Explain Itself | 32.8 | 31.5 | 30.2 | 33.0 | 31.5 | 32.2 |
| 4. The World Without Lies | 27.0 | 23.8 | 21.2 | 27.5 | 23.8 | 26.2 |
| 5. The Invisible Improvement | 30.2 | 26.5 | 24.0 | 29.2 | 29.0 | 31.0 |
| 6. The Society of Perfect Memory | 27.2 | 23.8 | 24.0 | 28.5 | 25.2 | 27.5 |
| 7. The One-Minute Democracy | 28.8 | 29.8 | 27.0 | 31.2 | 30.8 | 27.8 |
| 8. The Disappearing Expert | 26.5 | 25.8 | 25.2 | 30.0 | 28.2 | 23.5 |
| 9. The Perfectly Rational Stranger | 27.5 | 28.5 | 22.5 | 28.5 | 28.5 | 28.5 |
| 10. The City That Learns | 33.2 | 28.2 | 29.2 | 33.5 | 31.0 | 32.5 |
| 11. The Price of Certainty | 28.8 | 27.0 | 23.0 | 31.0 | 29.0 | 29.0 |
| 12. The Civilization of Specialists | 28.0 | 27.5 | 27.5 | 31.0 | 31.0 | 27.5 |
| 13. The Reverse Scientific Method | 31.8 | 28.5 | 25.8 | 32.5 | 33.2 | 30.5 |
| 14. The Market Without Ownership | 32.0 | 29.2 | 25.2 | 31.8 | 31.2 | 29.5 |
| 15. The Knowledge That Cannot Be Shared | 32.5 | 27.8 | 24.5 | 33.0 | 30.5 | 28.2 |
| 16. The Perfectly Fair Algorithm | 32.8 | 29.0 | 25.5 | 34.0 | 32.2 | 32.2 |
| 17. The World With Delayed Consequences | 27.0 | 26.5 | 21.5 | 29.8 | 25.2 | 27.0 |
| 18. The Intelligence Exchange | 33.0 | 28.5 | 27.0 | 34.0 | 33.2 | 31.8 |
| 19. The Civilization That Cannot Measure Time | 26.5 | 25.8 | 22.0 | 30.5 | 30.2 | 24.2 |
| 20. The Last Unsolved Problem | 31.0 | 27.8 | 25.0 | 31.5 | 30.2 | 29.0 |
Dimension × language
Mean score (0–5) on each criterion, over models and questions. Darker is higher.
| Dimension | English | Ukrainian | Polish | Russian |
|---|---|---|---|---|
| Depth of Understanding | 4.4 | 4.2 | 4.2 | 4.2 |
| Logical Reasoning | 4.2 | 4.1 | 4.1 | 4.1 |
| Creativity & Originality | 3.7 | 3.6 | 3.6 | 3.6 |
| Systems Thinking | 4.4 | 4.3 | 4.3 | 4.3 |
| Cognitive Flexibility | 3.7 | 3.5 | 3.6 | 3.5 |
| Clarity & Precision | 4.9 | 4.8 | 4.9 | 4.8 |
| Epistemic Maturity | 4.0 | 3.9 | 3.9 | 3.8 |
Dimension × model
Mean score (0–5) on each criterion, over the scored languages and questions.
| Dimension | Claude Sonnet | Codex Terra | Agy Gemini Flash | Claude Opus 5.5 | Codex Sol (high) | Claude Haiku 5.5 |
|---|---|---|---|---|---|---|
| Depth of Understanding | 4.4 | 4.0 | 4.0 | 4.7 | 4.2 | 4.2 |
| Logical Reasoning | 4.2 | 4.3 | 3.3 | 4.3 | 4.6 | 4.1 |
| Creativity & Originality | 3.7 | 3.2 | 3.9 | 4.2 | 3.6 | 3.4 |
| Systems Thinking | 4.5 | 4.0 | 4.1 | 4.7 | 4.3 | 4.3 | Cognitive Flexibility | 4.0 | 3.3 | 2.7 | 4.1 | 3.7 | 3.7 |
| Clarity & Precision | 5.0 | 5.0 | 4.3 | 5.0 | 5.0 | 5.0 |
| Epistemic Maturity | 4.2 | 3.9 | 2.9 | 4.2 | 4.2 | 4.1 |
Question × dimension
Mean score (0–5) per question and criterion, over models and scored languages.
| Question | Depth | Logical | Creativity | Systems | Cognitive | Clarity | Epistemic |
|---|---|---|---|---|---|---|---|
| 1. The Civilization Without Memory | 4.4 | 4.4 | 4.2 | 4.7 | 3.3 | 5.0 | 3.5 |
| 2. The Perfect Prediction Machine | 4.3 | 3.8 | 3.6 | 4.1 | 4.0 | 4.8 | 4.3 |
| 3. The Intelligence That Cannot Explain Itself | 4.3 | 4.8 | 3.6 | 4.4 | 4.9 | 4.9 | 4.9 |
| 4. The World Without Lies | 3.7 | 3.6 | 3.3 | 3.7 | 2.8 | 4.7 | 3.1 |
| 5. The Invisible Improvement | 4.2 | 4.5 | 3.0 | 3.8 | 3.5 | 4.8 | 4.5 |
| 6. The Society of Perfect Memory | 3.9 | 3.8 | 3.7 | 4.0 | 2.6 | 4.9 | 3.1 |
| 7. The One-Minute Democracy | 4.2 | 4.3 | 3.8 | 4.8 | 3.5 | 4.9 | 3.8 |
| 8. The Disappearing Expert | 4.0 | 3.9 | 3.5 | 4.3 | 2.8 | 4.9 | 3.2 |
| 9. The Perfectly Rational Stranger | 4.1 | 4.0 | 3.0 | 3.2 | 3.8 | 4.9 | 4.4 |
| 10. The City That Learns | 4.7 | 4.7 | 3.9 | 4.9 | 4.0 | 5.0 | 4.2 |
| 11. The Price of Certainty | 4.1 | 4.0 | 3.1 | 4.2 | 3.7 | 4.9 | 4.0 |
| 12. The Civilization of Specialists | 4.3 | 4.1 | 3.6 | 4.7 | 3.6 | 4.9 | 3.5 |
| 13. The Reverse Scientific Method | 4.5 | 4.2 | 4.2 | 4.5 | 3.8 | 4.8 | 4.2 |
| 14. The Market Without Ownership | 4.4 | 4.2 | 3.9 | 4.7 | 4.0 | 4.9 | 3.8 |
| 15. The Knowledge That Cannot Be Shared | 4.3 | 4.1 | 3.9 | 4.6 | 3.2 | 4.9 | 4.3 |
| 16. The Perfectly Fair Algorithm | 4.5 | 4.6 | 3.7 | 4.8 | 4.2 | 4.9 | 4.3 |
| 17. The World With Delayed Consequences | 4.0 | 3.5 | 3.8 | 4.1 | 3.0 | 4.7 | 3.1 |
| 18. The Intelligence Exchange | 4.6 | 4.5 | 4.0 | 4.3 | 4.5 | 4.8 | 4.5 |
| 19. The Civilization That Cannot Measure Time | 4.1 | 3.5 | 3.6 | 4.2 | 2.9 | 4.8 | 3.4 |
| 20. The Last Unsolved Problem | 4.5 | 4.2 | 3.6 | 4.1 | 3.8 | 4.8 | 4.0 |
What the experiment shows
Prompt language matters most for how much a model says, not for whether it can reason through the prompt at all. The three most verbose models produce noticeably shorter answers outside English; the concise models remain comparatively stable.
Model choice and question choice explain far more of the observed variation than language. Formatting habits, model rankings and which questions are hard transfer well across English, Ukrainian, Polish and Russian. The answers also stay in the requested language.
The small quality differences should be treated carefully. Ukrainian is the only non-English language with a drop that appears across both complete judge runs; Polish and Russian depend more on which judge is used. Since judges reward longer answers to some extent, part of the apparent quality penalty is likely a verbosity effect.
Practical takeaway: use the language your audience needs, but specify target length and structure when comparability matters. For high-stakes evaluations, normalize for prompt and answer length, use multiple judges, and do not mistake a shorter response for weaker reasoning.