15.11.2026  |  links

Does it matter in which language you ask an AI model a question?

—————

What I did

I used twenty open-ended, thought-experiment style questions (for example “The Civilization Without Memory”, “The Perfect Prediction Machine”, “The World Without Lies”). Each asks for reasoning rather than facts: hidden assumptions, second-order effects, counter-arguments and the limits of what can be known. English is the original; the Ukrainian, Polish and Russian versions are translations of the same questions.

The full list of questions, in the original English:

  1. The Civilization Without Memory: Imagine a civilization where everyone loses their personal memories every 30 days but retains their knowledge and skills. What institution would become essential for its survival, and what unexpected problems would that institution create?
  2. The Perfect Prediction Machine: A machine predicts human decisions with 100% accuracy, provided its predictions remain secret. How would you design an experiment to determine whether it predicts decisions or secretly influences them?
  3. The Intelligence That Cannot Explain Itself: Two AI systems solve scientific problems. One is correct 95% of the time but cannot explain its reasoning. The other is correct 80% of the time and always provides convincing explanations. Which would you invest in, and what evidence could reverse your decision?
  4. The World Without Lies: Tomorrow, every human permanently loses the ability to deliberately communicate false information. What major institution would become less effective rather than more effective, and why?
  5. The Invisible Improvement: You discover a technology that improves everyone’s reasoning ability by 20%, but nobody can perceive the improvement in themselves or others. How would you demonstrate that the technology works without relying on conventional intelligence tests?
  6. The Society of Perfect Memory: Everyone remembers every experience perfectly, including emotions and insignificant details. What human capability might deteriorate as a direct consequence, and how would society compensate?
  7. The One-Minute Democracy: A country replaces elections with a system in which every citizen can change their political representative once per minute. What would happen to political decision-making over the next decade, and what rules would you introduce to prevent the worst consequences?
  8. The Disappearing Expert: An organization discovers that whenever an employee becomes exceptionally good at their job, their expertise automatically transfers to a randomly selected colleague, leaving the original employee with average skills. How should the organization redesign its incentives and training?
  9. The Perfectly Rational Stranger: You encounter someone who always makes decisions that maximize their long-term well-being, yet their behavior appears consistently irrational to everyone else. How could you determine whether they are genuinely rational without knowing their goals?
  10. The City That Learns: A city can automatically redesign its roads, buildings, and public spaces every night based on the previous day’s behavior. What should it optimize for, and how could optimizing that metric eventually make the city worse?
  11. The Price of Certainty: You can eliminate uncertainty from one area of human life—medicine, economics, relationships, or politics—but doing so doubles uncertainty in another area. Which trade-off would produce the greatest long-term benefit, and how would you evaluate it?
  12. The Civilization of Specialists: Every person can master exactly one skill to an extraordinary level but becomes incapable of learning anything outside that specialty. Could such a civilization outperform ours technologically? What would determine its limits?
  13. The Reverse Scientific Method: Scientists discover that every theory explaining an observation makes that observation slightly less likely to occur in the future. How would you redesign scientific research to distinguish genuine discoveries from effects caused by studying them?
  14. The Market Without Ownership: Imagine an economy where nobody can own anything for longer than 24 hours, but everyone can freely exchange goods, services, and temporary usage rights. Which economic activities would flourish, which would collapse, and what new institutions would emerge?
  15. The Knowledge That Cannot Be Shared: You discover a scientific principle that can be understood through experimentation but cannot be communicated through language, mathematics, images, or demonstrations. How could you organize a research community to build upon this discovery?
  16. The Perfectly Fair Algorithm: An algorithm distributes resources so that everyone agrees the distribution procedure is fair, yet inequality continuously increases. How would you investigate whether the problem lies in the algorithm, the definition of fairness, or the behavior of participants?
  17. The World With Delayed Consequences: Every action produces its normal consequences exactly ten years later, while people remain aware of what they did. How would this change education, justice, business, and personal responsibility? Which existing institution would struggle most?
  18. The Intelligence Exchange: People can temporarily exchange their reasoning abilities with others, but their memories, personalities, and knowledge remain unchanged. How would you design an experiment to separate intelligence from expertise, and what results would challenge your understanding of intelligence?
  19. The Civilization That Cannot Measure Time: Humanity loses every reliable way to measure time, including clocks, calendars, astronomical cycles, and biological rhythms. People can still perceive sequences of events. How would civilization coordinate complex activities, and which technologies would become impossible?
  20. The Last Unsolved Problem: Humanity develops an AI capable of solving every precisely formulated problem, but it cannot decide which problems are worth solving. What would become the most valuable human intellectual ability, how would you recognize it, and could that ability itself eventually be automated?

Every question, in every language, was sent once to six model configurations through their command-line tools:

  • A – Claude Sonnet
  • B – Codex Terra
  • C – Agy Gemini Flash
  • D – Claude Opus 5.5
  • E – Codex Sol (high)
  • F – Claude Haiku 5.5

That gives 20 questions × 4 languages × 6 models = 480 answers. The models received only the question text; length, format and language of the answer were left to them.

How it was measured

Because one language effect can hide behind another, the answers were examined along seven separate dimensions:

  • D1 Length – characters and words per answer; the change versus English for the same model and the same question, corrected for the fact that the translated question itself is a different length.
  • D2 Structure – headings, list items, bold spans and table rows per answer.
  • D3 Language fidelity – does an answer stay in the language of the question? (Latin letters in Cyrillic text, letters that exist only in the neighbouring Slavic language, English function words.)
  • D4 Quality – a judge model rated each answer on seven criteria from 0 to 5: depth of understanding, logical reasoning, creativity and originality, systems thinking, cognitive flexibility, clarity and precision, epistemic maturity. The total is out of 35.
  • D5 Variance – how much of the variation in scores and lengths is explained by the question, the model, the language and the model × language interaction.
  • D6 Consistency – whether model rankings and question difficulty stay the same across languages.
  • D7 Judge reliability – whether the result is an artefact of the judge: agreement between judges, the language shift under each judge, and whether longer answers simply score higher.

Statistics. Every “change versus English” is a paired difference over the questions that have an answer in both languages (same model, same question). Confidence intervals are percentile bootstrap intervals with 10,000 resamples of the questions; a result is called distinguishable from zero when its 95% interval excludes zero. With 20 questions and one answer per cell, small effects cannot be detected, so model-level intervals are wide and the pooled estimates are more informative.

Judges. Answers were scored by three AI judges: codex (gpt-5.6-terra), claude (sonnet) and agy (gemini-3.8-flash). The codex and claude judges scored all 480 answers; agy scored only a part, so it is used as a cross-check only. The scoring prompt is the same for every language, and scores are attributed to a language by the text of the question, not by the file name.

Length: some models write less in every non-English language

Average answer length by language and model is shown below. The ordering of models is almost identical in every language (the verbose Opus and Agy at the top, the terse Codex Terra at the bottom; only Sonnet and Haiku swap places in Ukrainian), and English is the longest language for every model.

Average answer length, words
Table view: average answer length, words
ModelEnglishUkrainianPolishRussian
Claude Sonnet401320367340
Codex Terra158134155153
Agy Gemini Flash501381446432
Claude Opus 5.5536426451446
Codex Sol (high)225203207208
Claude Haiku 5.5354333338330

Raw differences mix two things: the models writing less, and the translated question itself being shorter or longer. The Ukrainian question is on average 5.5% shorter than the English one in characters, the Polish one 2.8% longer. Dividing the answer ratio by the question ratio gives a length change that is independent of the translation. Pooled over all models it is −7.1% for Ukrainian, −6.8% for Polish and −5.4% for Russian (characters); every interval excludes zero.

Structure: formatting is a model trait

Formatting survives translation much better than answer length. The number of headings, lists, bold spans and tables is overwhelmingly a model trait rather than a language trait: Claude Opus 5.5 and Agy Gemini Flash consistently use the most headings, while Codex Terra and Codex Sol remain terse in every language. Translation did not systematically flatten or enrich the answers’ structure.

Language fidelity: no drift into English or Russian

All models stayed in the language of the question. No response drifted into English or into a neighbouring Slavic language. The only visible exception was Agy Gemini Flash, which occasionally retained Latin-script technical terms in Cyrillic answers; these accounted for roughly 2% of letters and did not alter the answer language.

Answer quality: small, model-dependent, partly length-mediated

On the primary codex judge’s 35-point scale, the pooled shift versus English was −0.68 points for Ukrainian, −0.38 for Polish and −0.76 for Russian. These are small differences relative to the scale and to variation between questions. The observed pattern was strongest for the models whose answers became shorter outside English.

Language vs EnglishMean score shift95% confidence interval
Ukrainian−0.68[−1.18, −0.17]
Polish−0.38[−0.89, +0.12]
Russian−0.76[−1.27, −0.23]

Longer answers score higher even after accounting for model and language (partial correlation of log length with total score: r = 0.29, p < 0.001). Adding answer length as a covariate shrinks the estimated penalties to −0.24 for Ukrainian, −0.17 for Polish and −0.47 for Russian. Roughly half of the apparent score loss outside English may therefore be a consequence of the judge rewarding length rather than a loss of reasoning quality.

Where the variance comes from

Language explains only 0.7% of score variance. By comparison, the model explains 30.7% and the question explains 30.4%; the model × language interaction explains 1.0% and is not distinguishable from noise. For answer length, model choice explains 60.9%, question 24.0%, and language 1.1%.

TermScore shareLength share
Question30.4%24.0%
Model30.7%60.9%
Language0.7%1.1%
Model × language1.0%0.4%
Unexplained37.1%13.5%

Consistency: the same models win, the same questions are hard

Model rankings carry over almost perfectly between English and the other languages (rank correlation at least 0.94). Question difficulty is also stable across languages (correlation 0.85–0.91). Claude Opus 5.5 ranks first and Agy Gemini Flash last in every language. In other words, the language of the prompt does not meaningfully reorder the models or which questions they find difficult.

Judge reliability: is the language effect an artefact?

The judges correlate moderately, but they are not interchangeable. Claude’s scores correlate with codex at about r = 0.67 across the full-coverage languages and are on average roughly six points lower on the same answers. Crucially, both judges reproduce a Ukrainian drop, but they do not agree on Polish or Russian: claude reports −0.95 for Ukrainian, −0.04 for Polish and +0.03 for Russian. Therefore, only the Ukrainian result is robust across the two full-coverage judges.

Model profiles

ModelEnglish wordsTranslation-adjusted length changeProfile
Claude Opus 5.5536−11.3%language-sensitive
Agy Gemini Flash501−13.9%language-sensitive
Claude Sonnet401−11.3%language-sensitive
Claude Haiku 5.5354+0.2%language-robust
Codex Terra158−1.3%language-robust
Codex Sol (high)225+0.5%language-robust

The language-sensitive group writes longer, more heavily formatted English answers and loses 10–19% of its text in the other languages. Codex Terra, Codex Sol and Claude Haiku have stable length and scores, making them safer when outputs must be comparable across languages.

Limitations and recommendations

  • There is one answer per model, question and language, so sampling noise cannot be separated from the language effect.
  • The translations were not independently validated; prompt register and difficulty may differ.
  • Quality was judged by models, not humans, and the judges show a measurable verbosity bias.
  • The scoring prompt was Ukrainian for every language and judge-family preference was not controlled.
  • The 0–5 clarity criterion has a ceiling effect.

Practical advice. Normalise output length by translated-prompt length; report characters alongside words; treat differences under roughly one point out of 35 as noise at this sample size; ask explicitly for the desired length and structure when completeness matters; and use more than one judge.

Summary

  1. Length is the main language effect. After prompt-length correction, answers are 7.1% shorter in Ukrainian, 6.8% in Polish and 5.4% in Russian.
  2. Structure and language fidelity survive translation. Formatting is a model trait, and no answer switched languages.
  3. The quality penalty is small and judge-dependent. Only Ukrainian shows a consistent drop across the two complete judge runs.
  4. Language explains very little of the variance. Model and question matter far more.
  5. Part of the observed penalty is verbosity bias. Controlling for length shrinks language penalties by 38–64%.
  6. Rankings and question difficulty carry over. The same models lead and the same prompts remain hard.

What this study cannot say. It cannot cleanly separate sampling noise, translation effects, judge bias and genuine language effects on reasoning. The next useful study would repeat prompts, validate translations, include human ratings, and compare original and back-translated answers.

Additional figures and supporting tables

Translation-adjusted change in answer length by language and model, with 95% bootstrap intervals.
Translation-adjusted change in answer length. Filled dots indicate intervals that exclude zero.
ModelUkrainianPolishRussian
Claude Sonnet−20.2%−8.5%−15.1%
Codex Terra−7.8%−0.6%−1.5%
Agy Gemini Flash−20.7%−13.7%−7.4%
Claude Opus 5.5−17.2%−12.2%−12.8%
Codex Sol (high)−4.2%+1.1%+4.6%
Claude Haiku 5.5−0.4%+1.5%−0.4%
Change in total judged score versus English by language and model, with 95% bootstrap intervals.
Change in total quality score versus English, points out of 35.
ModelUkrainianPolishRussian
Claude Sonnet−1.30 [−2.4, −0.1]−0.65 [−1.6, +0.3]−1.05 [−1.9, −0.1]
Codex Terra−0.55 [−1.2, +0.1]+0.10 [−0.6, +0.8]+0.55 [−0.8, +2.0]
Agy Gemini Flash−0.35 [−2.0, +1.2]−0.25 [−1.8, +1.4]−1.60 [−3.1, −0.1]
Claude Opus 5.5−0.95 [−1.9, −0.1]−0.90 [−1.7, +0.0]−1.70 [−2.7, −0.8]
Codex Sol (high)−0.40 [−1.4, +0.7]+0.00 [−1.2, +1.2]−0.45 [−1.6, +0.8]
Claude Haiku 5.5−0.50 [−2.2, +1.1]−0.60 [−2.4, +1.1]−0.30 [−1.5, +1.1]
Change per quality criterion versus English, all models pooled.
Change per criterion versus English.
CriterionUkrainianPolishRussian
Depth of Understanding−0.15 [−0.25, −0.05]−0.12 [−0.23, −0.03]−0.12 [−0.23, −0.03]
Logical Reasoning−0.08 [−0.21, +0.04]−0.05 [−0.17, +0.07]−0.03 [−0.15, +0.08]
Creativity & Originality−0.09 [−0.19, +0.01]−0.07 [−0.17, +0.04]−0.09 [−0.20, +0.02]
Systems Thinking−0.09 [−0.19, +0.01]−0.03 [−0.14, +0.07]−0.07 [−0.18, +0.03]
Cognitive Flexibility−0.14 [−0.30, +0.02]−0.06 [−0.19, +0.07]−0.21 [−0.36, −0.06]
Clarity & Precision−0.04 [−0.10, +0.02]−0.01 [−0.07, +0.05]−0.08 [−0.14, −0.03]
Epistemic Maturity−0.07 [−0.20, +0.05]−0.04 [−0.17, +0.09]−0.14 [−0.28, −0.01]
Share of answer-score and answer-length variance explained by question, model, language and their interaction.
Variance decomposition: model and question explain much more than language.
JudgeLanguage vs EnglishnMean shift (95% CI)
codexUkrainian120−0.68 [−1.18, −0.17]
codexPolish120−0.38 [−0.89, +0.12]
codexRussian120−0.76 [−1.27, −0.23]
claudeUkrainian119−0.95 [−1.43, −0.48]
claudePolish119−0.04 [−0.48, +0.40]
claudeRussian119+0.03 [−0.45, +0.53]

Full table views

Change in answer length versus English, translation-adjusted

Characters, paired by question and adjusted for the translated prompt length. Red = shorter; blue = longer.

ModelUkrainianPolishRussian
Claude Sonnet−13%−10%−12%
Codex Terra−6%0%+2%
Agy Gemini Flash−19%−12%−11%
Claude Opus 5.5−11%−13%−11%
Codex Sol (high)+2%−1%0%
Claude Haiku 5.5+6%−4%−1%

Headings / list items / bold spans per answer

ModelEnglishUkrainianPolishRussian
Claude Sonnet4.2 / 13.9 / 15.44.5 / 14.2 / 14.24.9 / 15.3 / 16.44.1 / 14.8 / 13.5
Codex Terra0.0 / 5.4 / 1.20.0 / 5.8 / 0.70.0 / 6.0 / 1.70.0 / 5.6 / 0.0
Agy Gemini Flash7.0 / 18.4 / 21.86.2 / 15.5 / 15.76.5 / 16.4 / 19.47.0 / 17.2 / 17.9
Claude Opus 5.56.0 / 14.1 / 18.46.7 / 13.0 / 17.06.8 / 14.2 / 16.66.0 / 13.2 / 16.8
Codex Sol (high)0.1 / 8.3 / 4.20.6 / 9.0 / 5.00.2 / 8.3 / 4.80.2 / 8.6 / 3.3
Claude Haiku 5.51.8 / 9.1 / 11.32.6 / 9.6 / 13.72.8 / 11.9 / 14.02.5 / 9.9 / 12.9

Fidelity checks

ModelLatin letters in UkrainianLatin letters in RussianWrong-language letters / 1,000English function words / 1,000 (uk / pl / ru)
Claude Sonnet0.6%0.3%0.00.0 / 0.1 / 0.0
Codex Terra0.1%0.0%0.00.0 / 0.0 / 0.0
Agy Gemini Flash2.5%2.2%0.00.8 / 0.8 / 0.6
Claude Opus 5.50.2%0.3%0.00.0 / 0.0 / 0.0
Codex Sol (high)0.1%0.1%0.00.0 / 0.0 / 0.0
Claude Haiku 5.50.6%0.6%0.00.0 / 0.1 / 0.2

Mean total score by model and language

Higher scores are shown in deeper blue.

ModelEnglishUkrainianPolishRussian
Claude Sonnet30.6029.3029.9529.55
Codex Terra27.6027.0527.7028.15
Agy Gemini Flash25.7025.3525.4524.10
Claude Opus 5.532.1031.1531.2030.40
Codex Sol (high)29.9029.5029.9029.45
Claude Haiku 5.529.1528.6528.5528.85

Question × language

Mean total score (max 35) over the models, per question. Darker is higher.

QuestionEnglishUkrainianPolishRussian
1. The Civilization Without Memory29.830.228.729.3
2. The Perfect Prediction Machine29.228.329.229.5
3. The Intelligence That Cannot Explain Itself32.332.032.730.5
4. The World Without Lies24.824.726.024.2
5. The Invisible Improvement29.027.827.529.0
6. The Society of Perfect Memory26.825.725.326.3
7. The One-Minute Democracy29.030.029.728.2
8. The Disappearing Expert27.726.027.025.5
9. The Perfectly Rational Stranger26.727.328.327.0
10. The City That Learns31.531.032.030.7
11. The Price of Certainty29.328.027.527.0
12. The Civilization of Specialists29.728.528.828.0
13. The Reverse Scientific Method30.529.031.330.7
14. The Market Without Ownership30.729.328.830.5
15. The Knowledge That Cannot Be Shared30.528.028.830.3
16. The Perfectly Fair Algorithm31.331.030.031.5
17. The World With Delayed Consequences27.226.726.524.3
18. The Intelligence Exchange31.231.331.830.7
19. The Civilization That Cannot Measure Time26.725.527.027.0
20. The Last Unsolved Problem29.729.728.828.2

Question × model

Mean total score over the scored languages, per question and model. A dash means no scores.

QuestionClaude SonnetCodex TerraAgy Gemini FlashClaude Opus 5.5Codex Sol (high)Claude Haiku 5.5
1. The Civilization Without Memory30.027.529.232.529.028.8
2. The Perfect Prediction Machine30.529.823.231.230.529.0
3. The Intelligence That Cannot Explain Itself32.831.530.233.031.532.2
4. The World Without Lies27.023.821.227.523.826.2
5. The Invisible Improvement30.226.524.029.229.031.0
6. The Society of Perfect Memory27.223.824.028.525.227.5
7. The One-Minute Democracy28.829.827.031.230.827.8
8. The Disappearing Expert26.525.825.230.028.223.5
9. The Perfectly Rational Stranger27.528.522.528.528.528.5
10. The City That Learns33.228.229.233.531.032.5
11. The Price of Certainty28.827.023.031.029.029.0
12. The Civilization of Specialists28.027.527.531.031.027.5
13. The Reverse Scientific Method31.828.525.832.533.230.5
14. The Market Without Ownership32.029.225.231.831.229.5
15. The Knowledge That Cannot Be Shared32.527.824.533.030.528.2
16. The Perfectly Fair Algorithm32.829.025.534.032.232.2
17. The World With Delayed Consequences27.026.521.529.825.227.0
18. The Intelligence Exchange33.028.527.034.033.231.8
19. The Civilization That Cannot Measure Time26.525.822.030.530.224.2
20. The Last Unsolved Problem31.027.825.031.530.229.0

Dimension × language

Mean score (0–5) on each criterion, over models and questions. Darker is higher.

DimensionEnglishUkrainianPolishRussian
Depth of Understanding4.44.24.24.2
Logical Reasoning4.24.14.14.1
Creativity & Originality3.73.63.63.6
Systems Thinking4.44.34.34.3
Cognitive Flexibility3.73.53.63.5
Clarity & Precision4.94.84.94.8
Epistemic Maturity4.03.93.93.8

Dimension × model

Mean score (0–5) on each criterion, over the scored languages and questions.

DimensionClaude SonnetCodex TerraAgy Gemini FlashClaude Opus 5.5Codex Sol (high)Claude Haiku 5.5
Depth of Understanding4.44.04.04.74.24.2
Logical Reasoning4.24.33.34.34.64.1
Creativity & Originality3.73.23.94.23.63.4
Systems Thinking4.54.04.14.74.34.3
Cognitive Flexibility4.03.32.74.13.73.7
Clarity & Precision5.05.04.35.05.05.0
Epistemic Maturity4.23.92.94.24.24.1

Question × dimension

Mean score (0–5) per question and criterion, over models and scored languages.

QuestionDepthLogicalCreativitySystemsCognitiveClarityEpistemic
1. The Civilization Without Memory4.44.44.24.73.35.03.5
2. The Perfect Prediction Machine4.33.83.64.14.04.84.3
3. The Intelligence That Cannot Explain Itself4.34.83.64.44.94.94.9
4. The World Without Lies3.73.63.33.72.84.73.1
5. The Invisible Improvement4.24.53.03.83.54.84.5
6. The Society of Perfect Memory3.93.83.74.02.64.93.1
7. The One-Minute Democracy4.24.33.84.83.54.93.8
8. The Disappearing Expert4.03.93.54.32.84.93.2
9. The Perfectly Rational Stranger4.14.03.03.23.84.94.4
10. The City That Learns4.74.73.94.94.05.04.2
11. The Price of Certainty4.14.03.14.23.74.94.0
12. The Civilization of Specialists4.34.13.64.73.64.93.5
13. The Reverse Scientific Method4.54.24.24.53.84.84.2
14. The Market Without Ownership4.44.23.94.74.04.93.8
15. The Knowledge That Cannot Be Shared4.34.13.94.63.24.94.3
16. The Perfectly Fair Algorithm4.54.63.74.84.24.94.3
17. The World With Delayed Consequences4.03.53.84.13.04.73.1
18. The Intelligence Exchange4.64.54.04.34.54.84.5
19. The Civilization That Cannot Measure Time4.13.53.64.22.94.83.4
20. The Last Unsolved Problem4.54.23.64.13.84.84.0

What the experiment shows

Prompt language matters most for how much a model says, not for whether it can reason through the prompt at all. The three most verbose models produce noticeably shorter answers outside English; the concise models remain comparatively stable.

Model choice and question choice explain far more of the observed variation than language. Formatting habits, model rankings and which questions are hard transfer well across English, Ukrainian, Polish and Russian. The answers also stay in the requested language.

The small quality differences should be treated carefully. Ukrainian is the only non-English language with a drop that appears across both complete judge runs; Polish and Russian depend more on which judge is used. Since judges reward longer answers to some extent, part of the apparent quality penalty is likely a verbosity effect.

Practical takeaway: use the language your audience needs, but specify target length and structure when comparability matters. For high-stakes evaluations, normalize for prompt and answer length, use multiple judges, and do not mistake a shorter response for weaker reasoning.