How many real questions does proof settle?
Daniel's guess, stated before any counting: something proven to work settles at least 80% of everything, and the rest is guesses and luck. This page counts it on real questions people asked, and reports the number whatever it says.
I.The count so far
Measured on this sample of 100 questions, never proven. The interval is where the true share most likely sits (95%).
| Settled by | Share | 95% interval | Questions |
|---|---|---|---|
| Proven to work (both labelers agree) | 48.9% | 38.8 to 59% | 44 of 90 |
| Proven to work, or data and sources (both agree) | 86.7% | 78.1 to 92.2% | 78 of 90 |
| Proven to work, Sonnet alone | 50% | 40.4 to 59.6% | 50 of 100 |
| Proven to work, Haiku alone | 45% | 35.6 to 54.8% | 45 of 100 |
Proven to work alone: 48.9%, the whole interval is below 80%, so the guess does not hold there. Proven to work plus data and sources: 86.7%, and the interval (78.1 to 92.2%) spans 80%, so more questions are needed to tell.
II.How it is counted
The questions. 490 real questions drawn at random (seed 20261029) from 50 of the 151 main Stack Exchange sites, 100 questions before one random moment in each site's life. The first 100 are labeled so far; the rest follow.
The four buckets, written down before the first label: proven to work (proof, logic, computation, or a repeatable procedure known to work), data and sources, a choice of taste or values, and luck.
Two labelers, Sonnet and Haiku, each alone. They agree on 90% of questions (Cohen's kappa 0.837, where 1 is perfect and 0 is chance). Daniel also checks 20 at random by hand; his matches will be shown here.
III.Every disagreement
Sonnet: MATH, debug config, deterministic fix. Haiku: MEASURE, Cause depends on unseen config; needs observing and investigating.
Sonnet: MATH, reset steps for app setting. Haiku: MEASURE, Reset fix found by trying app settings and observing results.
Sonnet: CHOICE, recommend me suggestions. Haiku: MEASURE, Which ground covers grow in heavy shade is observed horticultural data.
Sonnet: MATH, defined syntax procedure. Haiku: MEASURE, Tridion's bracket syntax must be looked up in the platform.
Sonnet: MATH, definitions by logic. Haiku: MEASURE, Explains how the terms are used in sources, a description not a proof.
Sonnet: MATH, program output. Haiku: MEASURE, The 214 Kangxi radicals are a fixed published list.
Sonnet: MATH, transfer procedure. Haiku: MEASURE, Kindle book transfer rules depend on Amazon's current policy.
Sonnet: CHOICE, code review opinions. Haiku: MATH, Dockerfile build error has a checkable fix by build steps.
Sonnet: CHOICE, what to disable depends on needs. Haiku: MEASURE, Which Fedora services are safe to disable depends on the user's setup.
Sonnet: LUCK, unobservable afterlife. Haiku: CHOICE, Fate of the atman after unfulfilled desires is a belief question.
IV.Every run, oldest first
| Date | Questions | Proven to work | Plus data | Kappa |
|---|---|---|---|---|
| 2026-10-29 19:36 | 100 | 48.9% | 86.7% | 0.837 |