Research
What the universe holds, and how the published word lists compare when every list is measured against the same strings. Every figure here is computed from version 1.0's files by committed code.
Of the 9,725,285 single-token rows, 1,089,968 (11.2%) are listed by at least 1 published list and 8,033,152 (82.6%) rest on a corpus alone. Most listed rows are listed by 1 family: 43.7% of them by Wiktionary and no other. 417,864 rows, the consensus core, are listed by 2 or more independent families.
The sources compared
No single list can measure this, because none is measured against the others. Each figure counts the single-token rows, over the rows each list could hold; a list that cannot hold a class is never counted as not listing it.
Of the rows list A lists that list B can hold, the share B lists. Read across a row; the map is not symmetric, so nesting shows as a bright cell beside a dim one. Columns: WN WordNet, Wikt Wiktionary, US, GB, CA, and AU the hunspell dictionaries, EN ENABLE, SC SCOWL.
Point at or focus a cell: the row is list A, the column list B.
Shown: containment
Over the 9,538,019 rows all 5 families can hold; the 15 largest combinations. A filled dot marks a family that lists the rows.
Point at a column, or focus the first row of dots, for its count.
For each band of wordfreq's Zipf value, the share of the rows a list can hold that it lists. Most rows have no Zipf value; that band is stated under each chart.
WordNet
No Zipf value: 0.3% listed
Wiktionary (English)
No Zipf value: 7.5% listed
hunspell en_US
No Zipf value: 0.4% listed
hunspell en_GB
No Zipf value: 1.5% listed
hunspell en_CA
No Zipf value: 0.4% listed
hunspell en_AU
No Zipf value: 0.4% listed
ENABLE
No Zipf value: 0.9% listed
SCOWL
No Zipf value: 4.2% listed
Point at or focus a dot for its share, the rows behind it, and its 95% interval, which at these sizes is narrower than the line.
Shown: shares by strings
The listed rows all 5 families can hold, by the number of families that list them, in each band of wordfreq's Zipf value.
- 1 of 5
- 2 of 5
- 3 of 5
- 4 of 5
- 5 of 5
The darker the segment, the more families agree. Point at a segment for its count.
For each number of lists, the mean union over every choice of that many lists, and its smallest and largest.
All 8 lists list 1,089,968 rows. Point at a step for its range.
The technical report states the method and every figure: the sources compared.
In this section
-
The technical report
The dataset described and measured, with every figure from code.
-
Literature
Comparable resources, documentation standards, and contamination, each claim checked at its source.
-
The proposed study
A study design that has not been run: its question, design, instrument, and pilot.
Open questions
- Parts of speech. The universe records none; WordNet's listing covers only the rows it lists. A later version adds a part-of-speech layer, lemma and inflection relations, and WordNet's categories.
- Offensive content. Its share is not measured. The measure needs an offensive-term lexicon whose license allows the use.
- Recognition errors. How many corpus-only rows are errors of scanning would need a labeled sample, which this project chose not to make.
- The standard estimators of uncaptured strings. How far they miss on a closed frame inside the universe, where the answer is known, is planned as a calibration, never as a count of missing English strings.
- All 13 lists. The comparison over the withheld lists as well waits for the license review to confirm that aggregate counts may be published.
- Change over time. What each source adds and drops between versions needs a second snapshot of each.