Limitations

Read these before using the dataset.

• Most rows rest on a single corpus. 8,024,536 single-token rows (80.08%) were inherited from the EOL, are listed by none of the 8 published panel sets, and are Google Books 1-gram types. For 7,900,248 of them, Google Books is the only published containment, and it is a corpus built from scanned text, which carries recognition errors. In all, 8,761,768 rows are Google Books types.
• 602,165 single-token rows are listed by no published panel set and are in neither corpus. What supports them is their intake channel, where a published one is recorded.
• 16,373 rows (0.16%) have no published intake channel and no measured containment. Nothing published says how the string entered the pool, and no published panel set or corpus holds it.
• 330,386 rows (3.30%) were made by the EOL's synthetic engine. Its code does not survive, and its own per-row record of the method is empty, but its author describes the methods: affixes applied without a part-of-speech filter to the list as it then stood (plurals, past tenses, gerunds, agent nouns, comparatives and superlatives, and prefixed forms); chemical names varied by substituent, and short chains of root, infix, and suffix, made by substitution over a hand-written list of substituents and roots, with no tool that derives a name from a formula; medical and scientific prefixes; and Greek plurals. It gave each base word a domain label, which eol_synthetic_category carries where it survives, and it used NLTK, lemminflect, and spaCy. model_or_engine marks these rows engine_generated, and each row's method is inferred from its form (§8).
• 137,485 rows of the EOL's valid list were decided with a language model: 131,014 by the EOL's verification pipeline, which used one, and 6,471 by a model's verdict alone. model_or_engine marks them model_pipeline and model_alone (§12). The EOL's invalid list is not marked.
• The withholding changes what the panel columns count. The build measured 13 panel sets, and 5 of them are withheld (§6). panel_sets, panel_count, and provenance_class count the 8 published sets alone, so a row with panel_count 0 may still be listed by a withheld list.
• 274,471 rows of the shippable projection are not in this version. Once the withheld lists and channels are taken away, nothing published supports them, so they are held back in the private record (§6).
• A channel may be withheld for license reasons. 210,109 rows carry eol:channel-withheld in intake_channels: every channel recorded for them is withheld, because it names a withheld list or a source whose terms do not allow publishing the label.
• 12 of the feature layer's columns are withheld, because each was computed against a reference that holds a withheld list, or by a character model fitted on strings the withheld lists help choose (§6).
• The sources column has 2 meanings: a per-source containment list on the CEOL's own rows, and the dataset-level tag eol_universe@c1139698 on inherited rows, whose per-source listing is the evidence columns'.
• Null and NaN mean different things (§5), and the corpora do not measure multiword rows.
• eol is a fact at a single revision of the EOL, c1139698. The EOL's last published list adds 59 entries made after that revision; they are rows of this dataset with eol false.