Columns

Every column of every config: its type, what it means, and where it may be null. The table is generated from the dataset's columns module, so it cannot drift from the files.

Generated by pipeline/research/release/documents.py from the columns module (pipeline/research/release/columns.py), the public projection (pipeline/research/release/withholding.py), and the release record (pipeline/research/release/release_v1.0.json). Do not edit it by hand: make check compares it with what the code prints.

Every row is a candidate string, never a verdict. word is the join key of every config, and id is the first 16 hexadecimal digits of the sha256 of the word, the same for the same string in every version.

The configs

Config	File	Rows	Columns	Predicate
`universe`	`data/universe.parquet`	10,020,150	41	the pin's unrestricted rows, less the rows held back (card §6), in the public columns
`features`	`data/features.parquet`	9,725,285	11	token_count == 1 over universe; id, word, and the feature layer's columns
`panel-listed`	`data/panel-listed.parquet`	1,384,833	41	panel_count >= 1
`eol-valid`	`data/eol-valid.parquet`	344,713	41	eol == true
`spaced`	`data/spaced.parquet`	294,865	41	token_count > 1
`exceptions`	`data/exceptions.parquet`	209,883	41	classes carries any of hyphenated, accented, contraction, apostrophe-form, ligature, diaeresis
`sample`	`data/sample.parquet`	38,988	41	id starts with '00' (about 1 row in 256)

The filter configs (every config but universe and features) carry every column of universe, in its row order.

Null and NaN

Null means not measured by that source, never false. In the evidence columns a single-token row holds 1 (listed), 0 (not listed), or -1 (the source is structurally blind to the row's class, as a list without hyphens is to a hyphenated string). A spaced row holds null in every evidence column and in the corpus columns, because those sources were not measured over spaced rows.
NaN in the features config means the channel does not observe the string. It is a float value, not a null: no column of the features config holds a null.

The universe config

41 columns, in this order.

The key

Column	Type	Meaning	Null
`id`	string	the first 16 hexadecimal digits of the sha256 of the word's UTF-8 bytes; a function of the string alone, so the same string carries the same id in every version	never null
`word`	string	the candidate string, the output of the grammar: lower case, the 50-character charset, spaces single and interior; the join key of every layer	never null

The pin's columns

Column	Type	Meaning	Null
`eol`	bool	in the EOL's valid list at the pinned revision c1139698: a historical fact at a pin, carried as a covariate, never a CEOL verdict	never null
`eol_universe`	bool	in the EOL's pinned candidate universe (its valid or its invalid list); false for the CEOL's own 3 layers	never null
`classes`	string	the grammar's non-exclusive class flags joined by ';' (hyphenated, accented, diaeresis, ligature, apostrophe-form, contraction, s-final, spaced); the empty string on a plain-letter string	never null
`sources`	string	provenance at intake: for the CEOL's own layers the published sources whose derived sets contain the string, joined by ';', a withheld list's label removed (the empty string where each such source is withheld); for the strings inherited from the EOL the dataset-level tag eol_universe@c1139698 (their per-source listedness is the evidence columns')	never null
`intake_date`	string	the date the row entered a CEOL artifact (an ISO date): August 17, 2026 the EOL universe at the pinned revision, August 20, 2026 the census's exception candidates and the audit's additions, August 26, 2026 the spaced admission's multiword candidates	never null
`c4_addition`	bool	a row of the audit's additions to the single-token pin	never null
`is_restricted`	bool	attested only by NWL2023 or CSW21; false on every row of this dataset, because those rows never ship	never null
`token_count`	int32	the number of space-separated tokens	never null
`c015_addition`	bool	a row of the multiword candidates of the spaced admission	never null
`wn_instance`	bool	WordNet instance-synset membership, a properness signal recorded and never filtered on	null unless WordNet attests the row; measured on the spaced layer only; null means not measured, never false
`wn_capitalized`	bool	WordNet data-file capitalization, a properness signal	null unless WordNet attests the row; measured on the spaced layer only
`wk_proper_noun`	bool	a Wiktionary Proper-noun heading, a properness signal	null unless Wiktionary attests the row; measured on the spaced layer only
`wk_flags`	string	Wiktionary's category flags joined by ';' (idiomatic, phrasal-verb, phrasebook, proverb); may be the empty string	null unless Wiktionary attests the row; measured on the spaced layer only

The release layer

Column	Type	Meaning	Null
`class_explicit`	string	the pin's classes with the value 'plain' in place of the empty string; otherwise the flags unchanged	never null
`panel_sets`	string	the published panel sets that list the row, joined by ';' in the order wordnet, wiktionary_english, hunspell_en_US, hunspell_en_GB, hunspell_en_CA, hunspell_en_AU, enable1, scowl; on a single-token row a binary set lists the row where its evidence column is 1 and SCOWL where scowl_tier is above 0; on a spaced row the sets are the pin's published sources; the empty string where none lists the row. A withheld list is never named	never null
`variety_gb_not_us`	bool	true where hunspell_en_GB is 1 and hunspell_en_US is 0: a proxy for a variety difference between the 2 dictionaries, never a variety label	null where either hunspell column is -1 or null
`variety_us_not_gb`	bool	true where hunspell_en_US is 1 and hunspell_en_GB is 0: the reverse proxy	null where either hunspell column is -1 or null

The intake-provenance layer

Column	Type	Meaning	Null
`intake_layer`	string	the layer the row entered by: eol_valid, eol_invalid, c1_exceptions, c4_additions, spaced_c015	never null
`intake_kind`	string	obtained (an inherited string taken in through a channel), derived (a row of a CEOL layer derived from pinned sources), or generated (made by the EOL's lost synthetic engine)	never null
`intake_channels`	string	the first-seen channel labels joined by ';': the EOL's discovery log with its tiers removed, and the CEOL layers' own namespaces census:, audit:, spaced:, ledger:; a record of where a string was first seen, never an attestation. A label naming a withheld list is removed, and so is a channel withheld for license reasons; a row whose every label was withheld carries the single label eol:channel-withheld	never null
`eol_phase1_tier`	string	the tier suffix the EOL's labels on the row share: valid, unlikely, or none (mixed where the labels disagree)	never null
`eol_record`	string	which EOL dictionary holds the row's entry: valid-dict, invalid-dict, none	never null
`eol_synthetic_category`	string	the category recorded on a generated string (a domain label, or the EOL validation step's judgment)	null on every row that is not generated
`eol_synthetic_engine_inferred`	string	the engine and rule inferred from a generated string's form (affixation, chemical-variation, greek-plural, medical-scientific-prefixing, unresolved); an inference, so named	null on every row that is not generated
`panel_count`	int16	the number of the 8 published panel sets that list the row (the number of names in panel_sets, asserted equal on every row); a withheld list's listing is not counted	never null
`gb_type`	bool	the string is a Google Books English 1-gram type in the CEOL re-stream	null on spaced rows, which the corpora do not measure
`wordfreq_scored`	bool	wordfreq 3.1.1 scores the string	null on spaced rows, which the corpora do not measure
`wikt_title_any`	bool	some language's Wiktionary has a page titled with the string: a screen, not containment	null on spaced rows
`provenance_class`	string	exactly 1 of channel_and_containment, channel_only, containment_only, neither, over the published fields: whether the row has a published intake channel (a label other than eol:phase1-unrecorded and eol:channel-withheld) and whether a published panel set or a corpus contains it	never null

The evidence matrix

Column	Type	Meaning	Null
`wordnet`	int8	listed by WordNet 3.0's lemmas, re-derived under the CEOL grammar: 1 listed, 0 not listed, -1 structurally blind to the row's class	null on spaced rows, which the single-token sets were not measured over
`wiktionary_english`	int8	listed by Wiktionary pages with an English section (the August 4, 2026 dump): 1 listed, 0 not listed, -1 structurally blind to the row's class	null on spaced rows, which the single-token sets were not measured over
`hunspell_en_US`	int8	listed by the LibreOffice hunspell en_US dictionary, expanded: 1 listed, 0 not listed, -1 structurally blind to the row's class	null on spaced rows, which the single-token sets were not measured over
`hunspell_en_GB`	int8	listed by the LibreOffice hunspell en_GB dictionary, expanded: 1 listed, 0 not listed, -1 structurally blind to the row's class	null on spaced rows, which the single-token sets were not measured over
`hunspell_en_CA`	int8	listed by the LibreOffice hunspell en_CA dictionary, expanded: 1 listed, 0 not listed, -1 structurally blind to the row's class	null on spaced rows, which the single-token sets were not measured over
`hunspell_en_AU`	int8	listed by the LibreOffice hunspell en_AU dictionary, expanded: 1 listed, 0 not listed, -1 structurally blind to the row's class	null on spaced rows, which the single-token sets were not measured over
`enable1`	int8	listed by the ENABLE word list: 1 listed, 0 not listed, -1 structurally blind to the row's class	null on spaced rows, which the single-token sets were not measured over
`scowl_tier`	int16	the SCOWL size tier (10 to 95) of the smallest SCOWL list holding the string; 0 not listed; -1 not applicable to the row's class	null on spaced rows
`wikt_title_screen`	int8	1 where some language's Wiktionary has a page titled with the string, else 0: a screen, never a detector	null on spaced rows

Model and engine

Column	Type	Meaning	Null
`model_or_engine`	string	engine_generated on a row the EOL's synthetic engine made (intake_kind generated); on a row of the EOL's valid list, model_pipeline where the EOL listed it through its verification pipeline, which used a language model, and model_alone where a language model's verdict is its only basis; none on every other row. A record of how the EOL decided, never a verdict of this dataset's; the EOL's invalid list is not marked	never null

The features config

11 columns: the key and the feature layer's published columns, 1 row per single-token row of universe, in its order.

The feature layer, the features config

Column	Type	Meaning	Null
`id`	string	the first 16 hexadecimal digits of the sha256 of the word's UTF-8 bytes; a function of the string alone, so the same string carries the same id in every version	never null
`word`	string	the candidate string, the output of the grammar: lower case, the 50-character charset, spaces single and interior; the join key of every layer	never null
`gb_year_span`	int16	the years from the first to the last in which the string occurs in the Google Books English 1-grams (the 2020 release), inclusive	0 where the string is not a 1-gram type
`gb_span_fill`	float	the share of the span's years in which the string occurs	NaN where the string is not a 1-gram type
`gb_tokens_per_volume`	float	the total match count over the total volume count	NaN where no volume carries the string
`gb_volumes_per_year`	float	the total volume count over the years in which the string occurs	NaN where the string is not a 1-gram type
`gb_log10_match`	float	log10 of the total match count	NaN where the string is not a 1-gram type
`gb_log10_volume`	float	log10 of the total volume count	NaN where no volume carries the string
`zipf_log10_rank`	float	log10 of the string's frequency rank among the aggregate's types	NaN where the string is not a 1-gram type
`zipf_resid`	float	log10 of the match count minus the Zipf-Mandelbrot fit's prediction at that rank	NaN where the string is not a 1-gram type
`wordfreq_zipf`	float	the wordfreq 3.1.1 Zipf frequency (English) where wordfreq's list holds the string	NaN elsewhere, and on every hyphenated string (wordfreq is hyphen-blind)

What v1.0 withholds

The columns of 5 lists whose membership may not be published (see the card, section 6): nwl2023, csw21, sowpods_legacy, web2, web2a. Their labels are removed from sources, intake_channels, and panel_sets, and panel_sets, panel_count, and provenance_class are computed over the published panel sets alone.
Channel labels withheld for license reasons, removed from intake_channels; a row whose every label was withheld carries eol:channel-withheld.
12 feature columns (docs/licenses/FEATURE_COLUMNS.md): the 10 computed against a reference that holds a withheld list, ocr_neighbor_hits, ocr_neighbor_freq_ratio, ocr_apostrophe_ratio, ocr_ligature_ratio, ocr_hyphen_drop_ratio, ocr_strip_ratio, ocr_hyphen_junction_ratio, morph_productivity_vn, morph_has_parse, segment_attested_fraction; and the 2 of the character model, fitted on strings the withheld lists help choose, whose values pick out the strings those lists list: orthotactic_logp_total, orthotactic_logp_per_char.
The rows left with no public evidence once those are withheld, and the census and audit rows whose every source label names a withheld list; the card counts them by reason.