The full dataset is not published yet. A license review comes first, and the dataset will be released once it is finished; no date is promised. Until then the site shows the dataset's statistics and a sample, and lists cannot be built or downloaded.

Common English OpenList

A candidate, not a verdict

Every row of the universe is a candidate: a string, and a record of where it came from. No row says that a string is a word, that it is English, or that it is spelled right.

What a row says

A row records a string that passes the spelling grammar, and the evidence for it: which of the 8projection.published_sets published word lists list it, which way it came in, whether Google Books or wordfreq holds it, and, for a row of the English OpenList, what decided it there. That is all a row claims, and each claim can be checked against a pinned source.

Why no verdicts

Word lists disagree because they answer different questions. A spell-checker's dictionary, a tournament's word list, and a corpus's vocabulary each draw the line somewhere else, and none of them is wrong by its own rule. The universe keeps every candidate its sources gave and records each source's answer beside it, so a reader can apply a definition of their own and state exactly what it is.

That is what Compose a list is for: a definition becomes a set of criteria, the criteria a count, and the count a list with the code that reproduces it.

What a row rests on

Every row falls in 1 of 4 provenance classes, by whether a published channel brought it in and whether a published list or a corpus holds it.

Channel and containment
8,920,645rows.by_provenance_class.channel_and_containment
came in by a recorded channel, and a published list or a corpus holds it
Containment only
497,340rows.by_provenance_class.containment_only
a published list or a corpus holds it; no published channel is recorded
Channel only
585,792rows.by_provenance_class.channel_only
came in by a recorded channel; no published list or corpus holds it
Neither
16,373rows.by_provenance_class.neither
no published channel and no containment

Even the strongest class is evidence, not a ruling: a row that every published list lists is still a candidate, and a row that only a corpus holds may still be one a reader wants.

"Valid," in the English OpenList's sense

The English OpenList calls the entries of its main list valid. The universe records that as a fact about the English OpenList at revision c1139698: its eol column is true on the 344,713rows.eol rows that list held, and it means nothing more. Elsewhere on this site, a string of the universe is never called valid.

What the universe does not record

Part of speech (143,924 rows are WordNet-listed and could take a part of speech from WordNet); lemma and inflection relations; definitions and senses (they move with their sources and carry license obligations); pronunciation, syllables, etymology, and a first attestation year; variety and register labels (the 2 variety columns are proxies); a frequency drawn from many kinds of text for most rows (wordfreq scores 303,511; the Google Books counts cover 8,761,768 rows, but from scanned books alone); and validity, by design. The next major version plans parts of speech, lemma and inflection relations, and WordNet's categories.