The full dataset is not published yet. A license review comes first, and the dataset will be released once it is finished; no date is promised. Until then the site shows the dataset's statistics and a sample, and lists cannot be built or downloaded.

Common English OpenList
Sections

Literature

The resources a reader might compare with the universe, the standards its documentation answers to, and what is known about contamination. Every claim here was checked at its source; the claims ledger in the code repository gives each one's source, locator, and date.

This page sets the CEOL Universe beside the open English word lists and lexical databases a reader might compare it with, the documentation standards that bear on it, and what the literature says about contamination and canaries. Sizes and licenses are each resource's own statements, read on October 26, 2026, unless an entry gives another date. A string of the universe is a candidate; nothing here says whether any string is a word.

Comparable resources

Game-derived marks a list whose documentation, or its maintainer's documentation of the same source lists, says it takes its selection, in whole or in part, from a proprietary game word list: the American Official Scrabble Players Dictionary (OSPD), the British Official Scrabble Words (OSW), Collins Scrabble Words (CSW), which descends from both [1], or the Tournament Word List (TWL) [2]. The mark records what each list says about itself; it is not a finding about any entry.

Lexicons, Wiktionary, and its extractions

ResourceWhat it isSize, as statedLicense, as statedWhat it records per entryWhat it does not record
Princeton WordNet 3.0A lexical database of English that groups nouns, verbs, adjectives, and adverbs into synonym sets (synsets) linked by conceptual-semantic and lexical relations. Princeton states that it is no longer developed and names Open English WordNet and the Global WordNet Association as community-led resources that build on it [3].155,287 unique strings counted within each part of speech (147,278 distinct overall), 117,659 synsets, and 206,941 word-sense pairs [4].The WordNet 3.0 license: use, copying, modification, and distribution for any purpose without fee, provided the copyright notice and statements appear on all copies; Princeton's name may not be used in publicity [5].For each lemma and part of speech, its synsets, pointer (relation) types, and a count of tagged senses; for each synset, its words, lexicographer file, relations, and a gloss [6].Parts of speech other than the 4 above [4]; the database format has no pronunciation or etymology field [6].
Open English WordNet, 2025 editionA fork of Princeton WordNet developed under an open-source methodology, released in annual editions since 2019; from the 2025 release, proper nouns are in a separate resource, Open English Namenet [7].2025 edition: 135,969 words and 107,519 synsets; 2025 Plus edition, which adds curated proper nouns: 161,875 words and 120,564 synsets [7].CC BY 4.0 for the team's work, with the Princeton WordNet license applying to the underlying data; attribution to both is required [7].Lexical entries (lemma and part of speech) and senses, grouped into synsets that carry a part of speech, an Interlingual Index link, relations, and examples [7]; definitions [8]; pronunciation for nearly 35,000 entries and Wikidata links for over 6,000, as of the 2021 release [8].Proper nouns in the 2025 edition, which moved them to Open English Namenet [7].
English WiktionaryA collaboratively edited dictionary whose pages hold one section per language; an entry is a single language section [9]. The text is an unstructured wiki maintained by volunteers [10].English: 1,396,346 entries, 719,082 of them lemma entries, generated from the October 1, 2026 dump [9].CC BY-SA 4.0 and the GNU Free Documentation License, under the Wikimedia Terms of Use in effect since June 7, 2023 [11].Wikitext per language section; the next row lists what an extractor recovers from it [10].A fixed schema: it is an unstructured wiki [10].
Wiktextract and kaikki.orgAn extractor that expands Wiktionary's templates and Lua modules to produce structured JSON [10]; kaikki.org publishes its output [12].English: 1,396,111 distinct words, extracted October 21, 2026 from the English Wiktionary dump dated October 1, 2026, as the page stated on October 26, 2026 [12].The same licenses as Wiktionary, CC BY-SA and the GFDL, with no version stated [12].Lemmas, inflected forms, translations, etymology, usage examples, pronunciations (with sound-file URLs), lexical and semantic relations, and morphological, syntactic, semantic, topical, and dialectal annotations [10].A fixed release: the data is re-extracted regularly, usually at least once a week [12].
DBnaryLexical data extracted from 27 Wiktionary language editions and published as linked open data in the OntoLex model, one version per Wikimedia dump [13, 14].Not confirmed: the site was unreachable on October 26, 2026, and its home page states no counts [13].Stated 2 ways: CC BY-SA 3.0 on its site [13]; CC BY 4.0 in its Zenodo record for version 20260601 [14].Pages, entries, senses, translations, and lexical relations, with a set of disambiguated translations as an added layer [13].Not stated in the pages read.
MorphyNetA multilingual database of inflectional and derivational morphology drawn from Wiktionary editions [15].English: 396,772 words with 649,594 inflected forms; 67,412 words with 225,131 derived forms [15].CC BY-SA 3.0 [15].Inflection rows (lemma, form, features, morpheme segmentation); derivation rows (source and target word, their parts of speech, the affix, and its type) [15].Senses or pronunciation: its row formats have no such fields [15].
SPECIALIST Lexicon (National Library of Medicine)A syntactic lexicon of biomedical and general English, designed for the UMLS SPECIALIST Natural Language Processing System and released annually since 1994 [16].2027 release: 534,363 lexical items and 1,007,936 forms, 51.55% of the forms multiword [17].NLM's terms, not a Creative Commons license: no charge; redistributions keep the conditions; a publication must name the release number and date and describe any modifications; no proprietary rights may be claimed [18].Lexical records from which its tools retrieve syntactic category, inflectional and spelling variants, abbreviations, acronyms, and derivational variants; separate files of derivation pairs, synonyms, and antonyms [16].Not stated in the pages read.
CMU Pronouncing Dictionary (CMUdict) 0.7bA machine-readable pronunciation dictionary for North American English, maintained by Carnegie Mellon University's Speech Group [19, 20]."Over 134,000 words and their pronunciations" [19].A license file, Copyright 1993–2015 Carnegie Mellon University, that permits redistribution and use with or without modification if the copyright notice and disclaimer are kept [20]; the README calls research and commercial use completely unrestricted and asks users to acknowledge its origin [20].A pronunciation in the ARPAbet set of 39 phonemes, with a lexical stress marker on each vowel [19].Anything but the word and its pronunciation: its entries map words to pronunciations [19].

Word lists

ResourceWhat it isSize, as statedLicense, as statedWhat it records per entryWhat it does not record
SCOWL (SCOWLv1), release 2020.12.07. Partly game-derived at sizes 80 and above (see the last column).Spell Checking Oriented Word Lists: word lists split by size and spelling category, intended for use in spell checkers [21]. As of February 17, 2026, the original format is no longer maintained [22].The SCOWLv1 README (commit of July 22, 2024) counts 658,213 entries through size 95 in the American and English categories, names included [21].Copyright 2000–2018 Kevin Atkinson: permission to use, copy, modify, distribute, and sell, provided the notice appears in all copies; the notices of its components also apply, among them UKACD's, which requires that its notice be displayed and its text included verbatim on redistribution [23].Membership by spelling category (American, British, Canadian, Australian, and variant levels), sub-category (abbreviations, contractions, proper names, upper case, words), and size (10 to 95) [21].Hyphenated and open (spaced) compounds, which the original version excluded [24]. Note: size 80 includes all the lists of the ENABLE supplement package except one [23], and SCOWL's successor documents that package's lists of additional and stale OSPD words [25].
English Speller Database (ESDB, formerly SCOWLv2). Partly game-derived at sizes 80 and 85 (see the last column).SCOWL's successor: a SQLite database and text file on how common a word is, dialect spelling differences, variants, and basic part of speech and inflection [24]. It is a work in progress, and the database itself will not have an official release until it stabilizes [26].No entry count stated; the minimum size is 35, and size 95 is not included [24].Copyright 2000–2026 Kevin Atkinson, the same notice-retention grant, which the README calls "MIT-like"; further notices apply to Australian output, to generated lists larger than size 80 (UKACD), and possibly WordNet [24, 27].Size, region, spelling and variant level, a lemma with its part of speech, inflected forms, definition notes, usage notes (among them offensive and vulgar), and sometimes a tag naming the source list [24]. Unlike SCOWLv1, it includes hyphenated and open compounds [24].Note: its sources document lists additional OSPD words, added at size 80, and stale OSPD words, at size 85, both from the ENABLE2K supplement files [25].
LibreOffice hunspell dictionaries en_US, en_CA, en_AU, en_GBSpell-checker dictionaries. The en_US, en_CA, and en_AU README (version 2020.12.07) says these dictionaries are made by a SCOWL script and that the normal ones correspond to SCOWL size 60; en_GB began from a subset of Kevin Atkinson's list and has been extensively updated by others [28].No entry count stated in the READMEs [28].en_US, en_CA, en_AU: they come directly from SCOWL and are under SCOWL's copyright, with an affix file under Geoff Kuenning's BSD license; en_GB: "covered by his original LGPL licence", with the affix file also under the LGPL [28].Words and affix rules that generate inflected forms, 1 spelling variety per dictionary; en_GB adds hyphenated compounds, place names, possessives, plurals, and a morphological extension based on Atkinson's part-of-speech and inflection data [28].Most alternative spellings: the size-60 dictionaries generally include 1 variant per word [28].
ENABLE (ENABLE2K). A word-game list; it states it is not the tournament list.The Enhanced North American Benchmark Lexicon, a word list for word games researched and compiled by Alan Beale. Its README says the master list is not the proprietary TWL98, and that it reflects the OSPD changes in effect from January 1, 1998 [29].173,528 words in the Y2K update; the first edition had 100,701 [29].The master list is "formally released into the Public Domain"; credit to the originators is requested [29].Membership in a plain ASCII word list [29].Anything beyond the string [29].
12dicts 6.0.2 (June 2016). Partly game-derived (some inflections in 2of12inf).Alan Beale's word lists of common words; the classic lists record agreement among 12 source dictionaries of American English, 8 for learners of English and 4 desk dictionaries [30].Classic lists from about 22,000 to about 82,000 words [30].Released to the public domain, with acknowledgment requested; 2of12inf and the 2+2+3 lists depend on AGID, which keeps them out of the public domain [30].Membership in lists defined by how many source dictionaries hold a word; 2of12inf adds inflections, marking the plurals of uncountable nouns [30].Note: the README says some 2of12inf inflections came from Scrabble-related sources, and that no later list with inflections includes those plurals [30].
Moby Words II (Grady Ward). Game-derived: the 2 crossword files.Word lists by Grady Ward, released on Project Gutenberg on May 1, 2002 [31].Separate lists, among them 354,984 single words, 256,772 compound words, 74,550 common dictionary words, 21,986 names, 10,196 places, and 113,809 crossword-game words [31]."Public Domain material by grant from the author, January, 2001" [31].Membership in each list [31].Accents, which were stripped. The crossword file is described as compatible with the first edition of the OSPD, and a delta file makes it compatible with the second [31].
Moby Part-of-Speech II (Grady Ward)A part-of-speech list, the second edition, with about 15,000 new entries [32].No count stated in its documentation [32].The same 2001 public-domain grant [32].A word or phrase with 1 or more of 15 part-of-speech codes, the principal use first [32].Many -ed, -ing, -ly, and -ic forms, which are not listed explicitly [32].
YAWL (Yet Another Word List) 0.3.2.03. Game-derived.A word-game list by M. Leo Cooper, a superset of ENABLE, OSW, and lists by Alan Beale; rehosted on GitHub by Aaron Bull Schaefer [33].264,097 words [33].The list "is in the Public Domain" [33].Membership [33].Anything beyond the string [33].
Letterpress word list. Game-derived.The word list behind the Letterpress game, loosely based on other word lists; its rule is that a word allowed in another popular word game such as Scrabble should be allowed, and it names YAWL, ENABLE2K, and CSW12 as useful lists [34].Not stated [34].CC0 1.0 Universal [34].Membership in 1 sorted file per language; US, UK, and Australian spellings combined [34].Proper nouns and acronyms, excluded by rule with a few kept for historical reasons; which spelling variety an entry belongs to [34].
Wordnik wordlist. A word-game list; the source of its selection is not stated.A list of English words commonly used in word games, for game developers [35]. Wordnik's paid game dataset, which the README offers to those who need more data, states that it takes its playable words from ENABLE and the Tournament Word List [2].Not stated [35].MIT License, Copyright 2020 Wordnik [35].Membership [35].Definitions, parts of speech, frequency, and offensive-term labels, which the README offers in a licensed dataset [35].
dwyl/english-wordsA text file of English words extracted from a word list once published by Infochimps [36]."Over 466k" words in the README; 479k in the repository's description [36].The Unlicense, beside a README that says the copyright still belongs to the original publisher [36].1 string per line; a second file keeps only letters-only strings [36].Anything beyond the string [36].
Webster's Second word lists (web2, web2a)Word lists in FreeBSD's share/dict directory, introduced as Webster's Second International; web2a holds hyphenated terms and noun and adverbial phrases. FreeBSD maintains web2 as a list of current American English spellings and has removed a few entries [37].web2: 234,936 words; no count for web2a [37].No license stated; the README says the 1934 copyright "has lapsed, according to the supplier" [37]. The Catalog of Copyright Entries records renewals of the dictionary's 1934 copyrights on December 18, 1961 [38].Membership [37].Anything beyond the string [37].

Frequency resources and norms

ResourceWhat it isSize, as statedLicense, as statedWhat it records per entryWhat it does not record
wordfreq 3.1.1 (Robyn Speer)A Python library of word frequencies in over 40 languages, combining several sources; its data is a snapshot of usage through about 2021 and is unlikely to be updated again [39, 40]. 3.1.1, uploaded November 21, 2023, is the newest release on PyPI [41].No English word count stated; its table lists 7 sources for English [39].Code under the Apache license; data files that may be redistributed under CC BY-SA 4.0 [39]. The current README adds that converting the data to CSV does not follow the license, because CSV has no place for attribution [39].A frequency per word, also on the Zipf scale (the base-10 logarithm of occurrences per billion words) [39].Words below its cutoff: the lowest Zipf value in the large lists is 1.0, and a word not in the list returns a default value of 0, not a missing value [39].
Google Books Ngram, English, version 20200217N-gram counts from the books Google has scanned; the downloadable sets were generated in February 2020 (version 3), July 2012 (version 2), and July 2009 (version 1) [42].No English 1-gram type count stated [42]. The first corpus held 5,195,769 books, with 361 billion English words [43]; the 2012 edition held 8,116,746 books, the English part close to half a trillion words [44]."This compilation is licensed under a Creative Commons Attribution 3.0 Unported License" [42]; the Viewer's page says its graphs and data may be freely used for any purpose, with acknowledgment appreciated [45].Per n-gram and year, a match count and a volume count (the version 2 format) [42].N-grams that occur 40 times or fewer across the corpus, which versions 1 and 2 omit [42]; how often a book was read, since the corpus holds one copy of each book [46].
SUBTLEX-USWord frequencies from American film and television subtitles [47].74,286 words in the corpus, 60,384 of them with a frequency above 1 [48], from 51.0 million words in 8,388 films and episodes [47].The 2009 paper: freely available for research purposes [47]. The author's copy on OSF carries a license file dated May 7, 2024 stating CC BY-NC-SA 4.0 [49]. The Ghent University download page states no license [48].A frequency count, the number of films a word occurs in, a count of lowercase occurrences, and per-million values [48].Not stated in the pages read.
SUBTLEX-UKBritish English word frequencies from BBC subtitles broadcast from January 2010 to December 2012 [50].332,987 word types in 201.7 million tokens; the main file holds 160,022 types (159,235 single words and 787 hyphenated) [50].The author's OSF copy carries a license file dated May 7, 2024 stating CC BY-NC-SA 4.0 [51].Frequency, Zipf value, contextual diversity, the dominant part of speech and its lemma, and all parts of speech observed [50].Types seen in a single broadcast, and types with digits or other non-letter symbols, which the main file leaves out [50].
English Lexicon ProjectBehavioral and descriptive data for lexical decision and naming studies [52].40,481 words and 40,481 nonwords [52].No terms on the home page (archived October 14, 2026); an About page archived in 2019 says word lists from the site are for non-commercial research only and may not be used to develop speech technology [53]. The live site's certificate had expired when read on October 26, 2026.Lexical decision and naming responses (816 and 444 participants) and descriptive lexical statistics [52].Not stated in the pages read.
Web 1T 5-gram Version 1 (LDC2006T13). Not open.English word n-grams and their counts from about 1 trillion tokens of text from publicly accessible web pages, contributed by Google [54].13,588,391 unigrams; 1,024,908,267,229 tokens [54].The Web 1T 5-gram Version 1 Agreement of the Linguistic Data Consortium; fees apply [54].N-grams of 1 to 5 words with observed frequency counts [54].Not stated in the pages read.

Documentation standards

Each standard below asks a dataset to state certain things. The table gives the version read and the points that bear most on a candidate inventory.

StandardWhat it asks a dataset to state
Datasheets for Datasets (version 8, 2021)Questions in 7 sections: motivation, composition, collection process, preprocessing and labeling, uses, distribution, and maintenance [55]. Composition asks whether the data contains anything that might be offensive or insulting; distribution asks whether the dataset has a DOI; maintenance asks for an erratum, whether older versions stay available, and how others can contribute [55]. The questions are not prescriptive, those that do not apply may be skipped, and the process is not meant to be automated [55].
Data Statements (schema versions 1, 2, and 3)Version 1 has 9 elements, among them curation rationale, language variety (a BCP-47 tag with a prose description), speaker and annotator demographics, and a provenance appendix carrying the statements of source datasets; a short form of about 60 to 100 words should appear in any publication that uses the dataset to train, tune, or test a system [56]. Version 2 has 15 elements, adding a header, an executive summary of 60 to 100 words, limitations, metadata with errata, and disclosures; it says missing information should be marked as missing, not used as a reason to skip the statement [57]. Version 3 has 17 elements, adding distribution and maintenance [58].
Data Cards (2022)31 themes, among them the dataset's publishers and funding, updates and versions, differences across versions, upstream sources, inclusion and exclusion criteria, typical and outlier examples, and known patterns and biases, asked at 3 levels of detail [59].
Hugging Face dataset card templateSections, among them dataset details, uses and out-of-scope uses, structure, creation (curation rationale, source data, and optional annotations), bias, risks, and limitations, citation, and contact [60]. It has no section for versions, a change log, or maintenance, and its personal and sensitive information section sits inside the optional annotations section [60].
Croissant 1.0 and 1.1, with the Responsible AI extension 1.0A machine-readable metadata format for datasets [61], built on schema.org properties [62]. Version 1.0 was published on March 1, 2024 [63]. Version 1.1 (January 29, 2026) requires 9 properties, among them license, creator, and publication date; recommends a version number following MAJOR.MINOR.PATCH; strongly recommends a sha256 checksum for each file of a versioned dataset; and recommends the W3C PROV Ontology for provenance [62]. The Responsible AI extension (March 6, 2024) defines 20 properties, among them collection time frame, limitations, biases, use cases, personal and sensitive information, and a maintenance plan [64]. The NeurIPS 2026 Evaluations and Datasets track asks dataset authors for both the core and the Responsible AI fields [65].
FAIR Guiding Principles (2016)15 principles, among them a persistent identifier, metadata that stay accessible after the data are gone, a clear data usage license, detailed provenance, and domain community standards [66].

Lexicographic standards. The TEI Lex-0 guidelines (version 0.9.5, February 8, 2026) require a header with a file description and a profile description, a license recorded inside the metadata, and a description of the sources the resource was derived from, and they call a recorded revision history good practice [67]. The ELEXIS Lexicographic Data Seal of Compliance (2021) assesses 6 principles: trustworthiness, interoperability, stewardship, and citability for the base seal, and reciprocity and openness for the higher one [68]. Its trustworthiness criteria ask for the language varieties covered and the rights holder of the dataset and of its source; its citability criteria ask how to cite the whole and a single entry; its reciprocity criteria ask whether users can submit corrections [68]. The European Language Grid's minimal metadata ask, for a lexical resource, its basic unit of description and the kinds of information attached to it, and, for all data, language coverage and size and format per distribution [69]. The LRE Map asked authors at LREC conferences to describe each resource they built or used by its type, language, modality, use, availability, maturity, size, license, and documentation [70].

Documenting large corpora and citing evolving data. Documentation of the C4 web corpus examined where its text came from, what it contains, and what its filters removed, and found evaluation examples from other benchmarks inside it [71]. What's In My Big Data runs 16 analyses over 10 English corpora, benchmark contamination among them [72]. The Research Data Alliance's recommendations for citing evolving data ask for versioned and timestamped data, a timestamp and a checksum for each query result, and persistent identifiers for the queries [73]. The Tromsø recommendations for linguistic data ask a citation for author, date, title, publisher, and locator, and for the version where a dataset has versions [74].

None of the standards above asks a dataset to state its exposure to contamination or to carry a canary: a text search for both terms in each standard read (Datasheets, the 3 Data Statements schemas, Data Cards, the Hugging Face template, Croissant and its Responsible AI extension, FAIR, TEI Lex-0, the Seal of Compliance, the European Language Grid and META-SHARE schemas [75], the LRE Map, and the 2 citation recommendations) found neither.

Contamination and canaries

What contamination is. Sainz and colleagues separate guideline contamination (a model has seen a benchmark's annotation guidelines), raw-text contamination (it has seen the original text before annotation), and annotation contamination (it has seen the labels) [76]. Xu and colleagues grade contamination in 4 levels, from semantic (content on the same topic or from the same source) to label level [77].

What detection can show. Deng and colleagues organize detection methods into retrieval, temporal cutoff, masking, perturbation, canonical order, behavior manipulation, and membership inference [78]. Min-K% Prob, a method that needs no reference model, reaches an average AUC of 0.72 on the WikiMIA benchmark [79]. Guided instruction flags an instance when a model, given a dataset's name and the start of an instance, completes it exactly or nearly so [80]. The canonical-order test's false-positive guarantee rests on the dataset's order being exchangeable [81]. GPT-3's authors flagged test examples with a 13-gram overlap with the training data, and a bug left some overlaps in place [82]. Memorization grows log-linearly with model capacity, with how often an example is duplicated, and with the length of the prompt [83]. Xu and colleagues conclude that large-scale pretraining makes it nearly impossible to eliminate the risk fully [77].

Canaries. A canary in this sense is a marker placed in published material. BIG-bench puts a GUID canary in every task file so that researchers can filter its data out of web-scraped corpora, and so that a model can be tested afterward for training on it; it asks publications that quote its tasks to carry the string [84]. GPQA and BrowseComp each embed a canary that contains the BIG-bench string, to make their data easier to filter out of training corpora [85, 86], and a catalog of questionable practices in machine learning suggests that new benchmarks reuse the exact string [87]. Some developers report filtering on canaries. Anthropic's system card for Claude Opus 4.5 states that it dropped training documents containing the BIG-bench or Alignment Research Center canaries, reports that rephrased evaluation material persisted despite its filters, and suggests that writers of public evaluations attach canaries [88]. The PaLM report states that the BIG-bench canary does not appear in PaLM's training data [89]. The GPT-4 report says portions of BIG-bench were mixed into training and that it excluded BIG-bench from its results; it does not mention a canary [90].

What a canary cannot do. It works only where trainers know of it and respect it, and it does nothing against a developer who means to use the data [91]. It can be removed: third-party copies of GPQA and Humanity's Last Exam on Hugging Face were found with the canary sliced out [92], and removal is a stated limit of the method [93]. A memorization canary is a different object: the Secret Sharer study inserts random sequences into training data a chosen number of times and measures their exposure [94].

Other measures. Jacovi and colleagues propose encrypting public test data with a public key under a license that forbids derivatives, refusing to send test data to closed APIs that offer no training exclusion, and avoiding data that appears online with its solution [95]. Deng and colleagues add benchmarks built from recent text, dynamic refreshing, and protection of ground-truth labels [78]. Sainz and colleagues call for a registry of contamination cases [76]; the 2024 CONDA shared task's database held 566 entries over 91 contaminated sources from 23 contributors [96].

Dating a release. Several benchmarks make contamination questions answerable by date. LiveCodeBench tags each problem with its release date and, for a newer model, scores only the problems released after the model's cutoff [97]. LiveBench adds questions monthly and holds each month's new questions back for a month [98]. NewTerm collects the terms that the Cambridge, Collins, and Oxford online dictionaries add each year and plans annual updates [99]. WikiMIA uses Wikipedia timestamps and model release dates to separate text a model could have seen from text it could not [79]. Deng and colleagues caution that a time cutoff works only if data before and after it differ, since later text can repeat earlier text [78]. Resources date themselves too: wordfreq describes its data as a snapshot through about 2021 and gives text generated by language models as a reason it will not be updated [40].

How the universe applies this. The release's date and file hashes are recorded so that a later contamination question can be dated [100]. The dataset card carries a canary string, also recorded in the release record and in each Parquet file's metadata, so that a later test can check whether a model has memorized the dataset's documentation [100]. Each published file's sha256 is on the card [100], as Croissant recommends for versioned datasets [62].

What the universe adds, and what it lacks

What it records. For each candidate, the universe records its status in each of 8 published lists: WordNet 3.0, the English sections of a dated Wiktionary dump, the 4 LibreOffice hunspell dictionaries, ENABLE, and SCOWL [100]. For the 7 lists recorded as yes or no, each value is 1 (listed), 0 (not listed), -1 (the list cannot hold strings of that class, as a list without hyphens cannot hold a hyphenated string), or null (not measured); for SCOWL, the value is the size of the smallest SCOWL list that holds the string, 0 where none does, -1 where the class does not apply, and null where not measured [100]. It also records whether a string is a type of the 2020 Google Books English 1-grams and whether wordfreq 3.1.1 scores it [100]; the intake channel by which the string entered the pool, kept apart from containment [100]; and whether the English OpenList decided a row of its valid list with a language model [100]. Its candidates are the output of an executable grammar that admits hyphenated compounds, accented loanwords, contractions, apostrophe forms, ligature and diaeresis spellings, and multiword terms [100]. Each row's identifier is derived from the string alone, so the same string carries the same identifier in every version [100].

Against the resources above.

  • Per-list listing. The word lists above record their own membership. The nearest to a per-source record are ESDB, whose tags sometimes name the source list a word came from [24], and Wordnik's paid game dataset, which labels words by source [2].
  • Not measured is kept apart from not listed. wordfreq returns a default value of 0 for a word it does not hold [39]; in the universe, null means not measured by that source, never false [100].
  • Dated, fixed listings. Several comparable resources change between readings: kaikki.org re-extracts its data usually at least weekly [12], DBnary makes one version per Wikimedia dump [14], and Open English WordNet releases annual editions [7]. The universe's listings are fixed at pinned versions, and each published file's hash is on its card [100].
  • Strings with spaces and hyphens. SCOWLv1 excluded hyphenated and open compounds [24]; ESDB [24], Moby's compound list [31], and web2a [37] hold some. The universe states the grammar it admits strings under and marks each class [100].
  • Existence checks in benchmarks. Benchmarks of invented terms check existence when they are built and drop what the check finds: PhantomBench keeps a concept only if it has zero exact matches in the Dolma v1.7 corpus [101]; HypoTermQA removed every generated term with a web search result, keeping 790 of 1,000 [102]; and a vocabulary-test benchmark excluded 2 Spanish nonwords after finding them in dictionaries [103]. The universe records, per string, which published lists hold it and which were not measured, as a stored record rather than a filter [100].

What it lacks. The universe records no definitions or senses, which WordNet [6], Open English WordNet [8], Wiktionary and its extractions [10], and DBnary [13] record. It records no part of speech, though a candidate listed by WordNet could take one from WordNet [100]; WordNet [6], Open English WordNet [7], ESDB [24], Moby Part-of-Speech II [32], the SPECIALIST Lexicon [16], MorphyNet [15], and SUBTLEX-UK [50] each record one. It records no pronunciation, which CMUdict [19], Wiktextract [10], and part of Open English WordNet [8] record, and no etymology, which Wiktextract records [10]. It records no lemma or inflection relations [100], which ESDB [24], MorphyNet [15], and the SPECIALIST Lexicon [16] record. It records a count from the Google Books n-grams for most single-token rows, a corpus of scanned books, but a frequency drawn from many kinds of text, wordfreq's, for few rows [100]; wordfreq [39], the Google Books n-grams [42], and SUBTLEX [48, 50] supply frequencies. It records no behavioral measures, which the English Lexicon Project provides [52]; no spelling-variant levels, which SCOWL and ESDB record [21, 24]; and no variety labels, only 2 proxies from the hunspell dictionaries [100]. It records no validity: no row says that a string is a word [100].

References

  1. North American Scrabble Players Association, "SOWPODS," NASPA Wiki. https://www.scrabbleplayers.org/w/SOWPODS. Read October 26, 2026.
  2. Wordnik, "Wordnik For Game Developers," developer.wordnik.com. https://developer.wordnik.com/gameset. Read October 26, 2026.
  3. Princeton University, "WordNet: A Lexical Database for English," home page. https://wordnet.princeton.edu/. Read October 26, 2026, through the Internet Archive capture of October 6, 2026 (the live site refused the request).
  4. Princeton University, "wnstats(7WN): WordNet 3.0 database statistics," WordNet documentation. https://wordnet.princeton.edu/documentation/wnstats7wn. Read October 26, 2026, through the Internet Archive capture of October 3, 2026.
  5. Princeton University, "License and Commercial Use of WordNet," WordNet documentation. https://wordnet.princeton.edu/license-and-commercial-use. Read October 26, 2026, through the Internet Archive capture of October 4, 2026.
  6. Princeton University, "wndb(5WN): WordNet database files," WordNet documentation. https://wordnet.princeton.edu/documentation/wndb5wn. Read October 26, 2026, through the Internet Archive capture of May 10, 2026.
  7. Open English WordNet team, "english-wordnet: README.md, LICENSE.md, and FORMAT.md," GitHub repository globalwordnet/english-wordnet, branch main. https://github.com/globalwordnet/english-wordnet. Read October 26, 2026.
  8. Open English WordNet team, "Downloads," en-word.net. https://en-word.net/downloads. Read October 26, 2026.
  9. Wiktionary contributors, "Wiktionary:Statistics," English Wiktionary, revision 92513152, table generated from the October 1, 2026 dump. https://en.wiktionary.org/wiki/Wiktionary:Statistics. Read October 26, 2026.
  10. T. Ylonen, "Wiktextract: Wiktionary as Machine-Readable Structured Data," in Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), Marseille, pp. 1317–1325, 2022. https://aclanthology.org/2022.lrec-1.140/.
  11. Wikimedia Foundation, "Terms of Use," effective June 7, 2023, section 7, "Licensing of Content". https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use. Read October 26, 2026.
  12. kaikki.org, "English machine-readable dictionary; Raw data downloads; dictionary index," kaikki.org. https://kaikki.org/dictionary/English/index.html, https://kaikki.org/dictionary/rawdata.html, and https://kaikki.org/dictionary/index.html. Read October 26, 2026.
  13. G. Sérasset, "Dbnary – Wiktionary as Linguistic Linked Open Data," home page. http://kaiko.getalp.org/about-dbnary/. Read October 26, 2026, through the Internet Archive capture of October 11, 2026 (the live site did not respond).
  14. G. Sérasset, "DBnary - Wiktionary Lexical Data in RDF/Ontolex - All languages, version 20260601," Zenodo. https://doi.org/10.5281/zenodo.21496349 (concept record https://doi.org/10.5281/zenodo.10040982). Read October 26, 2026.
  15. MorphyNet project, "MorphyNet: README.md," GitHub repository kbatsuren/MorphyNet. https://github.com/kbatsuren/MorphyNet. Read October 26, 2026.
  16. National Library of Medicine, Lexical Systems Group, "The SPECIALIST Lexicon," lhncbc.nlm.nih.gov. https://lhncbc.nlm.nih.gov/LSG/Projects/lexicon/current/web/index.html. Read October 26, 2026.
  17. National Library of Medicine, Lexical Systems Group, "SPECIALIST Lexicon: Word Count Stats," lhncbc.nlm.nih.gov. https://lhncbc.nlm.nih.gov/LSG/Projects/lexicon/current/docs/designDoc/UDF/wordStats/index.html. Read October 26, 2026.
  18. National Library of Medicine, Lexical Systems Group, "Terms and Conditions for Use of the SPECIALIST NLP Tools," lhncbc.nlm.nih.gov. https://lhncbc.nlm.nih.gov/LSG/Projects/lexicon/current/web/termsAndConditions.html. Read October 26, 2026.
  19. Carnegie Mellon University Speech Group, "The CMU Pronouncing Dictionary," speech.cs.cmu.edu. http://www.speech.cs.cmu.edu/cgi-bin/cmudict. Read October 26, 2026.
  20. Carnegie Mellon University, "cmudict: README and LICENSE," GitHub repository cmusphinx/cmudict. https://github.com/cmusphinx/cmudict. Read October 26, 2026.
  21. K. Atkinson, "SCOWLv1 Readme," wordlist.aspell.net, commit b22230c of July 22, 2024. https://wordlist.aspell.net/scowl_v1-readme/. Read October 26, 2026.
  22. K. Atkinson, "News – ESDB," wordlist.aspell.net. https://wordlist.aspell.net/news/. Read October 26, 2026.
  23. K. Atkinson and others, "Copyright," SCOWL release 2020.12.07, GitHub repository en-wl/wordlist, tag rel-2020.12.07. https://raw.githubusercontent.com/en-wl/wordlist/rel-2020.12.07/scowl/Copyright. Read October 26, 2026.
  24. K. Atkinson, "English Speller Database (ESDB): README.md," GitHub repository en-wl/wordlist, branch v2. https://github.com/en-wl/wordlist/blob/v2/README.md. Read October 26, 2026.
  25. K. Atkinson, "Word Sources and Tags (docs/sources.md)," GitHub repository en-wl/wordlist, branch v2. https://github.com/en-wl/wordlist/blob/v2/docs/sources.md. Read October 26, 2026.
  26. K. Atkinson, "English Speller Database (ESDB)," home page. https://wordlist.aspell.net/. Read October 26, 2026.
  27. K. Atkinson, "Copyright," GitHub repository en-wl/wordlist, branch v2. https://github.com/en-wl/wordlist/blob/v2/Copyright. Read October 26, 2026.
  28. LibreOffice contributors, "README_en_US.txt, README_en_CA.txt, README_en_AU.txt, and README_en_GB.txt," GitHub repository LibreOffice/dictionaries, folder en, branch master. https://github.com/LibreOffice/dictionaries/tree/master/en. Read October 26, 2026.
  29. M. Cooper and A. Beale, "ENABLE (README-enable2k.txt)," mirrored in GitHub repository BartMassey/wordlists, commit 2845b64. https://raw.githubusercontent.com/BartMassey/wordlists/2845b642405f6e63db0af966a8260e99366d5b3e/README-enable2k.txt. Read October 26, 2026.
  30. A. Beale, "Version 6 of the 12dicts word lists (readme, release 6.0.2); 12Dicts Package," wordlist.aspell.net. https://wordlist.aspell.net/12dicts-readme/ and https://wordlist.aspell.net/12dicts/. Read October 26, 2026.
  31. G. Ward, "Moby Word Lists (Moby Words II documentation notes)," Project Gutenberg eBook 3201, released May 1, 2002. https://www.gutenberg.org/ebooks/3201. Read October 26, 2026.
  32. G. Ward, "Moby Part of Speech List (Moby Part-of-Speech II documentation notes)," Project Gutenberg eBook 3203. https://www.gutenberg.org/ebooks/3203. Read October 26, 2026.
  33. M. L. Cooper; rehosted by A. B. Schaefer, "Yet Another Word List (YAWL): README.md," GitHub repository elasticdog/yawl. https://github.com/elasticdog/yawl. Read October 26, 2026.
  34. lorenbrichter/Words maintainers, "Words: README.md and LICENSE," GitHub repository lorenbrichter/Words. https://github.com/lorenbrichter/Words. Read October 26, 2026.
  35. Wordnik, "wordlist: README.md and LICENSE," GitHub repository wordnik/wordlist. https://github.com/wordnik/wordlist. Read October 26, 2026.
  36. dwyl, "english-words: README.md, LICENSE.md, and repository description," GitHub repository dwyl/english-words. https://github.com/dwyl/english-words. Read October 26, 2026.
  37. J. A. Woods and the FreeBSD Project, "README (share/dict)," FreeBSD source tree. https://cgit.freebsd.org/src/plain/share/dict/README. Read October 26, 2026.
  38. Library of Congress, Copyright Office, "U.S. Copyright Renewals, 1961 July - December," Project Gutenberg eBook 11824, a transcription of the Catalog of Copyright Entries, lines 26239–26256. https://www.gutenberg.org/ebooks/11824. Read October 26, 2026.
  39. R. Speer, "wordfreq: README.md," GitHub repository rspeer/wordfreq, branch master. https://github.com/rspeer/wordfreq. Read October 26, 2026.
  40. R. Speer, "Why wordfreq will not be updated (SUNSET.md)," GitHub repository rspeer/wordfreq, branch master. https://github.com/rspeer/wordfreq/blob/master/SUNSET.md. Read October 26, 2026.
  41. Python Package Index, "wordfreq," project metadata. https://pypi.org/project/wordfreq/. Read October 26, 2026.
  42. Google Books Ngram Viewer team, "Google Books Ngram Exports," storage.googleapis.com. https://storage.googleapis.com/books/ngrams/books/datasetsv3.html. Read October 26, 2026.
  43. J.-B. Michel, Y. K. Shen, A. P. Aiden, A. Veres, M. K. Gray, W. Brockman, The Google Books Team, J. P. Pickett, D. Hoiberg, D. Clancy, P. Norvig, J. Orwant, S. Pinker, M. A. Nowak, and E. L. Aiden, "Quantitative analysis of culture using millions of digitized books," Science, vol. 331, no. 6014, pp. 176–182, 2011. https://doi.org/10.1126/science.1199644.
  44. Y. Lin, J.-B. Michel, E. L. Aiden, J. Orwant, W. Brockman, and S. Petrov, "Syntactic Annotations for the Google Books Ngram Corpus," in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, System Demonstrations, pp. 169–174, 2012. https://aclanthology.org/P12-3029/.
  45. Google Books Ngram Viewer team, "Google Books Ngram Viewer: About," books.google.com. https://books.google.com/ngrams/info. Read October 26, 2026.
  46. E. A. Pechenick, C. M. Danforth, and P. S. Dodds, "Characterizing the Google Books Corpus: Strong Limits to Inferences of Socio-Cultural and Linguistic Evolution," PLoS ONE, vol. 10, no. 10, e0137041, 2015. https://doi.org/10.1371/journal.pone.0137041.
  47. M. Brysbaert and B. New, "Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English," Behavior Research Methods, vol. 41, no. 4, pp. 977–990, 2009. https://doi.org/10.3758/BRM.41.4.977. (Accepted manuscript read.)
  48. Ghent University, Department of Experimental Psychology, "SUBTLEXus," ugent.be. https://www.ugent.be/pp/experimentele-psychologie/en/research/documents/subtlexus. Read October 26, 2026.
  49. M. Brysbaert, "SUBTLEX_US word frequency database (with license.txt of May 7, 2024)," OSF project djpqz. https://osf.io/djpqz/. Read October 26, 2026.
  50. W. J. B. van Heuven, P. Mandera, E. Keuleers, and M. Brysbaert, "SUBTLEX-UK: A new and improved word frequency database for British English," Quarterly Journal of Experimental Psychology, vol. 67, no. 6, pp. 1176–1190, 2014. https://doi.org/10.1080/17470218.2013.850521. (Author manuscript read.)
  51. M. Brysbaert, "SUBTLEX-UK word frequencies (with license.txt of May 7, 2024)," OSF project zq49t. https://osf.io/zq49t/. Read October 26, 2026.
  52. D. A. Balota, M. J. Yap, M. J. Cortese, K. A. Hutchison, B. Kessler, B. Loftis, J. H. Neely, D. L. Nelson, G. B. Simpson, and R. Treiman, "The English Lexicon Project," Behavior Research Methods, vol. 39, no. 3, pp. 445–459, 2007. https://doi.org/10.3758/BF03193014.
  53. English Lexicon Project, Washington University in St. Louis, "Home page; About The English Lexicon Project," elexicon.wustl.edu. https://elexicon.wustl.edu/. Read October 26, 2026, through the Internet Archive captures of October 14, 2026 (home page) and August 25, 2019 (About page).
  54. T. Brants and A. Franz, "Web 1T 5-gram Version 1 (LDC2006T13)," Linguistic Data Consortium catalog, 2006. https://catalog.ldc.upenn.edu/LDC2006T13. Read October 26, 2026.
  55. T. Gebru, J. Morgenstern, B. Vecchione, J. Wortman Vaughan, H. Wallach, H. Daumé III, and K. Crawford, "Datasheets for Datasets," arXiv:1803.09010v8, December 1, 2021; published in Communications of the ACM, December 2021. https://arxiv.org/abs/1803.09010.
  56. E. M. Bender and B. Friedman, "Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science," Transactions of the Association for Computational Linguistics, vol. 6, pp. 587–604, 2018. https://doi.org/10.1162/tacl_a_00041.
  57. E. M. Bender, B. Friedman, and A. McMillan-Major, "A Guide for Writing Data Statements for Natural Language Processing" (schema version 2), University of Washington Tech Policy Lab, 2021. https://techpolicylab.uw.edu/wp-content/uploads/2021/11/Data_Statements_Guide_V2.pdf.
  58. A. McMillan-Major and E. M. Bender, "Data Statements for Natural Language Processing: README.md and template-v3.md (schema version 3, 2024)," GitHub repository TechPolicyLab/Data-Statements. https://github.com/TechPolicyLab/Data-Statements. Read October 26, 2026.
  59. M. Pushkarna, A. Zaldivar, and O. Kjartansson, "Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI," arXiv:2204.01075v1, 2022 (ACM FAccT 2022). https://arxiv.org/abs/2204.01075.
  60. Hugging Face, "datasetcard_template.md," GitHub repository huggingface/huggingface_hub, branch main. https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md. Read October 26, 2026.
  61. M. Akhtar, O. Benjelloun, C. Conforti, L. Foschini, and others, "Croissant: A Metadata Format for ML-Ready Datasets," in Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), Datasets and Benchmarks Track, 2024; arXiv:2403.19546v3. https://arxiv.org/abs/2403.19546.
  62. O. Benjelloun and E. Simperl, editors, "Croissant Format Specification, Version 1.1 (published January 29, 2026)," MLCommons, GitHub repository mlcommons/croissant. https://github.com/mlcommons/croissant/blob/main/docs/croissant-spec-1.1.md. Read October 26, 2026.
  63. MLCommons, "Croissant Format Specification, Version 1.0 (published March 1, 2024)," GitHub repository mlcommons/croissant. https://github.com/mlcommons/croissant/blob/main/docs/croissant-spec.md. Read October 26, 2026.
  64. M. Akhtar, N. Jain, J. Giner-Miguelez, O. Benjelloun, E. Simperl, L. Aroyo, R. Shinde, and M. Kuchnik, "Croissant RAI Specification, Version 1.0 (published March 6, 2024)," MLCommons, GitHub repository mlcommons/croissant. https://github.com/mlcommons/croissant/blob/main/docs/croissant-rai-spec.md. Read October 26, 2026.
  65. NeurIPS 2026, "Call For Evaluations & Datasets 2026," neurips.cc. https://neurips.cc/Conferences/2026/CallForEvaluationsDatasets. Read October 26, 2026.
  66. M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, and others, "The FAIR Guiding Principles for scientific data management and stewardship," Scientific Data, vol. 3, 160018, 2016. https://doi.org/10.1038/sdata.2016.18.
  67. TEI Lex-0 contributors, "TEI Lex-0: A baseline encoding for lexicographic data, version 0.9.5 (February 8, 2026)," lex-0.org, chapter 1, "Header". https://lex-0.org/ and https://lex-0.org/header.html. Read October 26, 2026.
  68. T. Tasovac, L. Romary, E. Tóth-Czifra, and I. Marinski, "D6.4 Lexicographic Data Seal of Compliance," ELEXIS project deliverable, July 30, 2021. https://elex.is/wp-content/uploads/2021/08/ELEXIS_D6_4_Lexicographic_Data_Seal_of_Compliance.pdf.
  69. P. Labropoulou, K. Gkirtzou, M. Gavriilidou, M. Deligiannis, and others, "Making Metadata Fit for Next Generation Language Technology Platforms: The Metadata Schema of the European Language Grid," in Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp. 3428–3437, 2020. https://aclanthology.org/2020.lrec-1.420/.
  70. N. Calzolari, R. Del Gratta, G. Francopoulo, J. Mariani, F. Rubino, I. Russo, and C. Soria, "The LRE Map. Harmonising Community Descriptions of Resources," in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012), 2012. https://aclanthology.org/L12-1450/.
  71. J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner, "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus," arXiv:2104.08758v2, 2021 (EMNLP 2021). https://arxiv.org/abs/2104.08758.
  72. Y. Elazar, A. Bhagia, I. Magnusson, A. Ravichander, D. Schwenk, A. Suhr, P. Walsh, D. Groeneveld, L. Soldaini, S. Singh, H. Hajishirzi, N. A. Smith, and J. Dodge, "What's In My Big Data?," in Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024); arXiv:2310.20707v2. https://arxiv.org/abs/2310.20707.
  73. A. Rauber, A. Asmi, D. van Uytvanck, and S. Proell, "Data Citation of Evolving Data: Recommendations of the Working Group on Data Citation (WGDC)," Research Data Alliance, October 20, 2015. https://doi.org/10.15497/RDA00016.
  74. H. N. Andreassen, A. L. Berez-Kroeker, L. Collister, P. Conzett, C. Cox, K. De Smedt, B. McDonnell, and the Research Data Alliance Linguistic Data Interest Group, "Tromsø recommendations for citation of research data in linguistics," Research Data Alliance, December 4, 2019. https://doi.org/10.15497/RDA00040.
  75. M. Gavrilidou, P. Labropoulou, E. Desipri, S. Piperidis, H. Papageorgiou, M. Monachini, F. Frontini, T. Declerck, G. Francopoulo, V. Arranz, and V. Mapelli, "The META-SHARE Metadata Schema for the Description of Language Resources," in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012), 2012. https://aclanthology.org/L12-1593/.
  76. O. Sainz, J. A. Campos, I. García-Ferrero, J. Etxaniz, O. Lopez de Lacalle, and E. Agirre, "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark," in Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776–10787, 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.722.
  77. C. Xu, S. Guan, D. Greene, and M-T. Kechadi, "Benchmark Data Contamination of Large Language Models: A Survey," arXiv:2406.04244v1, 2024. https://arxiv.org/abs/2406.04244.
  78. C. Deng, Y. Zhao, Y. Heng, Y. Li, J. Cao, X. Tang, and A. Cohan, "Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation," in Findings of the Association for Computational Linguistics: ACL 2024, pp. 16078–16092, 2024. https://doi.org/10.18653/v1/2024.findings-acl.951.
  79. W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, "Detecting Pretraining Data from Large Language Models," in Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024); arXiv:2310.16789v3. https://arxiv.org/abs/2310.16789.
  80. S. Golchin and M. Surdeanu, "Time Travel in LLMs: Tracing Data Contamination in Large Language Models," in Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024); arXiv:2308.08493v3. https://arxiv.org/abs/2308.08493.
  81. Y. Oren, N. Meister, N. Chatterji, F. Ladhak, and T. B. Hashimoto, "Proving Test Set Contamination in Black Box Language Models," arXiv:2310.17623v2, 2023. https://arxiv.org/abs/2310.17623.
  82. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, and others, "Language Models are Few-Shot Learners," arXiv:2005.14165v4, 2020. https://arxiv.org/abs/2005.14165.
  83. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramèr, and C. Zhang, "Quantifying Memorization Across Neural Language Models," in Proceedings of the Eleventh International Conference on Learning Representations (ICLR 2023); arXiv:2202.07646v3. https://arxiv.org/abs/2202.07646.
  84. A. Srivastava, A. Rastogi, A. Rao, and others, "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models," Transactions on Machine Learning Research, 2023; arXiv:2206.04615v3. https://arxiv.org/abs/2206.04615.
  85. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, "GPQA: A Graduate-Level Google-Proof Q&A Benchmark," arXiv:2311.12022v1, 2023. https://arxiv.org/abs/2311.12022.
  86. J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, and others, "BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents," arXiv:2504.12516v1, 2025. https://arxiv.org/abs/2504.12516.
  87. G. Leech, J. J. Vazquez, N. Kupper, M. Yagudin, and L. Aitchison, "Questionable practices in machine learning," arXiv:2407.12220v2, 2024. https://arxiv.org/abs/2407.12220.
  88. Anthropic, "System Card: Claude Opus 4.5," November 2025, section 2.2, "Decontamination." https://www.anthropic.com/claude-opus-4-5-system-card.
  89. A. Chowdhery, S. Narang, J. Devlin, and others, "PaLM: Scaling Language Modeling with Pathways," arXiv:2204.02311v5, 2022. https://arxiv.org/abs/2204.02311.
  90. OpenAI, "GPT-4 Technical Report," arXiv:2303.08774v6, 2023 (revised March 4, 2024). https://arxiv.org/abs/2303.08774.
  91. S. Chen, Y. Chen, Z. Li, Y. Jiang, Z. Wan, Y. He, D. Ran, T. Gu, H. Li, T. Xie, and B. Ray, "Recent Advances in Large Language Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation," arXiv:2502.17521v2, 2025. https://arxiv.org/abs/2502.17521.
  92. Z. Han, M. Mankikar, J. Michael, and Z. Wang, "Search-Time Data Contamination," arXiv:2508.13180v1, 2025. https://arxiv.org/abs/2508.13180.
  93. T. Ishida, T. Lodkaew, and I. Yamane, "CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting," in Proceedings of the 43rd International Conference on Machine Learning, 2026; arXiv:2505.18102v7. https://arxiv.org/abs/2505.18102.
  94. N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song, "The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks," arXiv:1802.08232v3, 2019. https://arxiv.org/abs/1802.08232.
  95. A. Jacovi, A. Caciularu, O. Goldman, and Y. Goldberg, "Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks," in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5075–5084, 2023. https://doi.org/10.18653/v1/2023.emnlp-main.308.
  96. O. Sainz, I. García-Ferrero, A. Jacovi, J. A. Campos, Y. Elazar, E. Agirre, Y. Goldberg, and others, "Data Contamination Report from the 2024 CONDA Shared Task," arXiv:2407.21530v2, 2024. https://arxiv.org/abs/2407.21530.
  97. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code," arXiv:2403.07974v2, 2024. https://arxiv.org/abs/2403.07974.
  98. C. White, S. Dooley, M. Roberts, A. Pal, and others, "LiveBench: A Challenging, Contamination-Limited LLM Benchmark," in Proceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025); arXiv:2406.19314v2. https://arxiv.org/abs/2406.19314.
  99. H. Deng, W. Jiao, X. Liu, M. Zhang, and Z. Tu, "NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates," in Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), Datasets and Benchmarks Track; arXiv:2410.20814v1. https://arxiv.org/abs/2410.20814.
  100. ryanjosephkamp, "The CEOL Universe, version 1.0," dataset card, Hugging Face dataset ryanjosephkamp/ceol-universe, built October 25, 2026; the card's source is pipeline/research/release/CARD_v1.0.md in https://github.com/ryanjosephkamp/ceol-universe. Read October 26, 2026.
  101. H. Jung and H. Gonen, "PhantomBench: Benchmarking the Non-existential Threat of Language Models," arXiv:2606.11105v1, 2026. https://arxiv.org/abs/2606.11105.
  102. C. Uluoglakci and T. Taskaya Temizel, "HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs," in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 95–136, 2024. https://doi.org/10.18653/v1/2024.eacl-srw.9.
  103. G. Martínez, J. Conde, E. Merino-Gómez, B. Bermúdez-Margaretto, J. A. Hernández, P. Reviriego, and M. Brysbaert, "Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models," arXiv:2310.14703v2, 2024. https://arxiv.org/abs/2310.14703.