Sources and licenses
Where the strings come from, under which licenses, and how each row's way in is recorded.
The sources, their licenses, and the credits
The CEOL Universe is licensed CC BY-SA 4.0, the share-alike floor of its
Wiktionary-derived fields. NOTICES.md, beside the data files, carries
every notice, license, and credit the sources require, in their own words
where their terms require the text itself, and each Parquet file names it
in its metadata.
The 2nd column is the license the EOL's source manifest records; the 3rd is what the source's own files state, as the license review read them.
| Source | Recorded | What its own files state | What this dataset carries from it |
|---|---|---|---|
| Wiktionary, English sections and all titles (Wikimedia dumps) | CC-BY-SA-4.0 | CC BY-SA 4.0 and the GFDL, under the Wikimedia Terms of Use | wiktionary_english, wikt_title_screen, wikt_title_any, the spaced rows' listing, wk_proper_noun, wk_flags |
| WordNet 3.0 (Princeton University), as distributed with nltk | WordNet-3.0 | the WordNet 3.0 license, which NOTICES.md carries verbatim | wordnet, the spaced rows' listing, wn_instance, wn_capitalized |
| SCOWL (Kevin Atkinson; the December 7, 2020 archive) | permissive (SCOWL) | SCOWL's Copyright file, with its components' notices, which NOTICES.md carries verbatim | scowl_tier |
| LibreOffice hunspell dictionaries en_US, en_CA, en_AU | LGPL-2.1-or-later OR MPL-1.1 | their READMEs: they "come directly from SCOWL and is thus under the same copyright of SCOWL" | hunspell_en_US, hunspell_en_CA, hunspell_en_AU, the variety proxies |
| LibreOffice hunspell dictionary en_GB | LGPL-2.1-or-later OR MPL-1.1 | its README: begun from Kevin Atkinson's list and covered by his original LGPL license, since extensively updated by others; its credits include OpenStreetMap's contributors | hunspell_en_GB, the variety proxies |
| ENABLE (Alan Beale and M. Cooper) | public domain | its public-domain statement | enable1 |
| wordfreq 3.1.1 (Robyn Speer) | MIT | its README: the code under the Apache license, and data files that may be redistributed under CC BY-SA 4.0 | wordfreq_zipf, wordfreq_scored |
| Google Books Ngram corpus, English 1-grams, the 2020 release | CC-BY-3.0, aggregates only | its export page: a Creative Commons Attribution 3.0 Unported License | the gb_ columns, zipf_log10_rank, zipf_resid, gb_type |
The EOL universe at revision c1139698 | MIT | its card: the MIT License | the inherited strings, eol, eol_universe, the provenance layer's eol: channels |
Word-game material in 2 published lists. ENABLE's README says it
reflects the changes to the Official Scrabble Players Dictionary in effect
from January 1, 1998, and SCOWL's lists at size 80 and above include the
lists of the ENABLE supplement package, which SCOWL's successor documents
as holding additional and stale entries of that dictionary. Both lists'
own terms allow the use this dataset makes of them, so enable1 and
scowl_tier are published. Where they agree with a withheld game
list, the agreement is not independent evidence; a filter can leave out
ENABLE and SCOWL's sizes 80 and above to count without them.
Credits. Wiktionary's contributors, and the Free Dictionary API, which draws on Wiktionary; Princeton University for WordNet; Kevin Atkinson and the contributors named in SCOWL's copyright notice; the LibreOffice dictionary maintainers, among them Marco A.G. Pinto for en_GB; Alan Beale and M. Cooper for ENABLE; Robyn Speer for wordfreq (Zenodo, doi:10.5281/zenodo.7199437); the Google Books Ngram corpus (https://books.google.com/ngrams). wordfreq's data includes the SUBTLEX word lists of Marc Brysbaert and his colleagues, which wordfreq distributes by the authors' permission on 2 conditions: that wordfreq and code derived from it credit the SUBTLEX authors, and that it remain clear that SUBTLEX is freely available data. The page wordfreq's README names is no longer served; its last capture by the Internet Archive is of November 16, 2023 (https://web.archive.org/web/20231116144719/http://crr.ugent.be/programs-data/subtitle-frequencies). SUBTLEX-US is offered on Ghent University's pages (https://www.ugent.be/pp/experimentele-psychologie/en/research/documents/subtlexus) and SUBTLEX-UK on OSF (https://osf.io/zq49t/). The authors' current copies carry the license CC BY-NC-SA 4.0; this dataset's values come through wordfreq 3.1.1, under wordfreq's statement of the permission.
The intake provenance
The intake-provenance layer (pipeline/research/provenance/provenance.json)
records 2 senses of provenance and keeps them apart: the intake channel,
the route by which a string first entered the pool (for inherited strings,
the EOL's discovery log, which records where a string was first seen and is
never an attestation), and the measured containment in the published panel
sets and the 2 corpora. Over the published fields,
8,920,645 rows have both, 585,792 a
channel and no containment, 497,340 containment and no
channel, and 16,373 neither. The count of the last rests on a
reading of the EOL's correction ledgers that labels as generated the forms
the EOL demoted, which the same record states. 330,386 rows were
made by the EOL's synthetic engine; each carries intake_kind
generated, the category the EOL recorded, and an engine inferred from
its form and named as inferred.