The spelling grammar

The box runs the same rules as the dataset's code, pipeline/ceol/grammar.py, which is the grammar's specification. The site's tests compare the two on every vetted example, every sample row, and those strings altered by every rule.

The characters

47 letters and 3 marks. A character outside them fails the grammar instead of being changed into something close.

Two characters are rewritten, and nothing else

The typographic apostrophe, U+2019, becomes the plain one, U+0027. The no-break space, U+00A0, becomes the plain space, U+0020. Each is the same character in a different dress, and treating them apart would list a string twice.

Every other look-alike fails instead: the left quotation mark U+2018, the modifier apostrophe U+02BC, the grave and acute accents standing alone, the dashes U+2010, U+2013, and U+2014, and every other kind of space. Folding a character the grammar does not declare would invent data.

Composed, lowered, and never folded

A string is composed with Unicode's NFC form, so a letter and an accent typed separately become 1 character. It is never decomposed with NFKC, which would turn a presentation ligature such as U+FB03 into 3 letters; that ligature fails instead.

Letters are lowered, never case-folded. A case fold turns the German sharp s into "ss" and makes a foreign spelling look English; lowering leaves it in place, where it fails as an undeclared character. The string is composed again after lowering, because lowering can split a letter in two.

The form

• Every token holds at least 1 letter.
• A hyphen sits between 2 parts that each hold a letter: never first, never last, never 2 in a row.
• An apostrophe may begin a token, as in a leading elision, but never ends one and never doubles. So this version holds no plural possessive, which ends in an apostrophe; a form that ends in an apostrophe and s is held. Plural possessives may come in a later version, generated from their stems once the dataset has parts of speech, and that version would record the change to the grammar.
• Spaces are single and between tokens, never first or last, and each token passes these rules on its own.
• The underscore is never a character. Sources that write a space as an underscore are read with the space before the grammar sees them.

The spelling classes

A string that passes is given flags for what its characters and tokens hold. The flags combine: a string can be hyphenated and accented at once. A string with none is plain a to z.