[asciidoc-lang-dev] Unicode Issues

[Sylvain Leroux <sylvain@xxxxxxxxxxx>]

This probably need a closer attention, but as first toughs:

On 03/03/2021 05:51, Lex Trotman wrote:
> Just a first touch on encoding, I suggest that the standard require
> encodings to be either explicitly specified by the document (BOM or
> :encoding: attribute) or explicitly specified on the command line or be
> assumed to be UTF-8.  If an implementation picks up encodings from the
> environment that will make documents break if processed in another
> environment, so that should be actively discouraged.
Given you can write a valid Asciidoc document using _only_ the 7 bit
ASCII set, I would be more restrictive here:

"An AsciiDoc document is a stream of Unicode code point. Any conforming
processor MUST accept UTF-8 encoded input streams. As an extension,
processors MAY accept other encodings."

> So should AsciiDoc specify that the input must be normalised, or require
> implementations to do so adding cost scanning the document?
We must agree to one of the Unicode Normalization Form ("NFC", "NFD",
"NFKC", or "NFKD" [1]). The specs must be written assuming that
normalization form will be used for internal processing. A processor may
use another internal representation as long as the observable behavior
in conforming with the specs.

[1]: https://unicode.org/reports/tr15/

> 
> But accents can be stacked, especially in some languages past the
> Latin-1 set, and there is no single code point assigned to many such
> multi-combinations, so normalisation won't help, they will still exist
> as multiple code points in AsciiDoc input.  And combining code points
> can be applied to non-letter code points too.
> [...]
> But for the closing `*` the preceding code point of an
> unnormalised accented character is the combining code point class M not
> class L. 
Assuming the NFD normalization form, it's easy to define a LETTER as:

LETTER := UNICODE_L_CLASS_CP (UNICODE_M_CLASS_CP)*

Actually, I wonder if we couldn't simplify that as:

LETTER := UNICODE_L_CLASS_CP | UNICODE_M_CLASS_CP


> This is quite precise, but may become unwieldy as the
> various situations markups can be used in are mixed, ie nested quotes
> like _foo *blah*_ has the * between the _ and the letter.
Assuming a LL parser, "*blah*" was already recognized as an
"inline-strong" non-terminal when "_" is processed. So the rule won't
care about preceding character code:

inline := inline-emphasis | inline-strong | ANY_LETTER_OR_MARK
inline-emphasis := <???> "_" inline (ANY_SPACE* inline)* "_" <???>
inline-strong := <???> "*" inline (ANY_SPACE* inline)* "*" <???>

In my personal implementation, I defined "<???>" as
"not(ANY_LETTER_OR_MARK)". But I'm pretty sure it isn't compliant with
the actual Asciidoctor behavior.


> 
> This may all seem low level and picky, but its my experience that if
> Unicode's peculiarities are not addressed up front they will haunt you
> forever.  And that is of course particularly true of something that
> involves human language handling as AsciiDoc does.
I agree. This is something that has to be specified upfront.

- Sylvain

Attachment: signature.asc
Description: OpenPGP digital signature