Re: [asciidoc-lang-dev] Text Markup, syntax and parsing thereof

[Sylvain Leroux <sylvain@xxxxxxxxxxx>]

Hi Lex,

You brought up many great questions. I answer each main topic
individually as my reflection advance (and _if_ I have something to say).


On 07/03/2021 01:58, Lex Trotman wrote:>
> 1.a. The constraints are context around the markup, not part of the
> token, so it does not violate the requirement that markup be ASCII for
> the context to allow any Unicode.

Agreed.

>
> 1.b. As proposed in another thread initial "letter like" could be
> defined as a Unicode class L code point and final "letter like" could be
> defined as Unicode class L code point followed by any number of Unicode
> class M (combining characters such as accents) code points.

_Or_ as any Unicode class L code point or Unicode class M code point.
The current asciidoctor documentation explicitly states that: "An
AsciiDoc processor always assumes the content is UTF-8 encoded". Are
stray Unicode class M code point forbidden by Unicode? Is this something
we should enforce?

[1]: https://docs.asciidoctor.org/asciidoc/latest/normalization/
>
> 1.c. currently the punctuation allowed is (,;".?!) which are common
> English punctuations but do not include any non-English punctuation.
>
> 1.d. So should common non-English punctuation be allowed and which?
>
> 1.e. Should all Unicode category P punctuation be allowed?
>
> 1.e. Should punctuation be allowed before the initial constrained
markup?
>
> 1.f. Should only Unicode category Pi and Ps be allowed before and Pf and
> Pe and Po after?
>
> 1.g. What is "space" (here I'm talking in the context of constrained
> markup, there is another thread that addresses it more generally), eg
> Unicode category Zs (https://www.compart.com/en/unicode/category/Zs) and
> AsciiDoc line separators?
>
From i18l perspective, allowing "Unicode category Zs | Unicode category
Ps" ([2],[3])before an opening constrained markup seems reasonable. We
can hope this wouldn't raise the number of false positives in markup
detection dramatically. I have no opinion regarding the inclusion or not
of the Unicode category Pi ([4]) in that set.

I have the same reasoning regarding the closing markups, using the
complementary Unicode categories, of course.

[2]: https://www.compart.com/en/unicode/category/Zs
[3]: https://www.compart.com/en/unicode/category/Ps
[4]: https://www.compart.com/en/unicode/category/Pi


> 1.h. Or since unconstrained markup is available should the specification
> be conservative on what is allowed bounding unconstrained markup, the
> markups (*_`#~^) are uncommon in general English text, but tend to occur
> when talking about programming code and math, and I don't know how
> common they are in other languages.  The rules are intended to minimise
> nuisance recognition of such use-cases as markup, so the more situations
> that markup is allowed the more nuisance occurrences are likely.

This is the conservative approach, aiming toward maximizing the
compatibility with the existing implementation. But, even for Latin
scripts, the current implementation is not satisfactory. Think, for
example, to the inverted question and exclamation marks used in Spanish
[5]. So in fine, this has to evolve.

Should we consider that (1.h) option for the v1.x of the specs, knowing
we would evolve toward a more intl-aware solution based on the Unicode
categories in the v2.x? Or should we make the leap right now, assuming a
"good enough" compatibility with the existing document base?

[5]: https://en.wikipedia.org/wiki/Inverted_question_and_exclamation_marks

Attachment: signature.asc
Description: OpenPGP digital signature