[asciidoc-lang-dev] Whitespace handling

[Sylvain Leroux <sylvain@xxxxxxxxxxx>]

On 28/02/2021 11:37, Dan Allen wrote:
>> Lex:
>>   "Asciidoc spacing characters" is fine by me, if a bit long, but
> {asc} can fix that :-)
> 
> Excellent. We will certainly define it.

FWIW, it suits me well too ;)

>> > Dan wrote:
>> One thing we will need to be careful about, though, is that AsciiDoc
> doesn't support *all* spacing characters. So we'll just need to
> emphasize that in our definition / usage.
>> Lex wrote:
>> I wonder if the standard should consider changing that?
> 
> Perhaps. Though I think one of the values of AsciiDoc is that it's not
> cryptic. So limiting spacing characters to space, tab, and line feed has
> merit. When we start to allow other spacing characters, then I think it
> introduces inconsistently from one document to the next. I think a case
> has to be made for each individual character we permit. That way, we
> know why we permitted it.

I already discussed that idea with Dan elsewhere. I assume it's not a
problem if I quote him here: "A valid space character is a space, a tab,
or a line feed (aka newline). It's questionable whether a non-breaking
space should be allowed. But it definitely shouldn't extend beyond that."

My understanding at that time was a parser only has to deal with Ascii
characters (Formally, the C0 Controls and Basic Latin block [1]). So the
only spacing allowed between markups and user's content were \u0009 and
\u0020. That won't prevent any other spacing character inside the text,
at least up to the DOM.

Dan also reminded me at that time `asciidoctor` "normalizes" lines early
in the document processing to remove trailing spaces ([2]). Once again,
I understood "to remove trailing \u0009 and \u0020." That also means
trailing spaces will not make their way into the DOM.

We didn't discuss the case of the `\r`. AFAIK, asciidoctor only
recognizes `\n` as the line terminator. So, we might extend the
normalization process to remove `\r` at the end of a line.


[1]: https://unicode.org/charts/PDF/U0000.pdf
[2]:
https://discuss.asciidoctor.org/Section-without-a-title-tp8466p8468.html

- Sylvain

Attachment: signature.asc
Description: OpenPGP digital signature