| Re: [asciidoc-lang-dev] Whitespace handling |
On 12/03/2021 00:09, Lex Trotman wrote:> > > On Tue, 9 Mar 2021 at 19:30, Sylvain Leroux <sylvain@xxxxxxxxxxx > <mailto:sylvain@xxxxxxxxxxx>> wrote: > > On 09/03/2021 01:20, Lex Trotman wrote: > > Ahh yes another extension I did was that all non-terminals and tokens > > can return a numeric value (if they match) and I added a simple syntax > > for assigning that value if its needed, and comparing it to the > > parameter passed to the non-terminal. Tokens return a count of > relevant > > characters that varies from token to token but is effectively the > level > > for section/list tokens or length of delimiter for block delimiters. > > The (simplified) syntax for a "section" would be: > > > > Section(level) ::= token_level = <start_line_equals> > > (:token_level==level:) Markup_text_line Section_contents > *Section(level+1) > > > > where <line_start_equals> is the token from the lexer, > (:expression:) is > > a test that will fail the non-terminal if it fails, and name= assigns > > the value to name. These have the obvious implementation in code. > > > Yes, that's a problem for a PEG-based _implementation_. For the specs, I > suppose we can add such guard clauses without causing too much harm. > > > On thinking about it some more, I guess "can count" is true for any > recursive language, [...] "can count" is a statement I heard years ago in a CS course. To be a little bit more formal, as far as I remember: * a regular language is decidable by a finite state automaton. So it can count without bound a^n (one term repeated an arbitrary number of times). * a context-free language is decidable by a pushdown automaton, and thus it can count without bound (a^n)(b^n) (two terms repeated the same number of times). * a context-sensitive language is decidable by a linear bounded automaton. It can count without bounds (a^n)(b^n)(c^n) and similar language with more letters. Context-sensitive languages are a proper subset of recursive languages. And context-free languages are a proper subset of context-sensitive languages. So if you have an automaton for a recursive language, it's *largely* sufficient to recognize (a^n)(b^n) Take all that with the (huge) grain of salt it deserves, given my study years are far away. Now, back to our topic: > [...] if you recurse on each character of a block > delimiter then return on each character of the next you should be back > where you started, otherwise its not a matching delimiter. But > testing of section and list depth is trickier. Its the way some > functional languages do looping (and C++ Template Metaprogramming). But > that sort of recursion isn't really the sort of thing I would try using > in a specification, only full Computer Scientists would understand, and > we want the specification to be clear to writers checking edge cases of > their documents I would think. > I tried something like that a couple of days ago: block := '=' block '=' | block-content block-content := EOL block-list block-list := (text EOL | block EOL ) block-list | ε ^^^^^ As you said, the tricky part is when dealing with nested blocks. I didn't find a way to handle the constraint "the nested block delimiter's length must be different from the length of the delimiter for any enclosing block." For that you need to keep track of all open parent blocks and that didn't fit well with my approach of the problem. This issue is somewhat related to the question I asked yesterday (https://www.eclipse.org/lists/asciidoc-lang-dev/msg00141.html). At this point, I gave up trying to implement an RD parser for block parsing. I just tokenize the input and use a stateful object to construct the tree from the tokens stream. The good thing is, that way, I get rid of backtracking while parsing at the block level. And after all, this solution is not that inelegant ;) - Sylvain
Attachment:
signature.asc
Description: OpenPGP digital signature