David,
Thank you for sharing these thoughts. While this landmark is only the beginning of the journey, the preparation alone makes me feel like we've already accomplished so much. Just by extracting the documentation for the language, it has made it so much clearer what we're working towards.
As I mentioned in the kickoff email (
https://www.eclipse.org/lists/asciidoc-lang-dev/msg00001.html), either Sarah or I will be following up shortly with a thread about how the contribution process works so we can get on the same page about it and discuss how to collaborate most effectively. That should address your first set of questions about how and where to participate.
Before I respond to your language related questions, I want to remind everyone that the focus of the first version of the specification is to describe AsciiDoc as we know it today. Compatibility with existing content for version 1.0 is paramount. That way, changes to the language in subsequent versions can be acknowledged and traceable back to a well-defined starting point. We do anticipate making a handful of amendments to address outdated syntax, acute pain points, or aspects where it's more difficult to specify what exists than it is to just fix it. These are going to be tough calls, but there will be ample time to discuss each one.
With that said...
> David wrote:
> Open block syntax: ‘~’ as the open block delimiter.
My position is: absolutely. Not having a nestable open block is seriously holding back the language. And it's something we can add without impacting any existing content. If you don't mind, I would like to file this spec issue since I already have an existing proposal / parallel issue in the Asciidoctor project. (Feel free to nudge me if I forget).
> Macro attribute list parsing
We certainly need to tighten the rules for element attribute list parsing. Some of the oddies are due to "bridge syntax" that was introduced to help people transition when we added attribute list support to elements that didn't originally have them, like the link macro. By now, that migration has likely happened, so we can consider making attribute lists on elements more universal. In general, we should favor common usage rather than esoteric used edge cases. As you said, the simpler the better.
> File system” independence.
This is probably something that is going to be more post-1.0. We know we need a stronger referencing system and more extension points. But before we shoot for the moon, we need to define what is. If there's room to use more ambiguous language about how a reference is resolved, I think we should take that opportunity. (For example, we can say that an interdocument reference is a path that the implementation must locate). But we need to ensure that it does not impact existing content.
> “If it looks like an extension, it’s an extension” (except, presumably, for `include::`)
> There is a lot of built-in behavior that is triggered by syntax that looks like a block macro or inline macro.
Remember that an include directive is fundamentally a different thing that a block macro. And that isn't going to change (at least not in 1.0). The include directive and block macro share the same syntax for consistency. But the include directive is a preprocessor hook whereas the block macro is a block element.
One thing that's sorely missing, though, is an extension point that can supply the resolved lines for an include (an include resolver), but not have to deal with any other aspects of include, such as line and tag filtering. That's a pretty simple extension point to add and I think it should be considered for 1.0.
While on the topic of extensions, we may need to consider whether we should allow extensions to override built-in elements like image and stem. That has been requested many, many times. I don't see any reason why we can't say that an extension can claim a built-in name, thus intercepting it from the processor. The only reason we didn't do it in Asciidoctor is that it required moving some code around and we just never got to it. So that seems like a reasonable spec request.
> Grammar!
> How and where are we going to discuss what format our grammar will take and how to develop it?
I plan to start at least one thread, if not several, on this subject. I'm sure this is the question at the forefront of most peoples' minds when we bring up the topic of a specification. Generally speaking, I think we should start at a very high level with EBNF and see how far we can get (if only as an exercise). Once we've reached the limits of that notation, we should drop down into something more low-level / advanced like an ANTLR grammar. It will no doubt be iterative.
> AsciiDoc is not context-free, so a completely generated parser is not likely to be possible.
Agreed. I wouldn't describe the goal as trying to make an auto-generated implementation from a parser generator. I think the goal is to be able to describe in technical detail the rules for parsing. As you said, that may end up requiring considerable communication between the lexer and parser. And we will no doubt have moments where we realize we have a circular dependency in the parsing that we need to break (section titles with auto-generated IDs come to mind). That's part of what makes the AsciiDoc spec different from the more casual implementations that exist today. We may only be able to get halfway to a single grammar in 1.0, then get closer with each revision after that. Regardless, let's take up this topic in a dedicated thread. And, like I said, I think the first effort should be on a very high level (even if incomplete) EBNF grammar.
Best Regards,
-Dan
