[asciidoc-lang-dev] Include expansion and block parsing [Was: Extension

[Guillem Marpons <gmarpons@xxxxxxxxxxx>]

NOTE: I was alerted a few days ago that some users of this mailing list 
couldn't see my messages because they were getting caught by spam 
filters. Since then, some configuration changes have been applied to the 
list. If you happen to detect that my messages keep going to the spam 
folder, please let me know.


--


I take back this message from a few days ago to ask about the include:: 
directive. I haven't seen anyone question what David have said about it 
(which is congruent with my own experiments) and think that the 
consequences are quite far reaching. So I ask for clarification.

El 15/3/21 a las 18:26, David Jencks escribió:
> [...] 
>
> The include:: preprocessor instruction might be more complicated than the C preprocessor, which I’m not really familiar with.  You can use attributes in the include:: instruction, which can be set externally, in the document header, in the document body before the include:: instruction, and by block and block macro processors ‘invoked’ before the include instruction.  I don’t recall verifying that attributes added by block/block macro processors are taken into account by include::, but I think they should be for logical consistency.
> I think this means that the source has to be streamed through the parser perhaps line by line as each line may affect the content of the next line.  (lines may not be the correct unit here, perhaps blocks? include:: directives just need to start a line, but attributes can only be influenced at block boundaries). I think this also puts some constraints on how much lookahead is possible, since the entire “preprocessed source” isn’t known until parsing is finished.
>
> I’ll mention that I think the current IncludeProcessor and Preprocessor extensions don’t really make sense, but I don’t have a clear idea how to deal with these areas in a better way.

To elaborate on what David says: there is a circular dependency between 
include expansion, attribute resolution and block parsing:

* A line with contents

  :key: value

is considered an attribute entry only in some places (e.g., not inside a 
passthrough or verbatim delimited block), so parsing affects attribute 
setting and resolution.

* Attribute value affects include expansion, as David pointed out:

  include::{key}.adoc[]

* Include expansion affects both attribute resolution and parsing. An 
include file can left open any number of delimited blocks or other
constructions, thus affecting subsequent parsing, or the context in 
which a potential attribute entry is parsed. And attributes can be 
defined in and set in included files.

By some tests I've done, even comment strip-out cannot easily being done 
as an actual preprocess: an include can left open a comment block, 
affecting lines external to the included file.

All that means that in current AsciiDoc it's not possible to have an 
actual preprocessing as a separate step. Include expansion needs to be 
performed at the same time of (block) parsing (as David says). Or am I 
missing something?

How are the so called "preprocessor directives" implemented in 
Asciidoctor and other tools?


--
Guillem