By: Tom Sydney Kerckhove <syd@cs-syd.eu>
Add end-of-line comments and comments within declarations
A comment used to be a declaration of its own, so a comment written at the
end of a line, or between the lines of a transaction, was either a parse
error or was moved onto its own line by the formatter.
Every line can now carry a comment at the end of it, and comments on
indented lines of their own. Comments attach to the line above them rather
than the line below them, because the last line of a transaction has a line
above it to attach to but none below, and that keeps the formatter from
moving them.
A comment on a line of its own has to be indented to belong to the
declaration above it: a "--" at the first column starts a new declaration.
Without that distinction, a comment after the last posting of a transaction
is a shift/reduce conflict.
The one line that cannot carry a comment at the end of it is a description,
because a description is the whole rest of its line.
Fix two bugs in the process, both of which the extra comment positions would
otherwise have made much easier to hit:
* A comment whose text was only whitespace swallowed the declaration below
it. Nothing could match the rest of the line, so '$white+' matched the
newline instead and left the lexer in the comment start code, reading the
next declaration as comment text. A comment is now a single token, so
there is no start code left to be stranded in.
* A file path was lexed with a character set containing an unescaped '.',
which means "any character but a newline", so a path ran to the end of
the line. A trailing comment ended up inside the path. A path may now
contain spaces but not the "-- " that starts a comment.