The glcpp parser is line-based, so it needs to see a NEWLINE token at the end
of each line. This causes a trick for files that end without a final newline.
Previously, the lexer for glcpp punted in this case by unconditionally
returning a NEWLINE token at end-of-file, (causing most files to have an extra
blank line at the end). Here, we refine this by lexing end-of-file as a
NEWLINE token only if the immediately preceding token was not a NEWLINE token.
The patch is a minor change that only looks huge for two reasons:
1. Almost all glcpp test result ".expected" files are updated to drop
the extra newline.
2. All return statements from the lexer are adjusted to use a new
RETURN_TOKEN macro that tracks the last-token-was-a-newline state.
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Consider this test case:
#define EMPTY
int foo = 1+EMPTY+4;
The expression should compile as the sequence of tokens 1, PLUS,
UNARY_POSITIVE, 4. But glcpp has been failing for this case since it
results in the string "1++4" which a compiler correctly sees as a
syntax error, (1, POST_INCREMENT, 4).
We fix this by changing any macro with an empty definition to result
in a single SPACE token rather than nothing. This then gives "1+ +4"
which compiles correctly.
This commit does touch up the two existing test cases which already
have empty macros, (to add the space to the expected result).
It also adds a new test case to exercise the above scenario.
