fdo-mirrors/mesa
1
0
Fork 0
mirror of https://gitlab.freedesktop.org/mesa/mesa.git synced 2025-12-23 00:20:09 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
mesa/src/glsl/glcpp/glcpp-lex.l

468 lines
12 KiB
Text
Raw Normal View History

Add the tiniest shell of a flex/bison-based parser. It doesn't really *do* anything yet---merlely parsing a stream of whitespace-separated tokens, (and not interpreting them at all).
2010-05-10 11:44:09 -07:00
%{
/*
* Copyright © 2010 Intel Corporation
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice (including the next
* paragraph) shall be included in all copies or substantial portions of the
* Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
* DEALINGS IN THE SOFTWARE.
*/
#include <stdio.h>
#include <string.h>
glcpp: Add #error support.
2010-07-02 15:31:26 -07:00
#include <ctype.h>
Add the tiniest shell of a flex/bison-based parser. It doesn't really *do* anything yet---merlely parsing a stream of whitespace-separated tokens, (and not interpreting them at all).
2010-05-10 11:44:09 -07:00
Add some compiler warnings and corresponding fixes. Most of the current problems were (mostly) harmless things like missing declarations, but there was at least one real error, (reversed argument order for yyerrror).
2010-05-10 13:17:25 -07:00
#include "glcpp.h"
Add the tiniest shell of a flex/bison-based parser. It doesn't really *do* anything yet---merlely parsing a stream of whitespace-separated tokens, (and not interpreting them at all).
2010-05-10 11:44:09 -07:00
#include "glcpp-parse.h"
glcpp: Set line locations in the lexer.
2010-06-16 17:41:12 -07:00
glcpp-lex: Declare some generated functions to eliminate compiler warnings. It's really a bug in flex that these functions are generated with neither a declaration nor the 'static' keyword, but we can at least avoid the warnings this way.
2010-07-20 15:03:20 -07:00
/* Flex annoyingly generates some functions without making them
* static. Let's declare them here. */
int glcpp_get_column (yyscan_t yyscanner);
void glcpp_set_column (int column_no , yyscan_t yyscanner);
glsl: Define YY_NO_UNISTD_H on MSVC.
2011-03-04 12:49:55 +00:00
#ifdef _MSC_VER
#define YY_NO_UNISTD_H
#endif
glcpp: Avoid warnings in generated flex code. We define the YY_NO_INPUT macro to avoid one needless function being generated. for the other needless functions, (yyunput and yy_top_state), we add a new UNREACHABLE start condition and call these functions from an action there. This doesn't change functionality at all, (since we never enter the UNREACHABLE start condition), but makes the compiler stop complaining about these two functions being defined but not used.
2010-07-20 15:53:14 -07:00
#define YY_NO_INPUT
glsl: glcpp: Move handling of #line directives from lexer to parser. The GLSL specification requires that #line directives be interpreted after macro expansion. Our existing implementation of #line macros in the lexer prevents conformance on this point. Moving the handling of #line from the lexer to the parser gives us the macro expansion we need. An additional benefit is that the preprocessor also now supports comments on the same line as #line directives. Finally, the preprocessor now emits the (fully-macro-expanded) #line directives into the output. This allows the full GLSL compiler to also see and interpret these directives so it can also generate correct line numbers in error messages. Signed-off-by: Carl Worth <cworth@cworth.org> Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
2012-06-09 16:31:06 -07:00
#define YY_USER_ACTION \
do { \
if (parser->has_new_line_number) \
yylineno = parser->new_line_number; \
if (parser->has_new_source_number) \
yylloc->source = parser->new_source_number; \
yylloc->first_column = yycolumn + 1; \
glsl: Update lexers in glsl and glcpp to hande end position of token. Reviewed-by: Carl Worth <cworth@cworth.org>
2014-02-05 21:18:08 +06:00
yylloc->first_line = yylloc->last_line = yylineno; \
glsl: glcpp: Move handling of #line directives from lexer to parser. The GLSL specification requires that #line directives be interpreted after macro expansion. Our existing implementation of #line macros in the lexer prevents conformance on this point. Moving the handling of #line from the lexer to the parser gives us the macro expansion we need. An additional benefit is that the preprocessor also now supports comments on the same line as #line directives. Finally, the preprocessor now emits the (fully-macro-expanded) #line directives into the output. This allows the full GLSL compiler to also see and interpret these directives so it can also generate correct line numbers in error messages. Signed-off-by: Carl Worth <cworth@cworth.org> Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
2012-06-09 16:31:06 -07:00
yycolumn += yyleng; \
glsl: Update lexers in glsl and glcpp to hande end position of token. Reviewed-by: Carl Worth <cworth@cworth.org>
2014-02-05 21:18:08 +06:00
yylloc->last_column = yycolumn + 1; \
glsl: glcpp: Move handling of #line directives from lexer to parser. The GLSL specification requires that #line directives be interpreted after macro expansion. Our existing implementation of #line macros in the lexer prevents conformance on this point. Moving the handling of #line from the lexer to the parser gives us the macro expansion we need. An additional benefit is that the preprocessor also now supports comments on the same line as #line directives. Finally, the preprocessor now emits the (fully-macro-expanded) #line directives into the output. This allows the full GLSL compiler to also see and interpret these directives so it can also generate correct line numbers in error messages. Signed-off-by: Carl Worth <cworth@cworth.org> Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
2012-06-09 16:31:06 -07:00
parser->has_new_line_number = 0; \
parser->has_new_source_number = 0; \
} while(0);
glcpp: Fix source numbers set with "#line LINE_NUMBER SOURCE_NUMBER" Previously, the YY_USER_ACTION was overwriting the yylloc->source value in every action, (after that value had been carefully set by the handling of the #line directive). Instead, we want to initialize it once in YY_USER_INIT and then not touch it at all in YY_USER_ACTION.
2010-08-23 09:29:49 -07:00
#define YY_USER_INIT \
do { \
yylineno = 1; \
yycolumn = 1; \
yylloc->source = 0; \
} while(0)
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
/* It's ugly to have macros that have return statements inside of
* them, but flex-based lexer generation is all built around the
* return statement.
*
* To mitigate the ugliness, we defer as much of the logic as possible
* to an actual function, not a macro (see
* glcpplex_update_state_per_token) and we make the word RETURN
* prominent in all of the macros which may return.
*
* The most-commonly-used macro is RETURN_TOKEN which will perform all
* necessary state updates based on the provided token,, then
* conditionally return the token. It will not return a token if the
* parser is currently skipping tokens, (such as within #if
* 0...#else).
*
* The RETURN_TOKEN_NEVER_SKIP macro is a lower-level variant that
* makes the token returning unconditional. This is needed for things
* like #if and the tokens of its condition, (since these must be
* evaluated by the parser even when otherwise skipping).
*
* Finally, RETURN_STRING_TOKEN is a simple convenience wrapper on top
* of RETURN_TOKEN that performs a string copy of yytext before the
* return.
*/
#define RETURN_TOKEN_NEVER_SKIP(token) \
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
do { \
if (token == NEWLINE) \
parser->last_token_was_newline = 1; \
else \
parser->last_token_was_newline = 0; \
return (token); \
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
} while (0)
#define RETURN_TOKEN(token) \
do { \
if (! parser->skipping) { \
RETURN_TOKEN_NEVER_SKIP(token); \
} \
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
} while(0)
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
#define RETURN_STRING_TOKEN(token) \
do { \
if (! parser->skipping) { \
yylval->str = ralloc_strdup (yyextra, yytext); \
RETURN_TOKEN_NEVER_SKIP (token); \
} \
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
} while(0)
Add the tiniest shell of a flex/bison-based parser. It doesn't really *do* anything yet---merlely parsing a stream of whitespace-separated tokens, (and not interpreting them at all).
2010-05-10 11:44:09 -07:00
%}
glcpp: Add plumbing to support line locations.
2010-06-16 16:35:57 -07:00
%option bison-bridge bison-locations reentrant noyywrap
Convert lexer to talloc and add xtalloc wrappers. The lexer was previously using strdup (expecting the parser to free), but is now more consistent, easier to use, and slightly more efficent by using talloc along with the parser. Also, we add xtalloc and xtalloc_strdup wrappers around talloc and talloc_strdup to put all of the out-of-memory-checking code in one place.
2010-05-12 12:45:33 -07:00
%option extra-type="glcpp_parser_t *"
Specify %option prefix="glcpp_" in the source code, not the Makefile.
2010-06-16 11:51:43 -07:00
%option prefix="glcpp_"
glcpp: Set line locations in the lexer.
2010-06-16 17:41:12 -07:00
%option stack
glsl2: Use --nounistd to fix MSVC build Also remove the --never-interactive command line option for the preprocessor lexer. This was already done for main compiler lexer.
2010-08-13 13:08:54 -07:00
%option never-interactive
Add the tiniest shell of a flex/bison-based parser. It doesn't really *do* anything yet---merlely parsing a stream of whitespace-separated tokens, (and not interpreting them at all).
2010-05-10 11:44:09 -07:00
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
%x DONE COMMENT UNREACHABLE DEFINE NEWLINE_CATCHUP
glcpp: Handle missing newline at EOF. Fixes CorrectFuncOverload.vert.
2010-06-16 12:53:19 -07:00
Implment #define By using the recently-imported hash_table implementation.
2010-05-10 16:16:06 -07:00
SPACE [[:space:]]
NONSPACE [^[:space:]]
Fix defines involving both literals and other defined macros. We now store a list of tokens in our hash-table rather than a single string. This lets us replace each macro in the value as necessary. This code adds a link dependency on talloc which does exactly what we want in terms of memory management for a parser. The 3 tests added in the previous commit now pass.
2010-05-12 12:17:10 -07:00
NEWLINE [\n]
Implment #define By using the recently-imported hash_table implementation.
2010-05-10 16:16:06 -07:00
HSPACE [ \t]
Fix two whitespace bugs in the lexer. The first bug was not allowing whitespace between '#' and the directive name. The second bug was swallowing a terminating newline along with any trailing whitespace on a line. With these two fixes, and the previous commit to stop emitting SPACE tokens, the recently added extra-whitespace test now passes.
2010-05-14 17:29:24 -07:00
HASH ^{HSPACE}*#{HSPACE}*
Implment #define By using the recently-imported hash_table implementation.
2010-05-10 16:16:06 -07:00
IDENTIFIER [_a-zA-Z][_a-zA-Z0-9]*
glsl/glcpp: Fix glcpp to properly lex entire "preprocessing numbers" The preprocessor defines a notions of a "preprocessing number" that starts with either a digit or a decimal point, and continues with zero or more of digits, decimal points, identifier characters, or the sign symbols, ('-' and '+'). Prior to this change, preprocessing numbers were lexed as some combination of OTHER and IDENTIFIER tokens. This had the problem of causing undesired macro expansion in some cases. We add tests to ensure that the undesired macro expansion does not happen in cases such as: #define e +1 #define xyz -2 int n = 1e; int p = 1xyz; In either case these macro definitions have no effect after this change, so that the numeric literals, (whether valid or not), will be passed on as-is from the preprocessor to the compiler proper. This fixes the following Khronos GLES3 CTS tests: preprocessor.basic.correct_phases_vertex preprocessor.basic.correct_phases_fragment v2. Thanks to Anuj Phogat for improving the original regular expression, (which accepted a '+' or '-', where these are only allowed after one of [eEpP]. I also expanded the test to exercise this. v3. Also fixed regular expression to require at least one digit at the beginning (after an optional period). Otherwise, a string such as ".xyz" was getting sucked up as a preprocessing number, (where obviously this should be a field access). Again, I expanded the test to exercise this. Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 14:56:47 -07:00
PP_NUMBER [.]?[0-9]([._a-zA-Z0-9]|[eEpP][-+])*
Fix pass-through of '=' and add a test for it. Previously '=' was not included in our PUNCTUATION regeular expression, but it *was* excldued from our OTHER regular expression, so we were getting the default (and hamful) lex action of just printing it. The test we add here is named "punctuator" with the idea that we can extend it as needed for other punctuator testing.
2010-05-29 05:07:24 -07:00
PUNCTUATION [][(){}.&*~!/%<>^|;,=+-]
glcpp: Fix so that trailing punctuation does not prevent macro expansion The trick here is that flex always chooses the rule that matches the most text. So with a input text of "two:" which we want to be lexed as an IDENTIFIER token "two" followed by an OTHER token ":" the previous OTHER rule would match longer as a single token of "two:" which we don't want. We prevent this by forcing the OTHER pattern to never match any characters that appear in other constructs, (no letters, numbers, #, _, whitespace, nor any punctuation that appear in CPP operators). Fixes bug #44764: GLSL preprocessor doesn't replace defines ending with ":" https://bugs.freedesktop.org/show_bug.cgi?id=44764 Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> NOTE: This is a candidate for stable release branches.
2012-01-21 09:24:11 -08:00
/* The OTHER class is simply a catch-all for things that the CPP
parser just doesn't care about. Since flex regular expressions that
match longer strings take priority over those matching shorter
strings, we have to be careful to avoid OTHER matching and hiding
something that CPP does care about. So we simply exclude all
characters that appear in any other expressions. */
OTHER [^][_#[:space:]#a-zA-Z0-9(){}.&*~!/%<>^|;,=+-]
Fix defines involving both literals and other defined macros. We now store a list of tokens in our hash-table rather than a single string. This lets us replace each macro in the value as necessary. This code adds a link dependency on talloc which does exactly what we want in terms of memory management for a parser. The 3 tests added in the previous commit now pass.
2010-05-12 12:17:10 -07:00
glcpp: Fix handling of "#line 0" The existing DECIMAL_INTEGER pattern is the correct thing to use when looking for a C decimal integer, (that is, a digit-sequence not starting with 0 which would instead be an octal integer). But for #line, we really want to accept any digit sequence, (including "0"), and always interpret it as a decimal constant. So we add a new DIGITS pattern for this case. This should fix the compilation failure noted in bug #28138 https://bugs.freedesktop.org/show_bug.cgi?id=28138 (Though the generated file will not be updated until the next commit.)
2010-08-23 09:31:42 -07:00
DIGITS [0-9][0-9]*
Add support for octal and hexadecimal integer literals. In addition to the decimal literals which we already support. Note that we use strtoll here to get the large-width integers demanded by the specification.
2010-05-24 11:29:02 -07:00
DECIMAL_INTEGER [1-9][0-9]*[uU]?
OCTAL_INTEGER 0[0-7]*[uU]?
HEXADECIMAL_INTEGER 0[xX][0-9a-fA-F]+[uU]?
Add the tiniest shell of a flex/bison-based parser. It doesn't really *do* anything yet---merlely parsing a stream of whitespace-separated tokens, (and not interpreting them at all).
2010-05-10 11:44:09 -07:00
%%
glcpp: Add a more descriptive comment for the SKIP state manipulation Two things make this code confusing: 1. The uncharacteristic manipulation of lexer start state outside of flex rules. 2. The confusing semantics of the skip_stack (including the "lexing_if" override and the SKIP_NO_SKIP state). This new comment is intended to bring a bit more clarity for any readers. There is no intended beahvioral change to the code here. The actual code changes include better indentation to avoid an excessively-long line, and using the more descriptive INITIAL rather than 0. Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 15:14:19 -08:00
glcpp: Replace multi-line comment with a space (even as part of macro definition) The preprocessor has always replaced multi-line comments with a single space character, (as required by the specification), but as of commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b the lexer also emitted a NEWLINE token for each newline within the comment, (in order to preserve line numbers). The emitting of NEWLINE tokens within the comment broke the rule of "replace a multi-line comment with a single space" as could be exposed by code like the following: #define FOO a/* */b FOO Prior to commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b, this code defined the macro FOO as "a b" as desired. Since that commit, this code instead defines FOO as "a" and leaves a stray "b" in the output. In this commit, we fix this by not emitting the NEWLINE tokens while lexing the comment, but instead merely counting them in the commented_newlines variable. Then, when the lexer next encounters a non-commented newline it switches to a NEWLINE_CATCHUP state to emit as many NEWLINE tokens as necessary (so that subsequent parsing stages still generate correct line numbers). Of course, it would have been more clear if we could have written a loop to emit all the newlines, but flex conventions prevent that, (we must use "return" for each token we emit). It similarly would have been clear to have a new rule restricted to the <NEWLINE_CATCHUP> state with an action much like the body of this if condition. The problem with that is that this rule must not consume any characters. It might be possible to write a rule that matches a single lookahead of any character, but then we would also need an additional rule to ensure for the <EOF> case where there are no additional characters available for the lookahead to match. Given those considerations, and given that the SKIP-state manipulation already involves a code block at the top of the lexer function, before any rules, it seems best to me to go with the implementation here which adds a similar pre-rule code block for the NEWLINE_CATCHUP. Finally, this commit also changes the expected output of a few, existing glcpp tests. The change here is that the space character resulting from the multi-line comment is now emitted before the newlines corresponding to that comment. (Previously, the newlines were emitted first, and the space character afterward.) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=72686 Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 16:06:31 -08:00
glcpp_parser_t *parser = yyextra;
/* When we lex a multi-line comment, we replace it (as
* specified) with a single space. But if the comment spanned
* multiple lines, then subsequent parsing stages will not
* count correct line numbers. To avoid this problem we keep
* track of all newlines that were commented out by a
* multi-line comment, and we emit a NEWLINE token for each at
* the next legal opportunity, (which is when the lexer would
* be emitting a NEWLINE token anyway).
*/
if (YY_START == NEWLINE_CATCHUP) {
if (parser->commented_newlines)
parser->commented_newlines--;
if (parser->commented_newlines == 0)
BEGIN INITIAL;
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (NEWLINE);
glcpp: Replace multi-line comment with a space (even as part of macro definition) The preprocessor has always replaced multi-line comments with a single space character, (as required by the specification), but as of commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b the lexer also emitted a NEWLINE token for each newline within the comment, (in order to preserve line numbers). The emitting of NEWLINE tokens within the comment broke the rule of "replace a multi-line comment with a single space" as could be exposed by code like the following: #define FOO a/* */b FOO Prior to commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b, this code defined the macro FOO as "a b" as desired. Since that commit, this code instead defines FOO as "a" and leaves a stray "b" in the output. In this commit, we fix this by not emitting the NEWLINE tokens while lexing the comment, but instead merely counting them in the commented_newlines variable. Then, when the lexer next encounters a non-commented newline it switches to a NEWLINE_CATCHUP state to emit as many NEWLINE tokens as necessary (so that subsequent parsing stages still generate correct line numbers). Of course, it would have been more clear if we could have written a loop to emit all the newlines, but flex conventions prevent that, (we must use "return" for each token we emit). It similarly would have been clear to have a new rule restricted to the <NEWLINE_CATCHUP> state with an action much like the body of this if condition. The problem with that is that this rule must not consume any characters. It might be possible to write a rule that matches a single lookahead of any character, but then we would also need an additional rule to ensure for the <EOF> case where there are no additional characters available for the lookahead to match. Given those considerations, and given that the SKIP-state manipulation already involves a code block at the top of the lexer function, before any rules, it seems best to me to go with the implementation here which adds a similar pre-rule code block for the NEWLINE_CATCHUP. Finally, this commit also changes the expected output of a few, existing glcpp tests. The change here is that the space character resulting from the multi-line comment is now emitted before the newlines corresponding to that comment. (Previously, the newlines were emitted first, and the space character afterward.) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=72686 Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 16:06:31 -08:00
}
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
/* Set up the parser->skipping bit here before doing any lexing.
glcpp: Add a more descriptive comment for the SKIP state manipulation Two things make this code confusing: 1. The uncharacteristic manipulation of lexer start state outside of flex rules. 2. The confusing semantics of the skip_stack (including the "lexing_if" override and the SKIP_NO_SKIP state). This new comment is intended to bring a bit more clarity for any readers. There is no intended beahvioral change to the code here. The actual code changes include better indentation to avoid an excessively-long line, and using the more descriptive INITIAL rather than 0. Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 15:14:19 -08:00
*
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
* This bit controls whether tokens are skipped, (as implemented by
* RETURN_TOKEN), such as between "#if 0" and "#endif".
glcpp: Add a more descriptive comment for the SKIP state manipulation Two things make this code confusing: 1. The uncharacteristic manipulation of lexer start state outside of flex rules. 2. The confusing semantics of the skip_stack (including the "lexing_if" override and the SKIP_NO_SKIP state). This new comment is intended to bring a bit more clarity for any readers. There is no intended beahvioral change to the code here. The actual code changes include better indentation to avoid an excessively-long line, and using the more descriptive INITIAL rather than 0. Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 15:14:19 -08:00
*
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
* The parser maintains a skip_stack indicating whether we should be
* skipping, (and nested levels of #if/#ifdef/#ifndef/#endif) will
* push and pop items from the stack.
glcpp: Add a more descriptive comment for the SKIP state manipulation Two things make this code confusing: 1. The uncharacteristic manipulation of lexer start state outside of flex rules. 2. The confusing semantics of the skip_stack (including the "lexing_if" override and the SKIP_NO_SKIP state). This new comment is intended to bring a bit more clarity for any readers. There is no intended beahvioral change to the code here. The actual code changes include better indentation to avoid an excessively-long line, and using the more descriptive INITIAL rather than 0. Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 15:14:19 -08:00
*
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
* Here are the rules for determining whether we are skipping:
glcpp: Add a more descriptive comment for the SKIP state manipulation Two things make this code confusing: 1. The uncharacteristic manipulation of lexer start state outside of flex rules. 2. The confusing semantics of the skip_stack (including the "lexing_if" override and the SKIP_NO_SKIP state). This new comment is intended to bring a bit more clarity for any readers. There is no intended beahvioral change to the code here. The actual code changes include better indentation to avoid an excessively-long line, and using the more descriptive INITIAL rather than 0. Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 15:14:19 -08:00
*
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
* 1. If the skip stack is NULL, we are outside of all #if blocks
* and we are not skipping.
glcpp: Add a more descriptive comment for the SKIP state manipulation Two things make this code confusing: 1. The uncharacteristic manipulation of lexer start state outside of flex rules. 2. The confusing semantics of the skip_stack (including the "lexing_if" override and the SKIP_NO_SKIP state). This new comment is intended to bring a bit more clarity for any readers. There is no intended beahvioral change to the code here. The actual code changes include better indentation to avoid an excessively-long line, and using the more descriptive INITIAL rather than 0. Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 15:14:19 -08:00
*
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
* 2. If the skip stack is non-NULL, the type of the top node in
* the stack determines whether to skip. A type of
* SKIP_NO_SKIP is used for blocks wheere we are emitting
* tokens, (such as between #if 1 and #endif, or after the
* #else of an #if 0, etc.).
*
* 3. The lexing_directive bit overrides the skip stack. This bit
* is set when we are actively lexing the expression for a
* pre-processor condition, (such as #if, #elif, or #else). In
* this case, even if otherwise skipping, we need to emit the
* tokens for this condition so that the parser can evaluate
* the expression. (For, #else, there's no expression, but we
* emit tokens so the parser can generate a nice error message
* if there are any tokens here).
glcpp: Rework lexer to use a SKIP state rather than REJECT. Previously, the rule deleted by this commit was matched every single time (being the longest match). If not skipping, it used REJECT to continue on to the actual correct rule. The flex manual advises against using REJECT where possible, as it is one of the most expensive lexer features. So using it on every match seems undesirable. Perhaps more importantly, it made it necessary for the #if directive rules to contain a look-ahead pattern to make them as long as the (now deleted) "skip the whole line" rule. This patch introduces an exclusive start state, SKIP, to avoid REJECTs. Each time the lexer is called, the code at the top of the rules section will run, implicitly switching the state to the correct one. Fixes piglit tests 16384-consecutive-chars.frag and 16385-consecutive-chars.frag.
2011-03-01 12:24:58 -08:00
*/
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
if (parser->skip_stack &&
parser->skip_stack->type != SKIP_NO_SKIP &&
! parser->lexing_directive)
{
parser->skipping = 1;
} else {
parser->skipping = 0;
glcpp: Rework lexer to use a SKIP state rather than REJECT. Previously, the rule deleted by this commit was matched every single time (being the longest match). If not skipping, it used REJECT to continue on to the actual correct rule. The flex manual advises against using REJECT where possible, as it is one of the most expensive lexer features. So using it on every match seems undesirable. Perhaps more importantly, it made it necessary for the #if directive rules to contain a look-ahead pattern to make them as long as the (now deleted) "skip the whole line" rule. This patch introduces an exclusive start state, SKIP, to avoid REJECTs. Each time the lexer is called, the code at the top of the rules section will run, implicitly switching the state to the correct one. Fixes piglit tests 16384-consecutive-chars.frag and 16385-consecutive-chars.frag.
2011-03-01 12:24:58 -08:00
}
Add the tiniest shell of a flex/bison-based parser. It doesn't really *do* anything yet---merlely parsing a stream of whitespace-separated tokens, (and not interpreting them at all).
2010-05-10 11:44:09 -07:00
Implement comment handling in the lexer (with test). We support both single-line (//) and multi-line (/* ... */) comments and add a test for this, (trying to stress the rules just a bit by embedding one comment delimiter into a comment delimited with the other style, etc.). To keep the test suite passing we do now discard any output lines from glcpp that consist only of spacing, (in addition to blank lines as previously). We also discard any initial whitespace from gcc output. In neither case should the absence or presence of this whitespace affect correctness.
2010-06-01 12:18:43 -07:00
/* Single-line comments */
glcpp: Don't include the newline when discarding single-line comments Matching the newline here meant having to do some redundant work here, (incrementing line number, resetting column number, and returning a NEWLINE token), that could otherwise simply be left to the existing rule which matches a newline. Worse, when the comment rule matches the newline as well, the parser can lookahead and see a token for something that should actually be skipped. For example, in a case like this: #if 0 // comment here fail #else win #endif Both fail and win appear in the output, (not that the condition is being evaluated incorrectly---merely that one token after the comment's newline was being lexed/parse regardless of the condition). This commit fixes the above test case, (which is also remarkably similar to 087-if-comments which now passes).
2010-08-17 22:17:09 -07:00
"//"[^\n]* {
Implement comment handling in the lexer (with test). We support both single-line (//) and multi-line (/* ... */) comments and add a test for this, (trying to stress the rules just a bit by embedding one comment delimiter into a comment delimited with the other style, etc.). To keep the test suite passing we do now discard any output lines from glcpp that consist only of spacing, (in addition to blank lines as previously). We also discard any initial whitespace from gcc output. In neither case should the absence or presence of this whitespace affect correctness.
2010-06-01 12:18:43 -07:00
}
/* Multi-line comments */
glsl/glcpp: Add support for comments between #define and macro identifier The recent adddition of an error for "#define followed by a non-identifier" was a bit to aggressive since it used a regular expression in the lexer to flag any character that's not legal as the first character of an identifier. But we need to allow comments to appear here, (since we aren't removing comments in a preliminary pass). So we refine the error here to only flag characters that could not be an identifier, nor a comment, nor whitespace. We also augment the existing comment support to be active in the <DEFINE> state as well. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-19 11:57:06 -07:00
<DEFINE,INITIAL>"/*" { yy_push_state(COMMENT, yyscanner); }
glcpp: Set line locations in the lexer.
2010-06-16 17:41:12 -07:00
<COMMENT>[^*\n]*
glcpp: Replace multi-line comment with a space (even as part of macro definition) The preprocessor has always replaced multi-line comments with a single space character, (as required by the specification), but as of commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b the lexer also emitted a NEWLINE token for each newline within the comment, (in order to preserve line numbers). The emitting of NEWLINE tokens within the comment broke the rule of "replace a multi-line comment with a single space" as could be exposed by code like the following: #define FOO a/* */b FOO Prior to commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b, this code defined the macro FOO as "a b" as desired. Since that commit, this code instead defines FOO as "a" and leaves a stray "b" in the output. In this commit, we fix this by not emitting the NEWLINE tokens while lexing the comment, but instead merely counting them in the commented_newlines variable. Then, when the lexer next encounters a non-commented newline it switches to a NEWLINE_CATCHUP state to emit as many NEWLINE tokens as necessary (so that subsequent parsing stages still generate correct line numbers). Of course, it would have been more clear if we could have written a loop to emit all the newlines, but flex conventions prevent that, (we must use "return" for each token we emit). It similarly would have been clear to have a new rule restricted to the <NEWLINE_CATCHUP> state with an action much like the body of this if condition. The problem with that is that this rule must not consume any characters. It might be possible to write a rule that matches a single lookahead of any character, but then we would also need an additional rule to ensure for the <EOF> case where there are no additional characters available for the lookahead to match. Given those considerations, and given that the SKIP-state manipulation already involves a code block at the top of the lexer function, before any rules, it seems best to me to go with the implementation here which adds a similar pre-rule code block for the NEWLINE_CATCHUP. Finally, this commit also changes the expected output of a few, existing glcpp tests. The change here is that the space character resulting from the multi-line comment is now emitted before the newlines corresponding to that comment. (Previously, the newlines were emitted first, and the space character afterward.) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=72686 Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 16:06:31 -08:00
<COMMENT>[^*\n]*\n { yylineno++; yycolumn = 0; parser->commented_newlines++; }
glcpp: Set line locations in the lexer.
2010-06-16 17:41:12 -07:00
<COMMENT>"*"+[^*/\n]*
glcpp: Replace multi-line comment with a space (even as part of macro definition) The preprocessor has always replaced multi-line comments with a single space character, (as required by the specification), but as of commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b the lexer also emitted a NEWLINE token for each newline within the comment, (in order to preserve line numbers). The emitting of NEWLINE tokens within the comment broke the rule of "replace a multi-line comment with a single space" as could be exposed by code like the following: #define FOO a/* */b FOO Prior to commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b, this code defined the macro FOO as "a b" as desired. Since that commit, this code instead defines FOO as "a" and leaves a stray "b" in the output. In this commit, we fix this by not emitting the NEWLINE tokens while lexing the comment, but instead merely counting them in the commented_newlines variable. Then, when the lexer next encounters a non-commented newline it switches to a NEWLINE_CATCHUP state to emit as many NEWLINE tokens as necessary (so that subsequent parsing stages still generate correct line numbers). Of course, it would have been more clear if we could have written a loop to emit all the newlines, but flex conventions prevent that, (we must use "return" for each token we emit). It similarly would have been clear to have a new rule restricted to the <NEWLINE_CATCHUP> state with an action much like the body of this if condition. The problem with that is that this rule must not consume any characters. It might be possible to write a rule that matches a single lookahead of any character, but then we would also need an additional rule to ensure for the <EOF> case where there are no additional characters available for the lookahead to match. Given those considerations, and given that the SKIP-state manipulation already involves a code block at the top of the lexer function, before any rules, it seems best to me to go with the implementation here which adds a similar pre-rule code block for the NEWLINE_CATCHUP. Finally, this commit also changes the expected output of a few, existing glcpp tests. The change here is that the space character resulting from the multi-line comment is now emitted before the newlines corresponding to that comment. (Previously, the newlines were emitted first, and the space character afterward.) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=72686 Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 16:06:31 -08:00
<COMMENT>"*"+[^*/\n]*\n { yylineno++; yycolumn = 0; parser->commented_newlines++; }
glcpp: Set line locations in the lexer.
2010-06-16 17:41:12 -07:00
<COMMENT>"*"+"/" {
yy_pop_state(yyscanner);
Implement comment handling in the lexer (with test). We support both single-line (//) and multi-line (/* ... */) comments and add a test for this, (trying to stress the rules just a bit by embedding one comment delimiter into a comment delimited with the other style, etc.). To keep the test suite passing we do now discard any output lines from glcpp that consist only of spacing, (in addition to blank lines as previously). We also discard any initial whitespace from gcc output. In neither case should the absence or presence of this whitespace affect correctness.
2010-06-01 12:18:43 -07:00
if (yyextra->space_tokens)
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (SPACE);
Implement comment handling in the lexer (with test). We support both single-line (//) and multi-line (/* ... */) comments and add a test for this, (trying to stress the rules just a bit by embedding one comment delimiter into a comment delimited with the other style, etc.). To keep the test suite passing we do now discard any output lines from glcpp that consist only of spacing, (in addition to blank lines as previously). We also discard any initial whitespace from gcc output. In neither case should the absence or presence of this whitespace affect correctness.
2010-06-01 12:18:43 -07:00
}
glcpp: Reject #version and #line not followed by whitespace Fixes part of es3conform's preprocess16_frag test. Reviewed-by: Carl Worth <cworth@cworth.org>
2012-11-20 17:23:42 -08:00
{HASH}version{HSPACE}+ {
glcpp: Add __VERSION__ define to the current language version. Fixes: glsl-version-define glsl-version-define-110 glsl-version-define-120
2010-07-28 16:58:39 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (HASH_VERSION);
glcpp: Add __VERSION__ define to the current language version. Fixes: glsl-version-define glsl-version-define-110 glsl-version-define-120
2010-07-28 16:58:39 -07:00
}
glcpp: Pass #version, #extension, and #pragma directives through unchanged. Let the main compiler's lexer/parser handle them.
2010-06-16 12:21:17 -07:00
/* glcpp doesn't handle #extension, #version, or #pragma directives.
* Simply pass them through to the main compiler's lexer/parser. */
glsl/glcpp: Don't choke on an empty pragma The lexer was insisting that there be at least one character after "#pragma" and before the end of the line. This caused an error for a line consisting only of "#pragma" which volates at least the following sentence from the GLSL ES Specification 3.00.4: The scope as well as the effect of the optimize and debug pragmas is implementation-dependent except that their use must not generate an error. [Page 12 (Page 28 of PDF)] and likely the following sentence from that specification and also in GLSLangSpec 4.30.6: If an implementation does not recognize the tokens following #pragma, then it will ignore that pragma. Add a "make check" test to ensure no future regressions. This change fixes at least part of the following Khronos GLES3 CTS test: preprocessor.pragmas.pragma_vertex Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
2014-06-13 15:53:39 -07:00
{HASH}(extension|pragma)[^\n]* {
glcpp: Fix line and column numbering. Lines were off by one, and column numbering was completely daft.
2010-06-18 15:23:50 -07:00
yylineno++;
yycolumn = 0;
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (OTHER);
glcpp: Pass #version, #extension, and #pragma directives through unchanged. Let the main compiler's lexer/parser handle them.
2010-06-16 12:21:17 -07:00
}
glcpp: Reject #version and #line not followed by whitespace Fixes part of es3conform's preprocess16_frag test. Reviewed-by: Carl Worth <cworth@cworth.org>
2012-11-20 17:23:42 -08:00
{HASH}line{HSPACE}+ {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (HASH_LINE);
glcpp: Add basic #line support (adapted from the main compiler).
2010-08-18 17:38:05 -07:00
}
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
/* For the pre-processor directives, we return these tokens
* even when we are otherwise skipping. */
glcpp: Remove trailing contexts from #if rules. These are now unnecessary.
2011-03-03 09:52:36 -08:00
{HASH}ifdef {
glsl/glcpp: Fix glcpp to catch garbage after #if 1 ... #else Previously, a line such as: #else garbage would flag an error if it followed "#if 0", but not if it followed "#if 1". We fix this by setting a new bit of state (lexing_else) that allows the lexer to defer switching to the <SKIP> start state until after the NEWLINE following the #else directive. A new test case is added for: #if 1 #else garbage #endif which was untested before, (and did not generate the desired error). This fixes the following Khronos GLES3 CTS tests: tokens_after_else_vertex tokens_after_else_fragment Reviewed-by: Matt Turner <mattst88@gmail.com> Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 10:39:39 -07:00
yyextra->lexing_directive = 1;
glcpp: Actually support #ifdef and #ifndef. Strangely, the lexer never created these tokens, even though the parser already had code to handle them.
2010-06-17 14:36:34 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
RETURN_TOKEN_NEVER_SKIP (HASH_IFDEF);
glcpp: Actually support #ifdef and #ifndef. Strangely, the lexer never created these tokens, even though the parser already had code to handle them.
2010-06-17 14:36:34 -07:00
}
glcpp: Remove trailing contexts from #if rules. These are now unnecessary.
2011-03-03 09:52:36 -08:00
{HASH}ifndef {
glsl/glcpp: Fix glcpp to catch garbage after #if 1 ... #else Previously, a line such as: #else garbage would flag an error if it followed "#if 0", but not if it followed "#if 1". We fix this by setting a new bit of state (lexing_else) that allows the lexer to defer switching to the <SKIP> start state until after the NEWLINE following the #else directive. A new test case is added for: #if 1 #else garbage #endif which was untested before, (and did not generate the desired error). This fixes the following Khronos GLES3 CTS tests: tokens_after_else_vertex tokens_after_else_fragment Reviewed-by: Matt Turner <mattst88@gmail.com> Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 10:39:39 -07:00
yyextra->lexing_directive = 1;
glcpp: Actually support #ifdef and #ifndef. Strangely, the lexer never created these tokens, even though the parser already had code to handle them.
2010-06-17 14:36:34 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
RETURN_TOKEN_NEVER_SKIP (HASH_IFNDEF);
glcpp: Actually support #ifdef and #ifndef. Strangely, the lexer never created these tokens, even though the parser already had code to handle them.
2010-06-17 14:36:34 -07:00
}
glcpp: Remove trailing contexts from #if rules. These are now unnecessary.
2011-03-03 09:52:36 -08:00
{HASH}if/[^_a-zA-Z0-9] {
glsl/glcpp: Fix glcpp to catch garbage after #if 1 ... #else Previously, a line such as: #else garbage would flag an error if it followed "#if 0", but not if it followed "#if 1". We fix this by setting a new bit of state (lexing_else) that allows the lexer to defer switching to the <SKIP> start state until after the NEWLINE following the #else directive. A new test case is added for: #if 1 #else garbage #endif which was untested before, (and did not generate the desired error). This fixes the following Khronos GLES3 CTS tests: tokens_after_else_vertex tokens_after_else_fragment Reviewed-by: Matt Turner <mattst88@gmail.com> Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 10:39:39 -07:00
yyextra->lexing_directive = 1;
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
RETURN_TOKEN_NEVER_SKIP (HASH_IF);
Implement #if, #else, #elif, and #endif with tests. So far the only expression implemented is a single integer literal, but obviously that's easy to extend. Various things including nesting are tested here.
2010-05-20 22:27:07 -07:00
}
glcpp: Support #elif(expression) with no intervening space. And add test cases to ensure that this works - 110 verifies that glcpp rejects #elif<digits> which glcpp previously accepted. - 111 verifies that glcpp accepts #if followed immediately by (, +, -, !, or ~. - 112 does the same as 111 but for #elif. See 17f9beb6 for #if change. Reviewed-by: Carl Worth <cworth@cworth.org>
2012-11-26 11:53:45 -08:00
{HASH}elif/[^_a-zA-Z0-9] {
glsl/glcpp: Fix glcpp to catch garbage after #if 1 ... #else Previously, a line such as: #else garbage would flag an error if it followed "#if 0", but not if it followed "#if 1". We fix this by setting a new bit of state (lexing_else) that allows the lexer to defer switching to the <SKIP> start state until after the NEWLINE following the #else directive. A new test case is added for: #if 1 #else garbage #endif which was untested before, (and did not generate the desired error). This fixes the following Khronos GLES3 CTS tests: tokens_after_else_vertex tokens_after_else_fragment Reviewed-by: Matt Turner <mattst88@gmail.com> Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 10:39:39 -07:00
yyextra->lexing_directive = 1;
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
RETURN_TOKEN_NEVER_SKIP (HASH_ELIF);
Add support for octal and hexadecimal integer literals. In addition to the decimal literals which we already support. Note that we use strtoll here to get the large-width integers demanded by the specification.
2010-05-24 11:29:02 -07:00
}
glcpp: Remove trailing contexts from #if rules. These are now unnecessary.
2011-03-03 09:52:36 -08:00
{HASH}else {
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
RETURN_TOKEN_NEVER_SKIP (HASH_ELSE);
Add support for octal and hexadecimal integer literals. In addition to the decimal literals which we already support. Note that we use strtoll here to get the large-width integers demanded by the specification.
2010-05-24 11:29:02 -07:00
}
glcpp: Remove trailing contexts from #if rules. These are now unnecessary.
2011-03-03 09:52:36 -08:00
{HASH}endif {
stash
2010-05-26 09:32:12 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
RETURN_TOKEN_NEVER_SKIP (HASH_ENDIF);
glcpp: Replace multi-line comment with a space (even as part of macro definition) The preprocessor has always replaced multi-line comments with a single space character, (as required by the specification), but as of commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b the lexer also emitted a NEWLINE token for each newline within the comment, (in order to preserve line numbers). The emitting of NEWLINE tokens within the comment broke the rule of "replace a multi-line comment with a single space" as could be exposed by code like the following: #define FOO a/* */b FOO Prior to commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b, this code defined the macro FOO as "a b" as desired. Since that commit, this code instead defines FOO as "a" and leaves a stray "b" in the output. In this commit, we fix this by not emitting the NEWLINE tokens while lexing the comment, but instead merely counting them in the commented_newlines variable. Then, when the lexer next encounters a non-commented newline it switches to a NEWLINE_CATCHUP state to emit as many NEWLINE tokens as necessary (so that subsequent parsing stages still generate correct line numbers). Of course, it would have been more clear if we could have written a loop to emit all the newlines, but flex conventions prevent that, (we must use "return" for each token we emit). It similarly would have been clear to have a new rule restricted to the <NEWLINE_CATCHUP> state with an action much like the body of this if condition. The problem with that is that this rule must not consume any characters. It might be possible to write a rule that matches a single lookahead of any character, but then we would also need an additional rule to ensure for the <EOF> case where there are no additional characters available for the lookahead to match. Given those considerations, and given that the SKIP-state manipulation already involves a code block at the top of the lexer function, before any rules, it seems best to me to go with the implementation here which adds a similar pre-rule code block for the NEWLINE_CATCHUP. Finally, this commit also changes the expected output of a few, existing glcpp tests. The change here is that the space character resulting from the multi-line comment is now emitted before the newlines corresponding to that comment. (Previously, the newlines were emitted first, and the space character afterward.) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=72686 Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 16:06:31 -08:00
}
glcpp: Rework lexer to use a SKIP state rather than REJECT. Previously, the rule deleted by this commit was matched every single time (being the longest match). If not skipping, it used REJECT to continue on to the actual correct rule. The flex manual advises against using REJECT where possible, as it is one of the most expensive lexer features. So using it on every match seems undesirable. Perhaps more importantly, it made it necessary for the #if directive rules to contain a look-ahead pattern to make them as long as the (now deleted) "skip the whole line" rule. This patch introduces an exclusive start state, SKIP, to avoid REJECTs. Each time the lexer is called, the code at the top of the rules section will run, implicitly switching the state to the correct one. Fixes piglit tests 16384-consecutive-chars.frag and 16385-consecutive-chars.frag.
2011-03-01 12:24:58 -08:00
glcpp: Add #error support.
2010-07-02 15:31:26 -07:00
{HASH}error.* {
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
if (! parser->skipping) {
char *p;
for (p = yytext; !isalpha(p[0]); p++); /* skip " # " */
p += 5; /* skip "error" */
glcpp_error(yylloc, yyextra, "#error%s", p);
}
glcpp: Add #error support.
2010-07-02 15:31:26 -07:00
}
glsl/glcpp: Add support for comments between #define and macro identifier The recent adddition of an error for "#define followed by a non-identifier" was a bit to aggressive since it used a regular expression in the lexer to flag any character that's not legal as the first character of an identifier. But we need to allow comments to appear here, (since we aren't removing comments in a preliminary pass). So we refine the error here to only flag characters that could not be an identifier, nor a comment, nor whitespace. We also augment the existing comment support to be active in the <DEFINE> state as well. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-19 11:57:06 -07:00
/* After we see a "#define" we enter the <DEFINE> start state
* for the lexer. Within <DEFINE> we are looking for the first
* identifier and specifically checking whether the identifier
* is followed by a '(' or not, (to lex either a
* FUNC_IDENTIFIER or an OBJ_IDENITIFIER token).
*
* While in the <DEFINE> state we also need to explicitly
* handle a few other things that may appear before the
* identifier:
*
* * Comments, (handled above with the main support for
* comments).
*
* * Whitespace (simply ignored)
*
* * Anything else, (not an identifier, not a comment,
* and not whitespace). This will generate an error.
*/
glcpp: Don't use infinite lookhead for #define differentiation. Previously, we used lookahead patterns to differentiate: #define FOO(x) function macro #define FOO (x) object macro Unfortunately, our rule for function macros: {HASH}define{HSPACE}+/{IDENTIFIER}"(" relies on infinite lookahead, and apparently triggers a Flex bug where the generated code overflows a state buffer (see YY_STATE_BUF_SIZE). There's no need to use infinite lookahead. We can simply change state, match the identifier, and use a single character lookahead for the '('. This apparently makes Flex not generate the giant state array, which avoids the buffer overflow, and should be more efficient anyway. Fixes piglit test 17000-consecutive-chars-identifier.frag. NOTE: This is a candidate for every release branch ever. Signed-off-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Carl Worth <cworth@cworth.org>
2012-10-22 10:56:46 -07:00
{HASH}define{HSPACE}+ {
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
if (! parser->skipping) {
yyextra->space_tokens = 0;
yy_push_state(DEFINE, yyscanner);
RETURN_TOKEN (HASH_DEFINE);
}
stash
2010-05-26 09:32:12 -07:00
}
glsl/glcpp: Add support for comments between #define and macro identifier The recent adddition of an error for "#define followed by a non-identifier" was a bit to aggressive since it used a regular expression in the lexer to flag any character that's not legal as the first character of an identifier. But we need to allow comments to appear here, (since we aren't removing comments in a preliminary pass). So we refine the error here to only flag characters that could not be an identifier, nor a comment, nor whitespace. We also augment the existing comment support to be active in the <DEFINE> state as well. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-19 11:57:06 -07:00
/* An identifier immediately followed by '(' */
glcpp: Don't use infinite lookhead for #define differentiation. Previously, we used lookahead patterns to differentiate: #define FOO(x) function macro #define FOO (x) object macro Unfortunately, our rule for function macros: {HASH}define{HSPACE}+/{IDENTIFIER}"(" relies on infinite lookahead, and apparently triggers a Flex bug where the generated code overflows a state buffer (see YY_STATE_BUF_SIZE). There's no need to use infinite lookahead. We can simply change state, match the identifier, and use a single character lookahead for the '('. This apparently makes Flex not generate the giant state array, which avoids the buffer overflow, and should be more efficient anyway. Fixes piglit test 17000-consecutive-chars-identifier.frag. NOTE: This is a candidate for every release branch ever. Signed-off-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Carl Worth <cworth@cworth.org>
2012-10-22 10:56:46 -07:00
<DEFINE>{IDENTIFIER}/"(" {
yy_pop_state(yyscanner);
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (FUNC_IDENTIFIER);
glcpp: Don't use infinite lookhead for #define differentiation. Previously, we used lookahead patterns to differentiate: #define FOO(x) function macro #define FOO (x) object macro Unfortunately, our rule for function macros: {HASH}define{HSPACE}+/{IDENTIFIER}"(" relies on infinite lookahead, and apparently triggers a Flex bug where the generated code overflows a state buffer (see YY_STATE_BUF_SIZE). There's no need to use infinite lookahead. We can simply change state, match the identifier, and use a single character lookahead for the '('. This apparently makes Flex not generate the giant state array, which avoids the buffer overflow, and should be more efficient anyway. Fixes piglit test 17000-consecutive-chars-identifier.frag. NOTE: This is a candidate for every release branch ever. Signed-off-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Carl Worth <cworth@cworth.org>
2012-10-22 10:56:46 -07:00
}
glsl/glcpp: Add support for comments between #define and macro identifier The recent adddition of an error for "#define followed by a non-identifier" was a bit to aggressive since it used a regular expression in the lexer to flag any character that's not legal as the first character of an identifier. But we need to allow comments to appear here, (since we aren't removing comments in a preliminary pass). So we refine the error here to only flag characters that could not be an identifier, nor a comment, nor whitespace. We also augment the existing comment support to be active in the <DEFINE> state as well. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-19 11:57:06 -07:00
/* An identifier not immediately followed by '(' */
glcpp: Don't use infinite lookhead for #define differentiation. Previously, we used lookahead patterns to differentiate: #define FOO(x) function macro #define FOO (x) object macro Unfortunately, our rule for function macros: {HASH}define{HSPACE}+/{IDENTIFIER}"(" relies on infinite lookahead, and apparently triggers a Flex bug where the generated code overflows a state buffer (see YY_STATE_BUF_SIZE). There's no need to use infinite lookahead. We can simply change state, match the identifier, and use a single character lookahead for the '('. This apparently makes Flex not generate the giant state array, which avoids the buffer overflow, and should be more efficient anyway. Fixes piglit test 17000-consecutive-chars-identifier.frag. NOTE: This is a candidate for every release branch ever. Signed-off-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Carl Worth <cworth@cworth.org>
2012-10-22 10:56:46 -07:00
<DEFINE>{IDENTIFIER} {
yy_pop_state(yyscanner);
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (OBJ_IDENTIFIER);
stash
2010-05-26 09:32:12 -07:00
}
glsl/glcpp: Add support for comments between #define and macro identifier The recent adddition of an error for "#define followed by a non-identifier" was a bit to aggressive since it used a regular expression in the lexer to flag any character that's not legal as the first character of an identifier. But we need to allow comments to appear here, (since we aren't removing comments in a preliminary pass). So we refine the error here to only flag characters that could not be an identifier, nor a comment, nor whitespace. We also augment the existing comment support to be active in the <DEFINE> state as well. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-19 11:57:06 -07:00
/* Whitespace */
<DEFINE>{HSPACE}+ {
/* Just ignore it. Nothing to do here. */
}
/* '/' not followed by '*', so not a comment. This is an error. */
<DEFINE>[/][^*]{NONSPACE}* {
BEGIN INITIAL;
glcpp_error(yylloc, yyextra, "#define followed by a non-identifier: %s", yytext);
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (INTEGER_STRING);
glsl/glcpp: Add support for comments between #define and macro identifier The recent adddition of an error for "#define followed by a non-identifier" was a bit to aggressive since it used a regular expression in the lexer to flag any character that's not legal as the first character of an identifier. But we need to allow comments to appear here, (since we aren't removing comments in a preliminary pass). So we refine the error here to only flag characters that could not be an identifier, nor a comment, nor whitespace. We also augment the existing comment support to be active in the <DEFINE> state as well. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-19 11:57:06 -07:00
}
/* A character that can't start an identifier, comment, or
* space. This is an error. */
<DEFINE>[^_a-zA-Z/[:space:]]{NONSPACE}* {
glsl/glcpp: Emit proper error for #define with a non-identifier Previously, if the preprocessor encountered a #define with a non-identifier, such as: #define 123 456 The lexer had no explicit rules to match non-identifiers in the <DEFINE> start state. Because of this, flex's default rule was being invoked, (printing characters to stdout), and all text was being discarded by the compiler until the next identifier. As one can imagine, this led to all sorts of interesting and surprising results. Fix this by adding an explicit rule complementing the existing identifier-based rules that should catch all non-identifiers after #define and reliably give a well-formatted error message. A new test is added to "make check" to ensure this bug stays fixed. This commit also fixes the following Khronos GLES3 CTS test: define_non_identifier_vertex (The "fragment" variant was passing earlier only because the preprocessor was behaving so randomly and causing the compilation to fail. It's lucky, in fact, that the "vertex" version succesfully compiled so we could find and fix this bug.) Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-17 11:55:07 -07:00
BEGIN INITIAL;
glcpp_error(yylloc, yyextra, "#define followed by a non-identifier: %s", yytext);
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (INTEGER_STRING);
glsl/glcpp: Emit proper error for #define with a non-identifier Previously, if the preprocessor encountered a #define with a non-identifier, such as: #define 123 456 The lexer had no explicit rules to match non-identifiers in the <DEFINE> start state. Because of this, flex's default rule was being invoked, (printing characters to stdout), and all text was being discarded by the compiler until the next identifier. As one can imagine, this led to all sorts of interesting and surprising results. Fix this by adding an explicit rule complementing the existing identifier-based rules that should catch all non-identifiers after #define and reliably give a well-formatted error message. A new test is added to "make check" to ensure this bug stays fixed. This commit also fixes the following Khronos GLES3 CTS test: define_non_identifier_vertex (The "fragment" variant was passing earlier only because the preprocessor was behaving so randomly and causing the compilation to fail. It's lucky, in fact, that the "vertex" version succesfully compiled so we could find and fix this bug.) Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-17 11:55:07 -07:00
}
Fix #if-skipping to *really* skip the skipped group. Previously we were avoiding printing within a skipped group, but we were still evluating directives such as #define and #undef and still emitting diagnostics for things such as macro calls with the wrong number of arguments. Add a test for this and fix it with a high-priority rule in the lexer that consumes the skipped content.
2010-06-01 11:20:18 -07:00
{HASH}undef {
stash
2010-05-26 09:32:12 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (HASH_UNDEF);
stash
2010-05-26 09:32:12 -07:00
}
Starting over with the C99 grammar for the preprocessor. This is a fresh start with a much simpler approach for the flex/bison portions of the preprocessor. This isn't functional yet, (produces no output), but can at least read all of our test cases without any parse errors. The grammar here is based on the grammar provided for the preprocessor in the C99 specification.
2010-05-25 13:09:03 -07:00
{HASH} {
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
yyextra->space_tokens = 0;
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (HASH);
Implement #if, #else, #elif, and #endif with tests. So far the only expression implemented is a single integer literal, but obviously that's easy to extend. Various things including nesting are tested here.
2010-05-20 22:27:07 -07:00
}
stash
2010-05-26 09:32:12 -07:00
{DECIMAL_INTEGER} {
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (INTEGER_STRING);
stash
2010-05-26 09:32:12 -07:00
}
{OCTAL_INTEGER} {
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (INTEGER_STRING);
stash
2010-05-26 09:32:12 -07:00
}
{HEXADECIMAL_INTEGER} {
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (INTEGER_STRING);
stash
2010-05-26 09:32:12 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
"<<" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (LEFT_SHIFT);
Implement all operators specified for GLSL #if expressions (with tests). The operator coverage here is quite complete. The one big thing missing is that we are not yet doing macro expansion in #if lines. This makes the whole support fairly useless, so we plan to fix that shortcoming right away.
2010-05-24 10:37:38 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
">>" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (RIGHT_SHIFT);
Implement all operators specified for GLSL #if expressions (with tests). The operator coverage here is quite complete. The one big thing missing is that we are not yet doing macro expansion in #if lines. This makes the whole support fairly useless, so we plan to fix that shortcoming right away.
2010-05-24 10:37:38 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
"<=" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (LESS_OR_EQUAL);
Implement all operators specified for GLSL #if expressions (with tests). The operator coverage here is quite complete. The one big thing missing is that we are not yet doing macro expansion in #if lines. This makes the whole support fairly useless, so we plan to fix that shortcoming right away.
2010-05-24 10:37:38 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
">=" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (GREATER_OR_EQUAL);
Implement all operators specified for GLSL #if expressions (with tests). The operator coverage here is quite complete. The one big thing missing is that we are not yet doing macro expansion in #if lines. This makes the whole support fairly useless, so we plan to fix that shortcoming right away.
2010-05-24 10:37:38 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
"==" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (EQUAL);
Implement all operators specified for GLSL #if expressions (with tests). The operator coverage here is quite complete. The one big thing missing is that we are not yet doing macro expansion in #if lines. This makes the whole support fairly useless, so we plan to fix that shortcoming right away.
2010-05-24 10:37:38 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
"!=" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (NOT_EQUAL);
Implement all operators specified for GLSL #if expressions (with tests). The operator coverage here is quite complete. The one big thing missing is that we are not yet doing macro expansion in #if lines. This makes the whole support fairly useless, so we plan to fix that shortcoming right away.
2010-05-24 10:37:38 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
"&&" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (AND);
Implement all operators specified for GLSL #if expressions (with tests). The operator coverage here is quite complete. The one big thing missing is that we are not yet doing macro expansion in #if lines. This makes the whole support fairly useless, so we plan to fix that shortcoming right away.
2010-05-24 10:37:38 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
"||" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (OR);
Implement all operators specified for GLSL #if expressions (with tests). The operator coverage here is quite complete. The one big thing missing is that we are not yet doing macro expansion in #if lines. This makes the whole support fairly useless, so we plan to fix that shortcoming right away.
2010-05-24 10:37:38 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
"##" {
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
if (! parser->skipping) {
if (parser->is_gles)
glcpp_error(yylloc, yyextra, "Token pasting (##) is illegal in GLES");
RETURN_TOKEN (PASTE);
}
Implement #if, #else, #elif, and #endif with tests. So far the only expression implemented is a single integer literal, but obviously that's easy to extend. Various things including nesting are tested here.
2010-05-20 22:27:07 -07:00
}
stash
2010-05-26 09:32:12 -07:00
"defined" {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (DEFINED);
stash
2010-05-26 09:32:12 -07:00
}
Fix lexing of "defined" as an operator, not an identifier. Simply need to move the rule for IDENTIFIER to be after "defined" and everything is happy. With this change, tests 50 through 53 all pass now.
2010-05-26 09:35:34 -07:00
{IDENTIFIER} {
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (IDENTIFIER);
Fix lexing of "defined" as an operator, not an identifier. Simply need to move the rule for IDENTIFIER to be after "defined" and everything is happy. With this change, tests 50 through 53 all pass now.
2010-05-26 09:35:34 -07:00
}
glsl/glcpp: Fix glcpp to properly lex entire "preprocessing numbers" The preprocessor defines a notions of a "preprocessing number" that starts with either a digit or a decimal point, and continues with zero or more of digits, decimal points, identifier characters, or the sign symbols, ('-' and '+'). Prior to this change, preprocessing numbers were lexed as some combination of OTHER and IDENTIFIER tokens. This had the problem of causing undesired macro expansion in some cases. We add tests to ensure that the undesired macro expansion does not happen in cases such as: #define e +1 #define xyz -2 int n = 1e; int p = 1xyz; In either case these macro definitions have no effect after this change, so that the numeric literals, (whether valid or not), will be passed on as-is from the preprocessor to the compiler proper. This fixes the following Khronos GLES3 CTS tests: preprocessor.basic.correct_phases_vertex preprocessor.basic.correct_phases_fragment v2. Thanks to Anuj Phogat for improving the original regular expression, (which accepted a '+' or '-', where these are only allowed after one of [eEpP]. I also expanded the test to exercise this. v3. Also fixed regular expression to require at least one digit at the beginning (after an optional period). Otherwise, a string such as ".xyz" was getting sucked up as a preprocessing number, (where obviously this should be a field access). Again, I expanded the test to exercise this. Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 14:56:47 -07:00
{PP_NUMBER} {
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (OTHER);
glsl/glcpp: Fix glcpp to properly lex entire "preprocessing numbers" The preprocessor defines a notions of a "preprocessing number" that starts with either a digit or a decimal point, and continues with zero or more of digits, decimal points, identifier characters, or the sign symbols, ('-' and '+'). Prior to this change, preprocessing numbers were lexed as some combination of OTHER and IDENTIFIER tokens. This had the problem of causing undesired macro expansion in some cases. We add tests to ensure that the undesired macro expansion does not happen in cases such as: #define e +1 #define xyz -2 int n = 1e; int p = 1xyz; In either case these macro definitions have no effect after this change, so that the numeric literals, (whether valid or not), will be passed on as-is from the preprocessor to the compiler proper. This fixes the following Khronos GLES3 CTS tests: preprocessor.basic.correct_phases_vertex preprocessor.basic.correct_phases_fragment v2. Thanks to Anuj Phogat for improving the original regular expression, (which accepted a '+' or '-', where these are only allowed after one of [eEpP]. I also expanded the test to exercise this. v3. Also fixed regular expression to require at least one digit at the beginning (after an optional period). Otherwise, a string such as ".xyz" was getting sucked up as a preprocessing number, (where obviously this should be a field access). Again, I expanded the test to exercise this. Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 14:56:47 -07:00
}
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
{PUNCTUATION} {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (yytext[0]);
Perform "re lexing" on string list values rathern than on text. Previously, we would pass original strings back to the original lexer whenever we needed to re-lex something, (such as an expanded macro or a macro argument). Now, we instead parse the macro or argument originally to a string list, and then re-lex by simply returning each string from this list in turn. We do this in the recently added glcpp_parser_lex function that sits on top of the lower-level glcpp_lex that only deals with text. This doesn't change any behavior (at least according to the existing test suite which all still passes) but it brings us much closer to being able to "finalize" an unexpanded macro as required by the specification.
2010-05-19 13:28:24 -07:00
}
Make the lexer pass whitespace through (as OTHER tokens) for text lines. With this change, we can recreate the original text-line input exactly. Previously we were inserting a space between every pair of tokens so our output had a lot more whitespace than our input. With this change, we can drop the "-b" option to diff and match the input exactly.
2010-05-25 15:04:32 -07:00
{OTHER}+ {
glsl/glcpp: Abstract a bit of common code for returning string tokens Now that we have a common macro for returning tokens, it makes sense to perform some of the common work there, (such as copying string values). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 15:30:21 -07:00
RETURN_STRING_TOKEN (OTHER);
Make the lexer pass whitespace through (as OTHER tokens) for text lines. With this change, we can recreate the original text-line input exactly. Previously we were inserting a space between every pair of tokens so our output had a lot more whitespace than our input. With this change, we can drop the "-b" option to diff and match the input exactly.
2010-05-25 15:04:32 -07:00
}
glcpp: Do not remove spaces to preserve locations. After preprocessing by glcpp all adjacent spaces were replaced by single one and glsl parser received column-shifted shader source. It negatively affected ast location set up and produced wrong error messages for heavily-spaced shaders. Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
2014-02-05 20:15:56 +06:00
{HSPACE} {
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
if (yyextra->space_tokens) {
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
RETURN_TOKEN (SPACE);
Pass through literal space values from replacement lists. This makes test 15 pass and also dramatically simplifies the lexer. We were previously using a CONTROL state in the lexer to only emit SPACE tokens when on text lines. But that's not actually what we want. We need SPACE tokens in the replacement lists as well. Instead of a lexer state for this, we now simply set a "space_tokens" flag whenever we start constructing a pp_tokens list and clear the flag whenever we see a '#' introducing a directive. Much cleaner this way.
2010-05-25 16:59:02 -07:00
}
Simplify lexer significantly (remove all stateful lexing). We are able to remove all state by simply passing NEWLINE through as a token unconditionally (as opposed to only passing newline when on a driective line as we did previously).
2010-05-12 13:19:23 -07:00
}
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
/* We preserve all newlines, even between #if 0..#endif, so no
skipping.. */
\n {
glcpp: Replace multi-line comment with a space (even as part of macro definition) The preprocessor has always replaced multi-line comments with a single space character, (as required by the specification), but as of commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b the lexer also emitted a NEWLINE token for each newline within the comment, (in order to preserve line numbers). The emitting of NEWLINE tokens within the comment broke the rule of "replace a multi-line comment with a single space" as could be exposed by code like the following: #define FOO a/* */b FOO Prior to commit bd55ba568b301d0f764cd1ca015e84e1ae932c8b, this code defined the macro FOO as "a b" as desired. Since that commit, this code instead defines FOO as "a" and leaves a stray "b" in the output. In this commit, we fix this by not emitting the NEWLINE tokens while lexing the comment, but instead merely counting them in the commented_newlines variable. Then, when the lexer next encounters a non-commented newline it switches to a NEWLINE_CATCHUP state to emit as many NEWLINE tokens as necessary (so that subsequent parsing stages still generate correct line numbers). Of course, it would have been more clear if we could have written a loop to emit all the newlines, but flex conventions prevent that, (we must use "return" for each token we emit). It similarly would have been clear to have a new rule restricted to the <NEWLINE_CATCHUP> state with an action much like the body of this if condition. The problem with that is that this rule must not consume any characters. It might be possible to write a rule that matches a single lookahead of any character, but then we would also need an additional rule to ensure for the <EOF> case where there are no additional characters available for the lookahead to match. Given those considerations, and given that the SKIP-state manipulation already involves a code block at the top of the lexer function, before any rules, it seems best to me to go with the implementation here which adds a similar pre-rule code block for the NEWLINE_CATCHUP. Finally, this commit also changes the expected output of a few, existing glcpp tests. The change here is that the space character resulting from the multi-line comment is now emitted before the newlines corresponding to that comment. (Previously, the newlines were emitted first, and the space character afterward.) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=72686 Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
2013-12-19 16:06:31 -08:00
if (parser->commented_newlines) {
BEGIN NEWLINE_CATCHUP;
}
glsl/glcpp: Fix to emit spaces following directives The glcpp lexer and parser use the space_tokens state bit to avoid emitting tokens for spaces while parsing a directive. Previously, this bit was only being set again by the first non-space token following a directive. This led to a bug where a space, (or a comment that should emit a space), immediately following a directive, (optionally searated by newlines), would be omitted from the output. Here we fix the bug by also setting the space_tokens bit whenever we lex a newline in the standard start conditions.
2014-07-28 08:59:25 -07:00
yyextra->space_tokens = 1;
glsl/glcpp: Fix glcpp to catch garbage after #if 1 ... #else Previously, a line such as: #else garbage would flag an error if it followed "#if 0", but not if it followed "#if 1". We fix this by setting a new bit of state (lexing_else) that allows the lexer to defer switching to the <SKIP> start state until after the NEWLINE following the #else directive. A new test case is added for: #if 1 #else garbage #endif which was untested before, (and did not generate the desired error). This fixes the following Khronos GLES3 CTS tests: tokens_after_else_vertex tokens_after_else_fragment Reviewed-by: Matt Turner <mattst88@gmail.com> Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 10:39:39 -07:00
yyextra->lexing_directive = 0;
glcpp: Set line locations in the lexer.
2010-06-16 17:41:12 -07:00
yylineno++;
yycolumn = 0;
glsl/glcpp: Stop using a lexer start condition (<SKIP>) for token skipping. Here, "skipping" refers to the lexer not emitting any tokens for portions of the file within an #if condition (or similar) that evaluates to false. Previously, the lexer had a special <SKIP> start condition used to control this skipping. This start condition was not handled like a normal start condition. Instead, there was a particularly ugly block of code set to be included at the top of the generated lexing loop that would change from <INITIAL> to <SKIP> or from <SKIP> to <INITIAL> depending on various pieces of parser state, (such as parser->skip_state and parser->lexing_directive). Not only was that an ugly approach, but the <SKIP> start condition was complicating several glcpp bug fixes I attempted recently that want to use start conditions for other purposes, (such as a new <HASH> start condition). The recently added RETURN_TOKEN macro gives us a convenient way to implement skipping without using a lexer start condition. Now, at the top of the generated lexer, we examine all the necessary parser state and set a new parser->skipping bit. Then, in RETURN_TOKEN, we examine parser->skipping to determine whether to actually emit the token or not. Besides this, there are only a couple of other places where we need to examine the skipping bit (other than when returning a token): * To avoid emitting an error for #error if skipped. * To avoid entering the <DEFINE> start condition for a #define that is skipped. With all of this in place in the present commit, there are hopefully no behavioral changes with this patch, ("make check" still passes all of the glcpp tests at least). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 16:18:23 -07:00
RETURN_TOKEN_NEVER_SKIP (NEWLINE);
Make the lexer pass whitespace through (as OTHER tokens) for text lines. With this change, we can recreate the original text-line input exactly. Previously we were inserting a space between every pair of tokens so our output had a lot more whitespace than our input. With this change, we can drop the "-b" option to diff and match the input exactly.
2010-05-25 15:04:32 -07:00
}
glsl/glcpp: Add testing for EOF sans newline (and fix for <DEFINE>, <COMMENT>) The glcpp implementation has long had code to support a file that ends without a final newline. But we didn't have a "make check" test for this. Additionally, the <EOF> action was restricted only to the <INITIAL> state so it would fail to get invoked if the EOF was encountered in the <COMMENT> or the <DEFINE> case. Neither of these was a bug, per se, since EOF in either of these cases is an error anyway, (either "unterminated comment" or "missing macro name for #define"). But with the new explicit support for these cases, we not generate clean error messages in these cases, (rather than "unexpected $end" from before). Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:28:20 -07:00
<INITIAL,COMMENT,DEFINE><<EOF>> {
if (YY_START == COMMENT)
glcpp_error(yylloc, yyextra, "Unterminated comment");
if (YY_START == DEFINE)
glcpp_error(yylloc, yyextra, "#define without macro name");
glcpp: Handle missing newline at EOF. Fixes CorrectFuncOverload.vert.
2010-06-16 12:53:19 -07:00
BEGIN DONE; /* Don't keep matching this rule forever. */
glsl/glcpp: Fix glcpp to catch garbage after #if 1 ... #else Previously, a line such as: #else garbage would flag an error if it followed "#if 0", but not if it followed "#if 1". We fix this by setting a new bit of state (lexing_else) that allows the lexer to defer switching to the <SKIP> start state until after the NEWLINE following the #else directive. A new test case is added for: #if 1 #else garbage #endif which was untested before, (and did not generate the desired error). This fixes the following Khronos GLES3 CTS tests: tokens_after_else_vertex tokens_after_else_fragment Reviewed-by: Matt Turner <mattst88@gmail.com> Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
2014-06-12 10:39:39 -07:00
yyextra->lexing_directive = 0;
glsl/glcpp: Drop extra, final newline from most output The glcpp parser is line-based, so it needs to see a NEWLINE token at the end of each line. This causes a trick for files that end without a final newline. Previously, the lexer for glcpp punted in this case by unconditionally returning a NEWLINE token at end-of-file, (causing most files to have an extra blank line at the end). Here, we refine this by lexing end-of-file as a NEWLINE token only if the immediately preceding token was not a NEWLINE token. The patch is a minor change that only looks huge for two reasons: 1. Almost all glcpp test result ".expected" files are updated to drop the extra newline. 2. All return statements from the lexer are adjusted to use a new RETURN_TOKEN macro that tracks the last-token-was-a-newline state. Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2014-06-20 14:58:48 -07:00
if (! parser->last_token_was_newline)
RETURN_TOKEN (NEWLINE);
glcpp: Handle missing newline at EOF. Fixes CorrectFuncOverload.vert.
2010-06-16 12:53:19 -07:00
}
glcpp: Avoid warnings in generated flex code. We define the YY_NO_INPUT macro to avoid one needless function being generated. for the other needless functions, (yyunput and yy_top_state), we add a new UNREACHABLE start condition and call these functions from an action there. This doesn't change functionality at all, (since we never enter the UNREACHABLE start condition), but makes the compiler stop complaining about these two functions being defined but not used.
2010-07-20 15:53:14 -07:00
/* We don't actually use the UNREACHABLE start condition. We
only have this action here so that we can pretend to call some
generated functions, (to avoid "defined but not used"
warnings. */
<UNREACHABLE>. {
unput('.');
yy_top_state(yyextra);
}
Add the tiniest shell of a flex/bison-based parser. It doesn't really *do* anything yet---merlely parsing a stream of whitespace-separated tokens, (and not interpreting them at all).
2010-05-10 11:44:09 -07:00
%%
glcpp: Add support for lexing from a string. The standalone binary still reads from stdin, however.
2010-06-16 12:01:17 -07:00
void
glcpp_lex_set_source_string(glcpp_parser_t *parser, const char *shader)
{
yy_scan_string(shader, parser->scanner);
}
Reference in a new issue Copy permalink